We Keep Coding

How would you compress 256-byte string consists of only "F" and "G"?

Theoretically, how much you can compress this 256-byte string containing only "F" and "G"?
FGFFFFFFGFFFFGGGGGGGGGGGGGFFFFFGGGGGGGGGGGGFFGFGGGFFFGGGGGGGGFFFFFFFFFFFFFFFFFFFFFGGGGGGFFFGFGGFGFFFFGFFGFGGFFFGFGGFGFFFGFGGGGFGGGGGGGGGFFFFFFFFGGGGGGGFFFFFGFFGGGGGGGFFFGGGFFGGGGGGFFGGGGGGGGGFFGFFGFGFFGFFGFFFFGGGGFGGFGGGFFFGGGFFFGGGFFGGFFGGGGFFGFGGFFFGFGGF
While I don't see a real world application, it is intriguing that compression algorithms like gz, bzip2 and deflate have a disadvantage in this case.
Well, I have this answer and the C# code to demonstrate:
using System;
public class Program
{
public static void Main()
{
string testCase = "FGFFFFFFGFFFFGGGGGGGGGGGGGFFFFFGGGGGGGGGGGGFFGFGGGFFFGGGGGGGGFFFFFFFFFFFFFFFFFFFFFGGGGGGFFFGFGGFGFFFFGFFGFGGFFFGFGGFGFFFGFGGGGFGGGGGGGGGFFFFFFFFGGGGGGGFFFFFGFFGGGGGGGFFFGGGFFGGGGGGFFGGGGGGGGGFFGFFGFGFFGFFGFFFFGGGGFGGFGGGFFFGGGFFFGGGFFGGFFGGGGFFGFGGFFFGFGGF";
uint[] G = new uint[8]; // 256 bit
for (int i = 0; i < testCase.Length; i++)
G[(i / 32)] += (uint)(((testCase[i] & 1)) << (i % 32));
for (int i = 0; i < 8; i++)
Console.WriteLine(G[i]);
string gTestCase = string.Empty;
//G 71 0100 0111
//F 70 0100 0110
for (int i = 0; i < 256; i++)
gTestCase += (char)((((uint)G[i / 32] & (1 << (i % 32))) >> (i % 32)) | 70);
Console.WriteLine(testCase);
Console.WriteLine(gTestCase);
if (testCase == gTestCase)
Console.WriteLine("OK.");
}
}
It may sound silly, but as to how I can improve the algorithm so that this 256-bit decimal number can be further compressed, I have the following idea:
(Note: The following are different topics of discussion but related to compressing 256-byte further)
From my understanding of Microsoft's implementation of Decimal,
96-bit + 96-bit = 128-bit decimal.
Which implies that a 192-byte string containing of any two distinct characters can be encoded as 128-bit number instead of 192-bit number. Correct?
My questions are:
Can I do the same with 256-byte strings?
(by splitting each of them into a pair of two numbers before adding those two as a Decimal shorter than 256-bit)?
How do I decode the above-mentioned 128-bit Decimal back to a pair of two 96-bit numbers, while maintaining the compressed data size less than 192-bit?
Sorry for my previous rather vague question.
The following code would demonstrate how to add two 96-char "binary" strings as 128-char binary string.
public static string AddBinary(string a, string b) // 96-char binary strings
{
int[] x = { 0, 0, 0 };
int[] y = { 0, 0, 0 };
string c = String.Empty;
for (int z = 0; z < a.Length; z++)
x[(z / 32)] |= ((byte)(a[a.Length - z - 1]) & 1) << (z % 32);
for (int z = 0; z < b.Length; z++)
y[(z / 32)] |= ((byte)(b[b.Length - z - 1]) & 1) << (z % 32);
decimal m = new decimal(x[0], x[1], x[2], false, 0); //96-bit
decimal n = new decimal(y[0], y[1], y[2], false, 0); //96-bit
decimal k = decimal.Add(m, n);
int[] l = decimal.GetBits(k); //128-bit
Console.WriteLine(k);
for (int z = 127; z >= 0; z--)
c += (char)(((l[(z / 32)] & (1 << (z % 32))) >> (z % 32)) | 48);
return c.Contains("1") ? c.TrimStart('0') : "0";
}

96-bit + 96-bit = 128-bit decimal.
That is a misunderstanding. Decimal is 96bit integer/mantissa, a sign and an exponent from 0 to 28 (~5bit) to form a scaling factor for the mantissa.
Addition is from 2×(1+5+96) bits to 1×(1+5+96) bits, including inevitable rounding errors and overflow.
You can't get summands from a sum easily - for starters, addition is symmetrical, there is no earthly way of knowing which of two summands has been the first and which the second.
Paul Hankin mentioned the programmer's variant of compressibility: Kolmogorov complexity.
In all fairness, you'd have to add to the 256 bits of your recoding of the input string the size of a program to turn those bits into the original string.
(As would gz, bzip2, deflate(, LZW) - decoders for "pure LZ" can be very small. The usual escape is to define a file format, including a recognisably header.)
Lasse V. Karlsen mentioned one consequence of the Pigeon-hole principle: to tell each combination of 192 bits from every other one, you need no less than 2^192 codes.

fast way to convert integer array to byte array (11 bit)

I have integer array and I need to convert it to byte array
but I need to take (only and just only) first 11 bit of each element of the هinteger array
and then convert it to a byte array
I tried this code
// ***********convert integer values to byte values
//***********to avoid the left zero padding on the byte array
// *********** first step : convert to binary string
// ***********second step : convert binary string to byte array
// *********** first step
string ByteString = Convert.ToString(IntArray[0], 2).PadLeft(11,'0');
for (int i = 1; i < IntArray.Length; i++)
ByteString = ByteString + Convert.ToString(IntArray[i], 2).PadLeft(11, '0');
// ***********second step
int numOfBytes = ByteString.Length / 8;
byte[] bytes = new byte[numOfBytes];
for (int i = 0; i < numOfBytes; ++i)
{
bytes[i] = Convert.ToByte(ByteString.Substring(8 * i, 8), 2);
}
But it takes too long time (if the file size large , the code takes more than 1 minute)
I need a very very fast code (very few milliseconds only )
can any one help me ?

Basically, you're going to be doing a lot of shifting and masking. The exact nature of that depends on the layout you want. If we assume that we pack little-endian from each int, appending on the left, so two 11-bit integers with positions:
abcdefghijk lmnopqrstuv
become the 8-bit chunks:
defghijk rstuvabc 00lmnopq
(i.e. take the lowest 8 bits of the first integer, which leaves 3 left over, so pack those into the low 3 bits of the next byte, then take the lowest 5 bits of the second integer, then finally the remaining 6 bits, padding with zero), then something like this should work:
using System;
using System.Linq;
static class Program
{
static string AsBinary(int val) => Convert.ToString(val, 2).PadLeft(11, '0');
static string AsBinary(byte val) => Convert.ToString(val, 2).PadLeft(8, '0');
static void Main()
{
int[] source = new int[1432];
var rand = new Random(123456);
for (int i = 0; i < source.Length; i++)
source[i] = rand.Next(0, 2047); // 11 bits
// Console.WriteLine(string.Join(" ", source.Take(5).Select(AsBinary)));
var raw = Encode(source);
// Console.WriteLine(string.Join(" ", raw.Take(6).Select(AsBinary)));
var clone = Decode(raw);
// now prove that it worked OK
if (source.Length != clone.Length)
{
Console.WriteLine($"Length: {source.Length} vs {clone.Length}");
}
else
{
int failCount = 0;
for (int i = 0; i < source.Length; i++)
{
if (source[i] != clone[i] && failCount++ == 0)
{
Console.WriteLine($"{i}: {source[i]} vs {clone[i]}");
}
}
Console.WriteLine($"Errors: {failCount}");
}
}
static byte[] Encode(int[] source)
{
long bits = source.Length * 11;
int len = (int)(bits / 8);
if ((bits % 8) != 0) len++;
byte[] arr = new byte[len];
int bitOffset = 0, index = 0;
for (int i = 0; i < source.Length; i++)
{
// note: this encodes little-endian
int val = source[i] & 2047;
int bitsLeft = 11;
if(bitOffset != 0)
{
val = val << bitOffset;
arr[index++] |= (byte)val;
bitsLeft -= (8 - bitOffset);
val >>= 8;
}
if(bitsLeft >= 8)
{
arr[index++] = (byte)val;
bitsLeft -= 8;
val >>= 8;
}
if(bitsLeft != 0)
{
arr[index] = (byte)val;
}
bitOffset = bitsLeft;
}
return arr;
}
private static int[] Decode(byte[] source)
{
int bits = source.Length * 8;
int len = (int)(bits / 11);
// note no need to worry about remaining chunks - no ambiguity since 11 > 8
int[] arr = new int[len];
int bitOffset = 0, index = 0;
for(int i = 0; i < source.Length; i++)
{
int val = source[i] << bitOffset;
int bitsLeftInVal = 11 - bitOffset;
if(bitsLeftInVal > 8)
{
arr[index] |= val;
bitOffset += 8;
}
else if(bitsLeftInVal == 8)
{
arr[index++] |= val;
bitOffset = 0;
}
else
{
arr[index++] |= (val & 2047);
if(index != arr.Length) arr[index] = val >> 11;
bitOffset = 8 - bitsLeftInVal;
}
}
return arr;
}
}
If you need a different layout you'll need to tweak it.
This encodes 512 MiB in just over a second on my machine.
Overview to the Encode method:
The first thing is does is pre-calculate the amount of space that is going to be required, and allocate the output buffer; since each input contributes 11 bits to the output, this is just some modulo math:
long bits = source.Length * 11;
int len = (int)(bits / 8);
if ((bits % 8) != 0) len++;
byte[] arr = new byte[len];
We know the output position won't match the input, and we know we're going to be starting each 11-bit chunk at different positions in bytes each time, so allocate variables for those, and loop over the input:
int bitOffset = 0, index = 0;
for (int i = 0; i < source.Length; i++)
{
...
}
return arr;
So: taking each input in turn (where the input is the value at position i), take the low 11 bits of the value - and observe that we have 11 bits (of this value) still to write:
int val = source[i] & 2047;
int bitsLeft = 11;
Now, if the current output value is partially written (i.e. bitOffset != 0), we should deal with that first. The amount of space left in the current output is 8 - bitOffset. Since we always have 11 input bits we don't need to worry about having more space than values to fill, so: left-shift our value by bitOffset (pads on the right with bitOffset zeros, as a binary operation), and "or" the lowest 8 bits of this with the output byte. Essentially this says "if bitOffset is 3, write the 5 low bits of val into the 5 high bits of the output buffer"; finally, fixup the values: increment our write position, record that we have fewer bits of the current value still to write, and use right-shift to discard the 8 low bits of val (which is made of bitOffset zeros and 8 - bitOffset "real" bits):
if(bitOffset != 0)
{
val = val << bitOffset;
arr[index++] |= (byte)val;
bitsLeft -= (8 - bitOffset);
val >>= 8;
}
The next question is: do we have (at least) an entire byte of data left? We might not, if bitOffset was 1 for example (so we'll have written 7 bits already, leaving just 4). If we do, we can just stamp that down and increment the write position - then once again track how many are left and throw away the low 8 bits:
if(bitsLeft >= 8)
{
arr[index++] = (byte)val;
bitsLeft -= 8;
val >>= 8;
}
And it is possible that we've still got some left-over; for example, if bitOffset was 7 we'll have written 1 bit in the first chunk, 8 bits in the second, leaving 2 more to write - or if bitOffset was 0 we won't have written anything in the first chunk, 8 in the second, leaving 3 left to write. So, stamp down whatever is left, but do not increment the write position - we've written to the low bits, but the next value might need to write to the high bits. Finally, update bitOffset to be however many low bits we wrote in the last step (which could be zero):
if(bitsLeft != 0)
{
arr[index] = (byte)val;
}
bitOffset = bitsLeft;
The Decode operation is the reverse of this logic - again, calculate the sizes and prepare the state:
int bits = source.Length * 8;
int len = (int)(bits / 11);
int[] arr = new int[len];
int bitOffset = 0, index = 0;
Now loop over the input:
for(int i = 0; i < source.Length; i++)
{
...
}
return arr;
Now, bitOffset is the start position that we want to write to in the current 11-bit value, so if we start at the start, it will be 0 on the first byte, then 8; 3 bits of the second byte join with the first 11-bit integer, so the 5 bits become part of the second - so bitOffset is 5 on the 3rd byte, etc. We can calculate the number of bits left in the current integer by subtracting from 11:
int val = source[i] << bitOffset;
int bitsLeftInVal = 11 - bitOffset;
Now we have 3 possible scenarios:
1) if we have more than 8 bits left in the current value, we can stamp down our input (as a bitwise "or") but do not increment the write position (as we have more to write for this value), and note that we're 8-bits further along:
if(bitsLeftInVal > 8)
{
arr[index] |= val;
bitOffset += 8;
}
2) if we have exactly 8 bits left in the current value, we can stamp down our input (as a bitwise "or") and increment the write position; the next loop can start at zero:
else if(bitsLeftInVal == 8)
{
arr[index++] |= val;
bitOffset = 0;
}
3) otherwise, we have less than 8 bits left in the current value; so we need to write the first bitsLeftInVal bits to the current output position (incrementing the output position), and whatever is left to the next output position. Since we already left-shifted by bitOffset, what this really means is simply: stamp down (as a bitwise "or") the low 11 bits (val & 2047) to the current position, and whatever is left (val >> 11) to the next if that wouldn't exceed our output buffer (padding zeros). Then calculate our new bitOffset:
else
{
arr[index++] |= (val & 2047);
if(index != arr.Length) arr[index] = val >> 11;
bitOffset = 8 - bitsLeftInVal;
}
And that's basically it. Lots of bitwise operations - shifts (<< / >>), masks (&) and combinations (|).

If you wanted to store the least significant 11 bits of an int into two bytes such that the least significant byte has bits 1-8 inclusive and the most significant byte has 9-11:
int toStore = 123456789;
byte msb = (byte) ((toStore >> 8) & 7); //or 0b111
byte lsb = (byte) (toStore & 255); //or 0b11111111
To check this, 123456789 in binary is:
0b111010110111100110100010101
MMMLLLLLLLL
The bits above L are lsb, and have a value of 21, above M are msb and have a value of 5
Doing the work is the shift operator >> where all the binary digits are slid to the right 8 places (8 of them disappear from the right hand side - they're gone, into oblivion):
0b111010110111100110100010101 >> 8 =
0b1110101101111001101
And the mask operator & (the mask operator works by only keeping bits where, in each position, they're 1 in the value and also 1 in the mask) :
0b111010110111100110100010101 &
0b000000000000000000011111111 (255) =
0b000000000000000000000010101
If you're processing an int array, just do this in a loop:
byte[] bs = new byte[ intarray.Length*2 ];
for(int x = 0, b=0; x < intarray.Length; x++){
int toStore = intarray[x];
bs[b++] = (byte) ((toStore >> 8) & 7);
bs[b++] = (byte) (toStore & 255);
}

Set a range of bits in a circular bit field

I have a bit field consisting of 64 bits:
long bitfield = 0;
I can set the bit for a given index as follows:
void Set(long index)
{
bitfield |= 1L << (int)(index % 64);
}
i.e. if the index is 0, 64, 128, ... then bit 0 is set, if the index is 1, 65, 129, ... then bit 1 is set, and so on.
Question: given an index and a count (or a lower and upper index), how can I set the bits for all indexes in this range without using a loop?

long SetRangeMask(int lower, int upper) // 3..7
{
if (! (lower <= upper)) throw new ArgumentException("...");
int size = upper - lower + 1; // 7 - 3 + 1 = 5
if (size >= 64) return -1;
long mask = (1 << size) - 1; // #00100000 - 1 = #000011111
return mask << lower | mask >> -lower; // #00011111 << 3 = #011111000
}

You could use a lookup table for combined bit masks
A real simple approach with no thought to special cases or optimizations like these questions raised, would look like:
static readonly private long[] maskLUT = new long[64,64] { /* generated */ };
void SetRange(long lobit, long hibit)
{
lobit %= 64;
hibit %= 64;
bitfield |= lobit<hibit? maskLUT[lobit,hibit] : maskLUT[hibit,lobit];
}
Thoughts:
you might consider an optimization that given [lobit...hibit], if hibit-lobit>=64 you can set all bits at once.
There is a bit of thought to be put in the connected-ness of regions given the fact that both boundaries can wrap around (do you wrap-around both boundaries first, or do you wraparound lobit, and use the delta to find the hibit from the wrapped boundary, like with the optimization mentioned before?)

You can use 2x-1 to create a mask x bits long, then shift it and OR it in, like so:
void Set( int index, int count ) {
bitfield |= (long)(Math.Pow( 2, count ) - 1) << ((index-count) % 64);
}
Update: I like to think that Math.Pow optimizes powers of two to a left shift, but it may not. If that's the case, you can get a little more performance by replacing the call to Math.Pow with the corresponding left shift:
public void Set( int index, int count ) {
bitfield |= ((2 << count - 1) - 1) << ((index-count) % 64);
}

How to add even parity bit on 7-bit binary number

I am continuing from my previous question. I am making a c# program where the user enters a 7-bit binary number and the computer prints out the number with an even parity bit to the right of the number. I am struggling. I have a code, but it says BitArray is a namespace but is used as a type. Also, is there a way I could improve the code and make it simpler?
namespace BitArray
{
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Please enter a 7-bit binary number:");
int a = Convert.ToInt32(Console.ReadLine());
byte[] numberAsByte = new byte[] { (byte)a };
BitArray bits = new BitArray(numberAsByte);
int count = 0;
for (int i = 0; i < 8; i++)
{
if (bits[i])
{
count++;
}
}
if (count % 2 == 1)
{
bits[7] = true;
}
bits.CopyTo(numberAsByte, 0);
a = numberAsByte[0];
Console.WriteLine("The binary number with a parity bit is:");
Console.WriteLine(a);

Might be more fun to duplicate the circuit they use to do this..
bool odd = false;
for(int i=6;i>=0;i--)
odd ^= (number & (1 << i)) > 0;
Then if you want even parity set bit 7 to odd, odd parity to not odd.
or
bool even = true;
for(int i=6;i>=0;i--)
even ^= (number & (1 << i)) > 0;
The circuit is dual function returns 0 and 1 or 1 and 0, does more than 1 bit at a time as well, but this is a bit light for TPL....
PS you might want to check the input for < 128 otherwise things are going to go well wrong.
ooh didn't notice the homework tag, don't use this unless you can explain it.

Almost the same process, only much faster on a larger number of bits. Using only the arithmetic operators (SHR && XOR), without loops:
public static bool is_parity(int data)
{
//data ^= data >> 32; // if arg >= 64-bit (notice argument length)
//data ^= data >> 16; // if arg >= 32-bit
//data ^= data >> 8; // if arg >= 16-bit
data ^= data >> 4;
data ^= data >> 2;
data ^= data >> 1;
return (data & 1) !=0;
}
public static byte fix_parity(byte data)
{
if (is_parity(data)) return data;
return (byte)(data ^ 128);
}

Using a BitArray does not buy you much here, if anything it makes your code harder to understand. Your problem can be solved with basic bit manipulation with the & and | and << operators.
For example to find out if a certain bit is set in a number you can & the number with the corresponding power of 2. That leads to:
int bitsSet = 0;
for(int i=0;i<7;i++)
if ((number & (1 << i)) > 0)
bitsSet++;
Now the only thing remain is determining if bitsSet is even or odd and then setting the remaining bit if necessary.

Fastest way to calculate sum of bits in byte array

I have two byte arrays with the same length. I need to perform XOR operation between each byte and after this calculate sum of bits.
For example:
11110000^01010101 = 10100101 -> so 1+1+1+1 = 4
I need do the same operation for each element in byte array.

Use a lookup table. There are only 256 possible values after XORing, so it's not exactly going to take a long time. Unlike izb's solution though, I wouldn't suggest manually putting all the values in though - compute the lookup table once at startup using one of the looping answers.
For example:
public static class ByteArrayHelpers
{
private static readonly int[] LookupTable =
Enumerable.Range(0, 256).Select(CountBits).ToArray();
private static int CountBits(int value)
{
int count = 0;
for (int i=0; i < 8; i++)
{
count += (value >> i) & 1;
}
return count;
}
public static int CountBitsAfterXor(byte[] array)
{
int xor = 0;
foreach (byte b in array)
{
xor ^= b;
}
return LookupTable[xor];
}
}
(You could make it an extension method if you really wanted...)
Note the use of byte[] in the CountBitsAfterXor method - you could make it an IEnumerable<byte> for more generality, but iterating over an array (which is known to be an array at compile-time) will be faster. Probably only microscopically faster, but hey, you asked for the fastest way :)
I would almost certainly actually express it as
public static int CountBitsAfterXor(IEnumerable<byte> data)
in real life, but see which works better for you.
Also note the type of the xor variable as an int. In fact, there's no XOR operator defined for byte values, and if you made xor a byte it would still compile due to the nature of compound assignment operators, but it would be performing a cast on each iteration - at least in the IL. It's quite possible that the JIT would take care of this, but there's no need to even ask it to :)

Fastest way would probably be a 256-element lookup table...
int[] lut
{
/*0x00*/ 0,
/*0x01*/ 1,
/*0x02*/ 1,
/*0x03*/ 2
...
/*0xFE*/ 7,
/*0xFF*/ 8
}
e.g.
11110000^01010101 = 10100101 -> lut[165] == 4

This is more commonly referred to as bit counting. There are literally dozens of different algorithms for doing this. Here is one site which lists a few of the more well known methods. There are even CPU specific instructions for doing this.
Theorectically, Microsoft could add a BitArray.CountSetBits function that gets JITed with the best algorithm for that CPU architecture. I, for one, would welcome such an addition.

As I understood it you want to sum the bits of each XOR between the left and right bytes.
for (int b = 0; b < left.Length; b++) {
int num = left[b] ^ right[b];
int sum = 0;
for (int i = 0; i < 8; i++) {
sum += (num >> i) & 1;
}
// do something with sum maybe?
}

I'm not sure if you mean sum the bytes or the bits.
To sum the bits within a byte, this should work:
int nSum = 0;
for (int i=0; i<=7; i++)
{
nSum += (byte_val>>i) & 1;
}
You would then need the xoring, and array looping around this, of course.

The following should do
int BitXorAndSum(byte[] left, byte[] right) {
int sum = 0;
for ( var i = 0; i < left.Length; i++) {
sum += SumBits((byte)(left[i] ^ right[i]));
}
return sum;
}
int SumBits(byte b) {
var sum = 0;
for (var i = 0; i < 8; i++) {
sum += (0x1) & (b >> i);
}
return sum;
}

It can be rewritten as ulong and use unsafe pointer, but byte is easier to understand:
static int BitCount(byte num)
{
// 0x5 = 0101 (bit) 0x55 = 01010101
// 0x3 = 0011 (bit) 0x33 = 00110011
// 0xF = 1111 (bit) 0x0F = 00001111
uint count = num;
count = ((count >> 1) & 0x55) + (count & 0x55);
count = ((count >> 2) & 0x33) + (count & 0x33);
count = ((count >> 4) & 0xF0) + (count & 0x0F);
return (int)count;
}

A general function to count bits could look like:
int Count1(byte[] a)
{
int count = 0;
for (int i = 0; i < a.Length; i++)
{
byte b = a[i];
while (b != 0)
{
count++;
b = (byte)((int)b & (int)(b - 1));
}
}
return count;
}
The less 1-bits, the faster this works. It simply loops over each byte, and toggles the lowest 1 bit of that byte until the byte becomes 0. The castings are necessary so that the compiler stops complaining about the type widening and narrowing.
Your problem could then be solved by using this:
int Count1Xor(byte[] a1, byte[] a2)
{
int count = 0;
for (int i = 0; i < Math.Min(a1.Length, a2.Length); i++)
{
byte b = (byte)((int)a1[i] ^ (int)a2[i]);
while (b != 0)
{
count++;
b = (byte)((int)b & (int)(b - 1));
}
}
return count;
}

A lookup table should be the fastest, but if you want to do it without a lookup table, this will work for bytes in just 10 operations.
public static int BitCount(byte value) {
int v = value - ((value >> 1) & 0x55);
v = (v & 0x33) + ((v >> 2) & 0x33);
return ((v + (v >> 4) & 0x0F));
}
This is a byte version of the general bit counting function described at Sean Eron Anderson's bit fiddling site.