We Keep Coding

Removing leading 0s in a byte array

I have a byte array as follows -
byte[] arrByt = new byte[] { 0xF, 0xF, 0x11, 0x4 };
so in binary
arrByt = 00001111 00001111 00010001 000000100
Now I want to create a new byte array by removing leading 0s for each byte from arrByt
arrNewByt = 11111111 10001100 = { 0xFF, 0x8C };
I know that this can be done by converting the byte values into binary string values, removing the leading 0s, appending the values and converting back to byte values into the new array.
However this is a slow process for a large array.
Is there a faster way to achieve this (like logical operations, bit operations, or other efficient ways)?
Thanks.

This should do the job quite fast. At least only standard loops and operators. Give it a try, will also work for longer source arrays.
// source array of bytes
var arrByt = new byte[] {0xF, 0xF, 0x11, 0x4 };
// target array - first with the size of the source array
var targetArray = new byte[arrByt.Length];
// bit index in target array
// from left = byte 0, bit 7 = index 31; to the right = byte 4, bit 0 = index 0
var targetIdx = targetArray.Length * 8 - 1;
// go through all bytes of the source array from left to right
for (var i = 0; i < arrByt.Length; i++)
{
var startFound = false;
// go through all bits of the current byte from the highest to the lowest
for (var x = 7; x >= 0; x--)
{
// copy the bit if it is 1 or if there was already a 1 before in this byte
if (startFound || ((arrByt[i] >> x) & 1) == 1)
{
startFound = true;
// copy the bit from its position in the source array to its new position in the target array
targetArray[targetArray.Length - ((targetIdx / 8) + 1)] |= (byte) (((arrByt[i] >> x) & 1) << (targetIdx % 8));
// advance the bit + byte position in the target array one to the right
targetIdx--;
}
}
}
// resize the target array to only the bytes that were used above
Array.Resize(ref targetArray, (int)Math.Ceiling((targetArray.Length * 8 - (targetIdx + 1)) / 8d));
// write target array content to console
for (var i = 0; i < targetArray.Length; i++)
{
Console.Write($"{targetArray[i]:X} ");
}
// OUTPUT: FF 8C

If you are trying to find the location of the most-significant bit, you can do a log2() of the byte (and if you don't have log2, you can use log(x)/log(2) which is the same as log2(x))
For instance, the number 7, 6, 5, and 4 all have a '1' in the 3rd bit position (0111, 0110, 0101, 0100). The log2() of them are all between 2 and 2.8. Same thing happens for anything in the 4th bit, it will be a number between 3 and 3.9. So you can find out the Most Significant Bit by adding 1 to the log2() of the number (round down).
floor(log2(00001111)) + 1 == floor(3.9) + 1 == 3 + 1 == 4
You know how many bits are in a byte, so you can easily know the number of bits to shift left:
int numToShift = 8 - floor(log2(bytearray[0])) + 1;
shiftedValue = bytearray[0] << numToShift;
From there, it's just a matter of keeping track of how many outstanding bits (not pushed into a bytearray yet) you have, and then pushing some/all of them on.
The above code would only work for the first byte array. If you put this in a loop, the numToShift would maybe need to keep track of the latest empty slot to shift things into (you might have to shift right to fit in current byte array, and then use the leftovers to put into the start of the next byte array). So instead of doing "8 -" in the above code, you would maybe put the starting location. For instance, if only 3 bits were left to fill in the current byte array, you would do:
int numToShift = 3 - floor(log2(bytearray[0])) + 1;
So that number should be a variable:
int numToShift = bitsAvailableInCurrentByte - floor(log2(bytearray[0])) + 1;

Please check this code snippet. This might help you.
byte[] arrByt = new byte[] { 0xF, 0xF, 0x11, 0x4 };
byte[] result = new byte[arrByt.Length / 2];
var en = arrByt.GetEnumerator();
int count = 0;
byte result1 = 0;
int index = 0;
while (en.MoveNext())
{
count++;
byte item = (byte)en.Current;
if (count == 1)
{
while (item < 128)
{
item = (byte)(item << 1);
}
result1 ^= item;
}
if (count == 2)
{
count = 0;
result1 ^= item;
result[index] = result1;
index++;
result1 = 0;
}
}
foreach (var s in result)
{
Console.WriteLine(s.ToString("X"));
}

How to merge two bitmaps with specific shift(offset)?

Let say we have two bitmaps that are represented by unsigned long(64-bit) arrays. And I want to merge this two bitmaps using specific shift(offset).
For example merge bitmap1(bigger) into bitmap2(smaller) starting offset 3. Offset 3 mean that 3rd bit of bitmap1 corresponds to 0 bit of bitmap2.
By merge I mean logical Or operation. What is the cleanest way to do this?
Currently I have done this with simple uneffective for loop
const ulong BitsPerUlong = 64;
MergeAt(ulong startIndex, Bitmap bitmap2)
{
for (int i = startIndex; i < bitmap2.Capacity; i++)
{
bool newVal = bitmap2.GetAt(i) | bitmap1.GetAt(i)
bitmap2.SetAt(i, newVal)
}
}
bool GetAt(ulong index)
{
var dataOffset = BitOffsetToUlongOffset(index);
ulong mask = 0x1ul << ((int)(index % BitsPerUlong));
return (_data[dataOffset] & mask) == mask;
}
void SetAt(ulong index, bool value)
{
var dataOffset = BitOffsetToUlongOffset(index);
ulong mask = 0x1ul << ((int)(index % BitsPerUlong));
if (value)
{
_data[dataOffset] |= mask;
}
else
{
_data[dataOffset] &= ~mask;
}
}
ulong BitOffsetToUlongOffset(ulong index)
{
var dataOffset = index / BitsPerUlong;
return dataOffset;
}
(C/C++/C# accepted).

As you probably figured out yourself, if offset < BitsPerULong the first block can be merged with:
data1[0] |= data2[0] << offset;
Which leaves some bits in data2[0] unmerged, but you can get those with:
data2[0] >> (BitsPerULong - offset)
So the next merge for i > 0 becomes:
data1[i] |= (data2[i] << offset) | (data2[i-1] >> (BitsPerULong - offset));
from which you can construct a for-loop to merge all data. Of course, this still means a couple of bits from data2 will "fall off" but I think that's inherent to your problem description?
If you need a more generic solution where offset can also be greater than BitsPerULong, this needs a bit more work.

I presume you mean that you want to "merge" the smaller INTO the bigger.
Have you tried: bitmapLarger |= ( bitmapSmaller << 3 ) ?

Set a range of bits in a circular bit field

I have a bit field consisting of 64 bits:
long bitfield = 0;
I can set the bit for a given index as follows:
void Set(long index)
{
bitfield |= 1L << (int)(index % 64);
}
i.e. if the index is 0, 64, 128, ... then bit 0 is set, if the index is 1, 65, 129, ... then bit 1 is set, and so on.
Question: given an index and a count (or a lower and upper index), how can I set the bits for all indexes in this range without using a loop?

long SetRangeMask(int lower, int upper) // 3..7
{
if (! (lower <= upper)) throw new ArgumentException("...");
int size = upper - lower + 1; // 7 - 3 + 1 = 5
if (size >= 64) return -1;
long mask = (1 << size) - 1; // #00100000 - 1 = #000011111
return mask << lower | mask >> -lower; // #00011111 << 3 = #011111000
}

You could use a lookup table for combined bit masks
A real simple approach with no thought to special cases or optimizations like these questions raised, would look like:
static readonly private long[] maskLUT = new long[64,64] { /* generated */ };
void SetRange(long lobit, long hibit)
{
lobit %= 64;
hibit %= 64;
bitfield |= lobit<hibit? maskLUT[lobit,hibit] : maskLUT[hibit,lobit];
}
Thoughts:
you might consider an optimization that given [lobit...hibit], if hibit-lobit>=64 you can set all bits at once.
There is a bit of thought to be put in the connected-ness of regions given the fact that both boundaries can wrap around (do you wrap-around both boundaries first, or do you wraparound lobit, and use the delta to find the hibit from the wrapped boundary, like with the optimization mentioned before?)

You can use 2x-1 to create a mask x bits long, then shift it and OR it in, like so:
void Set( int index, int count ) {
bitfield |= (long)(Math.Pow( 2, count ) - 1) << ((index-count) % 64);
}
Update: I like to think that Math.Pow optimizes powers of two to a left shift, but it may not. If that's the case, you can get a little more performance by replacing the call to Math.Pow with the corresponding left shift:
public void Set( int index, int count ) {
bitfield |= ((2 << count - 1) - 1) << ((index-count) % 64);
}

Converting an Int to a BCD byte array [duplicate]

I want to convert an int to a byte[2] array using BCD.
The int in question will come from DateTime representing the Year and must be converted to two bytes.
Is there any pre-made function that does this or can you give me a simple way of doing this?
example:
int year = 2010
would output:
byte[2]{0x20, 0x10};

static byte[] Year2Bcd(int year) {
if (year < 0 || year > 9999) throw new ArgumentException();
int bcd = 0;
for (int digit = 0; digit < 4; ++digit) {
int nibble = year % 10;
bcd |= nibble << (digit * 4);
year /= 10;
}
return new byte[] { (byte)((bcd >> 8) & 0xff), (byte)(bcd & 0xff) };
}
Beware that you asked for a big-endian result, that's a bit unusual.

Use this method.
public static byte[] ToBcd(int value){
if(value<0 || value>99999999)
throw new ArgumentOutOfRangeException("value");
byte[] ret=new byte[4];
for(int i=0;i<4;i++){
ret[i]=(byte)(value%10);
value/=10;
ret[i]|=(byte)((value%10)<<4);
value/=10;
}
return ret;
}
This is essentially how it works.
If the value is less than 0 or greater than 99999999, the value won't fit in four bytes. More formally, if the value is less than 0 or is 10^(n*2) or greater, where n is the number of bytes, the value won't fit in n bytes.
For each byte:
Set that byte to the remainder of the value-divided-by-10 to the byte. (This will place the last digit in the low nibble [half-byte] of the current byte.)
Divide the value by 10.
Add 16 times the remainder of the value-divided-by-10 to the byte. (This will place the now-last digit in the high nibble of the current byte.)
Divide the value by 10.
(One optimization is to set every byte to 0 beforehand -- which is implicitly done by .NET when it allocates a new array -- and to stop iterating when the value reaches 0. This latter optimization is not done in the code above, for simplicity. Also, if available, some compilers or assemblers offer a divide/remainder routine that allows retrieving the quotient and remainder in one division step, an optimization which is not usually necessary though.)

Here's a terrible brute-force version. I'm sure there's a better way than this, but it ought to work anyway.
int digitOne = year / 1000;
int digitTwo = (year - digitOne * 1000) / 100;
int digitThree = (year - digitOne * 1000 - digitTwo * 100) / 10;
int digitFour = year - digitOne * 1000 - digitTwo * 100 - digitThree * 10;
byte[] bcdYear = new byte[] { digitOne << 4 | digitTwo, digitThree << 4 | digitFour };
The sad part about it is that fast binary to BCD conversions are built into the x86 microprocessor architecture, if you could get at them!

Here is a slightly cleaner version then Jeffrey's
static byte[] IntToBCD(int input)
{
if (input > 9999 || input < 0)
throw new ArgumentOutOfRangeException("input");
int thousands = input / 1000;
int hundreds = (input -= thousands * 1000) / 100;
int tens = (input -= hundreds * 100) / 10;
int ones = (input -= tens * 10);
byte[] bcd = new byte[] {
(byte)(thousands << 4 | hundreds),
(byte)(tens << 4 | ones)
};
return bcd;
}

maybe a simple parse function containing this loop
i=0;
while (id>0)
{
twodigits=id%100; //need 2 digits per byte
arr[i]=twodigits%10 + twodigits/10*16; //first digit on first 4 bits second digit shifted with 4 bits
id/=100;
i++;
}

More common solution
private IEnumerable<Byte> GetBytes(Decimal value)
{
Byte currentByte = 0;
Boolean odd = true;
while (value > 0)
{
if (odd)
currentByte = 0;
Decimal rest = value % 10;
value = (value-rest)/10;
currentByte |= (Byte)(odd ? (Byte)rest : (Byte)((Byte)rest << 4));
if(!odd)
yield return currentByte;
odd = !odd;
}
if(!odd)
yield return currentByte;
}

Same version as Peter O. but in VB.NET
Public Shared Function ToBcd(ByVal pValue As Integer) As Byte()
If pValue < 0 OrElse pValue > 99999999 Then Throw New ArgumentOutOfRangeException("value")
Dim ret As Byte() = New Byte(3) {} 'All bytes are init with 0's
For i As Integer = 0 To 3
ret(i) = CByte(pValue Mod 10)
pValue = Math.Floor(pValue / 10.0)
ret(i) = ret(i) Or CByte((pValue Mod 10) << 4)
pValue = Math.Floor(pValue / 10.0)
If pValue = 0 Then Exit For
Next
Return ret
End Function
The trick here is to be aware that simply using pValue /= 10 will round the value so if for instance the argument is "16", the first part of the byte will be correct, but the result of the division will be 2 (as 1.6 will be rounded up). Therefore I use the Math.Floor method.

I made a generic routine posted at IntToByteArray that you could use like:
var yearInBytes = ConvertBigIntToBcd(2010, 2);

static byte[] IntToBCD(int input) {
byte[] bcd = new byte[] {
(byte)(input>> 8),
(byte)(input& 0x00FF)
};
return bcd;
}

Fastest way to calculate sum of bits in byte array

I have two byte arrays with the same length. I need to perform XOR operation between each byte and after this calculate sum of bits.
For example:
11110000^01010101 = 10100101 -> so 1+1+1+1 = 4
I need do the same operation for each element in byte array.

Use a lookup table. There are only 256 possible values after XORing, so it's not exactly going to take a long time. Unlike izb's solution though, I wouldn't suggest manually putting all the values in though - compute the lookup table once at startup using one of the looping answers.
For example:
public static class ByteArrayHelpers
{
private static readonly int[] LookupTable =
Enumerable.Range(0, 256).Select(CountBits).ToArray();
private static int CountBits(int value)
{
int count = 0;
for (int i=0; i < 8; i++)
{
count += (value >> i) & 1;
}
return count;
}
public static int CountBitsAfterXor(byte[] array)
{
int xor = 0;
foreach (byte b in array)
{
xor ^= b;
}
return LookupTable[xor];
}
}
(You could make it an extension method if you really wanted...)
Note the use of byte[] in the CountBitsAfterXor method - you could make it an IEnumerable<byte> for more generality, but iterating over an array (which is known to be an array at compile-time) will be faster. Probably only microscopically faster, but hey, you asked for the fastest way :)
I would almost certainly actually express it as
public static int CountBitsAfterXor(IEnumerable<byte> data)
in real life, but see which works better for you.
Also note the type of the xor variable as an int. In fact, there's no XOR operator defined for byte values, and if you made xor a byte it would still compile due to the nature of compound assignment operators, but it would be performing a cast on each iteration - at least in the IL. It's quite possible that the JIT would take care of this, but there's no need to even ask it to :)

Fastest way would probably be a 256-element lookup table...
int[] lut
{
/*0x00*/ 0,
/*0x01*/ 1,
/*0x02*/ 1,
/*0x03*/ 2
...
/*0xFE*/ 7,
/*0xFF*/ 8
}
e.g.
11110000^01010101 = 10100101 -> lut[165] == 4

This is more commonly referred to as bit counting. There are literally dozens of different algorithms for doing this. Here is one site which lists a few of the more well known methods. There are even CPU specific instructions for doing this.
Theorectically, Microsoft could add a BitArray.CountSetBits function that gets JITed with the best algorithm for that CPU architecture. I, for one, would welcome such an addition.

As I understood it you want to sum the bits of each XOR between the left and right bytes.
for (int b = 0; b < left.Length; b++) {
int num = left[b] ^ right[b];
int sum = 0;
for (int i = 0; i < 8; i++) {
sum += (num >> i) & 1;
}
// do something with sum maybe?
}

I'm not sure if you mean sum the bytes or the bits.
To sum the bits within a byte, this should work:
int nSum = 0;
for (int i=0; i<=7; i++)
{
nSum += (byte_val>>i) & 1;
}
You would then need the xoring, and array looping around this, of course.

The following should do
int BitXorAndSum(byte[] left, byte[] right) {
int sum = 0;
for ( var i = 0; i < left.Length; i++) {
sum += SumBits((byte)(left[i] ^ right[i]));
}
return sum;
}
int SumBits(byte b) {
var sum = 0;
for (var i = 0; i < 8; i++) {
sum += (0x1) & (b >> i);
}
return sum;
}

It can be rewritten as ulong and use unsafe pointer, but byte is easier to understand:
static int BitCount(byte num)
{
// 0x5 = 0101 (bit) 0x55 = 01010101
// 0x3 = 0011 (bit) 0x33 = 00110011
// 0xF = 1111 (bit) 0x0F = 00001111
uint count = num;
count = ((count >> 1) & 0x55) + (count & 0x55);
count = ((count >> 2) & 0x33) + (count & 0x33);
count = ((count >> 4) & 0xF0) + (count & 0x0F);
return (int)count;
}

A general function to count bits could look like:
int Count1(byte[] a)
{
int count = 0;
for (int i = 0; i < a.Length; i++)
{
byte b = a[i];
while (b != 0)
{
count++;
b = (byte)((int)b & (int)(b - 1));
}
}
return count;
}
The less 1-bits, the faster this works. It simply loops over each byte, and toggles the lowest 1 bit of that byte until the byte becomes 0. The castings are necessary so that the compiler stops complaining about the type widening and narrowing.
Your problem could then be solved by using this:
int Count1Xor(byte[] a1, byte[] a2)
{
int count = 0;
for (int i = 0; i < Math.Min(a1.Length, a2.Length); i++)
{
byte b = (byte)((int)a1[i] ^ (int)a2[i]);
while (b != 0)
{
count++;
b = (byte)((int)b & (int)(b - 1));
}
}
return count;
}

A lookup table should be the fastest, but if you want to do it without a lookup table, this will work for bytes in just 10 operations.
public static int BitCount(byte value) {
int v = value - ((value >> 1) & 0x55);
v = (v & 0x33) + ((v >> 2) & 0x33);
return ((v + (v >> 4) & 0x0F));
}
This is a byte version of the general bit counting function described at Sean Eron Anderson's bit fiddling site.