We Keep Coding

Removing leading 0s in a byte array

I have a byte array as follows -
byte[] arrByt = new byte[] { 0xF, 0xF, 0x11, 0x4 };
so in binary
arrByt = 00001111 00001111 00010001 000000100
Now I want to create a new byte array by removing leading 0s for each byte from arrByt
arrNewByt = 11111111 10001100 = { 0xFF, 0x8C };
I know that this can be done by converting the byte values into binary string values, removing the leading 0s, appending the values and converting back to byte values into the new array.
However this is a slow process for a large array.
Is there a faster way to achieve this (like logical operations, bit operations, or other efficient ways)?
Thanks.

This should do the job quite fast. At least only standard loops and operators. Give it a try, will also work for longer source arrays.
// source array of bytes
var arrByt = new byte[] {0xF, 0xF, 0x11, 0x4 };
// target array - first with the size of the source array
var targetArray = new byte[arrByt.Length];
// bit index in target array
// from left = byte 0, bit 7 = index 31; to the right = byte 4, bit 0 = index 0
var targetIdx = targetArray.Length * 8 - 1;
// go through all bytes of the source array from left to right
for (var i = 0; i < arrByt.Length; i++)
{
var startFound = false;
// go through all bits of the current byte from the highest to the lowest
for (var x = 7; x >= 0; x--)
{
// copy the bit if it is 1 or if there was already a 1 before in this byte
if (startFound || ((arrByt[i] >> x) & 1) == 1)
{
startFound = true;
// copy the bit from its position in the source array to its new position in the target array
targetArray[targetArray.Length - ((targetIdx / 8) + 1)] |= (byte) (((arrByt[i] >> x) & 1) << (targetIdx % 8));
// advance the bit + byte position in the target array one to the right
targetIdx--;
}
}
}
// resize the target array to only the bytes that were used above
Array.Resize(ref targetArray, (int)Math.Ceiling((targetArray.Length * 8 - (targetIdx + 1)) / 8d));
// write target array content to console
for (var i = 0; i < targetArray.Length; i++)
{
Console.Write($"{targetArray[i]:X} ");
}
// OUTPUT: FF 8C

If you are trying to find the location of the most-significant bit, you can do a log2() of the byte (and if you don't have log2, you can use log(x)/log(2) which is the same as log2(x))
For instance, the number 7, 6, 5, and 4 all have a '1' in the 3rd bit position (0111, 0110, 0101, 0100). The log2() of them are all between 2 and 2.8. Same thing happens for anything in the 4th bit, it will be a number between 3 and 3.9. So you can find out the Most Significant Bit by adding 1 to the log2() of the number (round down).
floor(log2(00001111)) + 1 == floor(3.9) + 1 == 3 + 1 == 4
You know how many bits are in a byte, so you can easily know the number of bits to shift left:
int numToShift = 8 - floor(log2(bytearray[0])) + 1;
shiftedValue = bytearray[0] << numToShift;
From there, it's just a matter of keeping track of how many outstanding bits (not pushed into a bytearray yet) you have, and then pushing some/all of them on.
The above code would only work for the first byte array. If you put this in a loop, the numToShift would maybe need to keep track of the latest empty slot to shift things into (you might have to shift right to fit in current byte array, and then use the leftovers to put into the start of the next byte array). So instead of doing "8 -" in the above code, you would maybe put the starting location. For instance, if only 3 bits were left to fill in the current byte array, you would do:
int numToShift = 3 - floor(log2(bytearray[0])) + 1;
So that number should be a variable:
int numToShift = bitsAvailableInCurrentByte - floor(log2(bytearray[0])) + 1;

Please check this code snippet. This might help you.
byte[] arrByt = new byte[] { 0xF, 0xF, 0x11, 0x4 };
byte[] result = new byte[arrByt.Length / 2];
var en = arrByt.GetEnumerator();
int count = 0;
byte result1 = 0;
int index = 0;
while (en.MoveNext())
{
count++;
byte item = (byte)en.Current;
if (count == 1)
{
while (item < 128)
{
item = (byte)(item << 1);
}
result1 ^= item;
}
if (count == 2)
{
count = 0;
result1 ^= item;
result[index] = result1;
index++;
result1 = 0;
}
}
foreach (var s in result)
{
Console.WriteLine(s.ToString("X"));
}

How do I read an audio file into an array in C#

I am trying to read a WAV file into a buffer array in c# but am having some problems. I am using a file stream to manage the audio file. Here is what I have...
FileStream WAVFile = new FileStream(#"test.wav", FileMode.Open);
//Buffer for the wave file...
BinaryReader WAVreader = new BinaryReader(WAVFile);
//Read information from the header.
chunkID = WAVreader.ReadInt32();
chunkSize = WAVreader.ReadInt32();
RiffFormat = WAVreader.ReadInt32();
...
channels = WAVreader.ReadInt16();
samplerate = WAVreader.ReadInt32();
byteRate = WAVreader.ReadInt32();
blockAllign = WAVreader.ReadInt16();
bitsPerSample = WAVreader.ReadInt16();
dataID = WAVreader.ReadInt32();
dataSize = WAVreader.ReadInt32();
The above is reading data from the WAV file header. Then I have this:
musicalData = WAVreader.ReadBytes(dataSize);
...to read the actual sample data but this is only 26 bytes for 60 seconds of audio. Is this correct?
How would I go about converting the byte[] array to double[]?

This code should do the trick. It converts a wave file to a normalized double array (-1 to 1), but it should be trivial to make it an int/short array instead (remove the /32768.0 bit and add 32768 instead). The right[] array will be set to null if the loaded wav file is found to be mono.
I can't claim it's completely bullet proof (potential off-by-one errors), but after creating a 65536 sample array, and creating a wave from -1 to 1, none of the samples appear to go 'through' the ceiling or floor.
// convert two bytes to one double in the range -1 to 1
static double bytesToDouble(byte firstByte, byte secondByte)
{
// convert two bytes to one short (little endian)
short s = (secondByte << 8) | firstByte;
// convert to range from -1 to (just below) 1
return s / 32768.0;
}
// Returns left and right double arrays. 'right' will be null if sound is mono.
public void openWav(string filename, out double[] left, out double[] right)
{
byte[] wav = File.ReadAllBytes(filename);
// Determine if mono or stereo
int channels = wav[22]; // Forget byte 23 as 99.999% of WAVs are 1 or 2 channels
// Get past all the other sub chunks to get to the data subchunk:
int pos = 12; // First Subchunk ID from 12 to 16
// Keep iterating until we find the data chunk (i.e. 64 61 74 61 ...... (i.e. 100 97 116 97 in decimal))
while(!(wav[pos]==100 && wav[pos+1]==97 && wav[pos+2]==116 && wav[pos+3]==97))
{
pos += 4;
int chunkSize = wav[pos] + wav[pos + 1] * 256 + wav[pos + 2] * 65536 + wav[pos + 3] * 16777216;
pos += 4 + chunkSize;
}
pos += 8;
// Pos is now positioned to start of actual sound data.
int samples = (wav.Length - pos)/2; // 2 bytes per sample (16 bit sound mono)
if (channels == 2)
{
samples /= 2; // 4 bytes per sample (16 bit stereo)
}
// Allocate memory (right will be null if only mono sound)
left = new double[samples];
if (channels == 2)
{
right = new double[samples];
}
else
{
right = null;
}
// Write to double array/s:
int i=0;
while (pos < length)
{
left[i] = bytesToDouble(wav[pos], wav[pos + 1]);
pos += 2;
if (channels == 2)
{
right[i] = bytesToDouble(wav[pos], wav[pos + 1]);
pos += 2;
}
i++;
}
}
If you wanted to use plugins, then assuming your WAV file contains 16 bit PCM (which is the most common), you can use NAudio to read it out into a byte array, and then copy that into an array of 16 bit integers for convenience. If it is stereo, the samples will be interleaved left, right.
using (WaveFileReader reader = new WaveFileReader("myfile.wav"))
{
Assert.AreEqual(16, reader.WaveFormat.BitsPerSample, "Only works with 16 bit audio");
byte[] buffer = new byte[reader.Length];
int read = reader.Read(buffer, 0, buffer.Length);
short[] sampleBuffer = new short[read / 2];
Buffer.BlockCopy(buffer, 0, sampleBuffer, 0, read);
}
I personally try to avoid using third-party libraries as much as I can. But the option is still there if you'd like the code to look better and easier to handle.

It's been a good 10-15 years since I touched WAVE file processing, but unlike the first impression that most people get about wave files as simple fixed-size header followed by PCM encoded audio data, WAVE files are a bit more complex RIFF format files.
Instead of re-engineering RIFF file processing and various cases, I would suggest to use interop and call on APIs that deal with RIFF file format.
You can see example of how to open and get data buffer (and meta information about what buffer is) in this example. It's in C++, but it shows use of mmioOpen, mmioRead, mmioDescend, mmioAscend APIs that you would need to use to get your hands on a proper audio buffer.

Encoding an array of bytes similar to Base64, but with arbitrary radix

Does the procedure have a name, where you take a stream of 8-bit bytes and slice them into n-bit snippets stored in 8-bit containers?
The idea is very similar to Base64 encoding, where you split the stream of 1's and 0's into 6-bit chunks (instead of 8), meaning each chunk can have a decimal value of 0 - 63, each of which is assigned a unique human-readable character. In my case, I'm not looking to assign each chunk a specific character.
For example, the input 8-bit bytes:
11100101 01101100 01010011 00001100 11000000 10111101
become the 6-bit snippets:
111001 010110 110001 010011 000011 001100 000010 111101
which are subsequently stored as:
00111001 00010110 00110001 00010011 00000011 00001100 00000010 00111101
or, optionally, with an offset of 1 bit:
01110010 00101100 01100010 00100110 00000110 00011000 00000100 01111010
or and offset of 2 bits:
11100100 01011000 11000100 01001100 00001100 00110000 00001000 11110100
I was looking to write an algorithm in C# to encode a byte array to an arbitrary length with arbitrary offset, and another algorithm to convert it back again.
After quite a lot of headache, I thought I had successfully written the forward algorithm to encode an array of bytes. It worked for all my test cases, but when started writing the reverse algorithm I realised the whole problem was a lot more complicated than I thought it would be, and, in fact, my forward algorithm didn't work where n < 4.
I wanted to write the algorithms with bitwise operators, which is the more proper and elegant solution. The other way would have been to dump the byte array as a long string of 1's and 0's to slice, but that would have been much, much slower.
Here is my forward algorithm that works for cases where n >= 4:
public static byte[] EncodeForward(byte[] input, int n, int offset = 0)
{
byte[] output = new byte[(int)Math.Ceiling(input.Length * 8.0 / n)];
output[0] = (byte)(input[0] >> (8 - n));
int p = 1;
int r = 8 - n;
for (int i = 1; i < input.Length; i++)
{
output[p++] = (byte)((byte)((byte)(input[i - 1] << (8 - r)) | (byte)(input[i] >> r)) >> (8 - n - offset));
if ((r += (8 - n)) == n)
{
output[p++] = (byte)(input[i] & (byte)(0xFF >> (8 - n)));
r = 0;
}
}
return output;
}
I originally conceived it for just the case of n = 7, so each output byte would be composed by parts of at most 7 input bytes. However in the case where n < 4, each output byte would be composed by up to, I think, ceil(8/n) input bytes, so the process is a little more complex than above.
I was hoping to write the forward and reverse algorithms myself, but, honestly, after all this time debugging and testing what I've written and now finding this approach will never work for n < 4, I'm just looking for something that works. These two algorithms are just a very small piece of the project I'm working on.
Does this encoding/decoding procedure have a name, and is there either a built-in way to do it in C# or is there a library that will do it?

You are almost there. You just need and intermediate 16-bit buffer and an unprocessed bits counter. Disclaimer: I don't know C#. The (pseudo) code below is written with C in mind; you may need some tweaking.
For encoding,
uint16_t mask = 0xffff << (16 - width);
uint16_t buffer = (input[0] << 8) | uint[1];
i += 2;
int remaining = 16;
while (i < input.Length) {
while (remaining >= width) {
output[p++] = (buffer & mask) >> (16 - width);
buffer <<= width;
remaining -= width;
}
// Refill the buffer. Since it is 16-bit wide there is a room
// for an _entire_ input byte.
buffer |= input[i++] << (8 - remaining);
remaining += 8;
}
emit_remaining_bits(buffer, remaining);
For decoding:
uint16_t buffer = 0;
int remaining = 16;
while (i < input.Length) {
while (remaining > 8) {
buffer |= input[i++] << (remaining - width);
remaining += width;
}
output[p++] = (buffer >> 8) & 0x00ff;
buffer <<= 8;
remaining += 8;
}

How is PNG CRC calculated exactly?

For the past 4 hours I've been studying the CRC algorithm. I'm pretty sure I got the hang of it already.
I'm trying to write a png encoder, and I don't wish to use external libraries for the CRC calculation, nor for the png encoding itself.
My program has been able to get the same CRC's as the examples on tutorials. Like on Wikipedia:
Using the same polynomial and message as in the example, I was able to produce the same result in both of the cases. I was able to do this for several other examples as well.
However, I can't seem to properly calculate the CRC of png files. I tested this by creating a blank, one pixel big .png file in paint, and using it's CRC as a comparision. I copied the data (and chunk name) from the IDAT chunk of the png (which the CRC is calculated from), and calculated it's CRC using the polynomial provided in the png specification.
The polynomial provided in the png specification is the following:
x32 + x26 + x23 + x22 + x16 + x12 + x11 + x10 + x8 + x7 + x5 + x4 + x2 + x + 1
Which should translate to:
1 00000100 11000001 00011101 10110111
Using that polynomial, I tried to get the CRC of the following data:
01001001 01000100 01000001 01010100
00011000 01010111 01100011 11101000
11101100 11101100 00000100 00000000
00000011 00111010 00000001 10011100
This is what I get:
01011111 11000101 01100001 01101000 (MSB First)
10111011 00010011 00101010 11001100 (LSB First)
This is what is the actual CRC:
11111010 00010110 10110110 11110111
I'm not exactly sure how to fix this, but my guess would be I'm doing this part from the specification wrong:
In PNG, the 32-bit CRC is initialized to all 1's, and then the data from each byte is processed from the least significant bit (1) to the most significant bit (128). After all the data bytes are processed, the CRC is inverted (its ones complement is taken). This value is transmitted (stored in the datastream) MSB first. For the purpose of separating into bytes and ordering, the least significant bit of the 32-bit CRC is defined to be the coefficient of the x31 term.
I'm not completely sure I can understand all of that.
Also, here is the code I use to get the CRC:
public BitArray GetCRC(BitArray data)
{
// Prepare the divident; Append the proper amount of zeros to the end
BitArray divident = new BitArray(data.Length + polynom.Length - 1);
for (int i = 0; i < divident.Length; i++)
{
if (i < data.Length)
{
divident[i] = data[i];
}
else
{
divident[i] = false;
}
}
// Calculate CRC
for (int i = 0; i < divident.Length - polynom.Length + 1; i++)
{
if (divident[i] && polynom[0])
{
for (int j = 0; j < polynom.Length; j++)
{
if ((divident[i + j] && polynom[j]) || (!divident[i + j] && !polynom[j]))
{
divident[i + j] = false;
}
else
{
divident[i + j] = true;
}
}
}
}
// Strip the CRC off the divident
BitArray crc = new BitArray(polynom.Length - 1);
for (int i = data.Length, j = 0; i < divident.Length; i++, j++)
{
crc[j] = divident[i];
}
return crc;
}
So, how do I fix this to match the PNG specification?

You can find a complete implementation of the CRC calculation (and PNG encoding in general) in this public domain code:
static uint[] crcTable;
// Stores a running CRC (initialized with the CRC of "IDAT" string). When
// you write this to the PNG, write as a big-endian value
static uint idatCrc = Crc32(new byte[] { (byte)'I', (byte)'D', (byte)'A', (byte)'T' }, 0, 4, 0);
// Call this function with the compressed image bytes,
// passing in idatCrc as the last parameter
private static uint Crc32(byte[] stream, int offset, int length, uint crc)
{
uint c;
if(crcTable==null){
crcTable=new uint[256];
for(uint n=0;n<=255;n++){
c = n;
for(var k=0;k<=7;k++){
if((c & 1) == 1)
c = 0xEDB88320^((c>>1)&0x7FFFFFFF);
else
c = ((c>>1)&0x7FFFFFFF);
}
crcTable[n] = c;
}
}
c = crc^0xffffffff;
var endOffset=offset+length;
for(var i=offset;i<endOffset;i++){
c = crcTable[(c^stream[i]) & 255]^((c>>8)&0xFFFFFF);
}
return c^0xffffffff;
}
1 https://web.archive.org/web/20150825201508/http://upokecenter.dreamhosters.com/articles/png-image-encoder-in-c/

C# - Converting a Sequence of Numbers into Bytes

I am trying to send a UDP packet of bytes corresponding to the numbers 1-1000 in sequence. How do I convert each number (1,2,3,4,...,998,999,1000) into the minimum number of bytes required and put them in a sequence that I can send as a UDP packet?
I've tried the following with no success. Any help would be greatly appreciated!
List<byte> byteList = new List<byte>();
for (int i = 1; i <= 255; i++)
{
byte[] nByte = BitConverter.GetBytes((byte)i);
foreach (byte b in nByte)
{
byteList.Add(b);
}
}
for (int g = 256; g <= 1000; g++)
{
UInt16 st = Convert.ToUInt16(g);
byte[] xByte = BitConverter.GetBytes(st);
foreach (byte c in xByte)
{
byteList.Add(c);
}
}
byte[] sendMsg = byteList.ToArray();
Thank you.

You need to use :
BitConverter.GetBytes(INTEGER);

Think about how you are going to be able to tell the difference between:
260, 1 -> 0x1, 0x4, 0x1
1, 4, 1 -> 0x1, 0x4, 0x1
If you use one byte for numbers up to 255 and two bytes for the numbers 256-1000, you won't be able to work out at the other end which number corresponds to what.
If you just need to encode them as described without worrying about how they are decoded, it smacks to me of a contrived homework assignment or test, and I'm uninclined to solve it for you.

I think you are looking for something along the lines of a 7-bit encoded integer:
protected void Write7BitEncodedInt(int value)
{
uint num = (uint) value;
while (num >= 0x80)
{
this.Write((byte) (num | 0x80));
num = num >> 7;
}
this.Write((byte) num);
}
(taken from System.IO.BinaryWriter.Write(String)).
The reverse is found in the System.IO.BinaryReader class and looks something like this:
protected internal int Read7BitEncodedInt()
{
byte num3;
int num = 0;
int num2 = 0;
do
{
if (num2 == 0x23)
{
throw new FormatException(Environment.GetResourceString("Format_Bad7BitInt32"));
}
num3 = this.ReadByte();
num |= (num3 & 0x7f) << num2;
num2 += 7;
}
while ((num3 & 0x80) != 0);
return num;
}
I do hope this is not homework, even though is really smells like it.
EDIT:
Ok, so to put it all together for you:
using System;
using System.IO;
namespace EncodedNumbers
{
class Program
{
protected static void Write7BitEncodedInt(BinaryWriter bin, int value)
{
uint num = (uint)value;
while (num >= 0x80)
{
bin.Write((byte)(num | 0x80));
num = num >> 7;
}
bin.Write((byte)num);
}
static void Main(string[] args)
{
MemoryStream ms = new MemoryStream();
BinaryWriter bin = new BinaryWriter(ms);
for(int i = 1; i < 1000; i++)
{
Write7BitEncodedInt(bin, i);
}
byte[] data = ms.ToArray();
int size = data.Length;
Console.WriteLine("Total # of Bytes = " + size);
Console.ReadLine();
}
}
}
The total size I get is 1871 bytes for numbers 1-1000.
Btw, could you simply state whether or not this is homework? Obviously, we will still help either way. But we would much rather you try a little harder so you can actually learn for yourself.
EDIT #2:
If you want to just pack them in ignoring the ability to decode them back, you can do something like this:
protected static void WriteMinimumInt(BinaryWriter bin, int value)
{
byte[] bytes = BitConverter.GetBytes(value);
int skip = bytes.Length-1;
while (bytes[skip] == 0)
{
skip--;
}
for (int i = 0; i <= skip; i++)
{
bin.Write(bytes[i]);
}
}
This ignores any bytes that are zero (from MSB to LSB). So for 0-255 it will use one byte.
As states elsewhere, this will not allow you to decode the data back since the stream is now ambiguous. As a side note, this approach crams it down to 1743 bytes (as opposed to 1871 using 7-bit encoding).

A byte can only hold 256 distinct values, so you cannot store the numbers above 255 in one byte. The easiest way would be to use short, which is 16 bits. If you realy need to conserve space, you can use 10 bit numbers and pack that into a byte array ( 10 bits = 2^10 = 1024 possible values).

Naively (also, untested):
List<byte> bytes = new List<byte>();
for (int i = 1; i <= 1000; i++)
{
byte[] nByte = BitConverter.GetBytes(i);
foreach(byte b in nByte) bytes.Add(b);
}
byte[] byteStream = bytes.ToArray();
Will give you a stream of bytes were each group of 4 bytes is a number [1, 1000].
You might be tempted to do some work so that i < 256 take a single byte, i < 65535 take two bytes, etc. However, if you do this you can't read the values out of the stream. Instead, you'd add length encoding or sentinels bits or something of the like.
I'd say, don't. Just compress the stream, either using a built-in class, or gin up a Huffman encoding implementation using an agree'd upon set of frequencies.