I have a byte array. It contains 24 bit signed integers stored lsb to msb. The array could hold up to 4mb of data. The integers will be converted to 32 bit signed integers to be used in the application. I would like to hear about possible strategies for conversion and sampling of this data.
One thing I need to do with the data is graph it. With sequential sampling, I am worried about loosing some of the important peaks and valleys in the data. I also want to do some calculations to determine the highest and lowest values.
Given what I need to do, are there any algorithms or ways of doing things that will help me achieve my goal quickly and efficiently?
If your input has to be 3 byte ints, then you can convert to 4 byte ints as follows:
byte[] input = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9}; //sample data
byte[] buffer = new byte[4]; //4 byte buffer for conversion from 3-> 4 byte int
int[] output = new int[input.Length / 3];
for (int i = 0, j = 0; i < input.Length; i += 3, j++)
{
Buffer.BlockCopy(input, i, buffer, 0, 3);
int signed32 = BitConverter.ToInt32(buffer, 0);
output[j] = signed32;
}
Edit
Fixed block copy for little endian.
I would suggest you to convert the byte array to an int[]. That way, you can work with it easily and today's computers can work with 32-bit integers much better than if you had to work with bytes that represent 24-bit integers all the time.
You should use the regular sized ints.
Storage is cheap (especially if you only need ~4MB of data) and if you are going to convert them to int32's for manipulation it's better if they're in that format from the beginning.
If the conversion will actually produce another array of int32s then you've just doubled the memory footprint. If you convert individual elements you've just increased execution time.
Best use the native int size.
It might be easier to implement and for future developers to understand if you use the bytes directly (3 at a time).
// If you're reading from a file, you don't have to read the whole array.
// Just read a large chunk (like 3 * 1024) bytes at a time (so it's divisible by 3).
byte [] data = new []{1,2,3, 4,5,6, 7,8,9};
int [] values = new [data.Length /3];
int min = int.MaxValue;
int max = int.MaxValue;
for (int i = 0,j = 1; i < data.Length - 2; i += 3, j++)
{
byte b1 = data[i];
byte b2 = data[i+1];
byte b3 = data[i+2];
// Are we dealing with 2's compliment or a sign bit? Let's assume sign bit.
int sign = b3 >> 7 == 1 ? -1 : 1;
int value = sign * ((int) (b3 <1)>1)<<16 + b2 << 8 + b1;
values[j] = value;
max = max > value ? max : value;
min = min < value ? min : value;
}
Related
My application receives data from a serial port, which is send in packets. The packets are defined as following
1 byte - Identifier
2 bytes - lenght of data
n bytes - data
1 bytes - Checksum
For example if the length is specified as 508 there will be 508 bytes, which would be 127 uint32_t values.
Currently I use the following code to assemble the uint32_t values from the data that is sent in bytes:
private UInt32[] number_array = new UInt32[16384];
private void decodePacket(int startpos, byte[] data, int lenght)
{
/* Starting position */
int pos = startpos;
for(int i=0; i<lenght; i++)
{
/* Convert 4 bytes to one uint32_t value */
int value = data[i] | data[i + 1]<<8 | data[i + 2]<<16 | data[i + 3]<<24;
/* Write to array */
number_array[pos] = Convert.ToUInt32(value);
/* Advance i by 4 (bytes */
i += 4;
/* Advance pos */
pos++;
}
}
It does work fine, but I'm thinking it's very inefficient. There are usually 16384 uint32_t values to process, so this function is called a lot of times.
Is there a more efficient / faster way to do this?
Look at this simple code:
static void Main(string[] args)
{
byte[] data = new byte[]
{
1, 10, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 5
};
byte id = data[0];
byte[] len = new byte[4];
Array.Copy(data, 1, len, 0, 2);
int dataLen = BitConverter.ToInt32(len, 0);
byte[] dataRead = new byte[dataLen];
Array.Copy(data, 3, dataRead, 0, dataLen);
byte checksum = data[data.Length - 1];
Console.ReadKey();
}
Now, first you get identifier that is on the first pos.
Next, you get length of the data. You have to create 4 byte array to copy values from data array to this new array. It should be 4 bytes, because you would like to convert those bytes into Int32. You could have this array 2 bytes length and convert it to Int16, but Int32 should have better performance.
So, when you have length data in this len array, you can convert those values into Int32. But be careful of endianes. It may be different.
At the end you create an array that will contain all the REAL data. And then copy the real data to new array.
This solution should be faster than yours. But... Be careful of endianess.
Here is the best possible way using C#
private void decodePacket(int startpos, byte[] data, int lenght)
{
/* Starting position */
int pos = startpos;
for (int i = 0; i < lenght; i += 4)
{
/* Convert 4 bytes to one uint32_t value */
int value = BitConverter.ToInt32(data, i);
/* Write to array */
number_array[pos] = Convert.ToUInt32(value);
/* Advance pos */
pos++;
}
}
This is same code as in question but with two changes.
Index i was being incremented at two places which was resulting in increments of 5 instead of 4. Changed it to just one increment of 4.
Use of BitConveter instead of bitwise logic. Although it might not provide any significant performance boost. It is better to use BitConverter for platform independence.
UPDATE
In case your bytes are stored at 32-bit aligned memory addresses BitConverter provides you with maximum performance conversion. But in C# you cannot guarantee memory location alignment. In that case bit shifting is the only way.
In case of bit shifting BitConverter also uses same logic for little endian systems as shown in question. But, it can help keeping your code platform independent by using another bit shifting pattern for big endian systems.
I have two byte arrays, they have variable length but always add up to 8 bytes. These need to be combined into a long. I can do this with creating a byte array and copying the required data. But I was thinking that this should also be possible through bit-shifting. I've been trying this (simplified with just one length):
var bytes1 = new byte[] { 1, 2, 3, 4, 5, 6, 7 };
var bytes2 = new byte[] { 8 };
unsafe
{
fixed (byte* b1 = bytes1)
{
fixed (byte* b2 = bytes2)
{
ulong* bl1 = (ulong*)b1;
ulong v = (*bl1<< 8) | (*b2);
var bytes = bytes1.Concat(bytes2).ToArray();
// These two are different:
Console.WriteLine(v);
Console.WriteLine(BitConverter.ToUInt64(bytes, 0));
}
}
}
I'm aware that Concat works, but I'd like to this to work too.
First of all, (ulong*)b1 is an out of bounds read because the array has length 7 and sizeof(ulong) == 8. The next read is also broken in that way. Alignment is also a problem. I don't see a way to rescue that approach. You could read 4 bytes, then 2 bytes, then 1 byte if you really are looking for performance.
I'd loop over the arrays and shift in each byte:
ulong result = 0;
void MergeArray(byte[] bytes) {
foreach (var b in bytes) {
result = result << 8 | (ulong)b;
}
}
MergeArray(bytes1);
MergeArray(bytes2);
Using a local function for code sharing.
You can improve performance by taking 4 bytes as the first chunk if the array length supports a read of that size. Then, fetch 2, then fetch 1. That way there is not even a loop and the number of operations is minimized.
Whether this is good or not depends on your need for performance which must be traded off with code legibility.
I need to combine two Bytes into one int value.
I receive from my camera a 16bit Image were two successive bytes have the intensity value of one pixel. My goal is to combine these two bytes into one "int" vale.
I manage to do this using the following code:
for (int i = 0; i < VectorLength * 2; i = i + 2)
{
NewImageVector[ImagePointer] = ((int)(buffer.Array[i + 1]) << 8) | ((int)(buffer.Array[i]));
ImagePointer++;
}
My image is 1280*960 so VectorLength==1228800 and the incomming buffer size is 2*1228800=2457600 elements...
Is there any way that I can speed this up?
Maybe there is another way so I don't need to use a for-loop.
Thank you
You could use the equivalent to the union of c. Im not sure if faster, but more elegant:
[StructLayout(LayoutKind.Explicit)]
struct byte_array
{
[FieldOffset(0)]
public byte byte1;
[FieldOffset(1)]
public byte byte2;
[FieldOffset(0)]
public short int0;
}
use it like this:
byte_array ba = new byte_array();
//insert the two bytes
ba.byte1 = (byte)(buffer.Array[i]);
ba.byte2 = (byte)(buffer.Array[i + 1]);
//get the integer
NewImageVector[ImagePointer] = ba.int1;
You can fill your two bytes and use the int. To find the faster way take the StopWatch-Class and compare the two ways like this:
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//The code
stopWatch.Stop();
MessageBox.Show(stopWatch.ElapsedTicks.ToString()); //Or milliseconds ,...
Assuming you can (re-)define NewImageVector as a short[], and every two consecutive bytes in Buffer should be transformed into a short (which basically what you're doing now, only you cast to an int afterwards), you can use Buffer.BlockCopy to do it for you.
As the documentation tells, you Buffer.BlockCopy copies bytes from one array to another, so in order to copy your bytes in buffer you need to do the following:
Buffer.BlockCopy(Buffer, 0, NewImageVector, 0, [NumberOfExpectedShorts] * 2)
This tells BlockCopy that you want to start copying bytes from Buffer, starting at index 0, to NewImageVector starting at index 0, and you want to copy [NumberOfExpectedShorts] * 2 bytes (since every short is two bytes long).
No loops, but it does depend on the ability of using a short[] array instead of an int[] array (and indeed, on using an array to begin with).
Note that this also requires the bytes in Buffer to be in little-endian order (i.e. Buffer[index] contains the low byte, buffer[index + 1] the high byte).
You can achieve a small performance increase by using unsafe pointers to iterate the arrays. The following code assumes that source is the input byte array (buffer.Array in your case). It also assumes that source has an even number of elements. In production code you would obviously have to check these things.
int[] output = new int[source.Length / 2];
fixed (byte* pSource = source)
fixed (int* pDestination = output)
{
byte* sourceIterator = pSource;
int* destIterator = pDestination;
for (int i = 0; i < output.Length; i++)
{
(*destIterator) = ((*sourceIterator) | (*(sourceIterator + 1) << 8));
destIterator++;
sourceIterator += 2;
}
}
return output;
currently im working on a solution for a prime-number calculator/checker. The algorythm is already working and verry efficient (0,359 seconds for the first 9012330 primes). Here is a part of the upper region where everything is declared:
const uint anz = 50000000;
uint a = 3, b = 4, c = 3, d = 13, e = 12, f = 13, g = 28, h = 32;
bool[,] prim = new bool[8, anz / 10];
uint max = 3 * (uint)(anz / (Math.Log(anz) - 1.08366));
uint[] p = new uint[max];
Now I wanted to go to the next level and use ulong's instead of uint's to cover a larger area (you can see that already), where i tapped into my problem: the bool-array.
Like everybody should know, bool's have the length of a byte what takes a lot of memory when creating the array... So I'm searching for a more resource-friendly way to do that.
My first idea was a bit-array -> not byte! <- to save the bool's, but haven't figured out how to do that by now. So if someone ever did something like this, I would appreciate any kind of tips and solutions. Thanks in advance :)
You can use BitArray collection:
http://msdn.microsoft.com/en-us/library/system.collections.bitarray(v=vs.110).aspx
MSDN Description:
Manages a compact array of bit values, which are represented as Booleans, where true indicates that the bit is on (1) and false indicates the bit is off (0).
You can (and should) use well tested and well known libraries.
But if you're looking to learn something (as it seems to be the case) you can do it yourself.
Another reason you may want to use a custom bit array is to use the hard drive to store the array, which comes in handy when calculating primes. To do this you'd need to further split addr, for example lowest 3 bits for the mask, next 28 bits for 256MB of in-memory storage, and from there on - a file name for a buffer file.
Yet another reason for custom bit array is to compress the memory use when specifically searching for primes. After all more than half of your bits will be 'false' because the numbers corresponding to them would be even, so in fact you can both speed up your calculation AND reduce memory requirements if you don't even store the even bits. You can do that by changing the way addr is interpreted. Further more you can also exclude numbers divisible by 3 (only 2 out of every 6 numbers has a chance of being prime) thus reducing memory requirements by 60% compared to plain bit array.
Notice the use of shift and logical operators to make the code a bit more efficient.
byte mask = (byte)(1 << (int)(addr & 7)); for example can be written as
byte mask = (byte)(1 << (int)(addr % 8));
and addr >> 3 can be written as addr / 8
Testing shift/logical operators vs division shows 2.6s vs 4.8s in favor of shift/logical for 200000000 operations.
Here's the code:
void Main()
{
var barr = new BitArray(10);
barr[4] = true;
Console.WriteLine("Is it "+barr[4]);
Console.WriteLine("Is it Not "+barr[5]);
}
public class BitArray{
private readonly byte[] _buffer;
public bool this[long addr]{
get{
byte mask = (byte)(1 << (int)(addr & 7));
byte val = _buffer[(int)(addr >> 3)];
bool bit = (val & mask) == mask;
return bit;
}
set{
byte mask = (byte) ((value ? 1:0) << (int)(addr & 7));
int offs = (int)addr >> 3;
_buffer[offs] = (byte)(_buffer[offs] | mask);
}
}
public BitArray(long size){
_buffer = new byte[size/8 + 1]; // define a byte buffer sized to hold 8 bools per byte. The spare +1 is to avoid dealing with rounding.
}
}
I have this code for converting a byte[] to float[].
public float[] ConvertByteToFloat(byte[] array)
{
float[] floatArr = new float[array.Length / sizeof(float)];
int index = 0;
for (int i = 0; i < floatArr.Length; i++)
{
floatArr[i] = BitConverter.ToSingle(array, index);
index += sizeof(float);
}
return floatArr;
}
Problem is, I usually get a NaN result! Why should this be? I checked if there is data in the byte[] and the data seems to be fine. If it helps, an example of the values are:
new byte[] {
231,
255,
235,
255,
}
But this returns NaN (Not a Number) after conversion to float. What could be the problem? Are there other better ways of converting byte[] to float[]? I am sure that the values read into the buffer are correct since I compared it with my other program (which performs amplification for a .wav file).
If endianness is the problem, you should check the value of BitConverter.IsLittleEndian to determine if the bytes have to be reversed:
public static float[] ConvertByteToFloat(byte[] array) {
float[] floatArr = new float[array.Length / 4];
for (int i = 0; i < floatArr.Length; i++) {
if (BitConverter.IsLittleEndian) {
Array.Reverse(array, i * 4, 4);
}
floatArr[i] = BitConverter.ToSingle(array, i * 4);
}
return floatArr;
}
Isn't the problem that the 255 in the exponent represents NaN (see Wikipedia in the exponent section), so you should get a NaN. Try changing the last 255 to something else...
If it's the endianness that is wrong (you are reading big endian numbers) try these (be aware that they are "unsafe" so you have to check the unsafe flag in your project properties)
public static unsafe int ToInt32(byte[] value, int startIndex)
{
fixed (byte* numRef = &value[startIndex])
{
var num = (uint)((numRef[0] << 0x18) | (numRef[1] << 0x10) | (numRef[2] << 0x8) | numRef[3]);
return (int)num;
}
}
public static unsafe float ToSingle(byte[] value, int startIndex)
{
int val = ToInt32(value, startIndex);
return *(float*)&val;
}
I assume that your byte[] doesn't contain the binary representation of floats, but either a sequence of Int8s or Int16s. The examples you posted don't look like audio samples based on float (neither NaN nor -2.41E+24 are in the range -1 to 1. While some new audio formats might support floats outside that range, traditionally audio data consists of signed 8 or 16 bit integer samples.
Another thing you need to be aware of is that often the different channels are interleaved. For example it could contain the first sample for the left, then for the right channel, and then the second sample for both... So you need to separate the channels while parsing.
It's also possible, but uncommon that the samples are unsigned. In which case you need to remove the offset from the conversion functions.
So you first need to parse current position in the byte array into an Int8/16. And then convert that integer to a float in the range -1 to 1.
If the format is little endian you can use BitConverter. Another possibility that works with both endiannesses is getting two bytes manually and combining them with a bit-shift. I don't remember if little or big endian is common. So you need to try that yourself.
This can be done with functions like the following(I didn't test them):
float Int8ToFloat(Int8 i)
{
return ((i-Int8.MinValue)*(1f/0xFF))-0.5f;
}
float Int16ToFloat(Int16 i)
{
return ((i-Int16.MinValue)*(1f/0xFFFF))-0.5f;
}
Depends on how you want to convert the bytes to float. Start out by trying to pin-point what is actually stored in the byte array. Perhaps there is a file format specification? Other transformations in your code?
In the case each byte should be converted to a float between 0 and 255:
public float[] ConvertByteToFloat(byte[] array)
{
return array.Select(b => (float)b).ToArray();
}
If the bytes array contains binary representation of floats, there are several representation and if the representation stored in your file does not match the c# language standard floating point representation (IEEE 754) weird things like this will happen.