I have two byte arrays, they have variable length but always add up to 8 bytes. These need to be combined into a long. I can do this with creating a byte array and copying the required data. But I was thinking that this should also be possible through bit-shifting. I've been trying this (simplified with just one length):
var bytes1 = new byte[] { 1, 2, 3, 4, 5, 6, 7 };
var bytes2 = new byte[] { 8 };
unsafe
{
fixed (byte* b1 = bytes1)
{
fixed (byte* b2 = bytes2)
{
ulong* bl1 = (ulong*)b1;
ulong v = (*bl1<< 8) | (*b2);
var bytes = bytes1.Concat(bytes2).ToArray();
// These two are different:
Console.WriteLine(v);
Console.WriteLine(BitConverter.ToUInt64(bytes, 0));
}
}
}
I'm aware that Concat works, but I'd like to this to work too.
First of all, (ulong*)b1 is an out of bounds read because the array has length 7 and sizeof(ulong) == 8. The next read is also broken in that way. Alignment is also a problem. I don't see a way to rescue that approach. You could read 4 bytes, then 2 bytes, then 1 byte if you really are looking for performance.
I'd loop over the arrays and shift in each byte:
ulong result = 0;
void MergeArray(byte[] bytes) {
foreach (var b in bytes) {
result = result << 8 | (ulong)b;
}
}
MergeArray(bytes1);
MergeArray(bytes2);
Using a local function for code sharing.
You can improve performance by taking 4 bytes as the first chunk if the array length supports a read of that size. Then, fetch 2, then fetch 1. That way there is not even a loop and the number of operations is minimized.
Whether this is good or not depends on your need for performance which must be traded off with code legibility.
Related
My application receives data from a serial port, which is send in packets. The packets are defined as following
1 byte - Identifier
2 bytes - lenght of data
n bytes - data
1 bytes - Checksum
For example if the length is specified as 508 there will be 508 bytes, which would be 127 uint32_t values.
Currently I use the following code to assemble the uint32_t values from the data that is sent in bytes:
private UInt32[] number_array = new UInt32[16384];
private void decodePacket(int startpos, byte[] data, int lenght)
{
/* Starting position */
int pos = startpos;
for(int i=0; i<lenght; i++)
{
/* Convert 4 bytes to one uint32_t value */
int value = data[i] | data[i + 1]<<8 | data[i + 2]<<16 | data[i + 3]<<24;
/* Write to array */
number_array[pos] = Convert.ToUInt32(value);
/* Advance i by 4 (bytes */
i += 4;
/* Advance pos */
pos++;
}
}
It does work fine, but I'm thinking it's very inefficient. There are usually 16384 uint32_t values to process, so this function is called a lot of times.
Is there a more efficient / faster way to do this?
Look at this simple code:
static void Main(string[] args)
{
byte[] data = new byte[]
{
1, 10, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 5
};
byte id = data[0];
byte[] len = new byte[4];
Array.Copy(data, 1, len, 0, 2);
int dataLen = BitConverter.ToInt32(len, 0);
byte[] dataRead = new byte[dataLen];
Array.Copy(data, 3, dataRead, 0, dataLen);
byte checksum = data[data.Length - 1];
Console.ReadKey();
}
Now, first you get identifier that is on the first pos.
Next, you get length of the data. You have to create 4 byte array to copy values from data array to this new array. It should be 4 bytes, because you would like to convert those bytes into Int32. You could have this array 2 bytes length and convert it to Int16, but Int32 should have better performance.
So, when you have length data in this len array, you can convert those values into Int32. But be careful of endianes. It may be different.
At the end you create an array that will contain all the REAL data. And then copy the real data to new array.
This solution should be faster than yours. But... Be careful of endianess.
Here is the best possible way using C#
private void decodePacket(int startpos, byte[] data, int lenght)
{
/* Starting position */
int pos = startpos;
for (int i = 0; i < lenght; i += 4)
{
/* Convert 4 bytes to one uint32_t value */
int value = BitConverter.ToInt32(data, i);
/* Write to array */
number_array[pos] = Convert.ToUInt32(value);
/* Advance pos */
pos++;
}
}
This is same code as in question but with two changes.
Index i was being incremented at two places which was resulting in increments of 5 instead of 4. Changed it to just one increment of 4.
Use of BitConveter instead of bitwise logic. Although it might not provide any significant performance boost. It is better to use BitConverter for platform independence.
UPDATE
In case your bytes are stored at 32-bit aligned memory addresses BitConverter provides you with maximum performance conversion. But in C# you cannot guarantee memory location alignment. In that case bit shifting is the only way.
In case of bit shifting BitConverter also uses same logic for little endian systems as shown in question. But, it can help keeping your code platform independent by using another bit shifting pattern for big endian systems.
I have this List of integers { 20, 40 } and I want to get the length of the total number of bytes in the list
So since there are two integers, each integer is 4 bytes, therefore the total length of bytes in this case is 8.
If there was 3 integers the answer would be 12.
My idea was to loop through each integer and get the byte of the each integer using Bitconverter but I can't seem to append to the byte array.
var bytes = new byte[] { };
foreach (var num in nums)
{
bytes = BitConverter.GetBytes(num);
}
Then we would get the length of the bytes array but this method just does 1 value.
If you just want the number of bytes: sizeof(int) * nums.Count (or sizeof(int) * nums.Capacity if you want the size of the oversized backing array).
Assuming you want the data:
You can't use BitConverter to append bytes; you could copy each in turn, but honestly, that's not a good idea; you could loop applying offsets and using other APIs to write into the existing buffer, but - other tools exist; consider:
var nums = new List<int> { 20, 40 };
var arr = MemoryMarshal.Cast<int, byte>(CollectionsMarshal.AsSpan(nums)).ToArray();
// this is just a lazy way of showing the output
Console.WriteLine(BitConverter.ToString(arr));
Note that this is CPU-endian; if you need it to work identically on all CPUs, you'll need to use an endian-aware approach, for example BinaryPrimitives. For example:
byte[] arr = new byte[sizeof(int) * nums.Count];
var span = new Span<byte>(arr);
foreach (var num in nums)
{
BinaryPrimitives.WriteInt32BigEndian(span, num);
span = span.Slice(4);
}
I need to combine two Bytes into one int value.
I receive from my camera a 16bit Image were two successive bytes have the intensity value of one pixel. My goal is to combine these two bytes into one "int" vale.
I manage to do this using the following code:
for (int i = 0; i < VectorLength * 2; i = i + 2)
{
NewImageVector[ImagePointer] = ((int)(buffer.Array[i + 1]) << 8) | ((int)(buffer.Array[i]));
ImagePointer++;
}
My image is 1280*960 so VectorLength==1228800 and the incomming buffer size is 2*1228800=2457600 elements...
Is there any way that I can speed this up?
Maybe there is another way so I don't need to use a for-loop.
Thank you
You could use the equivalent to the union of c. Im not sure if faster, but more elegant:
[StructLayout(LayoutKind.Explicit)]
struct byte_array
{
[FieldOffset(0)]
public byte byte1;
[FieldOffset(1)]
public byte byte2;
[FieldOffset(0)]
public short int0;
}
use it like this:
byte_array ba = new byte_array();
//insert the two bytes
ba.byte1 = (byte)(buffer.Array[i]);
ba.byte2 = (byte)(buffer.Array[i + 1]);
//get the integer
NewImageVector[ImagePointer] = ba.int1;
You can fill your two bytes and use the int. To find the faster way take the StopWatch-Class and compare the two ways like this:
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//The code
stopWatch.Stop();
MessageBox.Show(stopWatch.ElapsedTicks.ToString()); //Or milliseconds ,...
Assuming you can (re-)define NewImageVector as a short[], and every two consecutive bytes in Buffer should be transformed into a short (which basically what you're doing now, only you cast to an int afterwards), you can use Buffer.BlockCopy to do it for you.
As the documentation tells, you Buffer.BlockCopy copies bytes from one array to another, so in order to copy your bytes in buffer you need to do the following:
Buffer.BlockCopy(Buffer, 0, NewImageVector, 0, [NumberOfExpectedShorts] * 2)
This tells BlockCopy that you want to start copying bytes from Buffer, starting at index 0, to NewImageVector starting at index 0, and you want to copy [NumberOfExpectedShorts] * 2 bytes (since every short is two bytes long).
No loops, but it does depend on the ability of using a short[] array instead of an int[] array (and indeed, on using an array to begin with).
Note that this also requires the bytes in Buffer to be in little-endian order (i.e. Buffer[index] contains the low byte, buffer[index + 1] the high byte).
You can achieve a small performance increase by using unsafe pointers to iterate the arrays. The following code assumes that source is the input byte array (buffer.Array in your case). It also assumes that source has an even number of elements. In production code you would obviously have to check these things.
int[] output = new int[source.Length / 2];
fixed (byte* pSource = source)
fixed (int* pDestination = output)
{
byte* sourceIterator = pSource;
int* destIterator = pDestination;
for (int i = 0; i < output.Length; i++)
{
(*destIterator) = ((*sourceIterator) | (*(sourceIterator + 1) << 8));
destIterator++;
sourceIterator += 2;
}
}
return output;
currently im working on a solution for a prime-number calculator/checker. The algorythm is already working and verry efficient (0,359 seconds for the first 9012330 primes). Here is a part of the upper region where everything is declared:
const uint anz = 50000000;
uint a = 3, b = 4, c = 3, d = 13, e = 12, f = 13, g = 28, h = 32;
bool[,] prim = new bool[8, anz / 10];
uint max = 3 * (uint)(anz / (Math.Log(anz) - 1.08366));
uint[] p = new uint[max];
Now I wanted to go to the next level and use ulong's instead of uint's to cover a larger area (you can see that already), where i tapped into my problem: the bool-array.
Like everybody should know, bool's have the length of a byte what takes a lot of memory when creating the array... So I'm searching for a more resource-friendly way to do that.
My first idea was a bit-array -> not byte! <- to save the bool's, but haven't figured out how to do that by now. So if someone ever did something like this, I would appreciate any kind of tips and solutions. Thanks in advance :)
You can use BitArray collection:
http://msdn.microsoft.com/en-us/library/system.collections.bitarray(v=vs.110).aspx
MSDN Description:
Manages a compact array of bit values, which are represented as Booleans, where true indicates that the bit is on (1) and false indicates the bit is off (0).
You can (and should) use well tested and well known libraries.
But if you're looking to learn something (as it seems to be the case) you can do it yourself.
Another reason you may want to use a custom bit array is to use the hard drive to store the array, which comes in handy when calculating primes. To do this you'd need to further split addr, for example lowest 3 bits for the mask, next 28 bits for 256MB of in-memory storage, and from there on - a file name for a buffer file.
Yet another reason for custom bit array is to compress the memory use when specifically searching for primes. After all more than half of your bits will be 'false' because the numbers corresponding to them would be even, so in fact you can both speed up your calculation AND reduce memory requirements if you don't even store the even bits. You can do that by changing the way addr is interpreted. Further more you can also exclude numbers divisible by 3 (only 2 out of every 6 numbers has a chance of being prime) thus reducing memory requirements by 60% compared to plain bit array.
Notice the use of shift and logical operators to make the code a bit more efficient.
byte mask = (byte)(1 << (int)(addr & 7)); for example can be written as
byte mask = (byte)(1 << (int)(addr % 8));
and addr >> 3 can be written as addr / 8
Testing shift/logical operators vs division shows 2.6s vs 4.8s in favor of shift/logical for 200000000 operations.
Here's the code:
void Main()
{
var barr = new BitArray(10);
barr[4] = true;
Console.WriteLine("Is it "+barr[4]);
Console.WriteLine("Is it Not "+barr[5]);
}
public class BitArray{
private readonly byte[] _buffer;
public bool this[long addr]{
get{
byte mask = (byte)(1 << (int)(addr & 7));
byte val = _buffer[(int)(addr >> 3)];
bool bit = (val & mask) == mask;
return bit;
}
set{
byte mask = (byte) ((value ? 1:0) << (int)(addr & 7));
int offs = (int)addr >> 3;
_buffer[offs] = (byte)(_buffer[offs] | mask);
}
}
public BitArray(long size){
_buffer = new byte[size/8 + 1]; // define a byte buffer sized to hold 8 bools per byte. The spare +1 is to avoid dealing with rounding.
}
}
I have a byte array. It contains 24 bit signed integers stored lsb to msb. The array could hold up to 4mb of data. The integers will be converted to 32 bit signed integers to be used in the application. I would like to hear about possible strategies for conversion and sampling of this data.
One thing I need to do with the data is graph it. With sequential sampling, I am worried about loosing some of the important peaks and valleys in the data. I also want to do some calculations to determine the highest and lowest values.
Given what I need to do, are there any algorithms or ways of doing things that will help me achieve my goal quickly and efficiently?
If your input has to be 3 byte ints, then you can convert to 4 byte ints as follows:
byte[] input = new byte[] {1, 2, 3, 4, 5, 6, 7, 8, 9}; //sample data
byte[] buffer = new byte[4]; //4 byte buffer for conversion from 3-> 4 byte int
int[] output = new int[input.Length / 3];
for (int i = 0, j = 0; i < input.Length; i += 3, j++)
{
Buffer.BlockCopy(input, i, buffer, 0, 3);
int signed32 = BitConverter.ToInt32(buffer, 0);
output[j] = signed32;
}
Edit
Fixed block copy for little endian.
I would suggest you to convert the byte array to an int[]. That way, you can work with it easily and today's computers can work with 32-bit integers much better than if you had to work with bytes that represent 24-bit integers all the time.
You should use the regular sized ints.
Storage is cheap (especially if you only need ~4MB of data) and if you are going to convert them to int32's for manipulation it's better if they're in that format from the beginning.
If the conversion will actually produce another array of int32s then you've just doubled the memory footprint. If you convert individual elements you've just increased execution time.
Best use the native int size.
It might be easier to implement and for future developers to understand if you use the bytes directly (3 at a time).
// If you're reading from a file, you don't have to read the whole array.
// Just read a large chunk (like 3 * 1024) bytes at a time (so it's divisible by 3).
byte [] data = new []{1,2,3, 4,5,6, 7,8,9};
int [] values = new [data.Length /3];
int min = int.MaxValue;
int max = int.MaxValue;
for (int i = 0,j = 1; i < data.Length - 2; i += 3, j++)
{
byte b1 = data[i];
byte b2 = data[i+1];
byte b3 = data[i+2];
// Are we dealing with 2's compliment or a sign bit? Let's assume sign bit.
int sign = b3 >> 7 == 1 ? -1 : 1;
int value = sign * ((int) (b3 <1)>1)<<16 + b2 << 8 + b1;
values[j] = value;
max = max > value ? max : value;
min = min < value ? min : value;
}