What is fastest way to write int[] array to byte[] array, but only LSB (last 8 bits)? I used for loop and bitmask bArray[i] = (byte)(iArray[i] & 0xFF), but array is very long (+900k) and this take about 50ms. Do you know maybe other faster option?
You can try parallelizing the workload:
Parallel.For(0,iArray.Length, i => bArray[i] = (byte)(iArray[i] & 0xFF));
This will spawn multiple threads to do the conversion. It tends to be faster on my machine, but, will sometimes take longer due to the overhead of spawning multiple threads.
What are you doing that 50ms is too slow?
Part of the slowness comes from the need to read more data than you need: when you do
array[i] & 0xFF
you read the entire four bytes of an int, only to drop three most significant ones.
You could avoid this overhead with unsafe code. Keep in mind that the approach below assumes little endian architecture:
static unsafe void CopyLsbUnsafe(int[] from, byte[] to) {
fixed (int* s = from) {
fixed (byte* d = to) {
byte* sb = (byte*) s;
int* db = (int*)d;
int* end = db + to.Length/4;
while (db != end) {
*db++ = (*(sb + 0) << 0)
| (*(sb + 4) << 8)
| (*(sb + 8) << 16)
| (*(sb + 12) << 24);
sb += 16;
}
}
}
}
The above code re-interprets the int array as an array of bytes, and the array of bytes as an array of integers. Then it reads every 4-th byte into the destination array using a pointer, writing to the destination in groups of four bytes using an integer assignment.
My testing shows a respectable 60% improvement over a simple loop.
Related
I need to combine two Bytes into one int value.
I receive from my camera a 16bit Image were two successive bytes have the intensity value of one pixel. My goal is to combine these two bytes into one "int" vale.
I manage to do this using the following code:
for (int i = 0; i < VectorLength * 2; i = i + 2)
{
NewImageVector[ImagePointer] = ((int)(buffer.Array[i + 1]) << 8) | ((int)(buffer.Array[i]));
ImagePointer++;
}
My image is 1280*960 so VectorLength==1228800 and the incomming buffer size is 2*1228800=2457600 elements...
Is there any way that I can speed this up?
Maybe there is another way so I don't need to use a for-loop.
Thank you
You could use the equivalent to the union of c. Im not sure if faster, but more elegant:
[StructLayout(LayoutKind.Explicit)]
struct byte_array
{
[FieldOffset(0)]
public byte byte1;
[FieldOffset(1)]
public byte byte2;
[FieldOffset(0)]
public short int0;
}
use it like this:
byte_array ba = new byte_array();
//insert the two bytes
ba.byte1 = (byte)(buffer.Array[i]);
ba.byte2 = (byte)(buffer.Array[i + 1]);
//get the integer
NewImageVector[ImagePointer] = ba.int1;
You can fill your two bytes and use the int. To find the faster way take the StopWatch-Class and compare the two ways like this:
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//The code
stopWatch.Stop();
MessageBox.Show(stopWatch.ElapsedTicks.ToString()); //Or milliseconds ,...
Assuming you can (re-)define NewImageVector as a short[], and every two consecutive bytes in Buffer should be transformed into a short (which basically what you're doing now, only you cast to an int afterwards), you can use Buffer.BlockCopy to do it for you.
As the documentation tells, you Buffer.BlockCopy copies bytes from one array to another, so in order to copy your bytes in buffer you need to do the following:
Buffer.BlockCopy(Buffer, 0, NewImageVector, 0, [NumberOfExpectedShorts] * 2)
This tells BlockCopy that you want to start copying bytes from Buffer, starting at index 0, to NewImageVector starting at index 0, and you want to copy [NumberOfExpectedShorts] * 2 bytes (since every short is two bytes long).
No loops, but it does depend on the ability of using a short[] array instead of an int[] array (and indeed, on using an array to begin with).
Note that this also requires the bytes in Buffer to be in little-endian order (i.e. Buffer[index] contains the low byte, buffer[index + 1] the high byte).
You can achieve a small performance increase by using unsafe pointers to iterate the arrays. The following code assumes that source is the input byte array (buffer.Array in your case). It also assumes that source has an even number of elements. In production code you would obviously have to check these things.
int[] output = new int[source.Length / 2];
fixed (byte* pSource = source)
fixed (int* pDestination = output)
{
byte* sourceIterator = pSource;
int* destIterator = pDestination;
for (int i = 0; i < output.Length; i++)
{
(*destIterator) = ((*sourceIterator) | (*(sourceIterator + 1) << 8));
destIterator++;
sourceIterator += 2;
}
}
return output;
I need to get values in UInt16 and UInt64 as Byte[]. At the moment I am using BitConverter.GetBytes, but this method gives me a new array instance each time.
I would like to use a method that allow me to "copy" those values to already existing arrays, something like:
.ToBytes(UInt64 value, Byte[] array, Int32 offset);
.ToBytes(UInt16 value, Byte[] array, Int32 offset);
I have been taking a look to the .NET source code with ILSpy, but I not very sure how this code works and how can I safely modify it to fit my requirement:
public unsafe static byte[] GetBytes(long value)
{
byte[] array = new byte[8];
fixed (byte* ptr = array)
{
*(long*)ptr = value;
}
return array;
}
Which would be the right way of accomplishing this?
Updated: I cannot use unsafe code. It should not create new array instances.
You can do like this:
static unsafe void ToBytes(ulong value, byte[] array, int offset)
{
fixed (byte* ptr = &array[offset])
*(ulong*)ptr = value;
}
Usage:
byte[] array = new byte[9];
ToBytes(0x1122334455667788, array, 1);
You can set offset only in bytes.
If you want managed way to do it:
static void ToBytes(ulong value, byte[] array, int offset)
{
byte[] valueBytes = BitConverter.GetBytes(value);
Array.Copy(valueBytes, 0, array, offset, valueBytes.Length);
}
Or you can fill values by yourself:
static void ToBytes(ulong value, byte[] array, int offset)
{
for (int i = 0; i < 8; i++)
{
array[offset + i] = (byte)value;
value >>= 8;
}
}
Now that .NET has added Span<T> support for better working with arrays, unmanaged memory, etc without excess allocations, they've also added System.Buffer.Binary.BinaryPrimitives.
This works as you would want, e.g. WriteUInt64BigEndian has this signature:
public static void WriteUInt64BigEndian (Span<byte> destination, ulong value);
which avoids allocating.
You say you want to avoid creating new arrays and you cannot use unsafe. Either use Ulugbek Umirov's answer with cached arrays (be careful with threading issues) or:
static void ToBytes(ulong value, byte[] array, int offset) {
unchecked {
array[offset + 0] = (byte)(value >> (8*7));
array[offset + 1] = (byte)(value >> (8*6));
array[offset + 2] = (byte)(value >> (8*5));
array[offset + 3] = (byte)(value >> (8*4));
//...
}
}
It seems that you wish to avoid, for some reason, creating any temporary new arrays. And you also want to avoid unsafe code.
You could pin the object and then copy to the array.
public static void ToBytes(ulong value, byte[] array, int offset)
{
GCHandle handle = GCHandle.Alloc(value, GCHandleType.Pinned);
try
{
Marshal.Copy(handle.AddrOfPinnedObject(), array, offset, 8);
}
finally
{
handle.Free();
}
}
BinaryWriter can be a good solution.
var writer = new BinaryWriter(new MemoryStream(yourbuffer, youroffset, yourbuffer.Length-youroffset));
writer.Write(someuint64);
It's useful when you need to convert a lot of data into a buffer continuously
var writer = new BinaryWriter(new MemoryStream(yourbuffer));
foreach(var value in yourints){
writer.Write(value);
}
or when you just want to write to a file, that's the best case to use BinaryWriter.
var writer = new BinaryWriter(yourFileStream);
foreach(var value in yourints){
writer.Write(value);
}
For anyone else that stumbles across this.
If Big Endian support is NOT required and Unsafe is allowed then I've written a library specifically to minimize GC allocations during serialization for single threaded serializers such as when serializing across TCP sockets.
Note: supports Little Endian ONLY example X86 / X64 (might add big endian support some day)
https://github.com/tcwicks/ChillX/tree/master/src/ChillX.Serialization
In the process also re-wrote BitConverter so that it works similar to BitPrimitives but with a lot of extras such as supporting arrays as well.
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Serialization/BitConverterExtended.cs
Random rnd = new Random();
RentedBuffer<byte> buffer = RentedBuffer<byte>.Shared.Rent(BitConverterExtended.SizeOfUInt64
+ (20 * BitConverterExtended.SizeOfUInt16)
+ (20 * BitConverterExtended.SizeOfTimeSpan)
+ (10 * BitConverterExtended.SizeOfSingle);
UInt64 exampleLong = long.MaxValue;
int startIndex = 0;
startIndex += BitConverterExtended.GetBytes(exampleLong, buffer.BufferSpan, startIndex);
UInt16[] shortArray = new UInt16[20];
for (int I = 0; I < shortArray.Length; I++) { shortArray[I] = (ushort)rnd.Next(0, UInt16.MaxValue); }
//When using reflection / expression trees CLR cannot distinguish between UInt16 and Int16 or Uint64 and Int64 etc...
//Therefore Uint methods are renamed.
startIndex += BitConverterExtended.GetBytesUShortArray(shortArray, buffer.BufferSpan, startIndex);
TimeSpan[] timespanArray = new TimeSpan[20];
for (int I = 0; I < timespanArray.Length; I++) { timespanArray[I] = TimeSpan.FromSeconds(rnd.Next(0, int.MaxValue)); }
startIndex += BitConverterExtended.GetBytes(timespanArray, buffer.BufferSpan, startIndex);
float[] floatArray = new float[10];
for (int I = 0; I < floatArray.Length; I++) { floatArray[I] = MathF.PI * rnd.Next(short.MinValue, short.MaxValue); }
startIndex += BitConverterExtended.GetBytes(floatArray, buffer.BufferSpan, startIndex);
//Do stuff with buffer and then
buffer.Return();
//Or
buffer = null;
//and let RentedBufferContract do this automatically
The underlying problem is actually two fold. Even if you use bitprimitives we still have the issue of the byte array buffers that we are writing to / reading from. The issue is further compounded if we have lots of array fields T[] for example int[]
Using standard BitConverter leads to situations like 80% time spent in GC.
bitprimitives is a much better but we still need to manage the byte[] buffers. If we create one byte[] buffer for each int in an array of 1000 ints that is 1000 byte[4] arrays to garbage collect which means rent a buffer big enough for all 1000 ints (int[4000]) from ArrayPool but then we have to manage returning these buffers.
Further complicating it is the fact that we have to track the length of each buffer that is actually used since ArrayPool will return arrays that are larger than that which is requested. System.Buffers.ArrayPool.Shared.Rent(4000); will usually return a byte[4096] or maybe even a byte[8192]
This serialization code:
uses a wrappers around ArrayPool in order make its usage transparent:
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Core/Structures/ManagedPool.cs
For managing renting and returning of ArrayPool buffers:
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Core/Structures/RentedBuffer.cs
With operator overloads on + operator to automatically assign from Span
For preventing memory leaks and also managing use and forget style guaranteed return of rented buffers:
https://github.com/tcwicks/ChillX/blob/master/src/ChillX.Core/Structures/RentedBufferContract.cs
Example:
//Assume ArrayProperty_Long is a long[] array.
RentedBuffer<long> ClonedArray = RentedBuffer<long>.Shared.Rent(ArrayProperty_Long.Length);
ClonedArray += ArrayProperty_Long;
ClonedArray will be returned to ArrayPool.Shared automatically when ClonedArray goes out of scope (handled by RentedBufferContract).
Or call ClonedArray.Return();
p.s. Any feedback is much appreciated.
Following benchmarks compare performance using rented buffers and this serializer implementation versus using MessagePack without rented buffers. Messagepack if micro benchmarked is a faster serializer. This performance difference is purely due to reducing GC collection overheads by pooling / renting buffers.
----------------------------------------------------------------------------------
Benchmarking ChillX Serializer: Test Object: Data class with 31 properties / fields of different types inlcuding multiple arrays of different types
Num Reps: 50000 - Array Size: 256
----------------------------------------------------------------------------------
Entity Size bytes: 20,678 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,025,523 - Mbps: 503.32 - Time: 00:00:01.9589880
Entity Size bytes: 20,678 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,026,129 - Mbps: 515.26 - Time: 00:00:01.9135886
Entity Size bytes: 20,678 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,026,291 - Mbps: 518.46 - Time: 00:00:01.9018027
----------------------------------------------------------------------------------
Benchmarking MessagePack Serializer: Test Object: Data class with 31 properties / fields of different types inlcuding multiple arrays of different types
Num Reps: 50000 - Array Size: 256
----------------------------------------------------------------------------------
Entity Size bytes: 19,811 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,012,386 - Mbps: 234.01 - Time: 00:00:04.0668261
Entity Size bytes: 19,811 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,012,285 - Mbps: 232.10 - Time: 00:00:04.0523329
Entity Size bytes: 19,811 - Count: 00,050,000 - Threads: 01 - Entities Per Second: 00,012,642 - Mbps: 238.84 - Time: 00:00:03.9811276
currently im working on a solution for a prime-number calculator/checker. The algorythm is already working and verry efficient (0,359 seconds for the first 9012330 primes). Here is a part of the upper region where everything is declared:
const uint anz = 50000000;
uint a = 3, b = 4, c = 3, d = 13, e = 12, f = 13, g = 28, h = 32;
bool[,] prim = new bool[8, anz / 10];
uint max = 3 * (uint)(anz / (Math.Log(anz) - 1.08366));
uint[] p = new uint[max];
Now I wanted to go to the next level and use ulong's instead of uint's to cover a larger area (you can see that already), where i tapped into my problem: the bool-array.
Like everybody should know, bool's have the length of a byte what takes a lot of memory when creating the array... So I'm searching for a more resource-friendly way to do that.
My first idea was a bit-array -> not byte! <- to save the bool's, but haven't figured out how to do that by now. So if someone ever did something like this, I would appreciate any kind of tips and solutions. Thanks in advance :)
You can use BitArray collection:
http://msdn.microsoft.com/en-us/library/system.collections.bitarray(v=vs.110).aspx
MSDN Description:
Manages a compact array of bit values, which are represented as Booleans, where true indicates that the bit is on (1) and false indicates the bit is off (0).
You can (and should) use well tested and well known libraries.
But if you're looking to learn something (as it seems to be the case) you can do it yourself.
Another reason you may want to use a custom bit array is to use the hard drive to store the array, which comes in handy when calculating primes. To do this you'd need to further split addr, for example lowest 3 bits for the mask, next 28 bits for 256MB of in-memory storage, and from there on - a file name for a buffer file.
Yet another reason for custom bit array is to compress the memory use when specifically searching for primes. After all more than half of your bits will be 'false' because the numbers corresponding to them would be even, so in fact you can both speed up your calculation AND reduce memory requirements if you don't even store the even bits. You can do that by changing the way addr is interpreted. Further more you can also exclude numbers divisible by 3 (only 2 out of every 6 numbers has a chance of being prime) thus reducing memory requirements by 60% compared to plain bit array.
Notice the use of shift and logical operators to make the code a bit more efficient.
byte mask = (byte)(1 << (int)(addr & 7)); for example can be written as
byte mask = (byte)(1 << (int)(addr % 8));
and addr >> 3 can be written as addr / 8
Testing shift/logical operators vs division shows 2.6s vs 4.8s in favor of shift/logical for 200000000 operations.
Here's the code:
void Main()
{
var barr = new BitArray(10);
barr[4] = true;
Console.WriteLine("Is it "+barr[4]);
Console.WriteLine("Is it Not "+barr[5]);
}
public class BitArray{
private readonly byte[] _buffer;
public bool this[long addr]{
get{
byte mask = (byte)(1 << (int)(addr & 7));
byte val = _buffer[(int)(addr >> 3)];
bool bit = (val & mask) == mask;
return bit;
}
set{
byte mask = (byte) ((value ? 1:0) << (int)(addr & 7));
int offs = (int)addr >> 3;
_buffer[offs] = (byte)(_buffer[offs] | mask);
}
}
public BitArray(long size){
_buffer = new byte[size/8 + 1]; // define a byte buffer sized to hold 8 bools per byte. The spare +1 is to avoid dealing with rounding.
}
}
I'm trying to debug some bit shifting operations and I need to visualize the bits as they exist before and after a Bit-Shifting operation.
I read from this answer that I may need to handle backfill from the shifting, but I'm not sure what that means.
I think that by asking this question (how do I print the bits in a int) I can figure out what the backfill is, and perhaps some other questions I have.
Here is my sample code so far.
static string GetBits(int num)
{
StringBuilder sb = new StringBuilder();
uint bits = (uint)num;
while (bits!=0)
{
bits >>= 1;
isBitSet = // somehow do an | operation on the first bit.
// I'm unsure if it's possible to handle different data types here
// or if unsafe code and a PTR is needed
if (isBitSet)
sb.Append("1");
else
sb.Append("0");
}
}
Convert.ToString(56,2).PadLeft(8,'0') returns "00111000"
This is for a byte, works for int also, just increase the numbers
To test if the last bit is set you could use:
isBitSet = ((bits & 1) == 1);
But you should do so before shifting right (not after), otherwise you's missing the first bit:
isBitSet = ((bits & 1) == 1);
bits = bits >> 1;
But a better option would be to use the static methods of the BitConverter class to get the actual bytes used to represent the number in memory into a byte array. The advantage (or disadvantage depending on your needs) of this method is that this reflects the endianness of the machine running the code.
byte[] bytes = BitConverter.GetBytes(num);
int bitPos = 0;
while(bitPos < 8 * bytes.Length)
{
int byteIndex = bitPos / 8;
int offset = bitPos % 8;
bool isSet = (bytes[byteIndex] & (1 << offset)) != 0;
// isSet = [True] if the bit at bitPos is set, false otherwise
bitPos++;
}
I'm currently trying to get this C code converted into C#.
Since I'm not really familiar with C I'd really apprecheate your help!
static unsigned char byte_table[2080] = {0};
First of, some bytearray gets declared but never filled which I'm okay with
BYTE* packet = //bytes come in here from a file
int unknownVal = 0;
int unknown_field0 = *(DWORD *)(packet + 0x08);
do
{
*((BYTE *)packet + i) ^= byte_table[(i + unknownVal) & 0x7FF];
++i;
}
while (i <= packet[0]);
But down here.. I really have no idea how to translate this into C#
BYTE = byte[] right?
DWORD = double?
but how can (packet + 0x08) be translated? How can I add a hex to a bytearray? Oo
I'd be happy about anything that helps! :)
In C, setting any set of memory to {0} will set the entire memory area to zeroes, if I'm not mistaken.
That bottom loop can be rewritten in a simpler, C# friendly fashion.
byte[] packet = arrayofcharsfromfile;
int field = packet[8]+(packet[9]<<8)+(packet[10]<<16)+(packet[11]<<24); //Assuming 32 bit little endian integer
int unknownval = 0;
int i = 0;
do //Why waste the newline? I don't know. Conventions are silly!
{
packet[i] ^= byte_table[(i+unknownval) & 0x7FF];
} while( ++i <= packet[0] );
field is set by taking the four bytes including and following index 8 and generating a 32 bit int from them.
In C, you can cast pointers to other types, as is done in your provided snippet. What they're doing is taking an array of bytes (each one 1/4 the size of a DWORD) and adding 8 to the index which advances the pointer by 8 bytes (since each element is a byte wide) and then treating that pointer as a DWORD pointer. In simpler terms, they're turning the byte array in to a DWORD array, and then taking index 2, as 8/4=2.
You can simulate this behavior in a safe fashion by stringing the bytes together with bitshifting and addition, as I demonstrated above. It's not as efficient and isn't as pretty, but it accomplishes the same thing, and in a platform agnostic way too. Not all platforms are little endian.