We Keep Coding

C# & C++ checksum calculation

I need your help with calculating checksum in C#. Code below is for application writen in C++. App is receiving packets from UDP. This is part of code where is receive and checksum calculation.
struct OPtoLSs {BlockReceivedS hdr; char data[0x100];}; //struct for received data
OPtoLSs mCKr;
#define hdrR mCKr.hdr.OPhdr
#define dataR mCKr.data
int const retHdr = recvfrom(sock, (char*)&hdrR, sizeof(hdrR), 0, (sockaddr*)(&from), &len);
int const retDat = recvfrom(sock, (char*)&dataR, hdrR.Length, 0, (sockaddr*)(&from), &len);
for(unsigned _int16 *p = (unsigned _int16*)&mCKr; p < (unsigned _int16*)&mCKr + ((retHdr+retDat)/2); p++)
{
ChkSum=(_int16)((*p+ChkSum)&0xffff);
}
assert_hard(ChkSum==0xffff);
I need to get right calculation for checksum in C#. I used code below, but when I send any msg from C# side to C++ side, it throws this error. Please can somebody help with this calculation in C# ? Thanks for all your responses.
// mCKr.hdr.OPhdr equivalent
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi, Size = 12)]
public struct OPpacketHdrS
{
public UInt32 packetMark;
public UInt32 OPpacketID;
public UInt16 ChKSum
{
get
{
UInt16 accumulator = 0;
unsafe
{
fixed (OPpacketHdrS* x = &this)
{
for (UInt16* p = (UInt16*)x; p < x + 1; ++p)
{
accumulator += *p;
}
}
}
return accumulator;
}
}
public UInt16 Length;
}
// mCKr.data equivalent
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Size = 20)]
public struct CommandKeyS
{
public OPpacketHdrS OPhdr;
public mHdr Hdr; // Hdr.ID=CommandKeyR
[MarshalAs(UnmanagedType.U2)]
public UInt16 cmdKey, unused;
}

The server calculates the checksum only of the half of the received header and data. Sent checksum should lead to calculated checksum become 0xffff.
I assume you are sending to the server CommandKeyS structure and it's size is constant. I assume ChkSum on the server is of type _int16 and initialized to 0.
I dont know C#, so replaced parts with C++ code hoping it's not too different. You can remove get function of ChKSum field and before sending the data call this function to initialize the checksum:
CalcChecksum( CommandKeyS* sentData )
{
//At this point all fields in sentData, except of OPhdr.ChKSum are properly initialized.
UInt16 accumulator = 0;
unsafe
{
//Calculate checksum of the half of the data without using ChKSum.
UInt32 chkSummedLen = sizeof( CommandKeyS )/2;
UInt8* chkSummedEnd = ((UInt8*)sentData) + chkSummedLen;
sentData->OPhdr.ChKSum = 0;
for (UInt16* p = (UInt16*)sentData; p < chkSummedEnd; ++p)
{
accumulator += *p;
}
//Calculate ChKSum to make the calculated checksum be equal to 0xffff.
sentData->OPhdr.ChKSum = 0xffff - accumulator;
}
}
The way of calculating checksums your server uses is not obvious and possibly not correct, since the bottom part of the data is not checksummed.

CRC-16/Modbus Implementation in C# malfunction

I'm currently setting up the communication between a controller for a step motor and a computer, coding an application in C# (it is the first time I use this programming language, and although I'm not a computer scientist but an industrial engineer, reason why I'm sure there are some ways of optimizing the function which I don't know, any recommendation on that matter would also be very appreciated). Therefore, I've been using the RS-485 that the controller has to communicate with it, and I've implemented an algorithm that generates the CRC(Cyclic Redundancy Check) bytes required.
And there is where my problem begins. I can't find the reason why my function doesn't generate the correct CRC value. I have checked with some online calculators of CRC and I've also used the example that appears in the Modbus Guide (where it also explains how is the code implemented).
Here is the code I've written for the calculus of the CRC:
class Program
{
static void Main(string[] args)
{
// 0x05, 0x06, 0x17, 0x70, 0x00, 0x01
byte[] prueba = new byte[] { 0x02, 0x07 };
byte[] result = Aux.CRC(prueba);
Console.WriteLine(result[0] + " " + result[1]);
}
}
class Aux{
public static byte[] CRC(byte[] to_evaluate)
{
byte[] CRC_Byte = new byte[2] { 0, 0 };
UInt16 CRC_Register = 0xFFFF; //16 bits 1111.1111.1111.1111
UInt16 CRC_pol = 0xa001; //16 bits 1010.0000.0000.0001
foreach (UInt16 byte_val in to_evaluate)
{
CRC_Register ^= byte_val;
Console.WriteLine("XOR inicial : {0:X}", CRC_Register);
for (byte i = 0; i < 8; i++)
{
CRC_Register >>= 1;
Console.WriteLine("Desplazamiento " + (i + 1) + ": {0:X}", CRC_Register);
if ((CRC_Register & 1) != 0)
{
CRC_Register ^= CRC_pol;
Console.WriteLine("XOR: {0:X}", CRC_Register);
}
}
}
Console.WriteLine("{0:X}",CRC_Register);
byte low_byte_CRC = (byte)((CRC_Register << 8) >> 8);
byte high_byte_CRC = (byte)(CRC_Register >> 8);
CRC_Byte[0] = low_byte_CRC;
CRC_Byte[1] = high_byte_CRC;
return CRC_Byte;
}
}
The expected result using the test array attached and the polinomial 0xa001 is 0x1241 for CRC_Register, and {0x41,0x12} for the CRC_Byte.

I had to implement a CRC check for PPP once in C# and it was absolutely no fun!
I found in this link the code that should correctly generate the CRC. It follows the CRC Generation procedure from section 6.2.2 on page 39 of the document you shared the link to.
// Compute the MODBUS RTU CRC
UInt16 ModRTU_CRC(byte[] buf, int len)
{
UInt16 crc = 0xFFFF;
for (int pos = 0; pos < len; pos++)
{
crc ^= (UInt16)buf[pos]; // XOR byte into least sig. byte of crc
for (int i = 8; i != 0; i--) // Loop over each bit
{
if ((crc & 0x0001) != 0) // If the LSB is set
{
crc >>= 1; // Shift right and XOR 0xA001
crc ^= 0xA001;
}
else // Else LSB is not set
{
crc >>= 1; // Just shift right
}
}
}
// Note, this number has low and high bytes swapped, so use it accordingly (or swap bytes)
return crc;
}

Is it possible to cast a pinned byte array to a struct so that bidirectional changes are possible in C#?

I've seen a number of methods to copy a byte[] to a struct, and vise-versa. However, I was wondering if it was possible to cast the byte[] pointer to a struct (like you do in C)..
I want to be able to cast a byte[] to a struct, make changes to the struct, and have the changes automatically appear in the byte[].
Thanks,
reza

You just cast the pointer (sometimes you need to go via void* in the middle):
struct Foo
{
public int Bar;
}
static unsafe void Main()
{
byte[] buffer = new byte[10];
fixed (byte* untyped = buffer)
{
var typed = (Foo*)untyped;
typed[0].Bar = 123;
}
// buffer has the changes
}
If you need to offset into the buffer, then use byte* untyped = &buffer[offset].
If you want a raw struct pointer, then:
fixed (byte* ptr = buffer)
{
var typed = (Foo*)ptr;
Foo* foo = &typed[0];
foo->Bar = 123;
}
However, note that you can't pass a Foo* to methods expecting a Foo or ref Foo.

I found this approach was the simplest to achieve what I wanted.. If you define the struct so that the byte[] overlaps with the elements, copies are effectively transparent between the struct and byte[] (assuming the endian-ness is what you expect; in my case, it is).
[StructLayout(LayoutKind.Explicit)]
public unsafe struct ListEntry {
[System.Runtime.InteropServices.FieldOffset(0)] public fixed byte raw[512];
[System.Runtime.InteropServices.FieldOffset(0)] public byte version;
[System.Runtime.InteropServices.FieldOffset(1)] public UInt16 magic;
[System.Runtime.InteropServices.FieldOffset(3)] public UInt32 start_time;
[System.Runtime.InteropServices.FieldOffset(7)] public UInt16 run_id;
[System.Runtime.InteropServices.FieldOffset(9)] public UInt16 channels;
[System.Runtime.InteropServices.FieldOffset(11)] public UInt16 sampling_rate;
[System.Runtime.InteropServices.FieldOffset(13)] public UInt32 start_sector;
[System.Runtime.InteropServices.FieldOffset(510)] public UInt16 checksum;
}

I would suggest that your best bet, if speed isn't critical, would probably be to have the data stored in a byte[], and have a class which holds an immutable reference to that byte[] and has properties whose get/set methods access the array. Property gets/sets would always reflect and be reflected in the state of the array, since the array itself would hold the state of the object. No "unsafe" code required.
Methods would probably look something like:
public static class IntPack
{ // All methods ssume unchecked arithmetic
public static Int16 FetchI16LE(this byte[] dat, int offset)
{
return (Int16)(dat[offset] + (dat[offset + 1] << 8));
}
public static Int32 FetchI32LE(this byte[] dat, int offset)
{
return dat[offset] + (dat[offset + 1] << 8) +
(dat[offset + 2] << 16) + (dat[offset + 3] << 24);
}
public static void StuffI16LE(this byte[] dat, int offset, int value)
{
dat[offset] = (byte)(value);
dat[offset+1] = (byte)(value >> 8);
}
public static void StuffI32LE(this byte[] dat, int offset, int value)
{
dat[offset] = (byte)(value);
dat[offset + 1] = (byte)(value >> 8);
dat[offset + 2] = (byte)(value >> 16);
dat[offset + 3] = (byte)(value >> 24);
}
}
The methods indicated assume little-endian ordering. One could easily write corresponding __BE methods methods for big-endian.

How to get little endian data from big endian in c# using bitConverter.ToInt32 method?

I am making application in C# which has a byte array containing hex values.
I am getting data as a big-endian but I want it as a little-endian and I am using Bitconverter.toInt32 method for converting that value to integer.
My problem is that before converting the value, I have to copy that 4 byte data into temporary array from source byte array and then reverse that temporary byte array.
I can't reverse source array because it also contains other data.
Because of that my application becomes slow.
In the code I have one source array of byte as waveData[] which contains a lot of data.
byte[] tempForTimestamp=new byte[4];
tempForTimestamp[0] = waveData[290];
tempForTimestamp[1] = waveData[289];
tempForTimestamp[2] = waveData[288];
tempForTimestamp[3] = waveData[287];
int number = BitConverter.ToInt32(tempForTimestamp, 0);
Is there any other method for that conversion?

Add a reference to System.Memory nuget and use BinaryPrimitives.ReverseEndianness().
using System.Buffers.Binary;
number = BinaryPrimitives.ReverseEndianness(number);
It supports both signed and unsigned integers (byte/short/int/long).

In modern-day Linq the one-liner and easiest to understand version would be:
int number = BitConverter.ToInt32(waveData.Skip(286).Take(4).Reverse().ToArray(), 0);
You could also...
byte[] tempForTimestamp = new byte[4];
Array.Copy(waveData, 287, tempForTimestamp, 0, 4);
Array.Reverse(tempForTimestamp);
int number = BitConverter.ToInt32(tempForTimestamp);
:)

If you know the data is big-endian, perhaps just do it manually:
int value = (buffer[i++] << 24) | (buffer[i++] << 16)
| (buffer[i++] << 8) | buffer[i++];
this will work reliably on any CPU, too. Note i is your current offset into the buffer.
Another approach would be to shuffle the array:
byte tmp = buffer[i+3];
buffer[i+3] = buffer[i];
buffer[i] = tmp;
tmp = buffer[i+2];
buffer[i+2] = buffer[i+1];
buffer[i+1] = tmp;
int value = BitConverter.ToInt32(buffer, i);
i += 4;
I find the first immensely more readable, and there are no branches / complex code, so it should work pretty fast too. The second could also run into problems on some platforms (where the CPU is already running big-endian).

Here you go
public static int SwapEndianness(int value)
{
var b1 = (value >> 0) & 0xff;
var b2 = (value >> 8) & 0xff;
var b3 = (value >> 16) & 0xff;
var b4 = (value >> 24) & 0xff;
return b1 << 24 | b2 << 16 | b3 << 8 | b4 << 0;
}

Declare this class:
using static System.Net.IPAddress;
namespace BigEndianExtension
{
public static class BigEndian
{
public static short ToBigEndian(this short value) => HostToNetworkOrder(value);
public static int ToBigEndian(this int value) => HostToNetworkOrder(value);
public static long ToBigEndian(this long value) => HostToNetworkOrder(value);
public static short FromBigEndian(this short value) => NetworkToHostOrder(value);
public static int FromBigEndian(this int value) => NetworkToHostOrder(value);
public static long FromBigEndian(this long value) => NetworkToHostOrder(value);
}
}
Example, create a form with a button and a multiline textbox:
using BigEndianExtension;
private void button1_Click(object sender, EventArgs e)
{
short int16 = 0x1234;
int int32 = 0x12345678;
long int64 = 0x123456789abcdef0;
string text = string.Format("LE:{0:X4}\r\nBE:{1:X4}\r\n", int16, int16.ToBigEndian());
text += string.Format("LE:{0:X8}\r\nBE:{1:X8}\r\n", int32, int32.ToBigEndian());
text += string.Format("LE:{0:X16}\r\nBE:{1:X16}\r\n", int64, int64.ToBigEndian());
textBox1.Text = text;
}
//Some code...

The most straightforward way is to use the BinaryPrimitives.ReadInt32BigEndian(ReadOnlySpan) Method introduced in .NET Standard 2.1
var number = BinaryPrimitives.ReadInt32BigEndian(waveData[297..291]);

If you won't ever again need that reversed, temporary array, you could just create it as you pass the parameter, instead of making four assignments. For example:
int i = 287;
int value = BitConverter.ToInt32({
waveData(i + 3),
waveData(i + 2),
waveData(i + 1),
waveData(i)
}, 0);

I use the following helper functions
public static Int16 ToInt16(byte[] data, int offset)
{
if (BitConverter.IsLittleEndian)
return BitConverter.ToInt16(BitConverter.IsLittleEndian ? data.Skip(offset).Take(2).Reverse().ToArray() : data, 0);
return BitConverter.ToInt16(data, offset);
}
public static Int32 ToInt32(byte[] data, int offset)
{
if (BitConverter.IsLittleEndian)
return BitConverter.ToInt32(BitConverter.IsLittleEndian ? data.Skip(offset).Take(4).Reverse().ToArray() : data, 0);
return BitConverter.ToInt32(data, offset);
}
public static Int64 ToInt64(byte[] data, int offset)
{
if (BitConverter.IsLittleEndian)
return BitConverter.ToInt64(BitConverter.IsLittleEndian ? data.Skip(offset).Take(8).Reverse().ToArray() : data, 0);
return BitConverter.ToInt64(data, offset);
}

You can also use Jon Skeet "Misc Utils" library, available at https://jonskeet.uk/csharp/miscutil/
His library has many utility functions. For Big/Little endian conversions you can check the MiscUtil/Conversion/EndianBitConverter.cs file.
var littleEndianBitConverter = new MiscUtil.Conversion.LittleEndianBitConverter();
littleEndianBitConverter.ToInt64(bytes, offset);
var bigEndianBitConverter = new MiscUtil.Conversion.BigEndianBitConverter();
bigEndianBitConverter.ToInt64(bytes, offset);
His software is from 2009 but I guess it's still relevant.

I dislike BitConverter, because (as Marc Gravell answered) it is specced to rely on system endianness, meaning you technically have to do a system endianness check every time you use BitConverter to ensure you don't have to reverse the array. And usually, with saved files, you generally know the endianness you're trying to read, and that might not be the same. You might just be handling file formats with big-endian values, too, like, for instance, PNG chunks.
Because of that, I just wrote my own methods for this, which take a byte array, the read offset and read length as arguments, as well as a boolean to specify the endianness handling, and which uses bit shifting for efficiency:
public static UInt64 ReadIntFromByteArray(Byte[] data, Int32 startIndex, Int32 bytes, Boolean littleEndian)
{
Int32 lastByte = bytes - 1;
if (data.Length < startIndex + bytes)
throw new ArgumentOutOfRangeException("startIndex", "Data array is too small to read a " + bytes + "-byte value at offset " + startIndex + ".");
UInt64 value = 0;
for (Int32 index = 0; index < bytes; index++)
{
Int32 offs = startIndex + (littleEndian ? index : lastByte - index);
value |= (((UInt64)data[offs]) << (8 * index));
}
return value;
}
This code can handle any value between 1 and 8 bytes, both little-endian and big-endian. The only small usage peculiarity is that you need to both give the amount of bytes to read, and need to specifically cast the result to the type you want.
Example from some code where I used it to read the header of some proprietary image type:
Int16 imageWidth = (Int16) ReadIntFromByteArray(fileData, hdrOffset, 2, true);
Int16 imageHeight = (Int16) ReadIntFromByteArray(fileData, hdrOffset + 2, 2, true);
This will read two consecutive 16-bit integers off an array, as signed little-endian values. You can of course just make a bunch of overload functions for all possibilities, like this:
public Int16 ReadInt16FromByteArrayLe(Byte[] data, Int32 startIndex)
{
return (Int16) ReadIntFromByteArray(data, startIndex, 2, true);
}
But personally I didn't bother with that.
And, here's the same for writing bytes:
public static void WriteIntToByteArray(Byte[] data, Int32 startIndex, Int32 bytes, Boolean littleEndian, UInt64 value)
{
Int32 lastByte = bytes - 1;
if (data.Length < startIndex + bytes)
throw new ArgumentOutOfRangeException("startIndex", "Data array is too small to write a " + bytes + "-byte value at offset " + startIndex + ".");
for (Int32 index = 0; index < bytes; index++)
{
Int32 offs = startIndex + (littleEndian ? index : lastByte - index);
data[offs] = (Byte) (value >> (8*index) & 0xFF);
}
}
The only requirement here is that you have to cast the input arg to 64-bit unsigned integer when passing it to the function.

public static unsafe int Reverse(int value)
{
byte* p = (byte*)&value;
return (*p << 24) | (p[1] << 16) | (p[2] << 8) | p[3];
}
If unsafe is allowed... Based on Marc Gravell's post

This will reverse the data inline if unsafe code is allowed...
fixed (byte* wavepointer = waveData)
new Span<byte>(wavepointer + offset, 4).Reverse();

Read or convert to Int32 from TWO byte arrays

I have the 4 bytes that represent an integer stored in 2 separate byte arrays. I would like to convert these into an Int32 WITHOUT copying to a third byte array and reading that using memorystream.
The reason the data is split across two byte arrays is because this is a simplified example of my issue which involves huge amounts of data that cannot fit into a single bytearray.
Is there any way to achieve this? I do not wish to concatenate the two byte arrays into a thrid because of the performance implications which are critical to me.
Moon

You can use a struct layout like this
[StructLayout(LayoutKind.Explicit, Size=4)]
struct UnionInt32Value
{
[FieldOffset(0)] public byte byte1;
[FieldOffset(1)] public byte byte2;
[FieldOffset(2)] public byte byte3;
[FieldOffset(3)] public byte byte4;
[FieldOffset(0)] public Int32 iVal;
}
Assign your bytes in the correct order then read your Int32 from iVal;
EDIT: Sample code
using System;
using System.Runtime.InteropServices;
namespace Test
{
class Program
{
[StructLayout(LayoutKind.Explicit, Size=4)]
struct UnionInt32Value
{
[FieldOffset(0)] public byte byte1;
[FieldOffset(1)] public byte byte2;
[FieldOffset(2)] public byte byte3;
[FieldOffset(3)] public byte byte4;
[FieldOffset(0)] public Int32 iVal;
}
public static void Main(string[] args)
{
UnionInt32Value v = new UnionInt32Value();
v.byte1=1;
v.byte2=0;
v.byte3=0;
v.byte4=0;
Console.WriteLine("this is one " + v.iVal);
v.byte1=0xff;
v.byte2=0xff;
v.byte3=0xff;
v.byte4=0xff;
Console.WriteLine("this is minus one " + v.iVal);
Console.Write("Press any key to continue . . . ");
Console.ReadKey(true);
}
}
}

Something like this?
int x = (array1[index] << 16) + array2[index];
Of course, you didn't specify a language, but that's the gist of it.

The BitConverter class is intended for this:
byte[] parts = { byte1, byte2, byte3, byte4 };
int value = BitConverter.ToInt32(parts, 0);

You can use BitConverter twice, like:
byte[] bytes0 = new byte[] { 255, 255 };
byte[] bytes1 = new byte[] { 0, 0 };
int res = BitConverter.ToInt16(bytes0, 0) << 16;
res |= BitConverter.ToUInt16(bytes1, 0);
Which yields -65536 (0b11111111 11111111 00000000 00000000)
If your integer parts isn't at position 0 in the array, you just replace the 0 in ToUint16 to change the position.
Little extension method:
public static class BitConverterExt
{
public static int ToInt32(byte[] arr0, int index0, byte[] arr1, int index1)
{
int partRes = BitConverter.ToInt16(arr1, index1) << 16;
return partRes | BitConverter.ToUInt16(arr0, index0);
}
}
Usage:
byte[] bytes0 = new byte[] { 0x0, 0xA };
byte[] bytes1 = new byte[] { 0x64, 0xFF };
int res = BitConverterExt.ToInt32(bytes0, 0, bytes1, 0);
//Res -10221056 (0xFF640A00)

If I understand correctly, you are having a problem whilst reading across the boundary of the two arrays. If that is so, this routine will read an integer anywhere in the two arrays, even if it is across the two of them.
int ReadInteger(byte[] array1, byte[] array2, int offset)
{
if (offset < 0 || (offset + 4) > (array1.Length + array2.Length))
throw new ArgumentOutOfRangeException();
if (offset <= (array1.Length - 4))
return BitConverter.ToInt32(array1, offset);
else if (offset >= array1.Length)
return BitConverter.ToInt32(array2, offset - array1.Length);
else
{
var buffer = new byte[4];
var numFirst = array1.Length - offset;
Array.Copy(array1, offset, buffer, 0, numFirst);
Array.Copy(array2, 0, buffer, numFirst, 4 - numFirst);
return BitConverter.ToInt32(buffer, 0);
}
}
Note: depending on how your integers are stored, you might want to change the order in which bytes are copied.