We Keep Coding

C# & C++ checksum calculation

I need your help with calculating checksum in C#. Code below is for application writen in C++. App is receiving packets from UDP. This is part of code where is receive and checksum calculation.
struct OPtoLSs {BlockReceivedS hdr; char data[0x100];}; //struct for received data
OPtoLSs mCKr;
#define hdrR mCKr.hdr.OPhdr
#define dataR mCKr.data
int const retHdr = recvfrom(sock, (char*)&hdrR, sizeof(hdrR), 0, (sockaddr*)(&from), &len);
int const retDat = recvfrom(sock, (char*)&dataR, hdrR.Length, 0, (sockaddr*)(&from), &len);
for(unsigned _int16 *p = (unsigned _int16*)&mCKr; p < (unsigned _int16*)&mCKr + ((retHdr+retDat)/2); p++)
{
ChkSum=(_int16)((*p+ChkSum)&0xffff);
}
assert_hard(ChkSum==0xffff);
I need to get right calculation for checksum in C#. I used code below, but when I send any msg from C# side to C++ side, it throws this error. Please can somebody help with this calculation in C# ? Thanks for all your responses.
// mCKr.hdr.OPhdr equivalent
[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi, Size = 12)]
public struct OPpacketHdrS
{
public UInt32 packetMark;
public UInt32 OPpacketID;
public UInt16 ChKSum
{
get
{
UInt16 accumulator = 0;
unsafe
{
fixed (OPpacketHdrS* x = &this)
{
for (UInt16* p = (UInt16*)x; p < x + 1; ++p)
{
accumulator += *p;
}
}
}
return accumulator;
}
}
public UInt16 Length;
}
// mCKr.data equivalent
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Size = 20)]
public struct CommandKeyS
{
public OPpacketHdrS OPhdr;
public mHdr Hdr; // Hdr.ID=CommandKeyR
[MarshalAs(UnmanagedType.U2)]
public UInt16 cmdKey, unused;
}

The server calculates the checksum only of the half of the received header and data. Sent checksum should lead to calculated checksum become 0xffff.
I assume you are sending to the server CommandKeyS structure and it's size is constant. I assume ChkSum on the server is of type _int16 and initialized to 0.
I dont know C#, so replaced parts with C++ code hoping it's not too different. You can remove get function of ChKSum field and before sending the data call this function to initialize the checksum:
CalcChecksum( CommandKeyS* sentData )
{
//At this point all fields in sentData, except of OPhdr.ChKSum are properly initialized.
UInt16 accumulator = 0;
unsafe
{
//Calculate checksum of the half of the data without using ChKSum.
UInt32 chkSummedLen = sizeof( CommandKeyS )/2;
UInt8* chkSummedEnd = ((UInt8*)sentData) + chkSummedLen;
sentData->OPhdr.ChKSum = 0;
for (UInt16* p = (UInt16*)sentData; p < chkSummedEnd; ++p)
{
accumulator += *p;
}
//Calculate ChKSum to make the calculated checksum be equal to 0xffff.
sentData->OPhdr.ChKSum = 0xffff - accumulator;
}
}
The way of calculating checksums your server uses is not obvious and possibly not correct, since the bottom part of the data is not checksummed.

Marshal a struct containing pointer to array

How to correctly marshal this C struct with pointer to array and pointer to pointer members in C# for use with 3rd party dll?
C:
typedef struct SomeStruct {
uint8_t *data[8];
int size[8];
uint8_t **extended_data;
};
Is it all just IntPtr and then you need to allocate the un-mamaged memory, copy data into it, and pin it? How would you do this if it's the case? The struct gets initialized through a function inside dll.
In python this is how I'd wrap and use this struct:
Python:
class SomeStruct(Structure):
_fields_ = [
('data', POINTER(c_uint8) * 8),
('size', c_int * 8),
('extended_data', POINTER(POINTER(c_uint8)))
]
# example use 1
dll = ctypes.CDLL("lib.dll")
some_struct = SomeStruct()
dll.init_struct(ctypes.byref(some_struct))
# or example 2
alloc_struct_fce = dll.alloc_struct
alloc_struct_fce.restype = ctypes.POINTER(SomeStruct) # specify return type
some_struct_ptr = alloc_struct_fce() # this gets passed to other dll functions
dll.some_processing(some_struct_ptr)
some_struct = some_struct_ptr.contents # dereference the pointer
Trying to find a C# equivalent of this code.
extended_data is a bonus if you know how to deal with it, it's dynamic size and I am not sure how I'd get its size.
A real example would be
struct AVFrame together with AVFrame *av_frame_alloc(void)
struct AVPacket with void av_init_packet(AVPacket *pkt) and AVPacket *av_packet_alloc(void)
The .dll provides methods to allocate and free these structures.

Given your struct, we could:
struct SomeStruct
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
public IntPtr[] data;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
public int[] size;
public IntPtr extended_data;
}
// I don't know the calling convention of your C method... You didn't show
// us its signature. It could be cdecl or stdcall
[DllImport(#"somedll.dll", CallingConvention = CallingConvention.Cdecl /* or StdCall */)]
public static extern void SomeStructMethod(out SomeStruct someStruct);
Then:
SomeStruct someStruct;
SomeStructMethod(out someStruct);
for (int i = 0; i < someStruct.data.Length; i++)
{
var array = new byte[someStruct.size[i]];
Marshal.Copy(someStruct.data[i], array, 0, array.Length);
Console.Write("> ");
for (int j = 0; j < array.Length; j++)
{
Console.Write($"{array[j]} ");
}
Console.WriteLine();
}
Note that at the end you should call some C method to free the memory allocated in someStruct otherwise you'll have a memory leak!
I can't help you about extended_data, because you haven't told us what it should be,

I woul use following code which is based on xanatos code :
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Runtime.InteropServices;
namespace ConsoleApplication1
{
class Program
{
[DllImport(#"somedll.dll", CallingConvention = CallingConvention.Cdecl /* or StdCall */)]
public static extern void SomeStructMethod(out SomeStruct someStruct);
static void Main(string[] args)
{
SomeStruct someStruct;
SomeStructMethod(out someStruct);
byte[] data = new byte[8];
Marshal.Copy(someStruct.data, data, 0, 8);
byte[][] extendedData = new byte[8][];
for(int i = 0; i < 8; i++)
{
extendedData[i] = new byte[someStruct.size[i]];
Marshal.Copy(someStruct.extended_data[i], extendedData[i], 0, someStruct.size[i]);
}
}
}
struct SomeStruct
{
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
public IntPtr data;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8)]
public int[] size;
public IntPtr[] extended_data;
}
}

PInvoke struct with nested struct array

I'm trying to PInvoke a method which has a struct parameter with nested struct array pointer. The c declaration looks like this:
duckdb_state duckdb_query(duckdb_connection connection, const char *query, duckdb_result *out_result);
typedef struct {
void *data;
bool *nullmask;
duckdb_type type;
char *name;
} duckdb_column;
typedef struct {
idx_t column_count;
idx_t row_count;
duckdb_column *columns;
char *error_message;
} duckdb_result;
I declared them in C# like this:
[DllImport("duckdb.dll", CallingConvention = CallingConvention.Cdecl, EntryPoint = "duckdb_query")]
public static extern DuckdbState DuckdbQuery(IntPtr connection, string query, out DuckdbResult result);
[StructLayout(LayoutKind.Sequential)]
public struct DuckdbColumn
{
IntPtr data;
bool nullmask;
DuckdbType type;
string name;
}
[StructLayout(LayoutKind.Sequential)]
public struct DuckdbResult
{
public long column_count;
public long row_count;
public IntPtr columns;
public string error_message;
}
But when I try to execute the query and read the columns I don't get any meaningful data:
result = DuckdbQuery(connection, "SELECT * FROM integers", out queryResult);
DuckdbColumn[] columns = new DuckdbColumn[queryResult.column_count];
var queryResultColumns = queryResult.columns;
var columnPointer = Marshal.ReadIntPtr(queryResultColumns);
var ptrToStructure = (DuckdbColumn)Marshal.PtrToStructure(columnPointer, typeof(DuckdbColumn));
How should I change the PInvoke declarations so that I can read the columns after executign the query?
There is example c code at: DuckDB c example
Update 1
I can get column names with the following code:
for (int i = 0; i < queryResult.column_count; i++)
{
var column = (DuckdbColumn)Marshal.PtrToStructure(queryResult.columns + 8 + (Marshal.SizeOf<DuckdbColumn>() + 8) * i, typeof(DuckdbColumn));
columns[i] = column;
}
but type field still says DUCKDB_TYPE_INVALID
Update 2
As suggested by David in his answer I changed bool nullmask;to IntPtr nullmask; and I can now read column information like this:
for (int i = 0; i < queryResult.column_count; i++)
{
var column = (DuckdbColumn)Marshal.PtrToStructure(queryResult.columns + Marshal.SizeOf<DuckdbColumn>() * i, typeof(DuckdbColumn));
columns[i] = column;
}

You have translated this field incorrectly
bool *nullmask
This is not a bool it's a pointer. Declare it as
IntPtr nullmask;
There could be other errors because we can't see all the translations. Additionally, the +8 in your array access pointer arithmetic looks suspicious.

faster way to return data as an array interoping c++

this is a very clean and nice solution to marsahall a struct array from unmanaged C++ code.
it is allmost perfect solution when it comes to simplicity, it took me a while to get to that level of understanding the concept, so that in few lines of code, as you can see C# Main(), i have a populated array of struct ready to be 'harvested'..
typedef struct {
int Id;
BSTR StrVal;
}Package;
extern "C" __declspec(dllexport) void dodata(int requestedLength,int StringSize, Package **Packs){
int count;
count=0;
*Packs = (Package*)LocalAlloc(0, requestedLength * sizeof(Package));
Package *Cur = *Packs;
while(count!= requestedLength)
{
Cur[count].StrVal = NULL;
Cur[count].Id = count;
Cur[count].StrVal=SysAllocString(L"abcdefghij");
Cur[count].StrVal[StringSize-1]=count+'0';
++count;
}
}
C#
[DllImport(#"ExportStructArr.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
public static extern void dodata(int requestedLength, int StringSize, out IntPtr csPkPtr);
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
public struct csPk
{
public int V;
[MarshalAsAttribute(UnmanagedType.BStr, SizeConst = 10)]
public string testStr;
}
static void Main(string[] args){
int ArrL = 16000;
csPk[] Cpk = new csPk[ArrL];
IntPtr CpkPtr = IntPtr.Zero;
int szPk = Marshal.SizeOf(typeof(csPk));
dodata(ArrL, 10, out CpkPtr);
}
now all i have to do is :
for (int i = 0; i < Cpk.Length; i++)
{
Cpk[i] = (csPk)Marshal.PtrToStructure(new IntPtr(CpkPtr.ToInt32() + (szPk * i)), typeof(csPk));
}
the solution is quite easy as you can see
the question is using unsafe or any kind of transformation to the data, even going down to bytes...
how could i optimize it to perform better returning the data ?
Edit:
links i have tried to learn from other answers here in SO:
answer from Hans Passant
answer from AbdElRaheim
answer from supercat
also tried google : wikipedia , a github post by stephentoub

this is a complete blazing fast solution to populate a list of objects, i did my best
and i will be happy to have comments and suggestions.
c++
typedef struct _DataPacket
{
BSTR buffer;
UINT size;
} DataPacket;
extern "C" __declspec(dllexport) void GetPacksUnsafe( int size, DataPacket** DpArray )
{
int szr = size;int count=0;
*DpArray = (DataPacket*)CoTaskMemAlloc( szr * sizeof( DataPacket ));
if ( DpArray != NULL )
{
DataPacket* CurPack = *DpArray;
for ( int i = 0; i < szr; i++, CurPack++ )
{
CurPack->size = i;
CurPack->buffer = SysAllocString(L"SomeText00");
CurPack->buffer[9]=i+'0';
}
}
}
C#
[DllImport(#"ExportStructArr.dll", CallingConvention = CallingConvention.Cdecl), SuppressUnmanagedCodeSecurity]
public static extern void GetPacksUnsafe(int size, PackU** outPackUArr);
[StructLayout(LayoutKind.Sequential)]
public unsafe struct PackU
{
public char* StrVal;
public int IntVal;
}
public static unsafe List<PackU> PopulateLstPackU(int ArrL)
{
PackU* PackUArrOut;
List<PackU> RtLstPackU = new List<PackU>(ArrL);
GetPacksUnsafe(ArrL, &PackUArrOut);
PackU* CurrentPack = PackUArrOut;
for (int i = 0; i < ArrL; i++, CurrentPack++)
{
RtLstPackU.Add(new PackU(){ StrVal = CurrentPack->StrVal, IntVal=CurrentPack->IntVal});
}
Marshal.FreeCoTaskMem((IntPtr)PackUArrOut);
for (int i = 0; i < 10; i++)
{
Console.WriteLine("{0}", new string(RtLstPackU[i].StrVal));
}
return RtLstPackU;
}
using the code is as simple as it could possibly be
static unsafe void Main(string[] args)
{
int ArrL = 100000;
List<PackU> LstPackU;
LstPackU = PopulateLstPackU(ArrL);
}
there you have a list of custom data as fast as a bullet..
EDIT
using pointers instead of strings :
typedef struct _DataPackCharPnt
{
char* buffer;
UINT IntVal;
} DataPackCharPnt;
extern "C" __declspec(dllexport) void GetPacksPnt( int size, DataPackCharPnt** DpArrPnt )
{
int count = 0;
int TmpStrSize = 10;
*DpArrPnt = (DataPackCharPnt*)CoTaskMemAlloc( size * sizeof( DataPackCharPnt ));
DataPackCharPnt* CurPackPnt = *DpArrPnt;
char dummyStringDataObject[]= "abcdefgHi";
for ( int i = 0; i < size; i++,CurPackPnt++ )
{
dummyStringDataObject[9] = i+'0';
CurPackPnt->IntVal=i;
CurPackPnt->buffer = (char*)malloc(sizeof(char)*TmpStrSize);
strcpy(CurPackPnt->buffer, dummyStringDataObject);
}
}
reduced the time taken from 11 to 7 ms populating 100k elements
is there any part of creating the buffer i could omit ?
the duty of dummyStringDataObject is to simulate work, say getting a file name then set the buffer with its value, so except for this extra time which is the whole purpose of this function, to return some unknown values and lengths of the strings...
could you optimize it even further ?

How to create an array of structures in C# using DLL (written in C)?

I want to fill a structure with data so I can pass it to my DLL functions. The structures are similar in both C# and C implementations. For instance something like (in C):
typedef struct{
apple[] apples;
leaf[] leaves;
} tree;
typedef struct{
int taste;
} apple;
typedef struct{
int color;
} leaf;
I want to create a tree structure on C#, fill it with apples and leaves and then send it to a function I have on a DLL.
How?

First of all, I think you should change the C struct a little bit:
typedef struct{
UINT cApples;
const apple *pApples;
UINT cLeaves;
const leaf *pLeaves;
} tree;
On the C# side:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct Tree
{
internal uint cApples;
internal IntPtr pApples;
internal uint cLeaves;
internal IntPtr pLeaves;
}
You can define Leaf and Apple similarly. Then you can populate them on C# side as follows:
private Tree CreateTree(Leaf[] leaves, Apple[] apples)
{
var result = new Tree();
result.cLeaves = (uint)leaves.Length;
result.cApples = (uint)apples.Length;
result.pLeaves = AllocateNativeMemory(leaves);
result.pApples = AllocateNativeMemory(apples);
return result;
}
private IntPtr AllocateNativeMemory<T>(T[] elements) where T : struct
{
int elementSize = Marshal.SizeOf(typeof(T));
IntPtr result = Marshal.AllocHGlobal(elements.Length * elementSize);
for (int i = 0; i < elements.Length; i++)
{
Marshal.StructureToPtr(
elements[i], new IntPtr((long)result + i * elementSize), false);
}
return result;
}
Now you can pass the result of the CreateTree method to the extern method that calls the C side.
Remark: The allocated memory should be freed; otherwise, your app will leak. If you decide that the C# side is responsible to free the allocated memory, you should do it at the end as follows:
private static void FreeTree(Tree tree)
{
FreeNativeMemory<Leaf>(tree.pLeaves, tree.cLeaves);
FreeNativeMemory<Apple>(tree.pApples, tree.cApples);
}
private static void FreeNativeMemory<T>(IntPtr arrayPtr, uint arrayLen) where T : struct
{
int elementSize = Marshal.SizeOf(typeof(T));
for (uint i = 0; i < arrayLen; i++)
{
Marshal.DestroyStructure(new IntPtr((long)arrayPtr + i * elementSize), typeof(T));
}
Marshal.FreeHGlobal(arrayPtr);
}