We Keep Coding

Marshalling C array in C# - Simple HelloWorld

Building off of my marshalling helloworld question, I'm running into issues marshalling an array allocated in C to C#. I've spent hours researching where I might be going wrong, but everything I've tried ends up with errors such as AccessViolationException.
The function that handles creating an array in C is below.
__declspec(dllexport) int __cdecl import_csv(char *path, struct human ***persons, int *numPersons)
{
int res;
FILE *csv;
char line[1024];
struct human **humans;
csv = fopen(path, "r");
if (csv == NULL) {
return errno;
}
*numPersons = 0; // init to sane value
/*
* All I'm trying to do for now is get more than one working.
* Starting with 2 seems reasonable. My test CSV file only has 2 lines.
*/
humans = calloc(2, sizeof(struct human *));
if (humans == NULL)
return ENOMEM;
while (fgets(line, 1024, csv)) {
char *tmp = strdup(line);
struct human *person;
humans[*numPersons] = calloc(1, sizeof(*person));
person = humans[*numPersons]; // easier to work with
if (person == NULL) {
return ENOMEM;
}
person->contact = calloc(1, sizeof(*(person->contact)));
if (person->contact == NULL) {
return ENOMEM;
}
res = parse_human(line, person);
if (res != 0) {
return res;
}
(*numPersons)++;
}
(*persons) = humans;
fclose(csv);
return 0;
}
The C# code:
IntPtr humansPtr = IntPtr.Zero;
int numHumans = 0;
HelloLibrary.import_csv(args[0], ref humansPtr, ref numHumans);
HelloLibrary.human[] humans = new HelloLibrary.human[numHumans];
IntPtr[] ptrs = new IntPtr[numHumans];
IntPtr aIndex = (IntPtr)Marshal.PtrToStructure(humansPtr, typeof(IntPtr));
// Populate the array of IntPtr
for (int i = 0; i < numHumans; i++)
{
ptrs[i] = new IntPtr(aIndex.ToInt64() +
(Marshal.SizeOf(typeof(IntPtr)) * i));
}
// Marshal the array of human structs
for (int i = 0; i < numHumans; i++)
{
humans[i] = (HelloLibrary.human)Marshal.PtrToStructure(
ptrs[i],
typeof(HelloLibrary.human));
}
// Use the marshalled data
foreach (HelloLibrary.human human in humans)
{
Console.WriteLine("first:'{0}'", human.first);
Console.WriteLine("last:'{0}'", human.last);
HelloLibrary.contact_info contact = (HelloLibrary.contact_info)Marshal.
PtrToStructure(human.contact, typeof(HelloLibrary.contact_info));
Console.WriteLine("cell:'{0}'", contact.cell);
Console.WriteLine("home:'{0}'", contact.home);
}
The first human struct gets marshalled fine. I get the access violation exceptions after the first one. I feel like I'm missing something with marshalling structs with struct pointers inside them. I hope I have some simple mistake I'm overlooking. Do you see anything wrong with this code?
See this GitHub gist for full source.

// Populate the array of IntPtr
This is where you went wrong. You are getting back a pointer to an array of pointers. You got the first one correct, actually reading the pointer value from the array. But then your for() loop got it wrong, just adding 4 (or 8) to the first pointer value. Instead of reading them from the array. Fix:
IntPtr[] ptrs = new IntPtr[numHumans];
// Populate the array of IntPtr
for (int i = 0; i < numHumans; i++)
{
ptrs[i] = (IntPtr)Marshal.PtrToStructure(humansPtr, typeof(IntPtr));
humansPtr = new IntPtr(humansPtr.ToInt64() + IntPtr.Size);
}
Or much more cleanly since marshaling arrays of simple types is already supported:
IntPtr[] ptrs = new IntPtr[numHumans];
Marshal.Copy(humansPtr, ptrs, 0, numHumans);
I found the bug by using the Debug + Windows + Memory + Memory 1. Put humansPtr in the Address field, switched to 4-byte integer view and observed that the C code was doing it correctly. Then quickly found out that ptrs[] did not contain the values I saw in the Memory window.
Not sure why you are writing code like this, other than as a mental exercise. It is not the correct way to go about it, you are for example completely ignoring the need to release the memory again. Which is very nontrivial. Parsing CSV files in C# is quite simple and just as fast as doing it in C, it is I/O bound, not execute-bound. You'll easily avoid these almost impossible to debug bugs and get lots of help from the .NET Framework.

Compare byte[] to T

I want to make a list of pointers to locations that contains a certain value in the process memory of another process. The value can be a short, int, long, string, bool or something else.
My idea is to use Generics for this. I have one problem with making it, how can I tell the compiler to what type he needs to convert the byte array?
This is what I made:
public List<IntPtr> ScanProccessFor<T>(T ItemToScanFor)
{
List<IntPtr> Output = new List<IntPtr>();
IntPtr StartOffset = SelectedProcess.MainModule.BaseAddress;
int ScanSize = SelectedProcess.MainModule.ModuleMemorySize;
for (int i = 0; i < ScanSize; i++)
if (ReadMemory(SelectedProcess, StartOffset + i, (UInt16)Marshal.SizeOf(ItemToScanFor)) == ItemToScanFor)
Output.Insert(Output.Count,StartOffset + i);
return Output;
}
How can I tell the compiler that he needs to convert the byte[] to type T?

Your question is a little bit confusing, but I'll try to answer what I can
Instead of taking a generic type, I would probably write a method that takes an instance of an interface like IConvertableToByteArray or something.
public IConvertableToByteArray
{
public byte[] ToByteArray();
}
Then If you needed to allow a specific type to be compatible with that method, you could make an encapsulating class
public IntConvertableToByteArray : IConvertableToByteArray
{
public int Value{get; set;}
public byte[] ToByteArray()
{
insert logic here
}
}

You could use Marshal.StructureToPtr to get an unmanaged representation of the structure (which has to be a 'simple' structure). You might need to special case strings though.
You should also think about the alignment constraints on what you are searching for -- advancing through memory 1 byte at a time will be very slow and wasteful if the item must be 4 or 8 byte aligned.

How to pinvoke a variable-length array of structs from GetTokenInformation() safely for 32-bit and 64-bit? C#

I'm following the pinvoke code provided here but am slightly scared by the marshalling of the variable-length array as size=1 and then stepping through it by calculating an offset instead of indexing into an array. Isn't there a better way? And if not, how should I do this to make it safe for 32-bit and 64-bit?
[StructLayout(LayoutKind.Sequential)]
public struct SID_AND_ATTRIBUTES
{
public IntPtr Sid;
public uint Attributes;
}
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_GROUPS
{
public int GroupCount;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 1)]
public SID_AND_ATTRIBUTES[] Groups;
};
public void SomeMethod()
{
IntPtr tokenInformation;
// ...
string retVal = string.Empty;
TOKEN_GROUPS groups = (TOKEN_GROUPS)Marshal.PtrToStructure(tokenInformation, typeof(TOKEN_GROUPS));
int sidAndAttrSize = Marshal.SizeOf(new SID_AND_ATTRIBUTES());
for (int i = 0; i < groups.GroupCount; i++)
{
// *** Scary line here:
SID_AND_ATTRIBUTES sidAndAttributes = (SID_AND_ATTRIBUTES)Marshal.PtrToStructure(
new IntPtr(tokenInformation.ToInt64() + i * sidAndAttrSize + IntPtr.Size),
typeof(SID_AND_ATTRIBUTES));
// ...
}
I see here another approach of declaring the length of the array as much bigger than it's likely to be, but that seemed to have its own problems.
As a side question: When I step through the above code in the debugger I'm not able to evaluate tokenInformation.ToInt64() or ToInt32(). I get an ArgumentOutOfRangeException. But the line of code executes just fine!? What's going on here?

Instead of guessing what the offset, is its generally better to use Marshal.OffsetOf(typeof(TOKEN_GROUPS), "Groups") to get the correct offset to the start of the array.

I think it looks okay -- as okay as any poking about in unmanaged land is, anyway.
However, I wonder why the start is tokenInformation.ToInt64() + IntPtr.Size and not tokenInformation.ToInt64() + 4 (as the GroupCount field type is an int and not IntPtr). Is this for packing/alignment of the structure or just something fishy? I do not know here.
Using tokenInformation.ToInt64() is important because on a 64-bit machine will explode (OverflowException) if the IntPtr value is larger than what an int can store. However, the CLR will handle a long just fine on both architectures and it doesn't change the actual value extracted from the IntPtr (and thus put back into the new IntPtr(...)).
Imagine this (untested) function as a convenience wrapper:
// unpacks an array of structures from unmanaged memory
// arr.Length is the number of items to unpack. don't overrun.
void PtrToStructureArray<T>(T[] arr, IntPtr start, int stride) {
long ptr = start.ToInt64();
for (int i = 0; i < arr.Length; i++, ptr += stride) {
arr[i] = (T)Marshal.PtrToStructure(new IntPtr(ptr), typeof(T));
}
}
var attributes = new SID_AND_ATTRIBUTES[groups.GroupCount];
PtrToStructureArray(attributes, new IntPtr(tokenInformation.ToInt64() + IntPtr.Size), sidAndAttrSize);
Happy coding.

Generic that takes only numeric types (int double etc)?

In a program I'm working on, I need to write a function to take any numeric type (int, short, long etc) and shove it in to a byte array at a specific offset.
There exists a Bitconverter.GetBytes() method that takes the numeric type and returns it as a byte array, and this method only takes numeric types.
So far I have:
private void AddToByteArray<T>(byte[] destination, int offset, T toAdd) where T : struct
{
Buffer.BlockCopy(BitConverter.GetBytes(toAdd), 0, destination, offset, sizeof(toAdd));
}
So basically my goal is that, for example, a call to AddToByteArray(array, 3, (short)10) would take 10 and store it in the 4th slot of array. The explicit cast exists because I know exactly how many bytes I want it to take up. There are cases where I would want a number that is small enough to be a short to really take up 4 bytes. On the flip side, there are times when I want an int to be crunched down to just a single byte. I'm doing this to create a custom network packet, if that makes any ideas pop in to your heads.
If the where clause of a generic supported something like "where T : int || long || etc" I would be ok. (And no need to explain why they don't support that, the reason is fairly obvious)
Any help would be greatly appreciated!
Edit: I realize that I could just do a bunch of overloads, one for each type I want to support... but I'm asking this question because I want to avoid precisely that :)

I disagree that this can't be done; it's just that the design I'd propose to do it is a little weird (and involved).
Here's the idea.
The Idea
Define an interface IBytesProvider<T>, with one method:
public interface IBytesProvider<T>
{
byte[] GetBytes(T value);
}
Then implement this in a BytesProvider<T> class with a static Default property.
If this sounds familiar, it's because it's exactly how the EqualityComparer<T> and Comparer<T> classes work (heavily used in plenty of LINQ extension methods).
The Implementation
Here's how I'd propose you set it up.
public class BytesProvider<T> : IBytesProvider<T>
{
public static BytesProvider<T> Default
{
get { return DefaultBytesProviders.GetDefaultProvider<T>(); }
}
Func<T, byte[]> _conversion;
internal BytesProvider(Func<T, byte[]> conversion)
{
_conversion = conversion;
}
public byte[] GetBytes(T value)
{
return _conversion(value);
}
}
static class DefaultBytesProviders
{
static Dictionary<Type, object> _providers;
static DefaultBytesProviders()
{
// Here are a couple for illustration. Yes, I am suggesting that
// in reality you would add a BytesProvider<T> for each T
// supported by the BitConverter class.
_providers = new Dictionary<Type, object>
{
{ typeof(int), new BytesProvider<int>(BitConverter.GetBytes) },
{ typeof(long), new BytesProvider<long>(BitConverter.GetBytes) }
};
}
public static BytesProvider<T> GetDefaultProvider<T>()
{
return (BytesProvider<T>)_providers[typeof(T)];
}
}
The Payoff
Now, finally, once you'd done all this, what you'd do is simply call:
byte[] bytes = BytesProvider<T>.Default.GetBytes(toAdd);
No overloads needed.

You can do this by first seperating the method into two parts, one to turn the value into an array of bytes, and another to insert them. Then just use overloads:
public static void AddToByteArray(byte[] destination, int offset, long value)
{ InsertBytes(destination, offset, BitConverter.GetBytes(value)); }
public static void AddToByteArray(byte[] destination, int offset, int value)
{ InsertBytes(destination, offset, BitConverter.GetBytes(value)); }
public static void AddToByteArray(byte[] destination, int offset, short value)
{ InsertBytes(destination, offset, BitConverter.GetBytes(value)); }
private static void InsertBytes(byte[] destination, int offset, byte[] bytes)
{
Buffer.BlockCopy(bytes, 0, destination, offset, bytes.Length);
}

This would not work anyway, because which overload of BitConverter.GetBytes() to use is resolved at compile time and not at runtime, so the generic argument passed as T will not be used to help determine the GetBytes() overload. Since there is no overload that accepts object, this approach could not work even if you could constrain T around some specific set of types. So you are doubly-screwed here.
Your only real option here is to overload your AddToByteArray method for each numeric type you want to accept. I know you don't want to do that, but there's little else you can do. (You could accept an argument of object and use reflection to invoke the specific overload of GetBytes() based on the argument type, but that would be dog slow due to reflection and boxing overhead...)

As a simplification of Dan Tao's solution in combination with SLaks' suggestion, here's a complete generic BitConverter:
public class BitConverter<T>
{
public static readonly Func<T, byte[]> GetBytes = x => new byte[] { };
static BitConverter()
{
BitConverter<byte>.GetBytes = x => new byte[] { x };
BitConverter<bool>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<char>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<double>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<Int16>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<Int32>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<Int64>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<Single>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<UInt16>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<UInt32>.GetBytes = x => BitConverter.GetBytes(x);
BitConverter<UInt64>.GetBytes = x => BitConverter.GetBytes(x);
}
}
Your example would then look like this:
private void AddToByteArray<T>(byte[] destination, int offset, T toAdd) where T : struct
{
Buffer.BlockCopy(BitConverter<T>.GetBytes(toAdd), 0, destination, offset, sizeof(toAdd));
}
You would simply have to provide a GetBytes() method for each expected type to the static constructor. Which, by the way, is not limited to struct types, for example:
BitConverter<MemoryStream>.GetBytes = x => x.ToArray();
BitConverter<string>.GetBytes = x => Encoding.Default.GetBytes(x);
If there's no GetBytes() method for the generic type, it will return an empty array (but you might want to change the code to throw an Exception instead!).

This is a real problem in C#, there is no common interface that all numeric types implement, you can restrict to struct and new(), but that still will allow structs with a parameterless constructor. If you really want to restrict it, you sadly have to use defined overloads for all numeric types.

If you only care about running on little-endian systems (like Windows), you could add a constraint on IConvertible (which I believe all the numeric types support), use that to convert the value to a 64-bit integer, get the bytes for that, and then throw away the bytes you know you don't need. Something like this:
private byte[] NumberToBytes<T>(T value)
where T : new(), struct, IConvertible
{
var longValue = value.ToUInt64();
var bytes = BitConverter.GetBytes(longValue);
Array.Resize(ref bytes, sizeof(T));
return bytes;
}
Of course, this assumes that you're only working with integers -- it wouldn't work for float, double, or decimal. And as noted above, it would only work on little-endian systems; for big-endian, you would need to keep the last sizeof(T) elements from the array, not the first.

Reading in a binary file containing an unknown quantity of structures (C#)

Ok, so I currently have a binary file containing an unknown number of structs like this:
private struct sTestStruct
{
public int numberOne;
public int numberTwo;
public int[] numbers; // This is ALWAYS 128 ints long.
public bool trueFalse;
}
So far, I use the following to read all the structs into a List<>:
List<sTestStruct> structList = new List<sTestStruct>();
while (binReader.BaseStream.Position < binReader.BaseStream.Length)
{
sTestStruct temp = new sTestStruct();
temp.numberOne = binReader.ReadInt32();
temp.numberTwo = binReader.ReadInt32();
temp.numbers = new int[128];
for (int i = 0; i < temp.numbers.Length; i++)
{
temp.numbers[i] = binReader.ReadInt32();
}
temp.trueFalse = binReader.ReadBoolean();
// Add to List<>
structList.Add(temp);
}
I don't really want to do this, as only one of the structs can be displayed to the user at once, so there is no point reading in more than one record at a time. So I thought that I could read in a specific record using something like:
fileStream.Seek(sizeof(sTestStruct) * index, SeekOrigin.Begin);
But it wont let me as it doesn't know the size of the sTestStruct, the structure wont let me predefine the array size, so how do I go about this??

The sTestStruct is not stored in one consecutive are of memory and sizeof(sTestStruct) is not directly related to the size of the records in the file. The numbers members is a reference to an array which you allocate youself in your reading code.
But you can easily specify the record size in code since it is a constant value. This code will seek to the record at index. You can then read one record using the body of your loop.
const Int32 RecordSize = (2 + 128)*sizeof(Int32) + sizeof(Boolean);
fileStream.Seek(RecordSize * index, SeekOrigin.Begin);
If you have many different fixed sized records and you are afraid that manually entering the record size for each record is error prone you could devise a scheme based on reflection and custom attributes.
Create an attribute to define the size of arrays:
[AttributeUsage(AttributeTargets.Field, AllowMultiple = false)]
sealed class ArraySizeAttribute : Attribute {
public ArraySizeAttribute(Int32 length) {
Length = length;
}
public Int32 Length { get; private set; }
}
Use the attribute on your record type:
private struct sTestStruct {
public int numberOne;
public int numberTwo;
[ArraySize(128)]
public int[] numbers; // This is ALWAYS 128 ints long.
public bool trueFalse;
}
You can then compute the size of the record using this sample code:
Int32 GetRecordSize(Type recordType) {
return recordType.GetFields().Select(fieldInfo => GetFieldSize(fieldInfo)).Sum();
}
Int32 GetFieldSize(FieldInfo fieldInfo) {
if (fieldInfo.FieldType.IsArray) {
// The size of an array is the size of the array elements multiplied by the
// length of the array.
var arraySizeAttribute = (ArraySizeAttribute) Attribute.GetCustomAttribute(fieldInfo, typeof(ArraySizeAttribute));
if (arraySizeAttribute == null)
throw new InvalidOperationException("Missing ArraySizeAttribute on array.");
return GetTypeSize(fieldInfo.FieldType.GetElementType())*arraySizeAttribute.Length;
}
else
return GetTypeSize(fieldInfo.FieldType);
}
Int32 GetTypeSize(Type type) {
if (type == typeof(Int32))
return 4;
else if (type == typeof(Boolean))
return 1;
else
throw new InvalidOperationException("Unexpected type.");
}
Use it like this:
var recordSize = GetRecordSize(typeof(sTestStruct));
fileStream.Seek(recordSize * index, SeekOrigin.Begin);
You will probably have to expand a little on this code to use it in production.

From everything I have read, the way you are doing it is the best method to read in binary data as it has the fewest gotchas where things can go wrong.

Define your struct like this:
struct sTestStruct
{
public int numberOne;
public int numberTwo;
[MarshalAs(UnmanagedType.ByValArray, SizeConst=128)]
public int[] numbers; // This is ALWAYS 128 ints long.
public bool trueFalse;
}
And use Marshal.Sizeof(typeof(sTestStruct)).