We Keep Coding

Byte Pointer NullReferenceException

So, long story short, this is obviously valid code. Working in Unity3D- runs fine on PC. Build for Android- NRE when trying to access past the specific address pinned- NRE where commented in the code block below.
public static unsafe byte[] SerialiseFloat(byte header, float val)
{
data = new byte[5];
fixed (byte* b_ptr = data)
{
*b_ptr = header;
*((float*)(b_ptr + 1)) = val; //NRE
return data;
}
}
And not to be offhand but because it seems to come up in every single question about unsafe context C#- yes, I need to be using it. No, it's not that important for this instance, I know it's an unnecessary use of an unsafe context in this example (this is from my old serialisation lib, contracts are generated which hand over to these methods to serialise members, it was just a quick put-together and these things need to be inlined so I'm not working with multiple arrays.
Anyhow, the point stands- NRE at the line commented above. Anybody know how this is even possible, and how I might begin to fix it? It is happening consistently, it's clearly allocated and pinned. Memory analysis shows that it's being referenced absolutely correctly, everything is aligned as expected. Reproduced this about 15 times now.

Thanks to GSerg for his comment pointing me towards this question. I now know of ARM's alignment constraints which come into play when working with floating-point values. I'll simply force my structure packing for strict alignment now that I'm aware of the cause. Thanks for the heads up- being new to mobile dev and ARM I had absolutely no idea about this and would have definitely lost some sleep, so a huge thanks from me!
EDIT: Just an update to confirm that everything is working properly after running through a few serialisation tests with my code generator now preprocessing generated contracts on Android to force floating point fields to always align to 4-byte memory addresses.

Clear structure in c#

Say I have a structure with several members eg.
private struct MyStats
{
public int packsGood, packsBad, packsTotal;
public bool haveAcceptedStart;
// ...and a bunch of other parameters
}
If I have a member variable of type MyStats, how do I use it for a while and then quickly clear it? In Delphi or C++ I might have used the following code:
memset(&m_stats, 0, sizeof(MyStats)); // C++
ZeroMemory(#m_stats, SizeOf(MyStats)); // Delphi
but that doesn't appear possible in C#. Surely I don't have to rattle through all members each time or PInvoke an API call?

Memory Management should generally not be one of your concerns in C#, as opposed to C or C++. There is the garbage collector to take care of memory clearing. As soon as all the references to your struct have disappeared, the garbage collector will reclaim the memory somewhere in the future. When exactly is irrelevant for a developer.

If you have a field in your class just to be used for a while an clearing it then your class isn't properly designed and does not comply with OOP principles.
If you are using that field just as an auxiliary value you should just replace it with a variable in every scope it is used; in this way you'll reduce the number of daunting side effects and since it's a value type it will be allocated on the stack and cleared as soon as the control flow leaves the current block (given you don't capture it in a lambda expression).

Prevent compiler/cpu instruction reordering c#

I have an Int64 containing two Int32 like this:
[StructLayout(LayoutKind.Explicit)]
public struct PackedInt64
{
[FieldOffset(0)]
public Int64 All;
[FieldOffset(0)]
public Int32 First;
[FieldOffset(4)]
public Int32 Second;
}
Now I want constructors (for all, first and second). However the struct requires all fields to be assigned before the constructor is exited.
Consider the all constructor.
public PackedInt64(Int64 all)
{
this.First = 0;
this.Second = 0;
Thread.MemoryBarrier();
this.All = all;
}
I want to be absolutely sure that this.All is assigned last in the constructor so that half of the field or more isn't overwritten in case of some compiler optimization or instruction reordering in the cpu.
Is Thread.MemoryBarrier() sufficient? Is it the best option?

Yes, this is the correct and best way of preventing reordering.
By executing Thread.MemoryBarrier() in your sample code, the processor will never be allowed to reorder instructions in such a way that the access/modification to First or Second will occur after the access/modification to All. Since they both occupy the same address space, you don't have to worry about your later changes being overwritten by your earlier ones.
Note that Thread.MemoryBarrier() only works for the current executing thread -- it isn't a type of lock. However, given that this code is running in a constructor and no other thread can yet have access to this data, this should be perfectly fine. If you do need cross-thread guarantee of operations, however, you'll need to use a locking mechanism to guarantee exclusive access.
Note that you may not actually need this instruction on x86 based machines, but I would still recommend the code in case you run on another platform one day (such as IA64). See the below chart for what platforms will reorder memory post-save, rather than just post-load.

The MemoryBarrier will prevent re-ordering, but this code is still broken.
LayoutKind.Explicit and FieldOffsetAttribute are documented as affecting the memory layout of the object when it is passed to unmanaged code. It can be used to interop with a C union, but it cannot be used to emulate a C union.
Even if it currently acts the way you expect, on the platform you tested, there is no guarantee that it will continue to do so. The only guarantee made is in the context of interop with unmanaged code (that is, p/invoke, COM interop, or C++/CLI it-just-works).
If you want to read a subset of bytes in a portable future-proof manner, you'll have to use bitwise operations or a byte array and BitConverter. Even if the syntax isn't as nice.

Check the remarks section of the following link: http://msdn.microsoft.com/en-us/library/system.threading.thread.memorybarrier.aspx
It says MemoryBarrier() is required only on multiprocessor systems with weak memory ordering. So, this is a sufficient option but whether this is the best option or not depends upon the system you are using.

First, I'm aware this answer doesn't really solve the reordering problem, but negates it. By using unsafe code, you can avoid writing to First and Second completely.
public unsafe PackedInt64(long all) {
fixed (PackedInt64* ptr = &this)
*(long*) ptr = all;
}
It's not meant to be the most elegant solution and probably doesn't pass most company policies regarding managed code, but it should work.

Access Violation Exception/Crash from C++ callback to C# function

So I have a native 3rd party C++ code base I am working with (.lib and .hpp files) that I used to build a wrapper in C++/CLI for eventual use in C#.
I've run into a particular problem when switching from Debug to Release mode, in that I get an Access Violation Exception when a callback's code returns.
The code from the original hpp files for callback function format:
typedef int (*CallbackFunction) (void *inst, const void *data);
Code from the C++/CLI Wrapper for callback function format:
(I'll explain why I declared two in a moment)
public delegate int ManagedCallbackFunction (IntPtr oInst, const IntPtr oData);
public delegate int UnManagedCallbackFunction (void* inst, const void* data);
--Quickly, the reason I declared a second "UnManagedCallbackFunction" is that I tried to create an "intermediary" callback in the wrapper, so the chain changed from Native C++ > C# to a version of Native C++ > C++/CLI Wrapper > C#...Full disclosure, the problem still lives, it's just been pushed to the C++/CLI Wrapper now on the same line (the return).
And finally, the crashing code from C#:
public static int hReceiveLogEvent(IntPtr pInstance, IntPtr pData)
{
Console.WriteLine("in hReceiveLogEvent...");
Console.WriteLine("pInstance: {0}", pInstance);
Console.WriteLine("pData: {0}", pData);
// provide object context for static member function
helloworld hw = (helloworld)GCHandle.FromIntPtr(pInstance).Target;
if (hw == null || pData == null)
{
Console.WriteLine("hReceiveLogEvent: received null instance pointer or null data\n");
return 0;
}
// typecast data to DataLogger object ptr
IntPtr ip2 = GCHandle.ToIntPtr(GCHandle.Alloc(new DataLoggerWrap(pData)));
DataLoggerWrap dlw = (DataLoggerWrap)GCHandle.FromIntPtr(ip2).Target;
//Do Logging Stuff
Console.WriteLine("exiting hReceiveLogEvent...");
Console.WriteLine("pInstance: {0}", pInstance);
Console.WriteLine("pData: {0}", pData);
Console.WriteLine("Setting pData to zero...");
pData = IntPtr.Zero;
pInstance = IntPtr.Zero;
Console.WriteLine("pData: {0}", pData);
Console.WriteLine("pInstance: {0}", pInstance);
return 1;
}
All writes to the console are done and then we see the dreaded crash on the return:
Unhandled exception at 0x04d1004c in
helloworld.exe: 0xC0000005: Access
violation reading location 0x04d1004c.
If I step into the debugger from here, all I see is that the last entry on the call stack is: > "04d1004c()" which evaluates to a decimal value of: 80805964
Which is only interesting if you look at the console which shows:
entering registerDataLogger
pointer to callback handle: 790848
fp for callback: 2631370
pointer to inst: 790844
in hReceiveLogEvent...
pInstance: 790844
pData: 80805964
exiting hReceiveLogEvent...
pInstance: 790844
pData: 80805964
Setting pData to zero...
pData: 0
pInstance: 0
Now, I know that between debug and release some things are quite different in the Microsoft world. I am, of course worried about byte padding and initialization of variables, so if there is something I am not providing here, just let me know and I'll add to the (already long) post. I also think the managed code may NOT be releasing all ownership and then the native C++ stuff (which I don't have the code for) may be trying to delete or kill off the pData object, thus crashing the app.
More full disclosure, it all works fine (seemingly) in Debug mode!
A real head scratch issue that would appreciate any help!

I think the stack got crushed because of mismatching calling conventions:
try out to put the attribute
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
on the callback delegate declaration.

This doesn't directly answer your question, but it may lead you in the right direction as far as debug mode okay vs. release mode not okay:
Since the debugger adds a lot of record-keeping information to the stack, generally padding out the size and layout of my program in memory, I was “getting lucky” in debug mode by scribbling over 912 bytes of memory that weren’t very important. Without the debugger, though, I was scribbling on top of rather important things, eventually walking outside of my own memory space, causing Interop to delete memory it didn’t own.
What is the definition of DataLoggerWrap? A char field may be too small for the data you are receiving.

I'm not sure what your are trying to achieve.
A few points:
1) The garbage collector is more aggressive in release mode so with bad ownership the behaviour you describe is not uncommon.
2) I don't understands what the below code is trying to do?
IntPtr ip2 = GCHandle.ToIntPtr(GCHandle.Alloc(new DataLoggerWrap(pData)));
DataLoggerWrap dlw = (DataLoggerWrap)GCHandle.FromIntPtr(ip2).Target;
You use GCHandle.Alloc to lock an instance of DataLoggerWrap in memory, but then you never pass it out to unmanaged - so why do you lock it?
You also never free it?
The second line then grabs back a reference - why the circular path? why the reference - you never use it?
3) You set the IntPtrs to null - why? - this will have no effect outside of the function scope.
4) You need to know what the contract of the callback is. Who owns pData the callback or the calling function?

I'm with #jdehaan, except CallingConvetion.StdCall could be the answer, especially when the 3rd party lib is written in BC++, for example.

Pinning a delegate within a struct before passing to unmanaged code

I'm trying to use an unmanaged C dll for loading image data into a C# application. The library has a fairly simple interface where you pass in a struct that contains three callbacks, one to receive the size of the image, one that receives each row of the pixels and finally one called when the load is completed. Like this (C# managed definition):
[System.Runtime.InteropServices.StructLayoutAttribute(System.Runtime.InteropServices.LayoutKind.Sequential)]
public struct st_ImageProtocol
{
public st_ImageProtocol_done Done;
public st_ImageProtocol_setSize SetSize;
public st_ImageProtocol_sendLine SendLine;
}
The types starting st_ImageProtocol are delgates:
public delegate int st_ImageProtocol_sendLine(System.IntPtr localData, int rowNumber, System.IntPtr pixelData);
With the test file that I'm using the SetSize should get called once, then the SendLine will get called 200 times (once for each row of pixels in the image), finally the Done callback gets triggered. What actually happens is that the SendLine is called 19 times and then a AccessViolationException is thrown claiming that the library tried to access protected memory.
I have access to the code of the C library (though I can't change the functionality) and during the loop where it calls the SendLine method it does not allocate or free any new memory, so my assumption is that the delegate itself is the issue and I need to pin it before I pass it in (I have no code inside the delegate itself currently, besides a counter to see how often it gets called, so I doubt I'm breaking anything on the managed side). The problem is that I don't know how to do this; the method I've been using to declare the structs in unmanaged space doesn't work with delegates (Marshal.AllocHGlobal()) and I can't find any other suitable method. The delegates themselves are static fields in the Program class so they shouldn't be being garbage collected, but I guess the runtime could be moving them.
This blog entry by Chris Brumme says that delegates don't need to be pinned before being passed into unmanaged code:
Clearly the unmanaged function pointer must refer to a fixed address. It would be a disaster if the GC were relocating that! This leads many applications to create a pinning handle for the delegate. This is completely unnecessary. The unmanaged function pointer actually refers to a native code stub that we dynamically generate to perform the transition & marshaling. This stub exists in fixed memory outside of the GC heap.
But I don't know if this holds true when the delegate is part of a struct. It does imply that it is possible to manually pin them though, and I'm interested in how to do this or any better suggestions as to why a loop would run 19 times then suddenly fail.
Thanks.
Edited to answer Johan's questions...
The code that allocates the struct is as follows:
_sendLineFunc = new st_ImageProtocol_sendLine(protocolSendLineStub);
_imageProtocol = new st_ImageProtocol()
{
//Set some other properties...
SendLine = _sendLineFunc
};
int protocolSize = Marshal.SizeOf(_imageProtocol);
_imageProtocolPtr = Marshal.AllocHGlobal(protocolSize);
Marshal.StructureToPtr(_imageProtocol, _imageProtocolPtr, true);
Where the _sendLineFunc and the _imageProtocol variables are both static fields of the Program class. If I understand the internals of this correctly, that means that I'm passing an unmanaged pointer to a copy of the _imageProtocol variable into the C library, but that copy contains a reference to the static _sendLineFunc. This should mean that the copy isn't touched by the GC - since it is unmanaged - and the delegate won't be collected since it is still in scope (static).
The struct actually gets passed to the library as a return value from another callback, but as a pointer:
private static IntPtr beginCallback(IntPtr localData, en_ImageType imageType)
{
return _imageProtocolPtr;
}
Basically there is another struct type that holds the image filename and the function pointer to this callback, the library figures out what type of image is stored in the file and uses this callback to request the correct protocol struct for the given type. My filename struct is declared and managed in the same way as the protocol one above, so probably contains the same mistakes, but since this delegate is only called once and called quickly I haven't had any problems with it yet.
Edited to update
Thanks to everybody for their responses, but after spending another couple of days on the problem and making no progress I decided to shelve it. In case anyone is interested I was attempting write a tool for users of the Lightwave 3D rendering application and a nice feature would have been the ability to view all the different image formats that Lightwave supports (some of which are fairly exotic). I thought that the best way to do this would be to write a C# wrapper for the plugin architecture that Lightwave uses for image manipulation so I could use their code to actually load the files. Unfortunately after trying a number of the plugins against my solution I had a variety of errors that I couldn't understand or fix and my guess is that Lightwave doesn't call the methods on the plugins in a standard way, probably to improve the security of running external code (wild stab in the dark, I admit). For the time being I'm going to drop the image feature and if I do decide to reinstate it I'll approach it in a different way.
Thanks again, I learnt a lot through this process even though I didn't get the result I wanted.

I had a similar problem when registering a callback delegate (it would be called, then poof!). My problem was that the object with the method being delegated was getting GC'ed. I created the object in a more global place so as to keep it from being GC'ed.
If something like that doesn't work, here are some other things to look at:
As additional info, take a look at GetFunctionPointerForDelegate from the Marshal class. That is another way you could do this. Just make sure that the delegates are not GC'ed. Then, instead of delegates in your struct, declare them as IntPtr.
That may not solve the pinning, but take a look at fixed keyword, even though that may not work for you since you are dealing with a longer lifetime than for what that is typically used.
Finally, look at stackalloc for creating non-GC memory. These methods will require the use of unsafe, and might therefore put some other constraints on your Assemblies.

It would be interesting to know a little more:
How do you create the ImageProtocol struct? Is it a local variable or a class member or do you allocate it in unmanaged memory with Marshal.AllocHGlobal?
How is it sent to the C function? Directly as stack variable or as a pointer?
A really tricky problem! It feels like the delegate data is moved around by the GC which causes the access violation. The interesting thing is that the delegate data type is a reference data type, which stores its data on the GC heap. This data contains things like the address of the function to call (function pointer) but also a reference to the object that contains the function. This should mean that even though the actual function code is stored outside of the GC heap, the data that holds the function pointer is stored in the GC heap and can hence be moved by the GC. I thought about the problem a lot last night but haven't come up with a solution....

You don't say exactly how the callback is declared in the C library. Unless it is explictly declared __stdcall you'll slowly corrupt your stack. You'll see your method get called (probably with the parameters reversed) but at some point in the future the program will crash.
So far as I know there is no way around that, other than writing another callback function in C that sits between the C# code and the library that wants a __cdecl callback.

If the c function is a __cdecl function then you have to use the Attribut
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
before the delegate declaration.