We Keep Coding

Efficient reading structured binary data from a file

I have the following code fragment that reads a binary file and validates it:
FileStream f = File.OpenRead("File.bin");
MemoryStream memStream = new MemoryStream();
memStream.SetLength(f.Length);
f.Read(memStream.GetBuffer(), 0, (int)f.Length);
f.Seek(0, SeekOrigin.Begin);
var r = new BinaryReader(f);
Single prevVal=0;
do
{
r.ReadUInt32();
var val = r.ReadSingle();
if (prevVal!=0) {
var diff = Math.Abs(val - prevVal) / prevVal;
if (diff > 0.25)
Console.WriteLine("Bad!");
}
prevVal = val;
}
while (f.Position < f.Length);
It unfortunately works very slowly, and I am looking to improve this. In C++, I would simply read the file into a byte array and then recast that array as an array of structures:
struct S{
int a;
float b;
}
How would I do this in C#?

define a struct (possible a readonly struct) with explicit layout ([StructLayout(LayoutKind.Explicit)]) that is precisely the same as your C++ code, then one of:
open the file as a memory-mapped file, get the pointer to the data; use either unsafe code on the raw pointer, or use Unsafe.AsRef<YourStruct> on the data, and Unsafe.Add<> to iterate
open the file as a memory-mapped file, get the pointer to the data; create a custom memory over the pointer (of your T), and iterate over the span
open the file as a byte[]; create a Span<byte> over the byte[], then use MemoryMarshal.Cast<,> to create a Span<YourType>, and iterate over that
open the file as a byte[]; use fixed to pin the byte* and get a pointer; use unsafe code to walk the pointer
something involve "pipelines" - a Pipe that is the buffer, maybe using StreamConnection on a FileStream for filling the pipe, and a worker loop that dequeues from the pipe; complication: the buffers can be discontiguous and may split at inconvenient places; solvable, but subtle code required whenever the first span isn't at least 8 bytes
(or some combination of those concepts)
Any of those should work much like your C++ version. The 4th is simple, but for very large data you probably want to prefer memory-mapped files

This is what we use (compatible with older versions of C#):
public static T[] FastRead<T>(FileStream fs, int count) where T: struct
{
int sizeOfT = Marshal.SizeOf(typeof(T));
long bytesRemaining = fs.Length - fs.Position;
long wantedBytes = count * sizeOfT;
long bytesAvailable = Math.Min(bytesRemaining, wantedBytes);
long availableValues = bytesAvailable / sizeOfT;
long bytesToRead = (availableValues * sizeOfT);
if ((bytesRemaining < wantedBytes) && ((bytesRemaining - bytesToRead) > 0))
{
Debug.WriteLine("Requested data exceeds available data and partial data remains in the file.");
}
T[] result = new T[availableValues];
GCHandle gcHandle = GCHandle.Alloc(result, GCHandleType.Pinned);
try
{
uint bytesRead;
if (!ReadFile(fs.SafeFileHandle, gcHandle.AddrOfPinnedObject(), (uint)bytesToRead, out bytesRead, IntPtr.Zero))
{
throw new IOException("Unable to read file.", new Win32Exception(Marshal.GetLastWin32Error()));
}
Debug.Assert(bytesRead == bytesToRead);
}
finally
{
gcHandle.Free();
}
GC.KeepAlive(fs);
return result;
}
[System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Interoperability", "CA1415:DeclarePInvokesCorrectly")]
[DllImport("kernel32.dll", SetLastError=true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool ReadFile
(
SafeFileHandle hFile,
IntPtr lpBuffer,
uint nNumberOfBytesToRead,
out uint lpNumberOfBytesRead,
IntPtr lpOverlapped
);
NOTE: This only works for structs that contain only blittable types, of course. And you must use [StructLayout(LayoutKind.Explicit)] and declare the packing to ensure that the struct layout is identical to the binary format of the data in the file.
For recent versions of C#, you can use Span as mentioned by Marc in the other answer!

Thank you everyone for very helpful comments and answers. Given this input, this is my preferred solution:
[StructLayout(LayoutKind.Sequential, Pack = 1)]
struct Data
{
public UInt32 dummy;
public Single val;
};
static void Main(string[] args)
{
byte [] byteArray = File.ReadAllBytes("File.bin");
ReadOnlySpan<Data> dataArray = MemoryMarshal.Cast<byte, Data>(new ReadOnlySpan<byte>(byteArray));
Single prevVal=0;
foreach( var v in dataArray) {
if (prevVal!=0) {
var diff = Math.Abs(v.val - prevVal) / prevVal;
if (diff > 0.25)
Console.WriteLine("Bad!");
}
prevVal = v.val;
}
}
}
It indeed works much faster than the original implementation.

You are actually not using the MemoryStream at all currently. Your BinaryReader accesses the file directly. To have the BinaryReader use the MemoryStream instead:
Replace
f.Seek(0, SeekOrigin.Begin);
var r = new BinaryReader(f);
...
while (f.Position < f.Length);
with
memStream.Seek(0, SeekOrigin.Begin);
var r = new BinaryReader(memStream);
...
while(r.BaseStream.Position < r.BaseStream.Length)

What is the C# equivalent of this C++ code (with InPtr)

I'm implementing a Custom Credential Provider in C#. I'm using a C++ project as example. This piece of C++ code provides an image to Windows. The way I see it phbmp is a pointer to the image-bitmap. The code either updates the pointer so it points to a new bitmap (read from Resource) or it loads the bitmap to the address pointed by phbmp. I'm not sure if the pointer itself is changed or not.
// Get the image to show in the user tile
HRESULT CSampleCredential::GetBitmapValue(DWORD dwFieldID, _Outptr_result_nullonfailure_ HBITMAP *phbmp)
{
HRESULT hr;
*phbmp = nullptr;
if ((SFI_TILEIMAGE == dwFieldID))
{
HBITMAP hbmp = LoadBitmap(HINST_THISDLL, MAKEINTRESOURCE(IDB_TILE_IMAGE));
if (hbmp != nullptr)
{
hr = S_OK;
*phbmp = hbmp;
}
else
{
hr = HRESULT_FROM_WIN32(GetLastError());
}
}
else
{
hr = E_INVALIDARG;
}
return hr;
}
Below is the C# equivalent I'm implementing:
public int GetBitmapValue(uint dwFieldID, IntPtr phbmp)
{
if (dwFieldID == 2)
{
Bitmap image = Resource1.TileImage;
ImageConverter imageConverter = new ImageConverter();
byte[] bytes = (byte[])imageConverter.ConvertTo(image, typeof(byte[]));
Marshal.Copy(bytes, 0, phbmp, bytes.Length);
return HResultValues.S_OK;
}
return HResultValues.E_INVALIDARG;
}
What I'm trying to do:
Load the image from resource (this works, it has the correct length)
Convert the Bitmap to an array of bytes
Copy these bytes to the address pointed by phbmp
This crashes, I assume because of memory-allocation.
The parameters in this method are defined by an interface (in CredentialProvider.Interop.dll, which is provided by Microsoft - I think). So I'm pretty sure it's correct and phbmp is not an out-parameter.
Because it is not an out-parameter I can not change phbmp to let it point to my bitmap, right? I have assigned phbmp to Bitmap.GetHbitmap() and that doesn't crash but it isn't working either. I assume that the change to phbmp is only local in this method.
I can understand that it is not possible to alloc memory to a predefined address. It's the other way around: you alloc memory and get an pointer to it. But then this change is local again. How does this work?

Although some people agreed that IntPtr should be an out-parameter (see comments in https://syfuhs.net/2017/10/15/creating-custom-windows-credential-providers-in-net/) the answer was actually:
var bmp = new Bitmap(imageStream);
Marshal.WriteIntPtr(phbmp, bmp.GetHbitmap());

Get IntPtr value from BitmapImage or stream in UWP

I'm working with image processing in WinForm and it work very well when I have Bitmap and BitmapData, I can easily get IntPtr from it. But in UWP, I have no way to get IntPtr from them. So do we have any way to do that?
UPDATE: If we cannot get IntPtr value, can we get the pointer address for that image? Something like this in WinForm:
byte* src = (byte*) BitmapData.Scan0.ToPointer( );

You could get pxiel data from file stream via BitmapDecoder and PixelDataProvider:
Windows.Storage.Streams.IRandomAccessStream random = await Windows.Storage.Streams.RandomAccessStreamReference.CreateFromUri(new Uri("ms-appx:///Assets/StoreLogo.png")).OpenReadAsync();
Windows.Graphics.Imaging.BitmapDecoder decoder = await Windows.Graphics.Imaging.BitmapDecoder.CreateAsync(random);
Windows.Graphics.Imaging.PixelDataProvider pixelData = await decoder.GetPixelDataAsync();
byte[] buffer = pixelData.DetachPixelData();
Then you could get Intptr from byte array via unsafe code
unsafe
{
fixed (byte* p = buffer)
{
IntPtr ptr = (IntPtr)p;
// do you stuff here
}
}
If compile unsafe code, you need to enable the Allow Unsafe Code option in project's build property.

MarshalAs(UnmanagedType.LPStr) - how does this convert utf-8 strings to char*

The question title is basically what I'd like to ask:
[MarshalAs(UnmanagedType.LPStr)] - how does this convert utf-8 strings to char* ?
I use the above line when I attempt to communicate between c# and c++ dlls;
more specifically, between:
somefunction(char *string) [c++ dll]
somefunction([MarshalAs(UnmanagedType.LPStr) string text) [c#]
When I send my utf-8 text (scintilla.Text) through c# and into my c++ dll,
I'm shown in my VS 10 debugger that:
the c# string was successfully converted to char*
the resulting char* properly reflects the corresponding utf-8 chars (including the bit in Korean) in the watch window.
Here's a screenshot (with more details):
As you can see, initialScriptText[0] returns the single byte(char): 'B' and the contents of char* initialScriptText are displayed properly (including Korean) in the VS watch window.
Going through the char pointer, it seems that English is saved as one byte per char, while Korean seems to be saved as two bytes per char. (the Korean word in the screenshot is 3 letters, hence saved in 6 bytes)
This seems to show that each 'letter' isn't saved in equal size containers, but differs depending on language. (possible hint on type?)
I'm trying to achieve the same result in pure c++: reading in utf-8 files and saving the result as char*.
Here's an example of my attempt to read a utf-8 file and convert to char* in c++:
observations:
loss in visual when converting from wchar_t* to char*
since result, s8 displays the string properly, I know I've converted the utf-8 file content in wchar_t* successfully to char*
since 'result' retains the bytes I've taken directly from the file, but I'm getting a different result from what I had through c# (I've used the same file), I've concluded that the c# marshal has put the file contents through some other procedure to further mutate the text to char*.
(the screenshot also shows my terrible failure in using wcstombs)
note: I'm using the utf8 header from (http://utfcpp.sourceforge.net/)
Please correct me on any mistakes in my code/observations.
I'd like to be able to mimic the result I'm getting through the c# marshal and I've realised after going through all this that I'm completely stuck. Any ideas?

[MarshalAs(UnmanagedType.LPStr)] - how does this convert utf-8 strings to char* ?
It doesn't. There is no such thing as a "utf-8 string" in managed code, strings are always encoded in utf-16. The marshaling from and to an LPStr is done with the default system code page. Which makes it fairly remarkable that you see Korean glyphs in the debugger, unless you use code page 949.
If interop with utf-8 is a hard requirement then you need to use a byte[] in the pinvoke declaration. And convert back and forth yourself with System.Text.Encoding.UTF8. Use its GetString() method to convert the byte[] to a string, its GetBytes() method to convert a string to byte[]. Avoid all this if possible by using wchar_t[] in the native code.

While the other answers are correct, there has been a major development in .NET 4.7. Now there is an option that does exactly what UTF-8 needs: UnmanagedType.LPUTF8Str. I tried it and it works like a Swiss chronometre, doing exactly what it sounds like.
In fact, I even used MarshalAs(UnmanagedType.LPUTF8Str) in one parameter and MarshalAs(UnmanagedType.LPStr) in another. Also works. Here is my method (takes in string parameters and returns a string via a parameter):
[DllImport("mylib.dll", ExactSpelling = true, CallingConvention = CallingConvention.StdCall)]
public static extern void ProcessContent([MarshalAs(UnmanagedType.LPUTF8Str)]string content,
[MarshalAs(UnmanagedType.LPUTF8Str), Out]StringBuilder outputBuffer,[MarshalAs(UnmanagedType.LPStr)]string settings);
Thanks, Microsoft! Another nuisance is gone.

ICustomMarshaler can be used, in case of using .NET Framework earlier than 4.7.
class UTF8StringCodec : ICustomMarshaler
{
public static ICustomMarshaler GetInstance(string cookie) => new UTF8StringCodec();
public void CleanUpManagedData(object ManagedObj)
{
// nop
}
public void CleanUpNativeData(IntPtr pNativeData)
{
Marshal.FreeCoTaskMem(pNativeData);
}
public int GetNativeDataSize()
{
throw new NotImplementedException();
}
public IntPtr MarshalManagedToNative(object ManagedObj)
{
var text = $"{ManagedObj}";
var bytes = Encoding.UTF8.GetBytes(text);
var ptr = Marshal.AllocCoTaskMem(bytes.Length + 1);
Marshal.Copy(bytes, 0, ptr, bytes.Length);
Marshal.WriteByte(ptr, bytes.Length, 0);
return ptr;
}
public object MarshalNativeToManaged(IntPtr pNativeData)
{
if (pNativeData == IntPtr.Zero)
{
return null;
}
var bytes = new MemoryStream();
var ofs = 0;
while (true)
{
var byt = Marshal.ReadByte(pNativeData, ofs);
if (byt == 0)
{
break;
}
bytes.WriteByte(byt);
ofs++;
}
return Encoding.UTF8.GetString(bytes.ToArray());
}
}
P/Invoke declaration:
[DllImport("native.dll", CallingConvention = CallingConvention.Cdecl)]
private extern static int NativeFunc(
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(UTF8StringCodec))] string path
);
Usage inside callback:
[StructLayout(LayoutKind.Sequential)]
struct Options
{
[MarshalAs(UnmanagedType.FunctionPtr)]
public CallbackFunc callback;
}
[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int CallbackFunc(
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(UTF8StringCodec))] string path
);

If you need to marshal UTF-8 string do it manually.
Define function with IntPtr instead of string:
somefunction(IntPtr text)
Then convert text to zero-terminated UTF8 array of bytes and write them to IntPtr:
byte[] retArray = Encoding.UTF8.GetBytes(text);
byte[] retArrayZ = new byte[retArray.Length + 1];
Array.Copy(retArray, retArrayZ, retArray.Length);
IntPtr retPtr = AllocHGlobal(retArrayZ.Length);
Marshal.Copy(retArrayZ, 0, retPtr, retArrayZ.Length);
somefunction(retPtr);

How to convert byte[] to IMediaSample on C#

Following some samples i saw on this site im trying the following code:
var byteArray = new byte[m.Length -10];
myMemoryStream.Position = 10;
myMemoryStream.Read(byteArray,0,(int)myMemoryStream.Length-10);
GCHandle pinnedArray = GCHandle.Alloc(byteArray, GCHandleType.Pinned);
IntPtr pointer = pinnedArray.AddrOfPinnedObject();
IMediaSample pSample = null;
Marshal.PtrToStructure( pointer,pSample);
pinnedArray.Free();
But i recieve a "NullArgumentException" when calling Marshal.PtrToStructure( pointer,pSample).
Of course i know the reason of that error, what i would like to know is the right way to do it.
If you need more information tell me and i will add it as fast as i can.
Please help.

You need to create the actual structure, which also implements the IMediaSample interface.
public struct IMediaSampleStructure : IMediaSample
{
// Implementation.
}
Then, you can do something like this:
IMediaSample pSample = new IMediaSampleStructure();
Marshal.PtrToStructure(pointer, (IMediaSampleStructure)pSample);