Related
I want to get the infos about the perfomance of each processor by using NtQuerySystemInformation from ntdll. Now i have the problem that it just runs trough all 5 tries and then returns.
NtQuerySystemInformation returns a NtStatus which is always InfoLengthMismatch which usually means i have to make my buffer larger. If i try it with only one processor (no array) and a buffersize of 0x10000 it works kinda fine, it gives me InfoLengthMismatch in the first try but 2nd try always works.
I tried to increase the buffer by 100 times, also 1000 times but nothing worked.
private _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION[] GetPerformanceInfo()
{
try
{
//Getting count of processors
SYSTEM_INFO sysInfo = new SYSTEM_INFO();
GetSystemInfo(out sysInfo);
int processors = (int)sysInfo.numberOfProcessors;
int tries = 0;
while (true)
{
//buffer size
uint infoLength = (uint)(Marshal.SizeOf(typeof(_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION)) * processors);
//buffer
IntPtr infoPtr = Marshal.AllocHGlobal((int)infoLength);
//Problem: result is always InfoLengthMismatch
NtStatus result = NtQuerySystemInformation(SYSTEM_INFORMATION_CLASS.SystemProcessorPerformanceInformation, infoPtr, infoLength, out infoLength);
//if success, get the array and return it
if (result == NtStatus.Success)
{
int szStruct = Marshal.SizeOf(typeof(_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION));
_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION[] localStructs = new _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION[processors];
for (uint i = 0; i < processors; i++)
localStructs[i] = (_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION)Marshal.PtrToStructure(new IntPtr(infoPtr.ToInt64() + (szStruct * processors)), typeof(_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION));
return localStructs;
}
//free ptr when fail
Marshal.FreeHGlobal(infoPtr);
if (++tries > 5)
return null;
//set ptr again, new try
infoPtr = Marshal.AllocHGlobal((int)infoLength);
}
}
catch (Exception e)
{
Console.WriteLine(e.Source + ": " + e.Message.ToString());
return null;
}
}
}
//struct of a large int
[StructLayout(LayoutKind.Explicit, Size = 8)]
public struct LARGE_INTEGER
{
[FieldOffset(0)] public Int64 QuadPart;
[FieldOffset(0)] public UInt32 LowPart;
[FieldOffset(4)] public Int32 HighPart;
}
//struct
public struct _SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION
{
public LARGE_INTEGER IdleTime;
public LARGE_INTEGER KernelTime;
public LARGE_INTEGER UserTime;
public LARGE_INTEGER Reserved1;
public ulong Reserved2;
}
EDIT: Wrong thing was this line:
uint infoLength = (uint)(Marshal.SizeOf(typeof(_SYSTEM_PROCESSOR_PERFORMANCE_INFORMATION)) * processors);
Usually if the size is wrong, it will out the right size to the infoLength var, but in this case I'm setting it every time at the beginning of the loop.
There are a couple things I've noticed. First off, you're allocating memory twice, and only freeing it once:
//buffer
IntPtr infoPtr = Marshal.AllocHGlobal((int)infoLength);
//...
//free ptr when fail
Marshal.FreeHGlobal(infoPtr);
//...
//set ptr again, new try
infoPtr = Marshal.AllocHGlobal((int)infoLength);
Secondly, When NtQuerySystemInformation returns an error, it will set the optional out parameter to the size required. Going off of what I see, the buffer size you're creating is 8 * 5 * processors. Or 40 * processors.
You can confirm that as well as the appropriate required size by outputting infoLength before and after the call.
// Should be 40 * processors
Console.WriteLine((int)infoLength);
//Problem: result is always InfoLengthMismatch
NtStatus result = NtQuerySystemInformation(SYSTEM_INFORMATION_CLASS.SystemProcessorPerformanceInformation,
infoPtr, infoLength, out infoLength);
// Will be bigger than 40 * processors
Console.WriteLine((int)infoLength);
I am currently checking that the digital signature of windows installer files (.msi) is valid in C# using the WinVerifyTrust api. I am also verifying that the thumbprint from the signature is from a known list.
I need to do the same for Mac OSX files (.dmg) in C# (on Windows). Is there any way to do this?
Any DMG file has 'Koly' block, I don't think you will easily find ready to go code on windows capable to read it in C#...But take a look here http://newosxbook.com/DMG.html
What you practically interested in is last 512 bytes of a file.
typedef struct {
char Signature[4]; // Magic ('koly')
uint32_t Version; // Current version is 4
uint32_t HeaderSize; // sizeof(this), always 512
uint32_t Flags; // Flags
uint64_t RunningDataForkOffset; //
uint64_t DataForkOffset; // Data fork offset (usually 0, beginning of file)
uint64_t DataForkLength; // Size of data fork (usually up to the XMLOffset, below)
uint64_t RsrcForkOffset; // Resource fork offset, if any
uint64_t RsrcForkLength; // Resource fork length, if any
uint32_t SegmentNumber; // Usually 1, may be 0
uint32_t SegmentCount; // Usually 1, may be 0
uuid_t SegmentID; // 128-bit GUID identifier of segment (if SegmentNumber !=0)
uint32_t DataChecksumType; // Data fork
uint32_t DataChecksumSize; // Checksum Information
uint32_t DataChecksum[32]; // Up to 128-bytes (32 x 4) of checksum
uint64_t XMLOffset; // Offset of property list in DMG, from beginning
uint64_t XMLLength; // Length of property list
uint8_t Reserved1[120]; // 120 reserved bytes - zeroed
uint32_t ChecksumType; // Master
uint32_t ChecksumSize; // Checksum information
uint32_t Checksum[32]; // Up to 128-bytes (32 x 4) of checksum
uint32_t ImageVariant; // Commonly 1
uint64_t SectorCount; // Size of DMG when expanded, in sectors
uint32_t reserved2; // 0
uint32_t reserved3; // 0
uint32_t reserved4; // 0
} __attribute__((__packed__)) UDIFResourceFile;
Consider the following lines of code as example of reading bytes
public static uint ToUInt32BigEndian(byte[] buffer, int offset)
{
uint val = (uint)(((buffer[offset + 0] << 24) & 0xFF000000U) | ((buffer[offset + 1] << 16) & 0x00FF0000U)
| ((buffer[offset + 2] << 8) & 0x0000FF00U) | ((buffer[offset + 3] << 0) & 0x000000FFU));
return val;
}
public static ulong ToUInt64BigEndian(byte[] buffer, int offset)
{
return ((ulong)ToUInt32BigEndian(buffer, offset + 0) << 32) | ToUInt32BigEndian(buffer, offset + 4);
}
internal class UdifChecksum : IByteArraySerializable
{
public uint ChecksumSize;
public byte[] Data;
public uint Type;
public int Size
{
get { return 136; }
}
public int ReadFrom(byte[] buffer, int offset)
{
Type = EndianUtilities.ToUInt32BigEndian(buffer, offset + 0);
ChecksumSize = EndianUtilities.ToUInt32BigEndian(buffer, offset + 4);
Data = EndianUtilities.ToByteArray(buffer, offset + 8, 128);
return 136;
}
public void WriteTo(byte[] buffer, int offset)
{
throw new NotImplementedException();
}
}
Here you can see example of reading all properties from file header
internal class UdifResourceFile : IByteArraySerializable
{
public UdifChecksum DataForkChecksum;
public ulong DataForkLength;
public ulong DataForkOffset;
public uint Flags;
public uint HeaderSize;
public uint ImageVariant;
public UdifChecksum MasterChecksum;
public ulong RsrcForkLength;
public ulong RsrcForkOffset;
public ulong RunningDataForkOffset;
public long SectorCount;
public uint SegmentCount;
public Guid SegmentGuid;
public uint SegmentNumber;
public uint Signature;
public uint Version;
public ulong XmlLength;
public ulong XmlOffset;
public bool SignatureValid
{
get { return Signature == 0x6B6F6C79; }
}
public int Size
{
get { return 512; }
}
public int ReadFrom(byte[] buffer, int offset)
{
Signature = EndianUtilities.ToUInt32BigEndian(buffer, offset + 0);
Version = EndianUtilities.ToUInt32BigEndian(buffer, offset + 4);
HeaderSize = EndianUtilities.ToUInt32BigEndian(buffer, offset + 8);
Flags = EndianUtilities.ToUInt32BigEndian(buffer, offset + 12);
RunningDataForkOffset = EndianUtilities.ToUInt64BigEndian(buffer, offset + 16);
DataForkOffset = EndianUtilities.ToUInt64BigEndian(buffer, offset + 24);
DataForkLength = EndianUtilities.ToUInt64BigEndian(buffer, offset + 32);
RsrcForkOffset = EndianUtilities.ToUInt64BigEndian(buffer, offset + 40);
RsrcForkLength = EndianUtilities.ToUInt64BigEndian(buffer, offset + 48);
SegmentNumber = EndianUtilities.ToUInt32BigEndian(buffer, offset + 56);
SegmentCount = EndianUtilities.ToUInt32BigEndian(buffer, offset + 60);
SegmentGuid = EndianUtilities.ToGuidBigEndian(buffer, offset + 64);
DataForkChecksum = EndianUtilities.ToStruct<UdifChecksum>(buffer, offset + 80);
XmlOffset = EndianUtilities.ToUInt64BigEndian(buffer, offset + 216);
XmlLength = EndianUtilities.ToUInt64BigEndian(buffer, offset + 224);
MasterChecksum = EndianUtilities.ToStruct<UdifChecksum>(buffer, offset + 352);
ImageVariant = EndianUtilities.ToUInt32BigEndian(buffer, offset + 488);
SectorCount = EndianUtilities.ToInt64BigEndian(buffer, offset + 492);
return Size;
}
public void WriteTo(byte[] buffer, int offset)
{
throw new NotImplementedException();
}
}
More details for code here (https://github.com/DiscUtils/DiscUtils)
If you do read XML definition of file (check XMLOffset and XMLLength) you should be able to identify file structure and extract files (and it's properties).
The XML Property list (which is uncompressed and easily viewable by seeking to the DOCTYPE declaration using more(1) or using tail(1)) is technically the resource fork of the DMG. The property list file contains, at a minimum, a "blkx" key, though it may contain other key/values, most commonly "plst", and sometimes a service level agreement (SLA) which will be displayed by the OS (specifically, /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/Resources/DiskImages UI Agent.app/Contents/MacOS/DiskImages UI Agent) as a pre-requisite to attaching the DMG*. Due to XML parser restrictions, data in the property list is 7-bit. This forces all binary (8-bit) data to be encoded using Base-64 encoding (a wiser choice would have been using CDATA blocks)
Since MAC is signing files (as far as I remember) the signature could be taken from there. Checksum at top level is protecting from package modification and should be made with same certificate that is used for signing.
Hope this helps.
Use OpenSsl, Bouncy Castle or System.Security.Cryptography (sha256 checksum or similar) to check the checksum using C# on your system. If you are the supplier you can generate a hash first and publish it on the download page for both .msi and .dmg files. I have tried this before and it works well. However, I have no code to attach to this answer at the moment, use ComputeHash function in System.Security.Cryptography.
If you don't have direct access to the files, you can download .dmg and create a hash from it using C#. A hash that will be correctly verified when checked unless manipulated that is. Creating a hash from all bytes of the file is way more secure than trust embedded data which can be replaced and signed to appear valid, unless you cross-check everything with the creator(s).
Hi I am trying to convert the C/C++ Strcut to C# and how to fill the structure member with address of another structure in C#?
C/C++ Struct looks like:
typedef struct _NDISUIO_QUERY_OID
{
NDIS_OID Oid;
PTCHAR ptcDeviceName;
UCHAR Data[sizeof(ULONG)];
} NDISUIO_QUERY_OID, *PNDISUIO_QUERY_OID;
typedef struct My_Struct
{
//les have 2 variables...
UINT a;
UINT b;
}My_STATS, *PMy_STATS;
PNDISUIO_QUERY_OID pQueryOid = NULL;
pQueryOid = (PNDISUIO_QUERY_OID)malloc(sizeof(NDISUIO_QUERY_OID)+ sizeof(My_STATS)) ;
PMy_STATS Statistics;
pQueryOid->Oid = ulOIDCode;//Required OID
pQueryOid->ptcDeviceName = AUB_NAME;//REquired STRING
memcpy(pQueryOid->Data, Statistics, sizeof(My_STATS));
My C# Struct is:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = sizeof(uint))]
public string Data;
};
Problem: How to copy the Statistics structure to Data array in C#??
Thanks :)
Here's my implementation (FYI, the SDF contains all of this code and a lot more)
internal class NDISQueryOid
{
protected const int NDISUIO_QUERY_OID_SIZE = 12;
protected byte[] m_data;
public int Size { get; private set; }
public NDISQueryOid(byte[] data)
{
int extrasize = data.Length;
Size = 8 + extrasize;
m_data = new byte[Size];
Buffer.BlockCopy(data, 0, m_data, DataOffset, data.Length);
}
public NDISQueryOid(int extrasize)
{
Size = NDISUIO_QUERY_OID_SIZE + extrasize;
m_data = new byte[Size];
}
protected const int OidOffset = 0;
public uint Oid
{
get { return BitConverter.ToUInt32(m_data, OidOffset); }
set
{
byte[] bytes = BitConverter.GetBytes(value);
Buffer.BlockCopy(bytes, 0, m_data, OidOffset, 4);
}
}
protected const int ptcDeviceNameOffset = OidOffset + 4;
public unsafe byte* ptcDeviceName
{
get
{
return (byte*)BitConverter.ToUInt32(m_data, ptcDeviceNameOffset);
}
set
{
byte[] bytes = BitConverter.GetBytes((UInt32)value);
Buffer.BlockCopy(bytes, 0, m_data, ptcDeviceNameOffset, 4);
}
}
protected const int DataOffset = ptcDeviceNameOffset + 4;
public byte[] Data
{
get
{
byte[] b = new byte[Size - DataOffset];
Array.Copy(m_data, DataOffset, b, 0, Size - DataOffset);
return b;
}
set
{
Size = 8 + value.Length;
m_data = new byte[Size];
Buffer.BlockCopy(value, 0, m_data, DataOffset, value.Length);
}
}
public byte[] getBytes()
{
return m_data;
}
public static implicit operator byte[](NDISQueryOid qoid)
{
return qoid.m_data;
}
}
Note that in my usage, the NDIS IOCT takes in a pointer (most of my NDIS work is all done as unsafe) so you'd have to do some adjustment there.
So if, for example, you're querying the BSSID, I know the BSSID data is 36 bytes, so I'd create something like this:
var queryOID = new NDISQueryOid(36);
then allocate the name and call NDIS (the production code has a lot more checking than this):
byte[] nameBytes = System.Text.Encoding.Unicode.GetBytes(adapterName + '\0');
fixed (byte* pName = &nameBytes[0])
{
queryOID.ptcDeviceName = pName;
queryOID.Oid = (uint)oid;
var bytes = queryOID.getBytes();
ndis.DeviceIoControl(IOCTL_NDISUIO_QUERY_OID_VALUE, bytes, bytes);
var result = new byte[queryOID.Data.Length];
Buffer.BlockCopy(queryOID.Data, 0, result, 0, result.Length);
}
EDIT
So the result member above is a byte array of the "result" of the query. What it means and how you interpret it depends on what the OID you queried was. For example, if you were querying the currently connected SSID (i.e. NDIS_OID.SSID), then that comes back as a 4-byte length followed by the ASCII-encoded name, so you'd decipher it like this:
int len = BitConverter.ToInt32(data, 0);
if (len > 0)
{
ssid = System.Text.Encoding.ASCII.GetString(data, 4, len);
}
But again, this is only for one specific OID. You have to handle every return case for every incoming OID you decide to support.
First you have the wrong translation of your C++ code: the C# equivalent of a C++ char[] is not a string, it's a byte[]. Once you have that, you just need to know, in general, how to copy a structure into a byte array. Here's a compilable example:
using System;
using System.Runtime.InteropServices;
struct Dest
{
public byte[] Data;
}
struct Src
{
public GCHandle StringHandle;
public long A;
public long B;
}
class Program
{
static void Main()
{
Copy();
}
static void Copy()
{
var str = "Hello";
var src = new Src {
A = 3,
B = 4,
StringHandle = GCHandle.Alloc(str, GCHandleType.Normal)
};
var dst = new Dest();
unsafe
{
Src* srcPtr = &src;
dst.Data = new byte[sizeof(Src)];
Marshal.Copy((IntPtr)srcPtr, dst.Data, 0, sizeof(Src));
}
// When you're sure no one can reference the string anymore
// (Including by accessing the data you put in dst.Data!)
src.StringHandle.Free();
}
EDIT: added example of how to deal with reference types such as strings.
Safely, you can't. .NET enforces type safety, which means that you simply can't force a string to be a structure. However, you can look at the data instead of doing unsafe type casts (why are you storing two uints in a string in the first place? And marshalling it as unicode?
First, you'll have to make Data a byte array. It might be possible to do this with a string as well, but that's just adding encoding issues to the mix; if you can, use byte[] instead. Also, if you don't need to have different kinds of data inside (it seems so), you could simply put the two uint fields right inside the struct and it should work just fine:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
public uint DataA;
public uint DataB;
};
The second approach would use a const-sized byte array, long enough to hold the two uints:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public struct _NDISUIO_QUERY_OID
{
public uint Oid;
[MarshalAs(UnmanagedType.LPWStr)]
public string ptcDeviceName;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = sizeof(ulong))]
public byte[] Data;
};
The first four bytes will be the first uint, the next will be the second.
And of course, you could also use a .NET struct the same way as in the original code - just make sure you use the correct datatype in _NDISUIO_QUERY_OID and it should work automagically.
One point to note though, it seems that the data returned isn't actually necessarily fixed-length. That is quite tricky and it basically means you'd have to deserialize the structure manually based on the pointer and length you get.
I am trying to use a DLL written in C in a C# application. I've created a simplified example that replicates the issue I am having.
The C code below creates an array of struct data's and assigns the array pointer to the array parameter passed to the get_data() function. The C# code is supposed to be the boilerplate code needed for marshalling the struct to be used in C#, but is presenting issues for me.
C code
#include <stdint.h>
#include <stdlib.h>
#include <string.h>
struct data {
int32_t value;
union {
struct person {
uint8_t first[10];
uint8_t last[10];
} person;
struct number {
int32_t imaginary;
int32_t real;
} number;
} type;
};
int get_data(int count, struct data ***array)
{
int i;
/* allocate pointers */
*array = calloc(count, sizeof(struct data*));
if (*array == NULL)
return 1;
for (i = 0; i < count; i++) {
/* allocate data struct */
struct data *data = calloc(1, sizeof(struct data));
if (data == NULL)
return 2;
if ((i % 2) == 0) {
/* if even, its human */
data->value = i;
memcpy(data->type.person.first, "john", 4);
memcpy(data->type.person.last, "doe", 3);
} else {
/* if odd its a number */
data->value = i;
data->type.number.imaginary = -1;
data->type.number.real = i + 1;
}
(*array)[i] = data;
}
return 0;
}
C# code
[DllImport("libdata.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern Int32 get_data(Int32 count, ref IntPtr array);
[StructLayout(LayoutKind.Sequential)]
public struct Person
{
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
public String first;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 10)]
public String last;
}
[StructLayout(LayoutKind.Sequential)]
public struct Number
{
public Int32 imaginary;
public Int32 real;
}
[StructLayout(LayoutKind.Explicit)]
public struct TypeUnion
{
[FieldOffset(0)]
public Person person;
[FieldOffset(0)]
public Number number;
}
[StructLayout(LayoutKind.Sequential)]
public struct Data
{
public Int32 value;
public TypeUnion type;
}
Right now when I run my test program I get an exception:
System.TypeLoadException was unhandled
Message=Could not load type 'WpfRibbonApplication1.TypeUnion' from assembly 'WpfRibbonApplication1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' because it contains an object field at offset 0 that is incorrectly aligned or overlapped by a non-object field.
I have tried several different ways of marshalling the Person strings, but get the exception whichever way I try (using this as reference). Am I missing something obvious? Could I get some help to properly read the array created in the C function within my C# application?
Edit (per David Heffernan's comment)
IntPtr arrayPtr = new IntPtr();
int count = 4;
int ret = LibData.get_data(count, ref arrayPtr);
Console.WriteLine("ret=" + ret);
for (int i = 0; i < count; i++)
{
IntPtr dataPtr = (IntPtr)Marshal.ReadIntPtr(arrayPtr) + (i * Marshal.SizeOf(typeof(IntPtr)));
Data data = (Data)Marshal.PtrToStructure(dataPtr, typeof(Data));
Console.WriteLine("value=" + data.value);
if ((i % 2) == 0)
{
// even is human
Console.WriteLine("first=" + data.type.first);
Console.WriteLine("last=" + data.type.last);
}
else
{
// odd is number
Console.WriteLine("imaginary=" + data.type.imaginary);
Console.WriteLine("real=" + data.type.real);
}
Console.WriteLine("");
}
The error message is telling you that you cannot overlay an object field with a non-object field. You are overlaying a string with an int.
There's no way you are going to get around that using FieldOffset to replicate the native union. As I see it you have two main options:
Stop using a union and include both Person and Number structs in the Data struct.
Continue using a union, but marshal it yourself. Use the Marshal class to read the data in the struct. For example, you'd use Marshal.ReadInt32 to read the integers, Marshal.Copy to read the character arrays and so on.
Question
I'm porting a C application into C#. The C app calls lots of functions from a 3rd-party DLL, so I wrote P/Invoke wrappers for these functions in C#. Some of these C functions allocate data which I have to use in the C# app, so I used IntPtr's, Marshal.PtrToStructure and Marshal.Copy to copy the native data (arrays and structures) into managed variables.
Unfortunately, the C# app proved to be much slower than the C version. A quick performance analysis showed that the above mentioned marshaling-based data copying is the bottleneck. I'm considering to speed up the C# code by rewriting it to use pointers instead. Since I don't have experience with unsafe code and pointers in C#, I need expert opinion regarding the following questions:
What are the drawbacks of using unsafe code and pointers instead of IntPtr and Marshaling? For example, is it more unsafe (pun intended) in any way? People seem to prefer marshaling, but I don't know why.
Is using pointers for P/Invoking really faster than using marshaling? How much speedup can be expected approximately? I couldn't find any benchmark tests for this.
Example code
To make the situation more clear, I hacked together a small example code (the real code is much more complex). I hope this example shows what I mean when I'm talking about "unsafe code and pointers" vs. "IntPtr and Marshal".
C library (DLL)
MyLib.h
#ifndef _MY_LIB_H_
#define _MY_LIB_H_
struct MyData
{
int length;
unsigned char* bytes;
};
__declspec(dllexport) void CreateMyData(struct MyData** myData, int length);
__declspec(dllexport) void DestroyMyData(struct MyData* myData);
#endif // _MY_LIB_H_
MyLib.c
#include <stdlib.h>
#include "MyLib.h"
void CreateMyData(struct MyData** myData, int length)
{
int i;
*myData = (struct MyData*)malloc(sizeof(struct MyData));
if (*myData != NULL)
{
(*myData)->length = length;
(*myData)->bytes = (unsigned char*)malloc(length * sizeof(char));
if ((*myData)->bytes != NULL)
for (i = 0; i < length; ++i)
(*myData)->bytes[i] = (unsigned char)(i % 256);
}
}
void DestroyMyData(struct MyData* myData)
{
if (myData != NULL)
{
if (myData->bytes != NULL)
free(myData->bytes);
free(myData);
}
}
C application
Main.c
#include <stdio.h>
#include "MyLib.h"
void main()
{
struct MyData* myData = NULL;
int length = 100 * 1024 * 1024;
printf("=== C++ test ===\n");
CreateMyData(&myData, length);
if (myData != NULL)
{
printf("Length: %d\n", myData->length);
if (myData->bytes != NULL)
printf("First: %d, last: %d\n", myData->bytes[0], myData->bytes[myData->length - 1]);
else
printf("myData->bytes is NULL");
}
else
printf("myData is NULL\n");
DestroyMyData(myData);
getchar();
}
C# application, which uses IntPtr and Marshal
Program.cs
using System;
using System.Runtime.InteropServices;
public static class Program
{
[StructLayout(LayoutKind.Sequential)]
private struct MyData
{
public int Length;
public IntPtr Bytes;
}
[DllImport("MyLib.dll")]
private static extern void CreateMyData(out IntPtr myData, int length);
[DllImport("MyLib.dll")]
private static extern void DestroyMyData(IntPtr myData);
public static void Main()
{
Console.WriteLine("=== C# test, using IntPtr and Marshal ===");
int length = 100 * 1024 * 1024;
IntPtr myData1;
CreateMyData(out myData1, length);
if (myData1 != IntPtr.Zero)
{
MyData myData2 = (MyData)Marshal.PtrToStructure(myData1, typeof(MyData));
Console.WriteLine("Length: {0}", myData2.Length);
if (myData2.Bytes != IntPtr.Zero)
{
byte[] bytes = new byte[myData2.Length];
Marshal.Copy(myData2.Bytes, bytes, 0, myData2.Length);
Console.WriteLine("First: {0}, last: {1}", bytes[0], bytes[myData2.Length - 1]);
}
else
Console.WriteLine("myData.Bytes is IntPtr.Zero");
}
else
Console.WriteLine("myData is IntPtr.Zero");
DestroyMyData(myData1);
Console.ReadKey(true);
}
}
C# application, which uses unsafe code and pointers
Program.cs
using System;
using System.Runtime.InteropServices;
public static class Program
{
[StructLayout(LayoutKind.Sequential)]
private unsafe struct MyData
{
public int Length;
public byte* Bytes;
}
[DllImport("MyLib.dll")]
private unsafe static extern void CreateMyData(out MyData* myData, int length);
[DllImport("MyLib.dll")]
private unsafe static extern void DestroyMyData(MyData* myData);
public unsafe static void Main()
{
Console.WriteLine("=== C# test, using unsafe code ===");
int length = 100 * 1024 * 1024;
MyData* myData;
CreateMyData(out myData, length);
if (myData != null)
{
Console.WriteLine("Length: {0}", myData->Length);
if (myData->Bytes != null)
Console.WriteLine("First: {0}, last: {1}", myData->Bytes[0], myData->Bytes[myData->Length - 1]);
else
Console.WriteLine("myData.Bytes is null");
}
else
Console.WriteLine("myData is null");
DestroyMyData(myData);
Console.ReadKey(true);
}
}
It's a little old thread, but I recently made excessive performance tests with marshaling in C#. I need to unmarshal lots of data from a serial port over many days. It was important to me to have no memory leaks (because the smallest leak will get significant after a couple of million calls) and I also made a lot of statistical performance (time used) tests with very big structs (>10kb) just for the sake of it (an no, you should never have a 10kb struct :-) )
I tested the following three unmarshalling strategies (I also tested the marshalling). In nearly all cases the first one (MarshalMatters) outperformed the other two.
Marshal.Copy was always slowest by far, the other two were mostly very close together in the race.
Using unsafe code can pose a significant security risk.
First:
public class MarshalMatters
{
public static T ReadUsingMarshalUnsafe<T>(byte[] data) where T : struct
{
unsafe
{
fixed (byte* p = &data[0])
{
return (T)Marshal.PtrToStructure(new IntPtr(p), typeof(T));
}
}
}
public unsafe static byte[] WriteUsingMarshalUnsafe<selectedT>(selectedT structure) where selectedT : struct
{
byte[] byteArray = new byte[Marshal.SizeOf(structure)];
fixed (byte* byteArrayPtr = byteArray)
{
Marshal.StructureToPtr(structure, (IntPtr)byteArrayPtr, true);
}
return byteArray;
}
}
Second:
public class Adam_Robinson
{
private static T BytesToStruct<T>(byte[] rawData) where T : struct
{
T result = default(T);
GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
try
{
IntPtr rawDataPtr = handle.AddrOfPinnedObject();
result = (T)Marshal.PtrToStructure(rawDataPtr, typeof(T));
}
finally
{
handle.Free();
}
return result;
}
/// <summary>
/// no Copy. no unsafe. Gets a GCHandle to the memory via Alloc
/// </summary>
/// <typeparam name="selectedT"></typeparam>
/// <param name="structure"></param>
/// <returns></returns>
public static byte[] StructToBytes<T>(T structure) where T : struct
{
int size = Marshal.SizeOf(structure);
byte[] rawData = new byte[size];
GCHandle handle = GCHandle.Alloc(rawData, GCHandleType.Pinned);
try
{
IntPtr rawDataPtr = handle.AddrOfPinnedObject();
Marshal.StructureToPtr(structure, rawDataPtr, false);
}
finally
{
handle.Free();
}
return rawData;
}
}
Third:
/// <summary>
/// http://stackoverflow.com/questions/2623761/marshal-ptrtostructure-and-back-again-and-generic-solution-for-endianness-swap
/// </summary>
public class DanB
{
/// <summary>
/// uses Marshal.Copy! Not run in unsafe. Uses AllocHGlobal to get new memory and copies.
/// </summary>
public static byte[] GetBytes<T>(T structure) where T : struct
{
var size = Marshal.SizeOf(structure); //or Marshal.SizeOf<selectedT>(); in .net 4.5.1
byte[] rawData = new byte[size];
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.StructureToPtr(structure, ptr, true);
Marshal.Copy(ptr, rawData, 0, size);
Marshal.FreeHGlobal(ptr);
return rawData;
}
public static T FromBytes<T>(byte[] bytes) where T : struct
{
var structure = new T();
int size = Marshal.SizeOf(structure); //or Marshal.SizeOf<selectedT>(); in .net 4.5.1
IntPtr ptr = Marshal.AllocHGlobal(size);
Marshal.Copy(bytes, 0, ptr, size);
structure = (T)Marshal.PtrToStructure(ptr, structure.GetType());
Marshal.FreeHGlobal(ptr);
return structure;
}
}
Considerations in Interoperability explains why and when Marshaling is required and at what cost. Quote:
Marshaling occurs when a caller and a callee cannot operate on the same instance of data.
repeated marshaling can negatively affect the performance of your application.
Therefore, answering your question if
... using pointers for P/Invoking really faster than using marshaling ...
first ask yourself a question if the managed code is able to operate on the unmanaged method return value instance. If the answer is yes then Marshaling and the associated performance cost is not required.
The approximate time saving would be O(n) function where n of the size of the marshalled instance.
In addition, not keeping both managed and unmanaged blocks of data in memory at the same time for the duration of the method (in "IntPtr and Marshal" example) eliminates additional overhead and the memory pressure.
What are the drawbacks of using unsafe code and pointers ...
The drawback is the risk associated with accessing the memory directly through pointers. There is nothing less safe to it than using pointers in C or C++. Use it if needed and makes sense. More details are here.
There is one "safety" concern with the presented examples: releasing of allocated unmanaged memory is not guaranteed after the managed code errors. The best practice is to
CreateMyData(out myData1, length);
if(myData1!=IntPtr.Zero) {
try {
// -> use myData1
...
// <-
}
finally {
DestroyMyData(myData1);
}
}
For anyone still reading,
Something I don't think I saw in any of the answers, - unsafe code does present something of a security risk. It's not a huge risk, it would be something quite challenging to exploit. However, if like me you work in a PCI compliant organization, unsafe code is disallowed by policy for this reason.
Managed code is normally very secure because the CLR takes care of memory location and allocation, preventing you from accessing or writing any memory you're not supposed to.
When you use the unsafe keyword and compile with '/unsafe' and use pointers, you bypass these checks and create the potential for someone to use your application to gain some level of unauthorized access to the machine it is running on. Using something like a buffer-overrun attack, your code could be tricked into writing instructions into an area of memory that might then be accessed by the program counter (i.e. code injection), or just crash the machine.
Many years ago, SQL server actually fell prey to malicious code delivered in a TDS packet that was far longer than it was supposed to be. The method reading the packet didn't check the length and continued to write the contents past the reserved address space. The extra length and content were carefully crafted such that it wrote an entire program into memory - at the address of the next method.
The attacker then had their own code being executed by the SQL server within a context that had the highest level of access. It didn't even need to break the encryption as the vulnerability was below this point in the transport layer stack.
Just wanted to add my experience to this old thread:
We used Marshaling in sound recording software - we received real time sound data from mixer into native buffers and marshaled it to byte[]. That was real performance killer. We were forced to move to unsafe structs as the only way to complete the task.
In case you don't have large native structs and don't mind that all data is filled twice - Marshaling is more elegant and much, much safer approach.
Two answers,
Unsafe code means it is not managed by the CLR. You need to take care of resources it uses.
You cannot scale the performance because there are so many factors effecting it. But definitely using pointers will be much faster.
Because you stated that your code calls to 3rd-party DLL, I think the unsafe code is more suited in you scenario. You ran into a particular situation of wapping variable-length array in a struct; I know, I know this kind of usage occurs all the time, but it's not always the case after all. You might want to have a look of some questions about this, for example:
How do I marshal a struct that contains a variable-sized array to C#?
If .. I say if .. you can modify the third party libraries a bit for this particular case, then you might consider the following usage:
using System.Runtime.InteropServices;
public static class Program { /*
[StructLayout(LayoutKind.Sequential)]
private struct MyData {
public int Length;
public byte[] Bytes;
} */
[DllImport("MyLib.dll")]
// __declspec(dllexport) void WINAPI CreateMyDataAlt(BYTE bytes[], int length);
private static extern void CreateMyDataAlt(byte[] myData, ref int length);
/*
[DllImport("MyLib.dll")]
private static extern void DestroyMyData(byte[] myData); */
public static void Main() {
Console.WriteLine("=== C# test, using IntPtr and Marshal ===");
int length = 100*1024*1024;
var myData1 = new byte[length];
CreateMyDataAlt(myData1, ref length);
if(0!=length) {
// MyData myData2 = (MyData)Marshal.PtrToStructure(myData1, typeof(MyData));
Console.WriteLine("Length: {0}", length);
/*
if(myData2.Bytes!=IntPtr.Zero) {
byte[] bytes = new byte[myData2.Length];
Marshal.Copy(myData2.Bytes, bytes, 0, myData2.Length); */
Console.WriteLine("First: {0}, last: {1}", myData1[0], myData1[length-1]); /*
}
else {
Console.WriteLine("myData.Bytes is IntPtr.Zero");
} */
}
else {
Console.WriteLine("myData is empty");
}
// DestroyMyData(myData1);
Console.ReadKey(true);
}
}
As you can see much of your original marshalling code is commented out, and declared a CreateMyDataAlt(byte[], ref int) for a coresponding modified external unmanaged function CreateMyDataAlt(BYTE [], int). Some of the data copy and pointer check turns to be unnecessary, that says, the code can be even simpler and probably runs faster.
So, what's so different with the modification? The byte array is now marshalled directly without warpping in a struct and passed to the unmanaged side. You don't allocate the memory within the unmanaged code, rather, just filling data to it(implementation details omitted); and after the call, the data needed is provided to the managed side. If you want to present that the data is not filled and should not be used, you can simply set length to zero to tell the managed side. Because the byte array is allocated within the managed side, it'll be collected sometime, you don't have to take care of that.
I had the same question today and I was looking for some concrete measurement values, but I couldn't find any. So I wrote my own tests.
The test is copying pixel data of a 10k x 10k RGB image. The image data is 300 MB (3*10^9 bytes). Some methods copy this data 10 times, others are faster and therefore copy it 100 times. The used copying methods include
array access via byte pointer
Marshal.Copy(): a) 1 * 300 MB, b) 1e9 * 3 bytes
Buffer.BlockCopy(): a) 1 * 300 MB, b) 1e9 * 3 bytes
Test environment:
CPU: Intel Core i7-3630QM # 2.40 GHz
OS: Win 7 Pro x64 SP1
Visual Studio 2015.3, code is C++/CLI, targeted .net version is 4.5.2, compiled for Debug.
Test results:
The CPU load is 100% for 1 core at all methods (equals 12.5% total CPU load).
Comparison of speed and execution time:
method speed exec.time
Marshal.Copy (1*300MB) 100 % 100%
Buffer.BlockCopy (1*300MB) 98 % 102%
Pointer 4.4 % 2280%
Buffer.BlockCopy (1e9*3B) 1.4 % 7120%
Marshal.Copy (1e9*3B) 0.95% 10600%
Execution times and calculated average throughput written as comments in the code below.
//------------------------------------------------------------------------------
static void CopyIntoBitmap_Pointer (array<unsigned char>^ i_aui8ImageData,
BitmapData^ i_ptrBitmap,
int i_iBytesPerPixel)
{
char* scan0 = (char*)(i_ptrBitmap->Scan0.ToPointer ());
int ixCnt = 0;
for (int ixRow = 0; ixRow < i_ptrBitmap->Height; ixRow++)
{
for (int ixCol = 0; ixCol < i_ptrBitmap->Width; ixCol++)
{
char* pPixel = scan0 + ixRow * i_ptrBitmap->Stride + ixCol * 3;
pPixel[0] = i_aui8ImageData[ixCnt++];
pPixel[1] = i_aui8ImageData[ixCnt++];
pPixel[2] = i_aui8ImageData[ixCnt++];
}
}
}
//------------------------------------------------------------------------------
static void CopyIntoBitmap_MarshallLarge (array<unsigned char>^ i_aui8ImageData,
BitmapData^ i_ptrBitmap)
{
IntPtr ptrScan0 = i_ptrBitmap->Scan0;
Marshal::Copy (i_aui8ImageData, 0, ptrScan0, i_aui8ImageData->Length);
}
//------------------------------------------------------------------------------
static void CopyIntoBitmap_MarshalSmall (array<unsigned char>^ i_aui8ImageData,
BitmapData^ i_ptrBitmap,
int i_iBytesPerPixel)
{
int ixCnt = 0;
for (int ixRow = 0; ixRow < i_ptrBitmap->Height; ixRow++)
{
for (int ixCol = 0; ixCol < i_ptrBitmap->Width; ixCol++)
{
IntPtr ptrScan0 = IntPtr::Add (i_ptrBitmap->Scan0, i_iBytesPerPixel);
Marshal::Copy (i_aui8ImageData, ixCnt, ptrScan0, i_iBytesPerPixel);
ixCnt += i_iBytesPerPixel;
}
}
}
//------------------------------------------------------------------------------
void main ()
{
int iWidth = 10000;
int iHeight = 10000;
int iBytesPerPixel = 3;
Bitmap^ oBitmap = gcnew Bitmap (iWidth, iHeight, PixelFormat::Format24bppRgb);
BitmapData^ oBitmapData = oBitmap->LockBits (Rectangle (0, 0, iWidth, iHeight), ImageLockMode::WriteOnly, oBitmap->PixelFormat);
array<unsigned char>^ aui8ImageData = gcnew array<unsigned char> (iWidth * iHeight * iBytesPerPixel);
int ixCnt = 0;
for (int ixRow = 0; ixRow < iHeight; ixRow++)
{
for (int ixCol = 0; ixCol < iWidth; ixCol++)
{
aui8ImageData[ixCnt++] = ixRow * 250 / iHeight;
aui8ImageData[ixCnt++] = ixCol * 250 / iWidth;
aui8ImageData[ixCnt++] = ixCol;
}
}
//========== Pointer ==========
// ~ 8.97 sec for 10k * 10k * 3 * 10 exec, ~ 334 MB/s
int iExec = 10;
DateTime dtStart = DateTime::Now;
for (int ixExec = 0; ixExec < iExec; ixExec++)
{
CopyIntoBitmap_Pointer (aui8ImageData, oBitmapData, iBytesPerPixel);
}
TimeSpan tsDuration = DateTime::Now - dtStart;
Console::WriteLine (tsDuration + " " + ((double)aui8ImageData->Length * iExec / tsDuration.TotalSeconds / 1e6));
//========== Marshal.Copy, 1 large block ==========
// 3.94 sec for 10k * 10k * 3 * 100 exec, ~ 7617 MB/s
iExec = 100;
dtStart = DateTime::Now;
for (int ixExec = 0; ixExec < iExec; ixExec++)
{
CopyIntoBitmap_MarshallLarge (aui8ImageData, oBitmapData);
}
tsDuration = DateTime::Now - dtStart;
Console::WriteLine (tsDuration + " " + ((double)aui8ImageData->Length * iExec / tsDuration.TotalSeconds / 1e6));
//========== Marshal.Copy, many small 3-byte blocks ==========
// 41.7 sec for 10k * 10k * 3 * 10 exec, ~ 72 MB/s
iExec = 10;
dtStart = DateTime::Now;
for (int ixExec = 0; ixExec < iExec; ixExec++)
{
CopyIntoBitmap_MarshalSmall (aui8ImageData, oBitmapData, iBytesPerPixel);
}
tsDuration = DateTime::Now - dtStart;
Console::WriteLine (tsDuration + " " + ((double)aui8ImageData->Length * iExec / tsDuration.TotalSeconds / 1e6));
//========== Buffer.BlockCopy, 1 large block ==========
// 4.02 sec for 10k * 10k * 3 * 100 exec, ~ 7467 MB/s
iExec = 100;
array<unsigned char>^ aui8Buffer = gcnew array<unsigned char> (aui8ImageData->Length);
dtStart = DateTime::Now;
for (int ixExec = 0; ixExec < iExec; ixExec++)
{
Buffer::BlockCopy (aui8ImageData, 0, aui8Buffer, 0, aui8ImageData->Length);
}
tsDuration = DateTime::Now - dtStart;
Console::WriteLine (tsDuration + " " + ((double)aui8ImageData->Length * iExec / tsDuration.TotalSeconds / 1e6));
//========== Buffer.BlockCopy, many small 3-byte blocks ==========
// 28.0 sec for 10k * 10k * 3 * 10 exec, ~ 107 MB/s
iExec = 10;
dtStart = DateTime::Now;
for (int ixExec = 0; ixExec < iExec; ixExec++)
{
int ixCnt = 0;
for (int ixRow = 0; ixRow < iHeight; ixRow++)
{
for (int ixCol = 0; ixCol < iWidth; ixCol++)
{
Buffer::BlockCopy (aui8ImageData, ixCnt, aui8Buffer, ixCnt, iBytesPerPixel);
ixCnt += iBytesPerPixel;
}
}
}
tsDuration = DateTime::Now - dtStart;
Console::WriteLine (tsDuration + " " + ((double)aui8ImageData->Length * iExec / tsDuration.TotalSeconds / 1e6));
oBitmap->UnlockBits (oBitmapData);
oBitmap->Save ("d:\\temp\\bitmap.bmp", ImageFormat::Bmp);
}
related information:
Why is memcpy() and memmove() faster than pointer increments?
Array.Copy vs Buffer.BlockCopy, Answer https://stackoverflow.com/a/33865267
https://github.com/dotnet/coreclr/issues/2430 "Array.Copy & Buffer.BlockCopy x2 to x3 slower < 1kB"
https://github.com/dotnet/coreclr/blob/master/src/vm/comutilnative.cpp, Line 718 at the time of writing: Buffer.BlockCopy() uses memmove