We Keep Coding

Exclude extra private field in struct with LayoutKind.Explicit from being part of the structure layout

Let's say we have one structure :
[StructLayout(LayoutKind.Explicit, Size=8)] // using System.Runtime.InteropServices;
public struct AirportHeader {
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
}
What I want to have : Both direct access to type string and int values, for the field Ident in this structure, without breaking the 8 bytes size of the structure, nor having to compute a string value each time from the int value.
The field Ident in that structure as int is interesting because I can fast compare with other idents if they match, other idents may come from datas that are unrelated to this structure, but are in the same int format.
Question : Is there a way to define a field that is not part of the struture layout ? Like :
[StructLayout(LayoutKind.Explicit, Size=8)]
public struct AirportHeader {
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
[NoOffset()] // <- is there something I can do the like of this
string _identStr;
public string IdentStr {
get { // EDIT ! missed the getter on this property
if (string.IsNullOrEmpty(_identStr)) _identStr =
System.Text.Encoding.ASCII.GetString(Ident.GetBytes());
// do the above only once. May use an extra private bool field to go faster.
return _identStr;
}
}
}
PS : I use pointers ('*' and '&', unsafe) because I need to deal with endianness (Local system, binary files/file format, network) and fast type conversions, fast arrays filling. I also use many flavours of Marshal methods (fixing structures on byte arrays), and a little of PInvoke and COM interop. Too bad some assemblies I'm dealing with doesn't have their dotNet counterpart yet.
TL;DR; For details only
The question is all it is about, I just don't know the answer. The following should answer most questions like "other approaches", or "why not do this instead", but could be ignored as the answer would be straightforward. Anyway, I preemptively put everything so it's clear from the start what am I trying to do. :)
Options/Workaround I'm currently using (or thinking of using) :
Create a getter (not a field) that computes the string value each time :
public string IdentStr {
get { return System.Text.Encoding.ASCII.GetString(Ident.GetBytes()); }
// where GetBytes() is an extension method that converts an int to byte[]
}
This approach, while doing the job, performs poorly : The GUI displays aircraft from a database of default flights, and injects other flights from the network with a refresh rate of one second (I should increase that to 5 seconds). I have around 1200 flights within a area, relating to 2400 airports (departure and arrival), meaning I have 2400 calls to the above code each second to display the ident in a DataGrid.
Create another struct (or class), which only purpose is to manage
data on GUI side, when not reading/writing to a stream or file. That means, read
the data with the explicit layout struct. Create another struct with
the string version of the field. Work with GUI. That will perform
better on an overall point of view, but, in the process of defining
structures for the game binaries, I'm already at 143 structures of
the kind (just with older versions of the game datas; there are a bunch I didn't write yet, and I plan to add structures for the newest datas types). ATM, more than half of them require one or more extra
fields to be of meaningful use. It's okay if I were the only one to use the assembly, but
other users will probably get lost with AirportHeader,
AirportHeaderEx, AirportEntry, AirportEntryEx,
AirportCoords, AirportCoordsEx.... I would avoid doing that.
Optimize option 1 to make computations perform faster (thanks to SO,
there are a bunch of ideas to look for - currently working on the idea). For the Ident field, I
guess I could use pointers (and I will). Already doing it for fields I must display in little endian and read/write in big
endian. There are other values, like 4x4 grid informations that are
packed in a single Int64 (ulong), that needs bit shifting to
expose the actual values. Same for GUIDs or objects pitch/bank/yaw.
Try to take advantage of overlapping fields (on study). That would work for GUIDs. Perhaps it may work for the Ident example, if MarshalAs can constrain the
value to an ASCII string. Then I just need to specify the same
FieldOffset, '0' in this case. But I'm unsure setting the field
value (entry.FieldStr = "FMEP";) actually uses the Marshal constrain on the managed code side. My undestanding is it will store the string in Unicode on managed side (?).
Furthermore, that wouldn't work for packed bits (bytes that contains
several values, or consecutive bytes hosting values that have to be
bit shifted). I believe it is impossible to specify value position, length and format
at bit level.
Why bother ? context :
I'm defining a bunch of structures to parse binary datas from array of bytes (IO.File.ReadAllBytes) or streams, and write them back, datas related to a game. Application logic should use the structures to quickly access and manipulate the datas on demand. Assembly expected capabilities is read, validate, edit, create and write, outside the scope of the game (addon building, control) and inside the scope of the game (API, live modding or monitoring). Other purpose is to understand the content of binaries (hex) and make use of that understanding to build what's missing in the game.
The purpose of the assembly is to provide a ready to use basis components for a c# addon contributor (I don't plan to make the code portable). Creating applications for the game or processing addon from source to compilation into game binaries. It's nice to have a class that loads the entire content of a file in memory, but some context require you to not do that, and only retrieve from the file what is necessary, hence the choice of the struct pattern.
I need to figure out the trust and legal issues (copyrighted data) but that's outside the scope of the main concern. If that matter, Microsoft did provide over the years public freely accessible SDKs exposing binaries structures on previous versions of the game, for the purpose of what I'm doing (I'm not the first and probably not the last to do so). Though, I wouldn't dare to expose undocumented binaries (for the latest game datas for instance), nor facilitate a copyright breach on copyrighted materials/binaries.
I'm just asking confirmation if there is a way or not to have private fields not being part of the structure layout. Naive belief ATM is "that's impossible, but there are workarounds". It's just that my c# experience is pretty sparce, so maybe I'm wrong, why I ask. Thanks !
As suggested, there are several ways to get the job done. Here are the getters/setters I came up with within the structure. I'll measure how each code performs on various scenarios later. The dict approach is very seducing as on many scenarios, I would need a directly accessible global database of (59000) airports with runways and parking spots (not just the Ident), but a fast check between struct fields is also interesting.
public string IdentStr_Marshal {
get {
var output = "";
GCHandle pinnedHandle; // CS0165 for me (-> c# v5)
try { // Fast if no exception, (very) slow if exception thrown
pinnedHandle = GCHandle.Alloc(this, GCHandleType.Pinned);
IntPtr structPtr = pinnedHandle.AddrOfPinnedObject();
output = Marshal.PtrToStringAnsi(structPtr, 4);
// Cannot use UTF8 because the assembly should work in Framework v4.5
} finally { if (pinnedHandle.IsAllocated) pinnedHandle.Free(); }
return output;
}
set {
value.PadRight(4); // Must fill the blanks - initial while loop replaced (Charlieface's)
IntPtr intValuePtr = IntPtr.Zero;
// Cannot use UTF8 because some users are on Win7 with FlightSim 2004
try { // Put a try as a matter of habit, but not convinced it's gonna throw.
intValuePtr = Marshal.StringToHGlobalAnsi(value);
Ident = Marshal.ReadInt32(intValuePtr, 0).BinaryConvertToUInt32(); // Extension method to convert type.
} finally { Marshal.FreeHGlobal(intValuePtr); // freeing the right pointer }
}
}
public unsafe string IdentStr_Pointer {
get {
string output = "";
fixed (UInt32* ident = &Ident) { // Fixing the field
sbyte* bytes = (sbyte*)ident;
output = new string(bytes, 0, 4, System.Text.Encoding.ASCII); // Encoding added (#Charlieface)
}
return output;
}
set {
// value must not exceed a length of 4 and must be in Ansi [A-Z,0-9,whitespace 0x20].
// value validation at this point occurs outside the structure.
fixed (UInt32* ident = &Ident) { // Fixing the field
byte* bytes = (byte*)ident;
byte[] asciiArr = System.Text.Encoding.ASCII.GetBytes(value);
if (asciiArr.Length >= 4) // (asciiArr.Length == 4) would also work
for (Int32 i = 0; i < 4; i++) bytes[i] = asciiArr[i];
else {
for (Int32 i = 0; i < asciiArr.Length; i++) bytes[i] = asciiArr[i];
for (Int32 i = asciiArr.Length; i < 4; i++) bytes[i] = 0x20;
}
}
}
}
static Dictionary<UInt32, string> ps_dict = new Dictionary<UInt32, string>();
public string IdentStr_StaticDict {
get {
string output; // logic update with TryGetValue (#Charlieface)
if (ps_dict.TryGetValue(Ident, out output)) return output;
output = System.Text.Encoding.ASCII.GetString(Ident.ToBytes(EndiannessType.LittleEndian));
ps_dict.Add(Ident, output);
return output;
}
set { // input can be "FMEE", "DME" or "DK". length of 2 characters is the minimum.
var bytes = new byte[4]; // Need to convert value to a 4 byte array
byte[] asciiArr = System.Text.Encoding.ASCII.GetBytes(value); // should be 4 bytes or less
// Put the valid ASCII codes in the array.
if (asciiArr.Length >= 4) // (asciiArr.Length == 4) would also work
for (Int32 i = 0; i < 4; i++) bytes[i] = asciiArr[i];
else {
for (Int32 i = 0; i < asciiArr.Length; i++) bytes[i] = asciiArr[i];
for (Int32 i = asciiArr.Length; i < 4; i++) bytes[i] = 0x20;
}
Ident = BitConverter.ToUInt32(bytes, 0); // Set structure int value
if (!ps_dict.ContainsKey(Ident)) // Add if missing
ps_dict.Add(Ident, System.Text.Encoding.ASCII.GetString(bytes));
}
}

As mentioned by others, it is not possible to exclude a field from a struct for marshalling.
You also cannot use a pointer as a string in most places.
If the number of different possible strings is relatively small (and it probably will be, given it's only 4 characters), then you could use a static Dictionary<int, string> as a kind of string-interning mechanism.
Then you write a property to add/retrieve the real string.
Note that dictionary access is O(1), and hashing an int just returns itself, so it will be very, very fast, but will take up some memory.
[StructLayout(LayoutKind.Explicit, Size=8)]
public struct AirportHeader
{
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
static Dictionary<int, string> _identStrings = new Dictionary<int, string>();
public string IdentStr =>
_identStrings.TryGetValue(Ident, out var ret) ? ret :
(_identStrings[Ident] = Encoding.ASCII.GetString(Ident.GetBytes());
}

This is not possible because a structure must contain all of its values ​​in a specific order. Usually this order is controlled by the CLR itself. If you want to change the order of the data order, you can use the StructLayout. However, you cannot exclude a field or that data would simply not exist in memory.
Instead of a string (which is a reference type) you can use a pointer to point directly to that string and use that in your structure in combination with the StructLayout. To get this string value, you can use a get-only property that reads directly from unmanaged memory.

How to use pythonnet to make C# function return bytearray parameter?

I am trying to return a bytearray from a function from a imported C# DLL. I do not know how to.
I have imported a C# Assembly into Python using pythonnet. I have to use C# because the C# dll is a third party dll for a piece of hardware I'm using. I cannot edit that dll.
From this function, I have successfully returned reference types that are not bytearray.
import clr
clr.AddReference(R'C:...\RFID Chip Sorter Project\UHFReader18CSharp.dll')
from ReaderB import StaticClassReaderB
fComAdr = 255
EPC = bytearray(13)
Totallen = int = 0
CardNum = int = 0
z = StaticClassReaderB.Inventory_G2(fComAdr, 0, 0, 0, EPC, Totallen, CardNum, 3)
print("z=", z)
print ("Z[0]=", z[0])
print("EPC=", EPC)
This is the function inside the dll when I use dotpeek:
[DllImport("Basic.dll", CallingConvention = CallingConvention.StdCall)]
public static extern int Inventory_G2(
ref byte ConAddr,
byte AdrTID,
byte LenTID,
byte TIDFlag,
byte[] EPClenandEPC,
ref int Totallen,
ref int CardNum,
int PortHandle);
This is the function description:
Function int StaticClassReaderB.Inventory_G2 (unsigned char *ComAdr, unsigned char AdrTID, unsigned char LenTID, unsigned char TIDFlag, unsigned char *EPClenandEPC, int * Totallen, int *CardNum, int FrmHandle);
Parameter:
ComAdr: Input, pointed to the address of the reader.
AdrTID:Input,query TID’s start address.
LenTID: Input,query TID’s data word number.
TIDFlag: Input,query TID’s flag.
TIDFlag=1:query TID.
TIDFlag=0:query EPC.
EPClenandEPC: Output, Pointed to the array storing the inventory result. It is the EPC data of tag Reader read. The unit of the array includes 1 byte EPCLen and N (the value of EPCLen) bytes EPC. It is the former high-word, low word in the EPC of each tag. It is the former high-byte, low byte in the each word.
Totallen: Output. Pointed to the byte count of the EPClenandEPC.
CardNum: Output. Pointed to the number of tag detected.
FrmHandle: Handle of the corresponding communication port the reader is connected. The handle value is got when calling function AutoOpenComPort or OpenComPort.
Returns:
Zero value when successfully, value:
0x01 Return before Inventory finished
0x02 the Inventory-scan-time overflow
0x03 More Data
0x04 Reader module MCU is Full
others when error occurred.
Actual results:
z = (1, 0, 13, 1)
z[0]= 1
EPC= bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
The first returned value should be the "Return value" of the function. Which is okay.
0, 13, 1 are ref values ComAdr, Totallen, and CardNum, respectively. However, what is clearly missing is "EPClenandEPC". EPClenandEPC is a bytearray. It doesn't have the "ref" and it is not returning for some reason.
When I typed print(EPC) I expected it to print the obtained bytearray, but clearly I am doing it wrong.
I also briefly tried using reflection but I am not familiar with it.
Thank you.

Marshaling a SAFEARRAY of VARIANTs that are BYTEs to C#

I create a SAFEARRAY storing VARIANTs that are BYTEs in C++.
When this structure is marshaled to C#, a weird thing happens.
If I print the content of this structure in C# to a WinForms ListBox, e.g.:
byte data[]
TestSafeArray(out data);
lstOutput.Items.Clear();
foreach (byte x in data)
{
lstOutput.Items.Add(x); // Strange numbers
}
I get some numbers that seem unrelated to the original ones. Moreover, each time I run the C# client for a new test, I get a different set of numbers.
Note that if I inspect the content of that data array with the Visual Studio debugger, I get the correct numbers, as the following screenshot shows:
However, if I CopyTo the marshaled data array to a new one, I get the correct numbers:
byte[] data;
TestSafeArray(out data);
// Copy to a new byte array
byte[] byteData = new byte[data.Length];
data.CopyTo(byteData, 0);
lstOutput.Items.Clear();
foreach (byte x in byteData)
{
lstOutput.Items.Add(x); // ** WORKS! **
}
This is the C++ repro code I use to build the SAFEARRAY (this function is exported from a native DLL):
extern "C" HRESULT __stdcall TestSafeArray(/* [out] */ SAFEARRAY** ppsa)
{
HRESULT hr = S_OK;
try
{
const std::vector<BYTE> v{ 11, 22, 33, 44 };
const int count = static_cast<int>(v.size());
CComSafeArray<VARIANT> sa(count);
for (int i = 0; i < count; i++)
{
CComVariant var(v[i]);
hr = sa.SetAt(i, var);
if (FAILED(hr))
{
return hr;
}
}
*ppsa = sa.Detach();
}
catch (const CAtlException& e)
{
hr = e;
}
return hr;
}
And this is the C# P/Invoke I used:
[DllImport("NativeDll.dll", PreserveSig = false)]
private static extern void TestSafeArray(
[Out, MarshalAs(UnmanagedType.SafeArray,
SafeArraySubType = VarEnum.VT_VARIANT)]
out byte[] result);
Note that if in C++ I create a SAFEARRAY storing BYTEs directly (instead of a SAFEARRAY(VARIANT)), I get the correct values immediately in C#, without the intermediate CopyTo operation.

[Out, MarshalAs(UnmanagedType.SafeArray,
SafeArraySubType = VarEnum.VT_VARIANT)]
out byte[] result);
You fibbed. You told the marshaller that you wanted an array of variants, indeed compatible with what the C++ compiler produced. It will dutifully produce an object[], object is the standard marshaling for a VARIANT. The elements of the array are boxed bytes.
That did not fool the debugger, it ignored the program declaration and looked at the array type and discovered object[], readily visible in your screenshot. So it properly accessed the boxed bytes. And it did not fool Array.CopyTo(), it takes an Array argument so is forced to look at the element type. And properly converted the boxed bytes to bytes, it knows how to do that.
But the C# compiler is fooled badly. It has no idea that it needs to emit an unbox instruction. Not actually sure what goes wrong, you are probably getting the low byte of the object address. Pretty random indeed :)
Fibbing in pinvoke declarations can be very useful. Works just fine in this particular case if the array contains actual objects, like strings, arrays or interface pointers. The probable reason the marshaller doesn't scream bloody murder. Not here, the boxing trips it up bad. You'll have to fix the declaration, use object[].

Convert piece of code from c++ to c#

i'm trying to convert code from c++ to c#.. but i'm stuck.
Can somebody let me know where i'm wrong?
C++ code is:
void CALLBACK Test(void *buffer, DWORD length)
{
BYTE *b=(BYTE*)buffer, temp[20000];
DWORD p=0;
while (p<length)
{
DWORD c=min(length-p, sizeof(temp));
memcpy(temp, b+p, c);
c=SendData(dummy, temp, c);
if (c==-1) break;
p+=c;
}
}
where is SendData(int, byte[], int)
And current c# code
void Test(IntPtr buffer, int length)
{
byte[] temp = new byte[20000];
byte[] b = new byte[length];
Marshal.Copy(buffer, b, 0, length);
long p=0;
while (p<length)
{
long c=Math.Min(length-p, 20000);
Array.Copy(temp, b+p, c);
c=SendData(dummy, temp, (int)c);
if (c==-1) break;
p+=c;
}
}
I'm not sure did I did it correctly, but I can see I cannot apply operator + to b+s because it's long and byte[].
Thanks

I think you want to use the 5-arg version of Array.Copy, which lets you pass offsets into both the start and destination arrays. This is the more "C#" way of doing things as opposed to pointer arithmetic.

In C++ you may add a DWORD (an int, really) to a pointer and the compiler knows to offset the address of the pointer by the specified amount. C# doesn't let you do that, normally. (There is a way to do pointer arithmetic in C#, but don't.) So, in the C# version, you need another way to copy at offset p relative to the start of b. One of the overloads of Array.Copy will let you do that. Something like:
Array.Copy(temp, 0, b, p, c);

You may want to look at: How to split a byte array, because that is in effect what you are trying to accomplish with the b+pportion.

C++ <--> C# modify a marshalled array of bytes

I have an unmanaged C++ function which is calling a managed C# method in a DLL. The purpose of the C# method is to take an array of bytes (allocated by the C++ caller), populate the array, and return it. I can get the array INTO the C# method, but the populated data are lost when they get back to the C++ function. Right now, this is my test code to debug the process:
C# DLL Method:
// Take an array of bytes and modify it
public ushort GetBytesFromBlaster([MarshalAs(UnmanagedType.LPArray)] byte[] dataBytes)
{
dataBytes[0] = (byte)'a';
dataBytes[1] = (byte)'b';
dataBytes[2] = (byte)'c';
return 3;
}
C++ function which calls the DLL:
// bytes[] has been already allocated by its caller
short int SimGetBytesP2P(unsigned char bytes[])
{
unsigned short int numBytes = 0;
bytes[0] = 'x';
bytes[1] = 'y';
bytes[2] = 'z';
// bytes[] are {'x', 'y', 'z'} here
guiPtr->GetBytesFromBlaster(bytes, &numBytes);
// bytes[] SHOULD be {'a', 'b', 'c'} here, but they are still {'x', 'y', 'z'}
return(numBytes);
}
I believe it has something to do with C# turning the C++ pointer into a new managed array, but modifying the original one. I have tried several variations using the "ref" modifyer, etc., but no luck. Also, these data are NOT null-terminated strings; the date bytes are raw 1-byte values, not null-terminated.
Can anyone please shed some light on this? Thanks!
Stuart

You could do the marshaling yourself. Have the C# function accept a parameter by value of type IntPtr. Also a second parameter indicating array length. No special marshaling attributes are needed or wanted.
Then, use Marshal.Copy and copy the array from the unmanaged pointer to a managed byte[] array that you allocated. Do your thing, and then when you're done, use Marshal.Copy to copy it back to the C++ unmanaged array.
These particular overloads should get you started:
http://msdn.microsoft.com/en-us/library/ms146625.aspx
http://msdn.microsoft.com/en-us/library/ms146631.aspx
For example:
public ushort GetBytesFromBlaster(IntPtr dataBytes, int arraySize)
{
byte[] managed = new byte[arraySize];
Marshal.Copy(dataBytes, managed, 0, arraySize);
managed[0] = (byte)'a';
managed[1] = (byte)'b';
managed[2] = (byte)'c';
Marshal.Copy(managed, 0, dataBytes, arraySize);
return 3;
}
Alternatively you could implement a custom marshaller as described in http://msdn.microsoft.com/en-us/library/w22x2hw6.aspx if the default one isn't doing what you need it to. But that looks like more work.

I believe that you just need to add a SizeConst attribute:
public ushort GetBytesFromBlaster(
[MarshalAs(UnmanagedType.LPArray, SizeConst=3)]
byte[] dataBytes
)
and the default marshaller should do the rest for you.