We Keep Coding

Exclude extra private field in struct with LayoutKind.Explicit from being part of the structure layout

Let's say we have one structure :
[StructLayout(LayoutKind.Explicit, Size=8)] // using System.Runtime.InteropServices;
public struct AirportHeader {
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
}
What I want to have : Both direct access to type string and int values, for the field Ident in this structure, without breaking the 8 bytes size of the structure, nor having to compute a string value each time from the int value.
The field Ident in that structure as int is interesting because I can fast compare with other idents if they match, other idents may come from datas that are unrelated to this structure, but are in the same int format.
Question : Is there a way to define a field that is not part of the struture layout ? Like :
[StructLayout(LayoutKind.Explicit, Size=8)]
public struct AirportHeader {
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
[NoOffset()] // <- is there something I can do the like of this
string _identStr;
public string IdentStr {
get { // EDIT ! missed the getter on this property
if (string.IsNullOrEmpty(_identStr)) _identStr =
System.Text.Encoding.ASCII.GetString(Ident.GetBytes());
// do the above only once. May use an extra private bool field to go faster.
return _identStr;
}
}
}
PS : I use pointers ('*' and '&', unsafe) because I need to deal with endianness (Local system, binary files/file format, network) and fast type conversions, fast arrays filling. I also use many flavours of Marshal methods (fixing structures on byte arrays), and a little of PInvoke and COM interop. Too bad some assemblies I'm dealing with doesn't have their dotNet counterpart yet.
TL;DR; For details only
The question is all it is about, I just don't know the answer. The following should answer most questions like "other approaches", or "why not do this instead", but could be ignored as the answer would be straightforward. Anyway, I preemptively put everything so it's clear from the start what am I trying to do. :)
Options/Workaround I'm currently using (or thinking of using) :
Create a getter (not a field) that computes the string value each time :
public string IdentStr {
get { return System.Text.Encoding.ASCII.GetString(Ident.GetBytes()); }
// where GetBytes() is an extension method that converts an int to byte[]
}
This approach, while doing the job, performs poorly : The GUI displays aircraft from a database of default flights, and injects other flights from the network with a refresh rate of one second (I should increase that to 5 seconds). I have around 1200 flights within a area, relating to 2400 airports (departure and arrival), meaning I have 2400 calls to the above code each second to display the ident in a DataGrid.
Create another struct (or class), which only purpose is to manage
data on GUI side, when not reading/writing to a stream or file. That means, read
the data with the explicit layout struct. Create another struct with
the string version of the field. Work with GUI. That will perform
better on an overall point of view, but, in the process of defining
structures for the game binaries, I'm already at 143 structures of
the kind (just with older versions of the game datas; there are a bunch I didn't write yet, and I plan to add structures for the newest datas types). ATM, more than half of them require one or more extra
fields to be of meaningful use. It's okay if I were the only one to use the assembly, but
other users will probably get lost with AirportHeader,
AirportHeaderEx, AirportEntry, AirportEntryEx,
AirportCoords, AirportCoordsEx.... I would avoid doing that.
Optimize option 1 to make computations perform faster (thanks to SO,
there are a bunch of ideas to look for - currently working on the idea). For the Ident field, I
guess I could use pointers (and I will). Already doing it for fields I must display in little endian and read/write in big
endian. There are other values, like 4x4 grid informations that are
packed in a single Int64 (ulong), that needs bit shifting to
expose the actual values. Same for GUIDs or objects pitch/bank/yaw.
Try to take advantage of overlapping fields (on study). That would work for GUIDs. Perhaps it may work for the Ident example, if MarshalAs can constrain the
value to an ASCII string. Then I just need to specify the same
FieldOffset, '0' in this case. But I'm unsure setting the field
value (entry.FieldStr = "FMEP";) actually uses the Marshal constrain on the managed code side. My undestanding is it will store the string in Unicode on managed side (?).
Furthermore, that wouldn't work for packed bits (bytes that contains
several values, or consecutive bytes hosting values that have to be
bit shifted). I believe it is impossible to specify value position, length and format
at bit level.
Why bother ? context :
I'm defining a bunch of structures to parse binary datas from array of bytes (IO.File.ReadAllBytes) or streams, and write them back, datas related to a game. Application logic should use the structures to quickly access and manipulate the datas on demand. Assembly expected capabilities is read, validate, edit, create and write, outside the scope of the game (addon building, control) and inside the scope of the game (API, live modding or monitoring). Other purpose is to understand the content of binaries (hex) and make use of that understanding to build what's missing in the game.
The purpose of the assembly is to provide a ready to use basis components for a c# addon contributor (I don't plan to make the code portable). Creating applications for the game or processing addon from source to compilation into game binaries. It's nice to have a class that loads the entire content of a file in memory, but some context require you to not do that, and only retrieve from the file what is necessary, hence the choice of the struct pattern.
I need to figure out the trust and legal issues (copyrighted data) but that's outside the scope of the main concern. If that matter, Microsoft did provide over the years public freely accessible SDKs exposing binaries structures on previous versions of the game, for the purpose of what I'm doing (I'm not the first and probably not the last to do so). Though, I wouldn't dare to expose undocumented binaries (for the latest game datas for instance), nor facilitate a copyright breach on copyrighted materials/binaries.
I'm just asking confirmation if there is a way or not to have private fields not being part of the structure layout. Naive belief ATM is "that's impossible, but there are workarounds". It's just that my c# experience is pretty sparce, so maybe I'm wrong, why I ask. Thanks !
As suggested, there are several ways to get the job done. Here are the getters/setters I came up with within the structure. I'll measure how each code performs on various scenarios later. The dict approach is very seducing as on many scenarios, I would need a directly accessible global database of (59000) airports with runways and parking spots (not just the Ident), but a fast check between struct fields is also interesting.
public string IdentStr_Marshal {
get {
var output = "";
GCHandle pinnedHandle; // CS0165 for me (-> c# v5)
try { // Fast if no exception, (very) slow if exception thrown
pinnedHandle = GCHandle.Alloc(this, GCHandleType.Pinned);
IntPtr structPtr = pinnedHandle.AddrOfPinnedObject();
output = Marshal.PtrToStringAnsi(structPtr, 4);
// Cannot use UTF8 because the assembly should work in Framework v4.5
} finally { if (pinnedHandle.IsAllocated) pinnedHandle.Free(); }
return output;
}
set {
value.PadRight(4); // Must fill the blanks - initial while loop replaced (Charlieface's)
IntPtr intValuePtr = IntPtr.Zero;
// Cannot use UTF8 because some users are on Win7 with FlightSim 2004
try { // Put a try as a matter of habit, but not convinced it's gonna throw.
intValuePtr = Marshal.StringToHGlobalAnsi(value);
Ident = Marshal.ReadInt32(intValuePtr, 0).BinaryConvertToUInt32(); // Extension method to convert type.
} finally { Marshal.FreeHGlobal(intValuePtr); // freeing the right pointer }
}
}
public unsafe string IdentStr_Pointer {
get {
string output = "";
fixed (UInt32* ident = &Ident) { // Fixing the field
sbyte* bytes = (sbyte*)ident;
output = new string(bytes, 0, 4, System.Text.Encoding.ASCII); // Encoding added (#Charlieface)
}
return output;
}
set {
// value must not exceed a length of 4 and must be in Ansi [A-Z,0-9,whitespace 0x20].
// value validation at this point occurs outside the structure.
fixed (UInt32* ident = &Ident) { // Fixing the field
byte* bytes = (byte*)ident;
byte[] asciiArr = System.Text.Encoding.ASCII.GetBytes(value);
if (asciiArr.Length >= 4) // (asciiArr.Length == 4) would also work
for (Int32 i = 0; i < 4; i++) bytes[i] = asciiArr[i];
else {
for (Int32 i = 0; i < asciiArr.Length; i++) bytes[i] = asciiArr[i];
for (Int32 i = asciiArr.Length; i < 4; i++) bytes[i] = 0x20;
}
}
}
}
static Dictionary<UInt32, string> ps_dict = new Dictionary<UInt32, string>();
public string IdentStr_StaticDict {
get {
string output; // logic update with TryGetValue (#Charlieface)
if (ps_dict.TryGetValue(Ident, out output)) return output;
output = System.Text.Encoding.ASCII.GetString(Ident.ToBytes(EndiannessType.LittleEndian));
ps_dict.Add(Ident, output);
return output;
}
set { // input can be "FMEE", "DME" or "DK". length of 2 characters is the minimum.
var bytes = new byte[4]; // Need to convert value to a 4 byte array
byte[] asciiArr = System.Text.Encoding.ASCII.GetBytes(value); // should be 4 bytes or less
// Put the valid ASCII codes in the array.
if (asciiArr.Length >= 4) // (asciiArr.Length == 4) would also work
for (Int32 i = 0; i < 4; i++) bytes[i] = asciiArr[i];
else {
for (Int32 i = 0; i < asciiArr.Length; i++) bytes[i] = asciiArr[i];
for (Int32 i = asciiArr.Length; i < 4; i++) bytes[i] = 0x20;
}
Ident = BitConverter.ToUInt32(bytes, 0); // Set structure int value
if (!ps_dict.ContainsKey(Ident)) // Add if missing
ps_dict.Add(Ident, System.Text.Encoding.ASCII.GetString(bytes));
}
}

As mentioned by others, it is not possible to exclude a field from a struct for marshalling.
You also cannot use a pointer as a string in most places.
If the number of different possible strings is relatively small (and it probably will be, given it's only 4 characters), then you could use a static Dictionary<int, string> as a kind of string-interning mechanism.
Then you write a property to add/retrieve the real string.
Note that dictionary access is O(1), and hashing an int just returns itself, so it will be very, very fast, but will take up some memory.
[StructLayout(LayoutKind.Explicit, Size=8)]
public struct AirportHeader
{
[FieldOffset(0)]
[MarshalAs(UnmanagedType.I4)]
public int Ident; // a 4 bytes ASCII : "FIMP" { 0x46, 0x49, 0x4D, 0x50 }
[FieldOffset(4)]
[MarshalAs(UnmanagedType.I4)]
public int Offset;
static Dictionary<int, string> _identStrings = new Dictionary<int, string>();
public string IdentStr =>
_identStrings.TryGetValue(Ident, out var ret) ? ret :
(_identStrings[Ident] = Encoding.ASCII.GetString(Ident.GetBytes());
}

This is not possible because a structure must contain all of its values ​​in a specific order. Usually this order is controlled by the CLR itself. If you want to change the order of the data order, you can use the StructLayout. However, you cannot exclude a field or that data would simply not exist in memory.
Instead of a string (which is a reference type) you can use a pointer to point directly to that string and use that in your structure in combination with the StructLayout. To get this string value, you can use a get-only property that reads directly from unmanaged memory.

C# Struct array but not class array

I was able to create an array of a struct I created, but I'm having trouble doing the same for an array of a class. I'm (faintly) aware that this probably isn't the best way to do this, but I'd appreciate help in figuring out what's going on.
I'm about 2 days into learning C#, and I'm navigating away from MS Office-VBA, if that gives you an idea of what I'm into. Anyway, I'm following an online reference, and along the way trying to play with what I've learned so far. This problem has come about as a result of my playing.
First, let me describe what I've done with the struct, and the array of that struct, with some code snippets.
I've been able to create a struct, called Machines...
// play with structs
struct Machines
{
// vars for struct
private string model, SN;
private int hours;
// assign values
public void AssignValues(string model_in, string SN_in, int hours_in)
{
model = model_in;
SN = SN_in;
hours = hours_in;
}
// display values
public void DisplayValues()
{
Console.WriteLine("Model: {0}", model);
Console.WriteLine("SN: {0}", SN);
Console.WriteLine("Hours: {0}", hours);
}
};
... things seem to work just fine:
public static void Main()
{
// play with structures
Machines machine1 = new Machines();
machine1.AssignValues("AA", "ABC01234", 34760);
machine1.DisplayValues();
Output is:
Model: AA
SN: ABC01234
Hours: 34760
Then, I can create an array of the struct, and things continue to go well:
// play with structures and arrays
// declare, create new instance
Machines [] MyArr = new Machines[10];
MyArr[0].AssignValues("AA", "ABC01235", 43000);
MyArr[0].DisplayValues();
But, when I attempt to do the same with a class, it's a different story. What's going on?
public class ArmstrongMachine
{
// vars for struct
private string model, SN;
private int hours;
// assign values
public void AssignValues(string model_in, string SN_in, int hours_in)
{
model = model_in;
SN = SN_in;
hours = hours_in;
}
// display values
public void DisplayValues()
{
Console.WriteLine("Model: {0}", model);
Console.WriteLine("SN: {0}", SN);
Console.WriteLine("Hours: {0}", hours);
}
};
...
// play with classes
ArmstrongMachine [] MyMachines = new ArmstrongMachine[10];
MyMachines[0].AssignValues("AA", "ABC01236", 51000);
MyMachines[0].DisplayValues();
The issue seems to begin with MyMachines[0].AssignValues.... If I comment out that line and the following, there are no problems (other than the warning that I've created a variable I'm not using).
Any ideas?
Also, please be aware that this is being compiled online.

A class gives you a reference type in C#.
The array thus holds references to objects, and not the objects themselves.
The array initially contains nothing, all zeroes, which means all the references will be null.
You need to initialize each element of the array to hold an object reference:
// play with classes
ArmstrongMachine [] MyMachines = new ArmstrongMachine[10];
MyMachines[0] = new ArmstrongMachine();
MyMachines[0].AssignValues("AA", "ABC01236", 51000);
MyMachines[0].DisplayValues();
MyMachines[1] = new ArmstrongMachine();
MyMachines[2] = new ArmstrongMachine();
...
MyMachines[9] = new ArmstrongMachine();
If the array holds value types, like the structs, then the array holds the struct values themselves, thus it works with the structs, and not with the objects.
Also note that you should emphatically not use mutable structs (structs you can change). There's tons of things that can go wrong and bite you in ... so you should not use them. Go with classes in this case.
Here's a video on the subject of mutable structs: Evil Structs, by Jon Skeet.

In simple words: A struct is only a way the memory is structured whereas a class is a real object. For the second example, you need to create instances of the class before you can make calls to it, because it only points to a memory location (which may be unassigned = a null reference):
ArmstrongMachine [] MyMachines = new ArmstrongMachine[10];
MyMachines[0] = new ArmstrongMachine();
MyMachines[0].AssignValues("AA", "ABC01236", 51000);
MyMachines[0].DisplayValues();

Compare byte[] to T

I want to make a list of pointers to locations that contains a certain value in the process memory of another process. The value can be a short, int, long, string, bool or something else.
My idea is to use Generics for this. I have one problem with making it, how can I tell the compiler to what type he needs to convert the byte array?
This is what I made:
public List<IntPtr> ScanProccessFor<T>(T ItemToScanFor)
{
List<IntPtr> Output = new List<IntPtr>();
IntPtr StartOffset = SelectedProcess.MainModule.BaseAddress;
int ScanSize = SelectedProcess.MainModule.ModuleMemorySize;
for (int i = 0; i < ScanSize; i++)
if (ReadMemory(SelectedProcess, StartOffset + i, (UInt16)Marshal.SizeOf(ItemToScanFor)) == ItemToScanFor)
Output.Insert(Output.Count,StartOffset + i);
return Output;
}
How can I tell the compiler that he needs to convert the byte[] to type T?

Your question is a little bit confusing, but I'll try to answer what I can
Instead of taking a generic type, I would probably write a method that takes an instance of an interface like IConvertableToByteArray or something.
public IConvertableToByteArray
{
public byte[] ToByteArray();
}
Then If you needed to allow a specific type to be compatible with that method, you could make an encapsulating class
public IntConvertableToByteArray : IConvertableToByteArray
{
public int Value{get; set;}
public byte[] ToByteArray()
{
insert logic here
}
}

You could use Marshal.StructureToPtr to get an unmanaged representation of the structure (which has to be a 'simple' structure). You might need to special case strings though.
You should also think about the alignment constraints on what you are searching for -- advancing through memory 1 byte at a time will be very slow and wasteful if the item must be 4 or 8 byte aligned.

How to count number of methods per java class using c sharp?

I have to complete a project in C# to find number of methods per java class.
I could find all methods in the .java file using c# regular expression, but what I want is to find the number of methods per each and every class, including inner classes. Can any one help me.
string[] lines = File.ReadAllLines(file);
int countLine = 0;
int AllCount = 0;
foreach (string line in lines)
{
countLine = MethodsCount(line);
AllCount = AllCount + countLine;
}
label5.Text = AllCount.ToString();
Here's the method-counting method.
private int MethodsCount (string LineOperator)
{
int count = 0;
string[] words = LineOperator.Split('{');
foreach (string word in words)
{
if (Regex.IsMatch(word, #"(static\s|final\s)?(public|private|internal)(\sstatic|\sfinal)?\s(int|boolean|void|double|String|long|String\[\]|String\[\]\[\])?\s([a-z]|[A-Z]+[a-z]+|[a-z]+[A-Z]+)+(\s)*\((\s|\n|\r)*"))
{
count = count + 1;
}
}
return count;
}
if we consider a class
public class vehicle {
public method1() {
---------
}
public method2() {
------
}
public class car extends vehicle {
public method3() {
-------
}
}
}
I want to get the output there are this number of methods in vehicle class,this number of methods in car class like wise.

Parsing a Java source file with just regex is flaky at best. There are so many different ways to format or annotate the code that you'll easily miss valid method declarations. The regex in your code sample does not handle line breaks, generic methods, the throws clause, arbitrary return types, the synchronized modifier or annotated arguments, two method declarations on the same line...
You'd have to drop regex and build (or reuse) a full parser to be sure you get everything. Fortunately, if you have access to a JDK you can take a shortcut.
First, compile the Java source code. This will give you an output directory full of .class files (inner classes get their own file). Scan the output directory to collect all the class names you need to analyze.
Then, for each class, run the javap command, which will give you this kind of output:
barend#TURMINDER-XUSS /tmp$ javap java.util.Iterator
Compiled from "Iterator.java"
public interface java.util.Iterator{
public abstract boolean hasNext();
public abstract java.lang.Object next();
public abstract void remove();
}
This is much easier to parse than a full Java source file. You can just count all lines containing \s[a-zA-Z][a-zA-Z0-9_]*\( and you have your method count. Use the 'compiled from' information to get to the method count per source file.
(edit) NOTE: javap doesn't print private methods by default. pass the -private argument to include these in the output.

Protobuf-net r282 having problems deserializing object serialized with r249

I've just updated from r249 to r282. Other than replacing the dll I've made no changes. Unfortunately, now deserializing the objects created before the update takes significantly longer. What used to take two seconds now takes five minutes.
Were there syntax changes between versions? Is there anything it no longer supports?
My classes are all using ProtoContract, ProtoMember, and ProtoInclude. I am running VS2010. As far as I was concerned there were no problems with my protocol buffer code. I'm only trying to upgrade because I figured it's good to have the most recent version.
Edit - 2010.09.09
One of the properties of my object is an array of ushorts. I've just noticed that this property did not serialize/deserialize properly with r282. The resulting values of the array are all zeros. The array had values before being serialized (r282) but not after deserialization (r282).

It turns out that despite my efforts, yes there was a breaking change in data format in one of the earlier builds. This only impacts ushort data, which was omitted from the handling at one point. This is regrettable, but the good news is that no data is lost - it is simply a bit inconvenient to access (it is essentially written via a string at the moment).
Here's my suggested workaround; for a member like:
[ProtoBuf.ProtoMember(1)]
public ushort[] Data {get;set;}
Replace that with:
[ProtoBuf.ProtoMember(1)]
private string[] LegacyData {get;set;}
private bool LegacyDataSpecified { get { return false; } set { } }
/* where 42 is just an unused new field number */
[ProtoBuf.ProtoMember(42, Options = MemberSerializationOptions.Packed)]
public ushort[] Data { get; set; }
[ProtoBuf.ProtoAfterDeserialization]
private void SerializationCallback()
{
if (LegacyData != null && LegacyData.Length > 0)
{
ushort[] parsed = Array.ConvertAll<string, ushort>(
LegacyData, ushort.Parse);
if (Data != null && Data.Length > 0)
{
int oldLen = parsed.Length;
Array.Resize(ref parsed, parsed.Length + Data.Length);
Array.Copy(Data, 0, parsed, oldLen, Data.Length);
}
Data = parsed;
}
LegacyData = null;
}
This imports old-style data into LegacyData and merges it during (after) serialization, or writes new-style data from Data. Faster, smaller, and supports both old and new data.