We Keep Coding

Integer to byte array in little endian way (generic function)

I have this method to convert long to little endian byte array
public static byte[] UnsignedIntegerToLEByteArray(ulong value)
{
// Value in bytes... in your system's endianness (let's say: little endian)
byte[] bytes = BitConverter.GetBytes(value);
// If it was big endian, reverse it
if (!BitConverter.IsLittleEndian)
Array.Reverse(bytes);
return bytes;
}
My goal is to use it for shorted data types too, like int, short, etc. See here:
byte a = 0xAA;
ushort b = 0xEEAA;
uint c = 0xAABBCCDD;
ulong d = 0xAABBCCDDAAAAAABB;
// If you passed short below, you are only interested
// in first two bytes of the array
byte []tmp = DppUtilities.UnsignedIntegerToLEByteArray(b);
This works if my machine is little endian.
Will it also work if it is run on big endian machine? I think yes, but I would like to verify.

You can play a trick with a help of IntPtr and Marshal in order to convert any struct (including byte, ushort and ulong):
// Disclaimer: the structure will be reversed as a whole, not field by field
public static byte[] ToLEByteArray<T>(T value) where T: struct {
int size = Marshal.SizeOf(typeof(T));
byte[] bytes = new byte[size];
IntPtr p = Marshal.AllocHGlobal(size);
try {
Marshal.StructureToPtr(value, p, true);
Marshal.Copy(p, bytes, 0, size);
}
finally {
Marshal.FreeHGlobal(p);
}
// If it was big endian, reverse it
if (!BitConverter.IsLittleEndian)
Array.Reverse(bytes);
return bytes;
}
....
Byte b = 123;
ushort s = 123;
ulong l = 123;
Byte[] result_byte = ToLEByteArray(b);
Byte[] result_ushort = ToLEByteArray(s);
Byte[] result_ulong = ToLEByteArray(l);
....
int i = 123456;
Byte[] result_int = ToLEByteArray(i);
EDIT: what's wrong with the implementation in the question? (from the comment).
Or, restating the question, what is that stuff with IntPtr, Marshal for?
The main issue of the question's implementation is the initial conversion to ulong:
// all the arguments will be converted to ulong
public static byte[] UnsignedIntegerToLEByteArray(ulong value)
In order to illustrate the problem, imagine, that we have two values
Byte x = 0x12; // 18
ulong u = 0x0000000000000012; // 18
and the expected output is
new byte[] {0x12}; // for a single byte
new byte[] {0x12, 0, 0, 0, 0, 0, 0, 0}; // for 8 bytes, i.e. ulong
however, the actual output will be
new byte[] {0x12, 0, 0, 0, 0, 0, 0, 0};
for both byte and ulong. This misbehaviour can lead to problems if you, say, want to write down numeric values (byte, short, ulong etc.), to a binary file, pass them to a binary stream etc:
using (Stream stm = ...) {
...
Byte[] buffer = UnsignedIntegerToLEByteArray(...);
stm.Write(buffer, offset, buffer.Length); // <- the (possibly!) erroneous write
...
}

I had the same thing and I tested it.
I can tell you it works 100%. If you want to check it out yourself, you can for example just pretend you need it the other way. So you reverse the byte array and reverse it if it's IsLittleEndian.
Your output is always in LittleEndian in your method, which is what you want.

Like BitConverter.GetBytes has different overloads for that, you should do it the same way:
public static byte[] UnsignedIntegerToLEByteArray(ulong value) { ...
public static byte[] UnsignedIntegerToLEByteArray(int value) { ...
public static byte[] UnsignedIntegerToLEByteArray(short value) { ...
The complier chooses the right overload when using it like you already did:
byte []tmp = DppUtilities.UnsignedIntegerToLEByteArray(b);
If you try to do this with a single method that try to check the zero bytes, it will produce wrong results:
How to tell if the number 00 00 12 34 should reverse 2 or 4 bytes?

Encode unicode string as byte array C++ and C#

I have C++ code which I want to rewrite to C#. This part
case ID_TYPE_UNICODE_STRING :
if(items[i].GetUString().length() > 0xFFFF)
throw dppError("error");
//GetUstring returns std::wstring type object
DataSize = (WORD) (sizeof(WCHAR)*(items[i].GetUString().length()));
blob.AppendData((const BYTE *) &DataSize, sizeof(WORD)); //blob is byte array
//GetUstring returns std::wstring type object
blob.AppendData((const BYTE *) items[i].GetUString().c_str(), DataSize);
break ;
basically serializes length in bytes of unicode string and string itself to byte array.
Here comes my problem (this code then sends this data to server). I don't know which encoding is used in above lines of code(UTF16, UTF8, etc.).
So I don't know what is the best way to reimplement it in C#.
How can I guess what encoding is used in this C++ project?
And if I can't find encoding used in C++ project, given endianness is same as stated in accepted answer of this question, do you think the two methods (GetBytes and GetString) in accepted answer will work for me (for serializing the unicode string as in C++ project and retrieving it back)? e.g.
these two:
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
static string GetString(byte[] bytes)
{
char[] chars = new char[bytes.Length / sizeof(char)];
System.Buffer.BlockCopy(bytes, 0, chars, 0, bytes.Length);
return new string(chars);
}
Or I am better of to learn what is the encoding used in C++ project?
I will then need to reconstruct the string in the same way from byte array too. And if I am better of learning which encoding was used in C++, how do I get the length of the string in bytes in C#, using System.Text.ASCII.WhateverEncodingWasUsedinC++.GetByteCount(string); ??
PS. Do you think the C++ code is working in encoding agnostic way? If yes, how can I repeat that also in C#?
UPDATE: I am guessing the encoding used is UTF16 because I saw that being mentioned in several variables names, so I think I will assume UTF16 is used, and if something doesn't work out during testing, look for alternative solutions. In that case, what is the best way to get the number of bytes of the UTF16 string? Is following method OK: System.Text.ASCII.Unicode.GetByteCount(string); ??
feedback and comments welcome. Am I wrong somewhere in my reasoning? Thanks

Change the method signature as like this for getting byte[] equivalent of input string.
static byte[] GetBytes(string str)
{
UnicodeEncoding uEncoding = new UnicodeEncoding();
byte[] stringContentBytes = uEncoding.GetBytes("Your string");
return stringContentBytes;
}
For reverse:
static string GetString(byte[] bytes)
{
UnicodeEncoding uEncoding = new UnicodeEncoding();
string stringContent=uEncoding.GetString(bytes);
return new string(stringContent);
}

Serializing a class to a byte array in c#

I am writing my custom serializer for my class which should serialize a class object to a byte array. Below is my sample code:
[StructLayout(LayoutKind.Sequential)]
public class ATTRIBUTE_STR
{
public uint atType;
public uint atLen;
public byte[] atData = new byte[3];
public byte[] serialize()
{
byte[] temp = new byte[Marshal.SizeOf(this)];
byte[] buffer = BitConverter.GetBytes(atType);
Array.Copy(buffer, 0, temp, 0, buffer.Length);
buffer = null;
buffer = BitConverter.GetBytes(atLen);
Array.Copy(buffer, 0, temp, 4, buffer.Length);
Array.Copy(this.atData, 0, temp, 8, this.atData.Length);
return temp;
}
};
However, the byte array atData because of how its stored in memory with byte alignment and all, it's not getting correctly into the temp byte array. Since the byte arrays have even byte alignment.
How can I serialize this object accounting for the alignment of members in memory?
EDIT: I know there are other options like Marshalling or ProtocolBuffers, but I would like using my custom serializer.

Your sizeof call is probably not returning what you think it is, but there hardly seems much point worrying about Sizeof if you're going to stick with hard-coded offsets for the output.]
So you might as well do something like this:
byte[] temp = new byte[8+atLen.Length];
BitConverter.GetBytes(atType).CopyTo(temp,0);
BitConverter.GetBytes(atLen).CopyTo(temp,4);
this.AtData.CopyTo(temp,8);
return temp;
Update: Sizeof returns the amount of unmanaged space necessary to store your object - but bear in mind that your atData array contents are not stored within your object, they are in an array object which is referred to by your object. So Marshal.Sizeof is not going to give you an answer which includes the size of your array - indeed, if you ask Marshal.Sizeof to give you Sizeof(atData), you'll get an exception.

Generic type pointers? and casting byte array to given type of generic?

Okay the basic idea what I'm trying to do is, converting byte array to something like short or int etc. etc.
A simple example might be:
unsafe
{
fixed (byte* byteArray = new byte[5] { 255, 255, 255, 126, 34 })
{
short shortSingle = *(short*)byteArray;
MessageBox.Show((shortSingle).ToString()); // works fine output is -1
}
}
Okay, so what I'm really trying to do is, make an extension to Stream class; extended read and write methods. I need help at the following code:
unsafe public static T Read<T>(this Stream stream)
{
int bytesToRead = sizeof(T); // ERROR: Cannot take the address of, get the size of, or declare a pointer to a managed type ('T')
byte[] buffer = new byte[bytesToRead];
if (bytesToRead != stream.Read(buffer, 0, bytesToRead))
{
throw new Exception();
}
fixed (byte* byteArray = buffer)
{
T typeSingle = *(T*)byteArray; // ERROR: Cannot take the address of, get the size of, or declare a pointer to a managed type ('T')
return typeSingle;
}
}
unsafe public static T[] Read<T>(this Stream stream, int count)
{
// haven't figured out it yet. This is where I read and return T arrays
}
I feel like I have to use pointers for speed because I will be working on writting and reading data from streams like NetworkStream classes. Thanks for your help!
EDIT:
And while I try to figure out how may I return T arrays, I've faced with this problem:
unsafe
{
fixed (byte* byteArray = new byte[5] { 0, 0, 255, 255, 34 })
{
short* shortArray = (short*)byteArray;
MessageBox.Show((shortArray[0]).ToString()); // works fine output is 0
MessageBox.Show((shortArray[1]).ToString()); // works fine output is -1
short[] managedShortArray = new short[2];
managedShortArray = shortArray; // The problem is, How may I convert pointer to a managed short array? ERROR: Cannot implicitly convert type 'short*' to 'short[]'
}
}
THE SUMMARY:
I have to convert from byte array to given type of T
OR
to given type of T array with given length

You can't make this function generic because of pointer restrictions in C#. Any of the following types may be a pointer type:
sbyte, byte, short, ushort, int, uint, long, ulong, char, float, double, decimal, or bool.
Any enum type.
Any pointer type.
Any user-defined struct type that contains fields of unmanaged types only.
But you can't set a restriction on T where T <can be pointer type>. where T : struct is very close, but not enough, because user-defined structs can contain fields of reference types.
There is a workaround - System.Runtime.InteropServices.Marshal.PtrToStructure() (it simply throws an exception if it is unable to work with specified object type), but it would also kill any achieved performance improvements.
I think the only way to do this is to create non-generic functions for all desired types.

Edit: unmanaged constraint added to C# 7.3.
Jumping in a bit late on this one, but with C# 7.3 comes the addition of the unmanaged type constraint.
With the unmanaged type constraint, you can use generic pointers (T*) among other things, provided the type passed in is unmanaged.
I tested the generic method you provided, and it does work now. Additionally, you can extend it to return an array like so:
public static unsafe T[] ReadAsArray<T>(this Stream stream) where T : unmanaged
{
var length = stream.Length;
var returnArray = new T[length];
for (var i = 0; i < length; i++)
{
int bytesToRead = sizeof(T); // no longer throws error
byte[] buffer = new byte[bytesToRead];
if (bytesToRead != stream.Read(buffer, 0, bytesToRead))
{
throw new Exception();
}
fixed (byte* byteArray = buffer)
{
T typeSingle = *(T*)byteArray; // no longer throws error
returnArray[i] = typeSingle;
}
}
return returnArray;
}
You can call it with the following code, which will print the contents of a file:
using (var sourceStream = File.Open(filename, FileMode.Open))
{
var byteArray = sourceStream.ReadAsArray<byte>();
Console.Write(new string(byteArray.Select(b => (char)b).ToArray()));
}

Is unsafe normal in .net C#?

I need do create a new instance of String from the array of sbytes (sbyte[]).
For that I need to convert sbyte[] into sbyte*
It is possible only using unsafe keyword.
is that okay or is there any other ways to create a String from array of sbytes?

First:
How to convert a sbyte[] to byte[] in C#?
sbyte[] signed = { -2, -1, 0, 1, 2 };
byte[] unsigned = (byte[]) (Array)signed;
Then:
string yourstring = UTF8Encoding.UTF8.GetString(unsigned);

Why are you using sbyte?
Encoding.Default.GetString() (and any other encoding) takes a byte[] Array as argument, so you could convert the sbyte[] Array using LINQ if all values are non-negative: array.Cast<byte>().ToArray().