I have an array of bytes that are actually 16-bit samples from a sound card. So 1000 bytes actually represents 500 Short (16-bit values).
Currently I'm converting them like this:
byte [] inputData = new byte[1000];
short [] convertedData = new short[500];
Buffer.BlockCopy(inputData, 0, convertedData , 0, 1000);
It works fine and it's pretty quick as it's a low-level byte copy.
However is there a way to do it without the copy? i.e. tell C# to treat this area of memory as an array of 500 shorts instead of 1000 bytes? I know that in C/C++ I could just cast the pointer and it would work.
This copy happens in a tight loop, up to 5000 times a second, so if I can remove the copy it would be worthwhile.
StructLayout lets you control the physical layout of the data fields in a class or structure. It is typically used when interfacing with unmanaged code which expects the data in a specific layout.
Give this a try:
[StructLayout(LayoutKind.Explicit)]
struct UnionArray
{
[FieldOffset(0)]
public Byte[] Bytes;
[FieldOffset(0)]
public short[] Shorts;
}
static void Main(string[] args)
{
var union = new UnionArray() {Bytes = new byte[1024]};
foreach (short s in union.Shorts)
{
Console.WriteLine(s);
}
}
Perhaps a C# analog of the C-language union would do the trick.
bytes[] inputData = Array.ConvertAll<short, bytes>(bytes, delegate(short[] convertedData) { return short.Parse(convertedData); } );
something like that, i didnt check, but maybe that'll help you out :)
Related
I know in C# we can always get the sub-array of a given array by using Array.Copy() method. However, this will consume more memory and processing time which is unnecessary in read-only situation. For example, I'm writing a heavy load network program which exchanges messages with other nodes in the cluster very frequently. The first 20 bytes of every message is the message header while the rest bytes make up the message body. Therefore, I will divide the received raw message into header byte array and body byte array in order to process them separately. However, this will obviously consume double memory and extra time. In C, we can easily use a pointer and assign offset to it to access different parts of the array.
For instance, in C language, if we have a char a[] = "ABCDEFGHIJKLMN", we can declare a char* ptr = a + 3 to represent the array DEFGHIJKLMN.
Is there a way to accomplish this in C#?
You might be interested in ArraySegments or unsafe.
ArraySegments delimits a section of a one-dimensional array.
Check ArraySegments in action
ArraySegments usage example:
int[] array = { 10, 20, 30 };
ArraySegment<int> segment = new ArraySegment<int>(array, 1, 2);
// The segment contains offset = 1, count = 2 and range = { 20, 30 }
Unsafe define an unsafe context in which pointers can be used.
Unsafe usage example:
int[] a = { 4, 5, 6, 7, 8 };
unsafe
{
fixed (int* c = a)
{
// use the pointer
}
}
First of all you must consider this as a premature optimization.
But you may use several ways to reduce memory consumption, if you sure you really need it:
1) You may use Flyweight pattern https://en.wikipedia.org/wiki/Flyweight_pattern to pool duplicated resources.
2) You may try to use unsafe directive and manual pointer management.
3) You may just switch to C for this functionality and just call native code from your C# program.
From my experience memory consumption for short-lived objects is not a big problem and I'd just write code with flyweight pattern and profile application afterwards.
Assuming you have a Message wrapper class in C#? Why not just add a property on it called header that returns the top 20 bytes.
You can easily accomplish this using skip and take suggested by Jonathon Reinhart above if you have the entire initial array in a memory array, but it sounds like you may have it in a network stream, which means the property might be a little more involved by doing a read of the initial 20 bytes from the the stream.
Something along the lines of:
class Message
{
private readonly Stream _stream;
private byte[] _inMemoryBytes;
public Message(Stream stream)
{
_stream = stream;
}
public IEnumerable<byte> Header
{
get
{
if (_inMemoryBytes.Length >= 20)
return _inMemoryBytes.Take(20);
_stream.Read(_inMemoryBytes, 0, 20);
return _inMemoryBytes.Take(20);
}
}
public IEnumerable<byte> FullMessage
{
get
{
// Read and return the whole message. You might want amend to data already read.
}
}
}
Say I have a struct like the following:
struct MyStruct {
Guid g;
}
Is it possible to get a Span<byte> that references the bytes of the struct?
struct MyStruct {
Guid g;
public void Foo() {
Span<byte> bytes = ???
}
}
Such that bytes would be a Span<byte> of length 16 that would allow reading and writing the individual bytes of the Guid field directly.
I can do something similar with unsafe code, but it seems that this should now be possible with safe code via span, but I can't figure out how to produce the span.
Edit: Clarify that I want a Span that points to the actual storage location of the Guid. Meaning new Span<byte>(g.ToByteArray)) is not what I'm looking for. That will allocate a new array, copy the bytes to the array, and create a Span referencing the newly allocated array. Modifying bytes via such a span will not modify the Guid.
You can force the Guid to an array
struct MyStruct {
public Guid g;
}
...
var s = new MyStruct();
var span = new Span<byte>(s.g.ToByteArray());
span[2] = 4;
Note : The above will not modify the original struct. However, as you pointed out, you can do this with unsafe
var s = new MyStruct();
var span = new Span<byte>(&s , Marshal.SizeOf(s));
// woah it just become mutable
span[2] = 4;
Try this in .Net Core 2.1 or above:
Span<MyStruct> valSpan = MemoryMarshal.CreateSpan(ref mystruct, 1);
Span<byte> span = MemoryMarshal.AsBytes(valSpan);
But use with caution, you should not contain any pointer or reference type in your struct, since GC can move reference type. Actually, it will do runtime check and throw an exception if you do that.
Look at the constructors of Span class, the only ways to create a Span is providing a managed array or a pointer.
But Guid is made up with 1 int, 2 shorts and 8 bytes, not a byte array.
So it's impossible (at least not now).
Some code I'm modifying makes extensive use of structs to communicate with some factory equipment by loading them to or from byte arrays.
Here's a made-up ultra-simplified example of such a struct.
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct K_FOO
{
public byte a_byte; // 1 byte
public BYTE3_TYPE b3; // 3 bytes
public int num; // happens to be 4 bytes
}
BYTE3_TYPE looks like this...
[StructLayout(LayoutKind.Sequential, Pack = 1)]
public class BYTE3_TYPE
{
[System.Runtime.InteropServices.MarshalAs(UnmanagedType.ByValArray, SizeConst = 3)]
public byte[] L = new byte[3];
}
If I just do a
K_FOO origFoo = new K_FOO();
the int and the byte are initialized to 0's, I assume because they are native types, but the byte array b3 is *un*initialized - all nulls. I have to explicitly load it, for example,
BYTE3_TYPE b3 = new BYTE3_TYPE();
origFoo.b3 = b3;
... and I couldn't think of any alternative because structs don't take parameterless constructors but the real structs are huge.
But I noticed something interesting. We have routines to copy these structs to and from byte arrays. For example . . .
public static T ByteArrayToStructure<T>(byte[] buffer) where T : struct
{
int length = buffer.Length;
IntPtr ptr = Marshal.AllocHGlobal(length); // allocate (length) bytes
Marshal.Copy(buffer, 0, ptr, length); // copies into UNmanaged space
T result = (T)Marshal.PtrToStructure(ptr, typeof(T));
Marshal.FreeHGlobal(ptr);
return result;
}
...and if I call it...
K_FOO retFoo = ByteArrayToStructure<K_FOO>(buffer.bytes);
... the resulting struct is returned fully initialized, the bytes in the byte array have all had their space allocated so they could be loaded, apparently in the PtrToStructure() call. This implies .Net "knows" how to initialize such a struct. So is there some way to get .Net to do that for me so I can avoid writing hundreds of lines of explicit initialization code? Thanks in advance!
If you make your BYTE3_TYPE a struct instead of a class, the default constructor (calling K_FOO origFoo = new K_FOO();) will initialize the entire thing to zero correctly.
This is also likely the correct approach if you're trying to match existing specifications and pass this to custom hardware.
This is related to my previous question, and thought will make this as a sparate question as it will make more sense.
I have created my Struct as :
public struct Smb_Parameters
{
public byte WordCount;
public ushort[] Words;
}
Without Assigning any Values when I try to get the size of the Struct it returns 4-Bytes:
Smb_Parameters smbParameter = new Smb_Parameters();
int len1 = Marshal.SizeOf(smbParameter );
MessageBox.Show(len1.ToString());
But When I assign the Values to structure fields :
Smb_Parameters smbParameter = new Smb_Parameters();
string myString= "String ll be converted to byte";
smbParameter.WordCount=0x00;
smbParameter .Words=Encoding.ASCII.GetBytes(myString);
int len1 = Marshal.SizeOf(smbParameter );
MessageBox.Show(len1.ToString());
Still now It shows the Length as 4-Bytes but I need the updated size.
If you wish to get the size as if it was an unmanaged type, you'd need to supply some information about its fields (e.g., the length of the array). Without it, the array length would not be taken into account.
e.g.,
public struct Smb_Parameters1
{
public byte WordCount; //1 byte
public ushort[] Words; //4 bytes (a "pointer")
}
Marshal.SizeOf(typeof(Smb_Parameters1)); //8 (with padding)
//I don't see how you get 4 unless you are on a 16-bit system maybe
[StructLayout(LayoutKind.Sequential)]
public struct Smb_Parameters2
{
public byte WordCount; //1 byte
[MarshalAs(UnmanagedType.ByValArray, SizeConst=10)]
public ushort[] Words; //20 bytes (2 * 10 bytes)
}
Marshal.SizeOf(typeof(Smb_Parameters2)); //22 (with padding)
Note that these are sizes are fixed (as if it was declared in a C/C++ program). The size reported by SizeOf() will only use these and not take into account what size array you store in Words.
Smb_Parameters1 s1 = new Smb_Parameters1() { Words = new ushort[] { 0, 1, 2 } };
Smb_Parameters2 s2 = new Smb_Parameters2() { Words = new ushort[] { 0, 1, 2 } };
Marshal.SizeOf(s1); //8 bytes
Marshal.SizeOf(s2); //22 bytes
The Words field is an array, and arrays are reference types. The structure doesn't actually contains the array items, it only contains a reference to the array, which is stored somewhere else (typically on the heap). So SizeOf always returns the same size, since the size of a reference doesn't depend on the size of the object pointed by that reference.
Marshal.SizeOf() returns a fixed size of an class/struct. The length does never depend on the contents of your object passed.
You could calculate the size with Marshal.SizeOf(typeof(byte))+Marshal.SizeOf(typeof(ushort))*yourarraylength
Another way would be to use the BinaryFormatter class to serialize your struct into binary form. It returns you a byte array. If you take the length of it you know its serialized size. Note that the result of BinaryFormatter cannot not be easily read by a non-.net language since its format is native to .net only.
I have a byte array that represents a complete TCP/IP packet. For clarification, the byte array is ordered like this:
(IP Header - 20 bytes)(TCP Header - 20 bytes)(Payload - X bytes)
I have a Parse function that accepts a byte array and returns a TCPHeader object. It looks like this:
TCPHeader Parse( byte[] buffer );
Given the original byte array, here is the way I'm calling this function right now.
byte[] tcpbuffer = new byte[ 20 ];
System.Buffer.BlockCopy( packet, 20, tcpbuffer, 0, 20 );
TCPHeader tcp = Parse( tcpbuffer );
Is there a convenient way to pass the TCP byte array, i.e., bytes 20-39 of the complete TCP/IP packet, to the Parse function without extracting it to a new byte array first?
In C++, I could do the following:
TCPHeader tcp = Parse( &packet[ 20 ] );
Is there anything similar in C#? I want to avoid the creation and subsequent garbage collection of the temporary byte array if possible.
A common practice you can see in the .NET framework, and that I recommend using here, is specifying the offset and length. So make your Parse function also accept the offset in the passed array, and the number of elements to use.
Of course, the same rules apply as if you were to pass a pointer like in C++ - the array shouldn't be modified or else it may result in undefined behavior if you are not sure when exactly the data will be used. But this is no problem if you are no longer going to be modifying the array.
I would pass an ArraySegment<byte> in this case.
You would change your Parse method to this:
// Changed TCPHeader to TcpHeader to adhere to public naming conventions.
TcpHeader Parse(ArraySegment<byte> buffer)
And then you would change the call to this:
// Create the array segment.
ArraySegment<byte> seg = new ArraySegment<byte>(packet, 20, 20);
// Call parse.
TcpHeader header = Parse(seg);
Using the ArraySegment<T> will not copy the array, and it will do the bounds checking for you in the constructor (so that you don't specify incorrect bounds). Then you change your Parse method to work with the bounds specified in the segment, and you should be ok.
You can even create a convenience overload that will accept the full byte array:
// Accepts full array.
TcpHeader Parse(byte[] buffer)
{
// Call the overload.
return Parse(new ArraySegment<byte>(buffer));
}
// Changed TCPHeader to TcpHeader to adhere to public naming conventions.
TcpHeader Parse(ArraySegment<byte> buffer)
If an IEnumerable<byte> is acceptable as an input rather than byte[], and you're using C# 3.0, then you could write:
tcpbuffer.Skip(20).Take(20);
Note that this still allocates enumerator instances under the covers, so you don't escape allocation altogether, and so for a small number of bytes it may actually be slower than allocating a new array and copying the bytes into it.
I wouldn't worry too much about allocation and GC of small temporary arrays to be honest though. The .NET garbage collected environment is extremely efficient at this type of allocation pattern, particularly if the arrays are short lived, so unless you've profiled it and found GC to be a problem then I'd write it in the most intuitive way and fix up performance issues when you know you have them.
If you really need these kind of control, you gotta look at unsafe feature of C#. It allows you to have a pointer and pin it so that GC doesn't move it:
fixed(byte* b = &bytes[20]) {
}
However this practice is not suggested for working with managed only code if there are no performance issues. You could pass the offset and length as in Stream class.
If you can change the parse() method, change it to accept the offset where the processing should begin.
TCPHeader Parse( byte[] buffer , int offset);
You could use LINQ to do something like:
tcpbuffer.Skip(20).Take(20);
But System.Buffer.BlockCopy / System.Array.Copy are probably more efficient.
This is how I solved it coming from being a c programmer to a c# programmer. I like to use MemoryStream to convert it to a stream and then BinaryReader to break apart the binary block of data. Had to add the two helper functions to convert from network order to little endian. Also for building a byte[] to send see
Is there a way cast an object back to it original type without specifing every case? which has a function that allow for converting from an array of objects to a byte[].
Hashtable parse(byte[] buf, int offset )
{
Hashtable tcpheader = new Hashtable();
if(buf.Length < (20+offset)) return tcpheader;
System.IO.MemoryStream stm = new System.IO.MemoryStream( buf, offset, buf.Length-offset );
System.IO.BinaryReader rdr = new System.IO.BinaryReader( stm );
tcpheader["SourcePort"] = ReadUInt16BigEndian(rdr);
tcpheader["DestPort"] = ReadUInt16BigEndian(rdr);
tcpheader["SeqNum"] = ReadUInt32BigEndian(rdr);
tcpheader["AckNum"] = ReadUInt32BigEndian(rdr);
tcpheader["Offset"] = rdr.ReadByte() >> 4;
tcpheader["Flags"] = rdr.ReadByte() & 0x3f;
tcpheader["Window"] = ReadUInt16BigEndian(rdr);
tcpheader["Checksum"] = ReadUInt16BigEndian(rdr);
tcpheader["UrgentPointer"] = ReadUInt16BigEndian(rdr);
// ignoring tcp options in header might be dangerous
return tcpheader;
}
UInt16 ReadUInt16BigEndian(BinaryReader rdr)
{
UInt16 res = (UInt16)(rdr.ReadByte());
res <<= 8;
res |= rdr.ReadByte();
return(res);
}
UInt32 ReadUInt32BigEndian(BinaryReader rdr)
{
UInt32 res = (UInt32)(rdr.ReadByte());
res <<= 8;
res |= rdr.ReadByte();
res <<= 8;
res |= rdr.ReadByte();
res <<= 8;
res |= rdr.ReadByte();
return(res);
}
I don't think you can do something like that in C#. You could either make the Parse() function use an offset, or create 3 byte arrays to begin with; one for the IP Header, one for the TCP Header and one for the Payload.
There is no way using verifiable code to do this. If your Parse method can deal with having an IEnumerable<byte> then you can use a LINQ expression
TCPHeader tcp = Parse(packet.Skip(20));
Some people who answered
tcpbuffer.Skip(20).Take(20);
did it wrong. This is excellent solution, but the code should look like:
packet.Skip(20).Take(20);
You should use Skip and Take methods on your main packet, and tcpbuffer should not be exist in the code you posted. Also you don't have to use then System.Buffer.BlockCopy.
JaredPar was almost correct, but he forgot the Take method
TCPHeader tcp = Parse(packet.Skip(20));
But he didn't get wrong with tcpbuffer.
Your last line of your posted code should look like:
TCPHeader tcp = Parse(packet.Skip(20).Take(20));
But if you want to use System.Buffer.BlockCopy anyway instead Skip and Take, because maybe it is better in performance as Steven Robbins answered : "But System.Buffer.BlockCopy / System.Array.Copy are probably more efficient", or your Parse function cannot deal with IEnumerable<byte>, or you are more used to System.Buffer.Block in your posted question, then I would recommend to simply just make tcpbuffer not local variable, but private or protected or public or internal and static or not field (in other words it should be defined and created outside method where your posted code is executed). Thus tcpbuffer will be created only once, and his values (bytes) will be set every time you pass the code you posted at System.Buffer.BlockCopy line.
This way your code can look like:
class Program
{
//Your defined fields, properties, methods, constructors, delegates, events and etc.
private byte[] tcpbuffer = new byte[20];
Your unposted method title(arguments/parameters...)
{
//Your unposted code before your posted code
//byte[] tcpbuffer = new byte[ 20 ]; No need anymore! this line can be removed.
System.Buffer.BlockCopy( packet, 20, this.tcpbuffer, 0, 20 );
TCPHeader tcp = Parse( this.tcpbuffer );
//Your unposted code after your posted code
}
//Your defined fields, properties, methods, constructors, delegates, events and etc.
}
or simply only the necessary part:
private byte[] tcpbuffer = new byte[20];
...
{
...
//byte[] tcpbuffer = new byte[ 20 ]; No need anymore! This line can be removed.
System.Buffer.BlockCopy( packet, 20, this.tcpbuffer, 0, 20 );
TCPHeader tcp = Parse( this.tcpbuffer );
...
}
If you did:
private byte[] tcpbuffer;
instead, then you must on your constructor/s add the line:
this.tcpbuffer = new byte[20];
or
tcpbuffer = new byte[20];
You know that you don't have to type this. before tcpbuffer, it is optional, but if you defined it static, then you cannot do that. Instead you'll have to type the class name and then the dot '.', or leave it (just type the name of the field and that's it all).
Why not flip the problem and create classes that overlay the buffer to pull bits out?
// member variables
IPHeader ipHeader = new IPHeader();
TCPHeader tcpHeader = new TCPHeader();
// passing in the buffer, an offset and a length allows you
// to move the header over the buffer
ipHeader.SetBuffer( buffer, 0, 20 );
if( ipHeader.Protocol == TCP )
{
tcpHeader.SetBuffer( buffer, ipHeader.ProtocolOffset, 20 );
}