I marshalled correctly to:
IntPtr buffer
Buffer is pointer to array of 2 pointers to arrays with respective data.
The problem is that I get not accurate data, like if there's something missing in the data retrieved (e.g. missimg samples from stream of audio data).
// length is parameter
IntPtr[] temp = new IntPtr[2];
Marshal.Copy(buffer, temp, 0, 2);
bufferedData = new byte[bufferSize];
byte[] a = new byte[length];
byte[] b = new byte[length];
Marshal.Copy(temp[0], a, 0, length);
Marshal.Copy(temp[1], b, 0, length);
edit: sorry I forgot to write those 2 lines :)
I finally solved it. I wasn't reading full input buffer by mistake. Thanks for all your help!
Yes, you would need to copy the byte buffers too :)
Update: That looks better!
If buffer is a pointer to an array, you would need to read the pointer once more.
Effectively:
buffer = Marshal.ReadIntPtr(buffer);
I don't know anything about C# so this is a complete guess, but - you seem to be copying from ints to bytes, is "length" the count in ints, or the count in bytes? Could there be a mixup there? This can be a problem in regular old C++ sometimes.
Related
Here by "directly" I mean without temporary byte[] array.
The problem is, for example I have array of ints or doubles on the disk, so currently I create two arrays -- byte array and int array (in case of ints). The former is just for reading, the latter is the actual output.
Since Stream can read only to byte array I read it to the first array, than copy all the data to the second. It works, but it really hurts me (I am not talking about performance here).
So, how to read the array without temporary array? Using C# unsafe context is fine for me.
So far I tried two approaches: I looked if it is possible to create an array reusing allocated memory and second which looked more promising -- I could get pointer to the result/second array and in unsafe context I could cast it to byte* pointer. Considering my needs it is 100% safe and valid, however byte* pointer is not a byte[] array in C# world and I cannot find the way to cast pointer to array.
Code:
void ReadStuff(Stream stream, double[] data)
{
var dataBytes = new byte[data.Length * sizeof(double)];
stream.Read(dataBytes, 0, dataBytes.Length);
Buffer.BlockCopy(dataBytes, 0, data, 0, dataBytes.Length);
// ...
}
There is no way I know of to copy data directly from a stream to a typed array. But you can process the data in chunks, limiting your memory overhead to a fixed amount. The memory will be copied twice, but this is unavoidable as far as I know.
For example:
public static void ReadArrayDataChunked(BinaryReader binaryReader, Array target, Type type, int bufferSize = 4096)
{
var buffer = new byte[bufferSize];
var tSize = Marshal.SizeOf(type) ;
var remainingBytes = target.Length * tSize;
var targetPosition = 0;
while (remainingBytes > 0)
{
var toRead = Math.Min(remainingBytes, buffer.Length);
var bytesRead = binaryReader.Read(buffer, 0, toRead);
Buffer.BlockCopy(buffer, 0, target, targetPosition, bytesRead);
targetPosition += bytesRead;
remainingBytes -= bytesRead;
}
}
Note that this only works for primitive types due to the BlockCopy, but this helps improve copy performance. You will need to read other types item by item.
For Serialization of Primitive Array, i'am wondering how to convert a Primitive[] to his corresponding byte[]. (ie an int[128] to a byte[512], or a ushort[] to a byte[]...)
The destination can be a Memory Stream, a network message, a file, anything.
The goal is performance (Serialization & Deserialization time), to be able to write with some streams a byte[] in one shot instead of loop'ing' through all values, or allocate using some converter.
Some already solution explored:
Regular Loop to write/read
//array = any int[];
myStreamWriter.WriteInt32(array.Length);
for(int i = 0; i < array.Length; ++i)
myStreamWriter.WriteInt32(array[i]);
This solution works for Serialization and Deserialization And is like 100 times faster than using Standard System.Runtime.Serialization combined with a BinaryFormater to Serialize/Deserialize a single int, or a couple of them.
But this solution becomes slower if array.Length contains more than 200/300 values (for Int32).
Cast?
Seems C# can't directly cast a Int[] to a byte[], or a bool[] to a byte[].
BitConverter.Getbytes()
This solution works, but it allocates a new byte[] at each call of the loop through my int[]. Performances are of course horrible
Marshal.Copy
Yup, this solution works too, but same problem as previous BitConverter one.
C++ hack
Because direct cast is not allowed in C#, i tryed some C++ hack after seeing into memory that array length is stored 4 bytes before array data starts
ARRAYCAST_API void Cast(int* input, unsigned char** output)
{
// get the address of the input (this is a pointer to the data)
int* count = input;
// the size of the buffer is located just before the data (4 bytes before as this is an int)
count--;
// multiply the number of elements by 4 as an int is 4 bytes
*count = *count * 4;
// set the address of the byte array
*output = (unsigned char*)(input);
}
and the C# that call:
byte[] arrayB = null;
int[] arrayI = new int[128];
for (int i = 0; i < 128; ++i)
arrayI[i] = i;
// delegate call
fptr(arrayI, out arrayB);
I successfully retrieve my int[128] into C++, switch the array length, and affecting the right adress to my 'output' var, but C# is only retrieving a byte[1] as return. It seems that i can't hack a managed variable like that so easily.
So i really start to think that all theses casts i want to achieve are just impossible in C# (int[] -> byte[], bool[] -> byte[], double[] -> byte[]...) without Allocating/copying...
What am-i missing?
How about using Buffer.BlockCopy?
// serialize
var intArray = new[] { 1, 2, 3, 4, 5, 6, 7, 8 };
var byteArray = new byte[intArray.Length * 4];
Buffer.BlockCopy(intArray, 0, byteArray, 0, byteArray.Length);
// deserialize and test
var intArray2 = new int[byteArray.Length / 4];
Buffer.BlockCopy(byteArray, 0, intArray2, 0, byteArray.Length);
Console.WriteLine(intArray.SequenceEqual(intArray2)); // true
Note that BlockCopy is still allocating/copying behind the scenes. I'm fairly sure that this is unavoidable in managed code, and BlockCopy is probably about as good as it gets for this.
I'm looking for a smooth/fast way to retrieve every nth short in a byte array and copy it to a new array.
l = lowerbyte
u = upperbyte
My data is of the following form:
byte[] bytes= {a0l, a0u, b0l, b0u, c0l, ..., n0l, n0l, a1l, a1u, b1l, ..., nXl, nXu}
What I need is to get get n byte arrays of length X (e.g., a[0..X], b[0..X], ... or M[a..n][0..X])
I was thinking of the following two steps:
convert values to short (=> short[] shorts= { a0, b0, c0, ... n0, a1, .. nX})
by using something like
short[] shorts= new short[(int)(Math.Ceiling(bytes.Length / 2.0)];
Buffer.BlockCopy(bytes, 0, shorts, 0, bytes.Length);
retrieve every second value from shorts
I'm looking for some fast code here... something like blockcopy with skip
I am completely aware that I could use a loop - but maybe there's a better
solution to it as I need to do this task for 80MB/s...
convert it back to byte arrays (same same - using blockcopy)
byte[] arrayX = new byte[shorts.Length * 2];
Buffer.BlockCopy(shorts, 0, arrayX , 0, arrayX .Length);
Thank you so much!
I think you might as just well copy the bytes directly to the new byte array, having calculated the correct offset for each short.
Converting it all to a separate short array and back to a byte array is a waste of time IMO.
Im trying to split up a memorystream into chunks by reading parts into a byte array but i think i have got something fundamentally wrong. I can read the first chunk but when i try to read rest of memorystream i get index out of bound even if there are more bytes to read. It seems that the issue is the size of the receving byte buffer that need to be as large as the memorystrem. I need to convert it into chunks as the code is in a webservice.
Anyone knows whats wrong with this code
fb.buffer is MemoryStream
long bytesLeft = fb.Buffer.Length;
fb.Buffer.Position = 0;
int offset =0;
int BUFF_SIZE = 8196;
while (bytesLeft > 0)
{
byte[] fs = new byte[BUFF_SIZE];
fb.Buffer.Read(fs, offset, BUFF_SIZE);
offset += BUFF_SIZE;
bytesLeft -= BUFF_SIZE;
}
offset here is the offset into the array. It should be zero here. You should also be looking at the return value from Read. It is not guaranteed to fill the buffer, even if more data is available.
However, if this is a MemoryStream - a better option might be ArraySegment<byte>, which requires no duplication of data.
Please look at this code for Stream.Read from MSDN from a glance - you shouldn't be incrementing offset - it should always be zero. Unless, of course, you happen to know for a fact the exact length of the stream in advance (therefore you would create the array the exact size).
You should also always grab the amount of bytes read from Read (the return value).
If you're looking to split it into 'chunks` do you mean you want n 8k chunks? Then you might want to do something like this:
List<byte[]> chunks = new List<byte[]>();
byte chunk = new byte[BUFF_SIZE];
int bytesRead = fb.Buffer.Read(chunk, 0, BUFF_SIZE);
while(bytesRead > 0)
{
if(bytesRead != BUFF_SIZE)
{
byte[] tail = new byte[bytesRead];
Array.Copy(chunk, tail, bytesRead);
chunk = tail;
}
chunks.Add(chunk);
bytesRead = fb.Buffer.Read(chunk, 0, BUFF_SIZE);
}
Note in particular that the last chunk is more than likely not going to be exactly BUFF_SIZE in length.
I'm currently reading a file and wanted to be able to convert the array of bytes obtained from the file into a short array.
How would I go about doing this?
Use Buffer.BlockCopy.
Create the short array at half the size of the byte array, and copy the byte data in:
short[] sdata = new short[(int)Math.Ceiling(data.Length / 2)];
Buffer.BlockCopy(data, 0, sdata, 0, data.Length);
It is the fastest method by far.
One possibility is using Enumerable.Select:
byte[] bytes;
var shorts = bytes.Select(b => (short)b).ToArray();
Another is to use Array.ConvertAll:
byte[] bytes;
var shorts = Array.ConvertAll(bytes, b => (short)b);
A shorthard is a compound of two bytes. If you are writing all the shorts to the file as true shorts then those conversions are wrong. You must use two bytes to get the true short value, using something like:
short s = (short)(bytes[0] | (bytes[1] << 8))
short value = BitConverter.ToInt16(bytes, index);
I dont know, but I would have expected another aproach to this question.
When converting a sequence of bytes into a sequence of shorts, i would have it done like #Peter did
short s = (short)(bytes[0] | (bytes[1] << 8))
or
short s = (short)((bytes[0] << 8) | bytes[1])
depending on endianess of the bytes in the file.
But the OP didnt mention his usage of the shorts or the definition of the shorts in the file.
In his case it would make no sense to convert the byte array to a short array, because it would take twice as much memory, and i doubt if a byte would be needed to be converted to a short when used elsewhere.
short[] wordArray = Array.ConvertAll(byteArray, (b) => (short)b);
byte[] bytes;
var shorts = bytes.Select(n => System.Convert.ToInt16(n)).ToArray();