We Keep Coding

Strange data corruption with Buffer.BlockCopy C#

BACKGROUND
I am writing a C# program which collects some information by data acquisition. It's quite complex so I won't detail it all here, but the data acquisition is instigated continuously and then, on an asynchronous thread, my program periodically visits the acquisition buffer and takes 100 samples from it. I then look inside the 100 samples for a trigger condition which I am interested in. If I see the trigger condition I collect a bunch of samples from a pre-trigger buffer, a bunch more from a post-trigger buffer, and assemble it all together into one 200-element array.
In my asynchronous thread I assemble my 200-element array (of type double) using the Buffer.BlockCopy method. The only specific reason I chose to use this method is that I need to be careful about how much data processing I do in my asynchronous thread; if I do too much I can end up over-filling the acquisition buffer because I am not visiting it often enough. Since Buffer.BlockCopy is much more efficient at pushing data from a source array into a destination array than a big 'for loop', that's the sole reason I decided to use it.
THE QUESTION
When I call the Buffer.BlockCopy method I do this:
Buffer.BlockCopy(newData, 0, myPulse, numSamplesfromPreTrigBuf, (trigLocation * sizeof(double));
Where;
newData is a double[] array containing new data (100 elements) (with typical data like 0.0034, 6.4342, etc ranging from 0 to 7).
myPulse is the destination array. It is instantiated with 200 elements.
numSamplesfromPreTrigBuf is an offset that I want to apply in this particular instance of the copy
trigLocation is the number of elements I want to copy in this particular instance.
The copy occurs without error, but the data written into myPulse is all screwed up; numbers such as -2.05E-289 and 5.72E+250. Either tiny numbers or massive numbers. These numbers do not occur in my source array.
I have resolved the issue simply by using Array.Copy() instead, with no other source-code modification except for removing the need to calculate the number of elements to copy by multiplying by sizeof(double). But I did spend two hours trying to debug the Buffer.BlockCopy() method with absolutely no idea why the copy is garbage.
Would any body have an idea, from my example usage of Buffer.BlockCopy (which I believe is the correct usage), how garbage data might be copied across?

I assume your offset is wrong - it's also a byte-offset, so you need to multiply it by sizeof(double), just like with the length.
Be careful about using BlockCopy and similar methods - you lose some of the safety of .NET. Unlike outright unsafe methods, it does check array bounds, but you can still produce some pretty weird results (and I assume you could e.g. produce invalid references - a big problem EDIT: fortunately, BlockCopy only works on primitive typed arrays).
Also, BlockCopy isn't thread-safe, so you want to synchronize access to the shared buffer, if you're accessing it from more than one thread at a time.

Indeed, Buffer.BlockCopy allows the source Array and destination Array to have different element types, so long as each element type is primitive. Either way, as you can see from the mscorlib.dll source code for ../vm/comutilnative.cpp, the copy is just a direct imaging operation which never interprets the copied bytes in any way (i.e., as 'logical' or 'numeric' values). It basically calls the C-language classic, memmove. So don't expect this:
var rgb = new byte[] { 1, 2, 3 };
var rgl = new long[3];
Buffer.BlockCopy(rgb, 0, rgl, 0, 3); // likely ERROR: integers never widened or narrowed
// INTENTION?: rgl = { 1, 2, 3 }
// RESULT: rgl = { 0x0000000000030201, 0, 0 }
Now given that Buffer.BlockCopy takes just a single count argument, allowing differently-sized element types introduces the fundamental semantic ambiguity of whether that single count argument would be counting the total bytes in terms or source or destination elements. Solutions to this might include:
Add a second count property so you'd have one each for src and dst; (no...)
Arbitrarily select src vs. dst for expressing count--and document the choice; (no...)
Always express count in bytes, since element size "1" is the (only) common denominator that's suitable to arbitrary different-sized element types. (yes)
Since (1.) is complex (possibly adding even more confusion), and the arbitrary symmetry-breaking of (2.) is poorly self-documenting, the choice taken here was (3.), meaning the count argument must always be specified in bytes.
Because the situation for the srcOffset and dstOffset arguments isn't as critical (on account of there being independent arguments for each 'offset', whereby each c̲o̲u̲l̲d̲ be indexed relative to its respective Array; spoiler alert: ...they aren't), it's less-widely mentioned that these arguments are also always expressed in bytes. From the documentation (emphasis added):
Buffer.BlockCopyhttps://learn.microsoft.com/en-us/dotnet/api/system.buffer.blockcopyParameters
src Array The source buffer.
srcOffset Int32 The zero-based byte offset into src.
dst Array The destination buffer.
dstOffset Int32 The zero-based byte offset into dst.
count Int32 The number of bytes to copy.
The fact that the srcOffset and dstOffset are byte-offsets leads to the strange situations under discussion on this page. For one thing, it entails that the copy of the first and/or last element can be partial:
var rgb = new byte[] { 0xFF, 1, 2, 3, 4, 5, 6, 7, 8, 0xFF };
var rgl = new long[10];
Buffer.BlockCopy(rgb, 1, rgl, 1, 8); // likely ERROR: does not target rgl[1], but
// rather *parts of* both rgl[0] and rgl[1]
// INTENTION? (see above) rgl = { 0L, 1L, 2L, 3L, 4L, 5L, 6L, 7L, 8L, 0L } ✘
// INTENTION? (little-endian) rgl = { 0L, 0x0807060504030201L, 0L, ... } ✘
// INTENTION? (big-endian) rgl = { 0L, 0x0102030405060708L, 0L, ... } ✘
// ACTUAL RESULT: rgl = { 0x0706050403020100L, 8L, 0L, 0L, ... } ?
// ^^-- this byte not copied (see text)
Here, we see that instead of (perhaps) copying something into rgl[1], the element at index 1, and then (maybe) continuing on from there, the copy targeted byte-offset 1 within the first element rgl[0], and led to a partial copy--and surely unintended corruption--of that element. Specifically, byte 0 of rgl[0]--the least-significant-byte of a little-endian long--was not copied.
Continuing with the example, the long value at index 1 is a̲l̲s̲o̲ incompletely written, this time storing value '8' into its least-significant-byte, notably without affecting its other (upper) 7 bytes.
Because I didn't craft my example well enough to explicitly show it, let me be clear about this last point: for these partially-copied long values, the parts that are not copied are not zeroed out as might normally be expected from a proper long store of a byte value. So for the discussion of Buffer.BlockCopy, "partially-copied" means that the un-copied bytes of any multi-byte primitive (e.g. long) value are retained unaltered from before the operation, and thus become "merged" into the new value in some endianness-dependent--and thus likely (and hopefully) unintentional--manner.
To "fix" the example code, the offset supplied for each Array must be pre-multiplied by its respective element size to convert it to a byte offset. This will "correct" the above code to the only sensible operation Buffer.​BlockCopy might reasonably perform here, namely a little-endian copy between (one or more) source and (one or more) destination elements, taking care to ensure that no element is partially- or incompletely-copied, respective to its size.
Buffer.BlockCopy(rgb, 1 * sizeof(byte), rgl, 1 * sizeof(long), 8); // CORRECTED (?)
// CORRECT RESULT: rgl = { 0L, 0x0807060504030201L, 0L, ... } ✔
// repaired code shows a proper little-endian store of eight consecutive bytes from a
// byte[] into exactly one complete element of a long[].
In the fixed example, 8 complete byte elements are copied to 1 complete long element. For simplicity, this is a "many-to-1" copy, but you can imagine more elaborate scenarios as well (not shown). In fact, with respect to element count from source-to-destination, a single call to Buffer.BlockCopy can deploy any of five operational patterns: { nop, 1-to-1, 1-to-many, many-to-1, many-to-many }.
The code also illustrates how concerns of endianness are implicated by Buffer.BlockCopy accepting arrays of differently-sized elements. Indeed, the repaired example seems to entail that the code now inherently incorporates a (correctness-)​dependency on the endianness of the CPU on which it happens to be running. Combine this with the fact that realistic use-cases seem scarce or obscure, especially remembering the very real and error-prone partial-copying hazard discussed above.
Considering these points would suggest that the technique of mixing source/destination element sizes within a single call to Buffer.BlockCopy, while allowed by the API, should be avoided. In any case, use mixed element sizes with special caution, if at all.

How to fastly copy only a portion of an array to another array?

I'm writing a high-performance data structure. One problem I came across is there doesn't seem to be anyway to copy only a portion of an array to another array (preferably as quickly as possible). I also use generics, so I'm not really sure how I'd use Buffer.BlockCopy since it demands byte addresses and it appears to be impossible to objectively determine the size of an object. I know Buffer.BlockCopy works at a byte-level, but does it also count padding as a byte?
Example:
var tmo=new T[5];
var source = new T[10];
for(int i=5;i<source.Length;i++)
{
tmp[i-5]=source[i];
}
How would I do this in a faster way like Array.CopyTo?

You can use Array.Copy().
Array.Copy(source , 5, tmp, 0, tmp.Length);

Resizing big arrays

I have an array which size is like 2 GB (filled with audio samples). Now I want to apply a filter for that array. This filter is generating like 50% more samples than input source. So now I need to create new array which size is 3 GB. Now I gave 5 GB of memory used. But if this filter can operate only at that source array and only need some more space in this array.
Question: can I allocate a memory in C# that can be resized w/o creating a second memory block, then removing that first one?
I just thought, If memory in PC's is divided into 4 kB pages (or more), so why C# cannot (?) use that good feature?

If your filter can work in-place just allocate 50% more space at the beginning. All you need to know is the actual length of the original sample.
If that code doesn't work always and you don't want to consume more memory beforehand, you can allocate half of the original array (the extension array) and check which part your access relates to:
byte[] myOriginalArray = new byte[2GB]; // previously allocated
byte[] myExtensionArray = new byte[1GB]; // 50% of the original
for(... my processing code of the array ...)
{
byte value = read(index);
... process the index and the value here
store(index, value);
}
byte read(int index)
{
if(index < 2GB) return myOriginalArray[index];
return myExtensionArray[index - 2GB];
}
void store(int index, byte value)
{
if(index < 2GB) myOriginalArray[index] = value;
myExtensionArray[index - 2GB] = value;
}
You add index check and subtraction overhead for each access to the array. That could also be made smarter for certain cases. For instance for the portion you do not need to access extension you can use your faster loop and for the part where you need to write to extension part you can use the slower version (two consecutive loops).

Question: can I allocate a memory in C# that can be resized w/o creating a second memory block, then removing that first one?
No, you cannot resize an array in .NET. If you want to increase the size of an array you will have to create a new and bigger array and copy all the data from the existing array to the new array.
To get around this problem you could provide your own "array" implementation based on allocating smaller chunks of memory but presenting it as one big buffer of data. An example of this is StringBuilder that is based on an implementation of chunks of characters, each chunk being a separate Char[] array.
Another option is to use P/Invoke to get access to low level memory management functions like VirtualAlloc that allows you to reserve pages of memory in advance. You need to do this in a 64 bit process because the virtual address space of a 32 bit process is only 4 GB. You probably also need to work with unsafe code and pointers.

Prepend to a C# Array

Given a populated byte[] values in C#, I want to prepend the value (byte)0x00 to the array. I assume this will require making a new array and adding the contents of the old array. Speed is an important aspect of my application. What is the best way to do this?
-- EDIT --
The byte[] is used to store DSA (Digital Signature Algorithm) parameters. The operation will only need to be performed once per array, but speed is important because I am potentially performing this operation on many different byte[]s.

If you are only going to perform this operation once then there isn't a whole lot of choices. The code provided by Monroe's answer should do just fine.
byte[] newValues = new byte[values.Length + 1];
newValues[0] = 0x00; // set the prepended value
Array.Copy(values, 0, newValues, 1, values.Length); // copy the old values
If, however, you're going to be performing this operation multiple times you have some more choices. There is a fundamental problem that prepending data to an array isn't an efficient operation, so you could choose to use an alternate data structure.
A LinkedList can efficiently prepend data, but it's less efficient in general for most tasks as it involves a lot more memory allocation/deallocation and also looses memory locallity, so it may not be a net win.
A double ended queue (known as a deque) would be a fantastic data structure for you. You can efficiently add to the start or the end, and efficiently access data anywhere in the structure (but you can't efficiently insert somewhere other than the start or end). The major problem here is that .NET doesn't provide an implementation of a deque. You'd need to find a 3rd party library with an implementation.
You can also save yourself a lot when copying by keeping track of "data that I need to prepend" (using a List/Queue/etc.) and then waiting to actually prepend the data as long as possible, so that you minimize the creation of new arrays as much as possible, as well as limiting the number of copies of existing elements.
You could also consider whether you could adjust the structure so that you're adding to the end, rather than the start (even if you know that you'll need to reverse it later). If you are appending a lot in a short space of time it may be worth storing the data in a List (which can efficiently add to the end) and adding to the end. Depending on your needs, it may even be worth making a class that is a wrapper for a List and that hides the fact that it is reversed. You could make an indexer that maps i to Count-i, etc. so that it appears, from the outside, as though your data is stored normally, even though the internal List actually holds the data backwards.

Ok guys, let's take a look at the perfomance issue regarding this question.
This is not an answer, just a microbenchmark to see which option is more efficient.
So, let's set the scenario:
A byte array of 1,000,000 items, randomly populated
We need to prepend item 0x00
We have 3 options:
Manually creating and populating the new array
Manually creating the new array and using Array.Copy (#Monroe)
Creating a list, loading the array, inserting the item and converting the list to an array
Here's the code:
byte[] byteArray = new byte[1000000];
for (int i = 0; i < byteArray.Length; i++)
{
byteArray[i] = Convert.ToByte(DateTime.Now.Second);
}
Stopwatch stopWatch = new Stopwatch();
//#1 Manually creating and populating a new array;
stopWatch.Start();
byte[] extendedByteArray1 = new byte[byteArray.Length + 1];
extendedByteArray1[0] = 0x00;
for (int i = 0; i < byteArray.Length; i++)
{
extendedByteArray1[i + 1] = byteArray[i];
}
stopWatch.Stop();
Console.WriteLine(string.Format("#1: {0} ms", stopWatch.ElapsedMilliseconds));
stopWatch.Reset();
//#2 Using a new array and Array.Copy
stopWatch.Start();
byte[] extendedByteArray2 = new byte[byteArray.Length + 1];
extendedByteArray2[0] = 0x00;
Array.Copy(byteArray, 0, extendedByteArray2, 1, byteArray.Length);
stopWatch.Stop();
Console.WriteLine(string.Format("#2: {0} ms", stopWatch.ElapsedMilliseconds));
stopWatch.Reset();
//#3 Using a List
stopWatch.Start();
List<byte> byteList = new List<byte>();
byteList.AddRange(byteArray);
byteList.Insert(0, 0x00);
byte[] extendedByteArray3 = byteList.ToArray();
stopWatch.Stop();
Console.WriteLine(string.Format("#3: {0} ms", stopWatch.ElapsedMilliseconds));
stopWatch.Reset();
Console.ReadLine();
And the results are:
#1: 9 ms
#2: 1 ms
#3: 6 ms
I've run it multiple times and I got different numbers, but the proportion is always the same: #2 is always the most efficient choice.
My conclusion: arrays are more efficient then Lists (although they provide less functionality), and somehow Array.Copy is really optmized (would like to understand that, though).
Any feedback will be appreciated.
Best regards.
PS: this is not a swordfight post, we are at a Q&A site to learn and teach. And learn.

The easiest and cleanest way for .NET 4.7.1 and above is to use the side-effect free Prepend().
Adds a value to the beginning of the sequence.
Example
// Creating an array of numbers
var numbers = new[] { 1, 2, 3 };
// Trying to prepend any value of the same type
var results = numbers.Prepend(0);
// output is 0, 1, 2, 3
Console.WriteLine(string.Join(", ", results ));

As you surmised, the fastest way to do this is to create new array of length + 1 and copy all the old values over.
If you are going to be doing this many times, then I suggest using a List<byte> instead of byte[], as the cost of reallocating and copying while growing the underlying storage is amortized more effectively; in the usual case, the underlying vector in the List is grown by a factor of two each time an addition or insertion is made to the List that would exceed its current capacity.
...
byte[] newValues = new byte[values.Length + 1];
newValues[0] = 0x00; // set the prepended value
Array.Copy(values, 0, newValues, 1, values.Length); // copy the old values

When I need to append data frequently but also want O(1) random access to individual elements, I'll use an array that is over allocated by some amount of padding for quickly adding new values. This means you need to store the actual content length in another variable, as the array.length will indicate the length + the padding. A new value gets appended by using one slot of the padding, no allocation & copy are necessary until you run out of padding. In effect, allocation is amortized over several append operations. There are speed space trade offs, as if you have many of these data structures you could have a fair amount of padding in use at any one time in the program.
This same technique can be used in prepending. Just as with appending, you can introduce an interface or abstraction between the users and the implementation: you can have several slots of padding so that new memory allocation is only necessary occasionally. As some above suggested, you can also implement a prepending interface with an appending data structure that reverses the indexes.
I'd package the data structure as an implementation of some generic collection interface, so that the interface appears fairly normal to the user (such as an array list or something).
(Also, if removal is supported, it's probably useful to clear elements as soon as they are removed to help reduce gc load.)
The main point is to consider the implementation and the interface separately, as decoupling them gives you the flexibility to choose varied implementations or to hide implementation details using a minimal interface.
There are many other data structures you could use depending on the applicability to your domain. Ropes or Gap Buffer; see What is best data structure suitable to implement editor like notepad?; Trie's do some useful things, too.

I know this is a VERY old post but I actually like using lambda. Sure my code may NOT be the most efficient way but its readable and in one line. I use a combination of .Concat and ArraySegment.
string[] originalStringArray = new string[] { "1", "2", "3", "5", "6" };
int firstElementZero = 0;
int insertAtPositionZeroBased = 3;
string stringToPrepend = "0";
string stringToInsert = "FOUR"; // Deliberate !!!
originalStringArray = new string[] { stringToPrepend }
.Concat(originalStringArray).ToArray();
insertAtPositionZeroBased += 1; // BECAUSE we prepended !!
originalStringArray = new ArraySegment<string>(originalStringArray, firstElementZero, insertAtPositionZeroBased)
.Concat(new string[] { stringToInsert })
.Concat(new ArraySegment<string>(originalStringArray, insertAtPositionZeroBased, originalStringArray.Length - insertAtPositionZeroBased)).ToArray();

The best choice depends on what you're going to be doing with this collection later on down the line. If that's the only length-changing edit that will ever be made, then your best bet is to create a new array with one additional slot and use Array.Copy() to do the rest. No need to initialize the first value, since new C# arrays are always zeroed out:
byte[] PrependWithZero(byte[] input)
{
var newArray = new byte[input.Length + 1];
Array.Copy(input, 0, newArray, 1, input.Length);
return newArray;
}
If there are going to be other length-changing edits that might happen, the most performant option might be to use a List<byte> all along, as long as the additions aren't always to the beginning. (If that's the case, even a linked list might not be an option that you can dismiss out of hand.):
var list = new List<byte>(input);
list.Insert(0, 0);

I am aware this is over 4-year-old accepted post, but for those who this might be relevant Buffer.BlockCopy would be faster.

Array of shortened integers

Just to avoid inventing hot-water, I am asking here...
I have an application with lots of arrays, and it is running out of memory.
So the thought is to compress the List<int> to something else, that would have same interface (IList<T> for example), but instead of int I could use shorter integers.
For example, if my value range is 0 - 100.000.000 I need only ln2(1000000) = 20 bits. So instead of storing 32 bits, I can trim the excess and reduce memory requirements by 12/32 = 37.5%.
Do you know of an implementation of such array. c++ and java would be also OK, since I could easily convert them to c#.
Additional requirements (since everyone is starting to getting me OUT of the idea):
integers in the list ARE unique
they have no special property so they aren't compressible in any other way then reducing the bit count
if the value range is one million for example, lists would be from 2 to 1000 elements in size, but there will be plenty of them, so no BitSets
new data container should behave like re-sizable array (regarding method O()-ness)
EDIT:
Please don't tell me NOT to do it. The requirement for this is well thought-over, and it is the ONLY option that is left.
Also, 1M of value range and 20 bit for it is ONLY AN EXAMPLE. I have cases with all different ranges and integer sizes.
Also, I could have even shorter integers, for example 7 bit integers, then packing would be
00000001
11111122
22222333
33334444
444.....
for first 4 elements, packed into 5 bytes.
Almost done coding it - will be posted soon...

Since you can only allocate memory in byte quantums, you are essentially asking if/how you can fit the integers in 3 bytes instead of 4 (but see #3 below). This is not a good idea.
Since there is no 3-byte sized integer type, you would need to use something else (e.g. an opaque 3-byte buffer) in its place. This would require that you wrap all access to the contents of the list in code that performs the conversion so that you can still put "ints" in and pull "ints" out.
Depending on both the architecture and the memory allocator, requesting 3-byte chunks might not affect the memory footprint of your program at all (it might simply litter your heap with unusable 1-byte "holes").
Reimplementing the list from scratch to work with an opaque byte array as its backing store would avoid the two previous issues (and it can also let you squeeze every last bit of memory instead of just whole bytes), but it's a tall order and quite prone to error.
You might want instead to try something like:
Not keeping all this data in memory at the same time. At 4 bytes per int, you 'd need to reach hundreds of millions of integers before memory runs out. Why do you need all of them at the same time?
Compressing the dataset by not storing duplicates if possible. There are bound to be a few of them if you are up to hundreds of millions.
Changing your data structure so that it stores differences between successive values (deltas), if that is possible. This might be not very hard to achieve, but you can only realistically expect something at the ballpark of 50% improvement (which may not be enough) and it will totally destroy your ability to index into the list in constant time.

One option that will get your from 32 bits to 24bits is to create a custom struct that stores an integer inside of 3 bytes:
public struct Entry {
byte b1; // low
byte b2; // middle
byte b3; // high
public void Set(int x) {
b1 = (byte)x;
b2 = (byte)(x >> 8);
b3 = (byte)(x >> 16);
}
public int Get() {
return (b3 << 16) | (b2 << 8) | b1;
}
}
You can then just create a List<Entry>.
var list = new List<Entry>();
var e = new Entry();
e.Set(12312);
list.Add(e);
Console.WriteLine(list[0].Get()); // outputs 12312

This reminds me of base64 and similar kinds of binary-to-text encoding.
They take 8 bit bytes and do a bunch of bit-fiddling to pack them into 4-, 5-, or 6-bit printable characters.
This also reminds me of the Zork Standard Code for Information Interchange (ZSCII), which packs 3 letters into 2 bytes, where each letter occupies 5 bits.
It sounds like you want to taking a bunch of 10- or 20-bit integers and pack them into a buffer of 8-bit bytes.
The source code is available for many libraries that handle a packed array of single bits
(a
b
c
d
e).
Perhaps you could
(a) download that source code and modify the source (starting from some BitArray or other packed encoding), recompiling to create a new library that handles packing and unpacking 10- or 20-bit integers rather than single bits.
It may take less programming and testing time to
(b) write a library that, from the outside, appears to act just like (a), but internally it breaks up 20-bit integers into 20 separate bits, then stores them using an (unmodified) BitArray class.

Edit: Given that your integers are unique you could do the following: store unique integers until the number of integers you're storing is half the maximum number. Then switch to storing the integers you don't have. This will reduce the storage space by 50%.
Might be worth exploring other simplification techniques before trying to use 20-bit ints.
How do you treat duplicate integers? If have lots of duplicates you could reduce the storage size by storing the integers in a Dictionary<int, int> where keys are unique integers and values are corresponding counts. Note this assumes you don't care about the order of your integers.
Are your integers all unique? Perhaps you're storing lots of unique integers in the range 0 to 100 mil. In this case you could try storing the integers you don't have. Then when determining if you have an integer i just ask if it's not in your collection.