We Keep Coding

Fastest way to copy data from ReadOnlySpan to output with pixel conversion

I'm having performance issue when I copy the data from input (ReadOnlySpan) to output (Span) using (Loops like 'for')
there is Span.CopyTo, it's perfect and very fast
but for now it's useless without converting the pixels.
below is the code, I have feeling that there is some short way to do that instead of the current process:
public unsafe void UpdateFromOutput(CanvasDevice device, ReadOnlySpan<byte> data, uint width, uint height, uint pitch)
{
using (var renderTargetMap = new BitmapMap(device, RenderTarget))
{
var inputPitch = (int)pitch;
var mapPitch = (int)renderTargetMap.PitchBytes;
var mapData = new Span<byte>(new IntPtr(renderTargetMap.Data).ToPointer(), (int)RenderTarget.Size.Height * mapPitch);
switch (CurrentPixelFormat)
{
case PixelFormats.RGB0555:
FramebufferConverter.ConvertFrameBufferRGB0555ToXRGB8888(width, height, data, inputPitch, mapData, mapPitch);
break;
case PixelFormats.RGB565:
FramebufferConverter.ConvertFrameBufferRGB565ToXRGB8888(width, height, data, inputPitch, mapData, mapPitch);
break;
}
}
}
then inside function like ConvertFrameBufferRGB0555ToXRGB8888
I will go through width and height like below:
var castInput = MemoryMarshal.Cast<byte, ushort>(input);
var castInputPitch = inputPitch / sizeof(ushort);
var castOutput = MemoryMarshal.Cast<byte, uint>(output);
var castOutputPitch = outputPitch / sizeof(uint);
castOutput.Fill(0);
for (var i = 0; i < height;i++)
{
var inputLine = castInput.Slice(i * castInputPitch, castInputPitch);
var outputLine = castOutput.Slice(i * castOutputPitch, castOutputPitch);
for (var j = 0; j < width;j++)
{
outputLine[j] = ConverToRGB888(inputLine[j]);
}
}
The code above working but slow in some cases.
Please note: I'm modifying a project so the code above was written by the original developer and I need help because I don't understand how the process is working, still very confused.. specially in the Slice part.
Tried as test only to copy the input to output directly data.CopyTo(mapData); and I got this (as expected):
Hope there is some solution with Marshal and Span functions
Many thanks.
Update regarding (ConverToRGB888)
As for ConverToRGB888, the original code contains RGB565LookupTable:
private const uint LookupTableSize = ushort.MaxValue + 1;
private static uint[] RGB565LookupTable = new uint[LookupTableSize];
public static void SetRGB0565LookupTable()
{
uint r565, g565, b565;
double red = 255.0;
double green = 255.0;
double blue = 255.0;
for (uint i = 0; i < LookupTableSize; i++)
{
//RGB565
r565 = (i >> 11) & 0x1F;
g565 = (i >> 5) & 0x3F;
b565 = (i & 0x1F);
r565 = (uint)Math.Round(r565 * red / 31.0);
g565 = (uint)Math.Round(g565 * green / 63.0);
b565 = (uint)Math.Round(b565 * blue / 31.0);
RGB565LookupTable[i] = (0xFF000000 | r565 << 16 | g565 << 8 | b565);
}
}
private static uint ConverToRGB888(ushort x)
{
return RGB565LookupTable[x];
}
SetRGB0565LookupTable() will be called only once to fill the values.
Conclusion:
The Fill(0) was not important and it was causing delay
The unsafe version (accepted answer) was clear for me and a bit faster
Avoiding For partially even faster like Here [Tested]
Pre-Lookup table is very helpful and made the conversion faster
Memory helpers like Span.CopyTo, Buffer.MemoryCopy source available Here
Using Parallel.For is faster in some cases with help of UnsafeMemory
If you have input with (Pixels Type) supported by Win2D there is possibility to avoid loops like:
byte[] dataBytes = new byte[data.Length];
fixed (byte* inputPointer = &data[0])
Marshal.Copy((IntPtr)inputPointer, dataBytes, 0, data.Length);
RenderTarget = CanvasBitmap.CreateFromBytes(renderPanel, dataBytes, (int)width, (int)height, DirectXPixelFormat.R8G8UIntNormalized, 92, CanvasAlphaMode.Ignore);
but not sure from the last point as I wasn't able to test on 565,555.
Thanks for DekuDesu the explanation and simplified version he provide helped me to do more tests.

I'm having performance issue when I copy the data from input (ReadOnlySpan) to output (Span) using (Loops like 'for')
The the code you provided is already pretty safe and has the best complexity you're going get for pixel-by-pixel operations. The presence of nested for loops does not necessarily correspond to performance issues or increased complexity.
I need help because I don't understand how the process is working, still very confused.. specially in the Slice part.
This code looks like it's meant to convert one bitmap format to another. Bitmaps come in varying sizes and formats. Because of this they include an additional piece of information along with width and height, pitch.
Pitch is the distance in bytes between two lines of pixel information, this is used to account for formats that don't include full 32/64bit color information.
Knowing this I commented the method in question as to help explain what it's doing.
public static void ConvertFrameBufferRGB565ToXRGB8888(uint width, uint height, ReadOnlySpan<byte> input, int inputPitch, Span<byte> output, int outputPitch)
{
// convert the span of bytes into a span of ushorts
// so we can use span[i] to get a ushort
var castInput = MemoryMarshal.Cast<byte, ushort>(input);
// pitch is the number of bytes between the first byte of a line and the first byte of the next line
// convert the pitch from bytes into ushort pitch
var castInputPitch = inputPitch / sizeof(ushort);
// convert the span of bytes into a span of ushorts
// so we can use span[i] to get a ushort
var castOutput = MemoryMarshal.Cast<byte, uint>(output);
var castOutputPitch = outputPitch / sizeof(uint);
for (var i = 0; i < height; i++)
{
// get a line from the input
// remember that pitch is the number of ushorts between lines
// so i * pitch here gives us the index of the i'th line, and we don't need the padding
// ushorts at the end so we only take castInputPitch number of ushorts
var inputLine = castInput.Slice(i * castInputPitch, castInputPitch);
// same thing as above but for the output
var outputLine = castOutput.Slice(i * castOutputPitch, castOutputPitch);
for (var j = 0; j < width; j++)
{
// iterate through the line, converting each pixel and storing it in the output span
outputLine[j] = ConverToRGB888(inputLine[j]);
}
}
}
Fastest way to copy data from ReadOnlySpan to output with pixel conversion
Honestly the method you provided is just fine, it's safe and fast ish. Keep in mind that copying data like bitmaps linearly on CPU's is an inherently slow process. The most performance savings you could hope for is avoiding copying data redundantly. Unless this needs absolutely blazing speed I would not recommend changes other than removing .fill(0) since it's probably unnecessary, but you would have to test that.
If you ABSOLUTELY must get more performance out of this you may want to consider something like what I've provided below. I caution you however, unsafe code like this is well.. unsafe and is prone to errors. It has almost no error checking and makes a LOT of assumptions, so that's up for you to implement.
If it's still not fast enough consider writing a .dll in C and use interop maybe.
public static unsafe void ConvertExtremelyUnsafe(ulong height, ref byte inputArray, ulong inputLength, ulong inputPitch, ref byte outputArray, ulong outputLength, ulong outputPitch)
{
// pin down pointers so they dont move on the heap
fixed (byte* inputPointer = &inputArray, outputPointer = &outputArray)
{
// since we have to account for padding we should go line by line
for (ulong y = 0; y < height; y++)
{
// get a pointer for the first byte of the line of the input
byte* inputLinePointer = inputPointer + (y * inputPitch);
// get a pointer for the first byte of the line of the output
byte* outputLinePointer = outputPointer + (y * outputPitch);
// traverse the input line by ushorts
for (ulong i = 0; i < (inputPitch / sizeof(ushort)); i++)
{
// calculate the offset for the i'th ushort,
// becuase we loop based on the input and ushort we dont need an index check here
ulong inputOffset = i * sizeof(ushort);
// get a pointer to the i'th ushort
ushort* rgb565Pointer = (ushort*)(inputLinePointer + inputOffset);
ushort rgb565Value = *rgb565Pointer;
// convert the rgb to the other format
uint rgb888Value = ConverToRGB888(rgb565Value);
// calculate the offset for i'th uint
ulong outputOffset = i * sizeof(uint);
// at least attempt to avoid overflowing a buffer, not that the runtime would let you do that, i would hope..
if (outputOffset >= outputLength)
{
throw new IndexOutOfRangeException($"{nameof(outputArray)}[{outputOffset}]");
}
// get a pointer to the i'th uint
uint* rgb888Pointer = (uint*)(outputLinePointer + outputOffset);
// write the bytes of the rgb888 to the output array
*rgb888Pointer = rgb888Value;
}
}
}
}
disclaimer: I wrote this on mobile

C# repeat data, reset on boundary

I'm trying to write some repeated data into a byte array, it looks like this:
byte[] bytes = Encoding.ASCII.GetBytes("UNKNOWN");
int count = 0;
for (int i = 0; i < several_MB_worth_of_bytes; i++)
{
output[i] = bytes[count];
count++;
if (count >= bytes.Length) count = 0;
}
This works, however, I need to reset the count variable if I've written exactly 1MB worth of bytes, so the next byte after the 1MB boundary will the the first 'U' in the string. This needs to happen on every MB boundary.
I can't seem to quite figure out the best way of handling the reset. I've taken a look at the ByteSize library for .net, but I'm still not sure how that's going to help me.

You could check whether i is a multiple of 1 MB inside the same if statement where you are resetting the counter:
if (count >= bytes.Length || i % (1024*1024) == 0) count = 0;

Convert 2 successive Bytes to one int value Increase speed in C#

I need to combine two Bytes into one int value.
I receive from my camera a 16bit Image were two successive bytes have the intensity value of one pixel. My goal is to combine these two bytes into one "int" vale.
I manage to do this using the following code:
for (int i = 0; i < VectorLength * 2; i = i + 2)
{
NewImageVector[ImagePointer] = ((int)(buffer.Array[i + 1]) << 8) | ((int)(buffer.Array[i]));
ImagePointer++;
}
My image is 1280*960 so VectorLength==1228800 and the incomming buffer size is 2*1228800=2457600 elements...
Is there any way that I can speed this up?
Maybe there is another way so I don't need to use a for-loop.
Thank you

You could use the equivalent to the union of c. Im not sure if faster, but more elegant:
[StructLayout(LayoutKind.Explicit)]
struct byte_array
{
[FieldOffset(0)]
public byte byte1;
[FieldOffset(1)]
public byte byte2;
[FieldOffset(0)]
public short int0;
}
use it like this:
byte_array ba = new byte_array();
//insert the two bytes
ba.byte1 = (byte)(buffer.Array[i]);
ba.byte2 = (byte)(buffer.Array[i + 1]);
//get the integer
NewImageVector[ImagePointer] = ba.int1;
You can fill your two bytes and use the int. To find the faster way take the StopWatch-Class and compare the two ways like this:
Stopwatch stopWatch = new Stopwatch();
stopWatch.Start();
//The code
stopWatch.Stop();
MessageBox.Show(stopWatch.ElapsedTicks.ToString()); //Or milliseconds ,...

Assuming you can (re-)define NewImageVector as a short[], and every two consecutive bytes in Buffer should be transformed into a short (which basically what you're doing now, only you cast to an int afterwards), you can use Buffer.BlockCopy to do it for you.
As the documentation tells, you Buffer.BlockCopy copies bytes from one array to another, so in order to copy your bytes in buffer you need to do the following:
Buffer.BlockCopy(Buffer, 0, NewImageVector, 0, [NumberOfExpectedShorts] * 2)
This tells BlockCopy that you want to start copying bytes from Buffer, starting at index 0, to NewImageVector starting at index 0, and you want to copy [NumberOfExpectedShorts] * 2 bytes (since every short is two bytes long).
No loops, but it does depend on the ability of using a short[] array instead of an int[] array (and indeed, on using an array to begin with).
Note that this also requires the bytes in Buffer to be in little-endian order (i.e. Buffer[index] contains the low byte, buffer[index + 1] the high byte).

You can achieve a small performance increase by using unsafe pointers to iterate the arrays. The following code assumes that source is the input byte array (buffer.Array in your case). It also assumes that source has an even number of elements. In production code you would obviously have to check these things.
int[] output = new int[source.Length / 2];
fixed (byte* pSource = source)
fixed (int* pDestination = output)
{
byte* sourceIterator = pSource;
int* destIterator = pDestination;
for (int i = 0; i < output.Length; i++)
{
(*destIterator) = ((*sourceIterator) | (*(sourceIterator + 1) << 8));
destIterator++;
sourceIterator += 2;
}
}
return output;

Byte/char buffer to long and/or double

In my code I need to convert string representation of integers to long and double values.
String representation is a byte array (byte[]). For example, for a number 12345 string representation is { 49, 50, 51, 52, 53 }
Currently, I use following obvious code for conversion to long (and almost the same code for conversion to double)
private long bytesToIntValue()
{
string s = System.Text.Encoding.GetEncoding("Latin1").GetString(bytes);
return long.Parse(s, CultureInfo.InvariantCulture);
}
This code works as expected, but in my case I want something better. It's because currently I must convert bytes to string first.
In my case, bytesToIntValue() gets called about 12 million times and about 25% of all memory allocations are made in this method.
Sure, I want to optimize this part. I want to perform conversions without intermediate string (+ speed, - allocation).
What would you recommend? How can I perform conversions without intermediate strings? Is there a faster method to perform conversions?
EDIT:
Byte arrays I am dealing with are always contain ASCII-encoded data. Numbers can be negative. For double values exponential format is allowed. Hexadecimal integers are not allowed.

How can I perform conversions without intermediate strings?
Well you can easily convert each byte to a char. For example - untested:
private static long ConvertAsciiBytesToInt32(byte[] bytes)
{
long value = 0;
foreach (byte b in bytes)
{
value *= 10L;
char c = b; // Implicit conversion; effectively ISO-8859-1
if (c < '0' || c > '9')
{
throw new ArgumentException("Bytes contains non-digit: " + c);
}
value += (c - '0');
}
return value;
}
Note that this really does assume it's ASCII (or compatible) - if your byte array is actually UTF-16 (for example) then it will definitely do the wrong thing.
Also note that this doesn't perform any sort of length validation or overflow checking... and it doesn't cope with negative numbers. You could add all of these if you want, but we don't know enough about your requirements to know if it's worth adding the complexity.

I'm not sure that there is a easy way to do that,
Please note that it won't work with other encodings, The test shown on my computer that this is only 3 times faster (I don't think it worth it).
The code + test :
class MainClass
{
public static void Main(string[] args)
{
string str = "12341234";
byte[] buffer = Encoding.ASCII.GetBytes(str);
Stopwatch sw = Stopwatch.StartNew();
for(int i = 0; i < 1000000 ;i ++)
{
long val = BufferToLong.GetValue(buffer);
}
Console.WriteLine (sw.ElapsedMilliseconds);
sw.Restart();
for (int i = 0 ; i < 1000000 ; i++)
{
string valStr = Encoding.ASCII.GetString(buffer);
long val = long.Parse(valStr);
}
Console.WriteLine (sw.ElapsedMilliseconds);
}
}
static class BufferToLong
{
public static long GetValue(Byte[] buffer) {
long number = 0;
foreach (byte currentByte in buffer) {
char currentChar = (char)currentByte;
int currentDigit = currentChar - '0';
number *= 10 ;
number += currentDigit;
}
return number;
}
}

In the end, I created C# version of strol function. This function comes with CRT and source code of CRT comes with Visual Studio.
The resulting method is almost the same as code provided by #Jon Skeet in his answer but also contains some checks for overflow.
In my case all the changes proved to be very useful in terms of speed and memory.

How to set each bit in a byte array

How do I set each bit in the following byte array which has 21 bytes or 168 bits to either zero or one?
byte[] logonHours
Thank you very much

Well, to clear every bit to zero you can just use Array.Clear:
Array.Clear(logonHours, 0, logonHours.Length);
Setting each bit is slightly harder:
for (int i = 0; i < logonHours.Length; i++)
{
logonHours[i] = 0xff;
}
If you find yourself filling an array often, you could write an extension method:
public static void FillArray<T>(this T[] array, T value)
{
// TODO: Validation
for (int i = 0; i < array.Length; i++)
{
array[i] = value;
}
}

BitArray.SetAll:
System.Collections.BitArray a = new System.Collections.BitArray(logonHours);
a.SetAll(true);
Note that this copies the data from the byte array. It's not just a wrapper around it.

This may be more than you need, but ...
Usually when dealing with individual bits in any data type, I define a const for each bit position, then use the binary operators |, &, and ^.
i.e.
const byte bit1 = 1;
const byte bit2 = 2;
const byte bit3 = 4;
const byte bit4 = 8;
.
.
const byte bit8 = 128;
Then you can turn whatever bits you want on and off using the bit operations.
byte byTest = 0;
byTest = byTest | bit4;
would turn bit 4 on but leave the rest untouched.
You would use the & and ^ to turn them off or do more complex exercises.
Obviously, since you only want to turn all bits up or down then you can just set the byte to 0 or 255. That would turn them all off or on.