We Keep Coding

Fastest way to copy data from ReadOnlySpan to output with pixel conversion

I'm having performance issue when I copy the data from input (ReadOnlySpan) to output (Span) using (Loops like 'for')
there is Span.CopyTo, it's perfect and very fast
but for now it's useless without converting the pixels.
below is the code, I have feeling that there is some short way to do that instead of the current process:
public unsafe void UpdateFromOutput(CanvasDevice device, ReadOnlySpan<byte> data, uint width, uint height, uint pitch)
{
using (var renderTargetMap = new BitmapMap(device, RenderTarget))
{
var inputPitch = (int)pitch;
var mapPitch = (int)renderTargetMap.PitchBytes;
var mapData = new Span<byte>(new IntPtr(renderTargetMap.Data).ToPointer(), (int)RenderTarget.Size.Height * mapPitch);
switch (CurrentPixelFormat)
{
case PixelFormats.RGB0555:
FramebufferConverter.ConvertFrameBufferRGB0555ToXRGB8888(width, height, data, inputPitch, mapData, mapPitch);
break;
case PixelFormats.RGB565:
FramebufferConverter.ConvertFrameBufferRGB565ToXRGB8888(width, height, data, inputPitch, mapData, mapPitch);
break;
}
}
}
then inside function like ConvertFrameBufferRGB0555ToXRGB8888
I will go through width and height like below:
var castInput = MemoryMarshal.Cast<byte, ushort>(input);
var castInputPitch = inputPitch / sizeof(ushort);
var castOutput = MemoryMarshal.Cast<byte, uint>(output);
var castOutputPitch = outputPitch / sizeof(uint);
castOutput.Fill(0);
for (var i = 0; i < height;i++)
{
var inputLine = castInput.Slice(i * castInputPitch, castInputPitch);
var outputLine = castOutput.Slice(i * castOutputPitch, castOutputPitch);
for (var j = 0; j < width;j++)
{
outputLine[j] = ConverToRGB888(inputLine[j]);
}
}
The code above working but slow in some cases.
Please note: I'm modifying a project so the code above was written by the original developer and I need help because I don't understand how the process is working, still very confused.. specially in the Slice part.
Tried as test only to copy the input to output directly data.CopyTo(mapData); and I got this (as expected):
Hope there is some solution with Marshal and Span functions
Many thanks.
Update regarding (ConverToRGB888)
As for ConverToRGB888, the original code contains RGB565LookupTable:
private const uint LookupTableSize = ushort.MaxValue + 1;
private static uint[] RGB565LookupTable = new uint[LookupTableSize];
public static void SetRGB0565LookupTable()
{
uint r565, g565, b565;
double red = 255.0;
double green = 255.0;
double blue = 255.0;
for (uint i = 0; i < LookupTableSize; i++)
{
//RGB565
r565 = (i >> 11) & 0x1F;
g565 = (i >> 5) & 0x3F;
b565 = (i & 0x1F);
r565 = (uint)Math.Round(r565 * red / 31.0);
g565 = (uint)Math.Round(g565 * green / 63.0);
b565 = (uint)Math.Round(b565 * blue / 31.0);
RGB565LookupTable[i] = (0xFF000000 | r565 << 16 | g565 << 8 | b565);
}
}
private static uint ConverToRGB888(ushort x)
{
return RGB565LookupTable[x];
}
SetRGB0565LookupTable() will be called only once to fill the values.
Conclusion:
The Fill(0) was not important and it was causing delay
The unsafe version (accepted answer) was clear for me and a bit faster
Avoiding For partially even faster like Here [Tested]
Pre-Lookup table is very helpful and made the conversion faster
Memory helpers like Span.CopyTo, Buffer.MemoryCopy source available Here
Using Parallel.For is faster in some cases with help of UnsafeMemory
If you have input with (Pixels Type) supported by Win2D there is possibility to avoid loops like:
byte[] dataBytes = new byte[data.Length];
fixed (byte* inputPointer = &data[0])
Marshal.Copy((IntPtr)inputPointer, dataBytes, 0, data.Length);
RenderTarget = CanvasBitmap.CreateFromBytes(renderPanel, dataBytes, (int)width, (int)height, DirectXPixelFormat.R8G8UIntNormalized, 92, CanvasAlphaMode.Ignore);
but not sure from the last point as I wasn't able to test on 565,555.
Thanks for DekuDesu the explanation and simplified version he provide helped me to do more tests.

I'm having performance issue when I copy the data from input (ReadOnlySpan) to output (Span) using (Loops like 'for')
The the code you provided is already pretty safe and has the best complexity you're going get for pixel-by-pixel operations. The presence of nested for loops does not necessarily correspond to performance issues or increased complexity.
I need help because I don't understand how the process is working, still very confused.. specially in the Slice part.
This code looks like it's meant to convert one bitmap format to another. Bitmaps come in varying sizes and formats. Because of this they include an additional piece of information along with width and height, pitch.
Pitch is the distance in bytes between two lines of pixel information, this is used to account for formats that don't include full 32/64bit color information.
Knowing this I commented the method in question as to help explain what it's doing.
public static void ConvertFrameBufferRGB565ToXRGB8888(uint width, uint height, ReadOnlySpan<byte> input, int inputPitch, Span<byte> output, int outputPitch)
{
// convert the span of bytes into a span of ushorts
// so we can use span[i] to get a ushort
var castInput = MemoryMarshal.Cast<byte, ushort>(input);
// pitch is the number of bytes between the first byte of a line and the first byte of the next line
// convert the pitch from bytes into ushort pitch
var castInputPitch = inputPitch / sizeof(ushort);
// convert the span of bytes into a span of ushorts
// so we can use span[i] to get a ushort
var castOutput = MemoryMarshal.Cast<byte, uint>(output);
var castOutputPitch = outputPitch / sizeof(uint);
for (var i = 0; i < height; i++)
{
// get a line from the input
// remember that pitch is the number of ushorts between lines
// so i * pitch here gives us the index of the i'th line, and we don't need the padding
// ushorts at the end so we only take castInputPitch number of ushorts
var inputLine = castInput.Slice(i * castInputPitch, castInputPitch);
// same thing as above but for the output
var outputLine = castOutput.Slice(i * castOutputPitch, castOutputPitch);
for (var j = 0; j < width; j++)
{
// iterate through the line, converting each pixel and storing it in the output span
outputLine[j] = ConverToRGB888(inputLine[j]);
}
}
}
Fastest way to copy data from ReadOnlySpan to output with pixel conversion
Honestly the method you provided is just fine, it's safe and fast ish. Keep in mind that copying data like bitmaps linearly on CPU's is an inherently slow process. The most performance savings you could hope for is avoiding copying data redundantly. Unless this needs absolutely blazing speed I would not recommend changes other than removing .fill(0) since it's probably unnecessary, but you would have to test that.
If you ABSOLUTELY must get more performance out of this you may want to consider something like what I've provided below. I caution you however, unsafe code like this is well.. unsafe and is prone to errors. It has almost no error checking and makes a LOT of assumptions, so that's up for you to implement.
If it's still not fast enough consider writing a .dll in C and use interop maybe.
public static unsafe void ConvertExtremelyUnsafe(ulong height, ref byte inputArray, ulong inputLength, ulong inputPitch, ref byte outputArray, ulong outputLength, ulong outputPitch)
{
// pin down pointers so they dont move on the heap
fixed (byte* inputPointer = &inputArray, outputPointer = &outputArray)
{
// since we have to account for padding we should go line by line
for (ulong y = 0; y < height; y++)
{
// get a pointer for the first byte of the line of the input
byte* inputLinePointer = inputPointer + (y * inputPitch);
// get a pointer for the first byte of the line of the output
byte* outputLinePointer = outputPointer + (y * outputPitch);
// traverse the input line by ushorts
for (ulong i = 0; i < (inputPitch / sizeof(ushort)); i++)
{
// calculate the offset for the i'th ushort,
// becuase we loop based on the input and ushort we dont need an index check here
ulong inputOffset = i * sizeof(ushort);
// get a pointer to the i'th ushort
ushort* rgb565Pointer = (ushort*)(inputLinePointer + inputOffset);
ushort rgb565Value = *rgb565Pointer;
// convert the rgb to the other format
uint rgb888Value = ConverToRGB888(rgb565Value);
// calculate the offset for i'th uint
ulong outputOffset = i * sizeof(uint);
// at least attempt to avoid overflowing a buffer, not that the runtime would let you do that, i would hope..
if (outputOffset >= outputLength)
{
throw new IndexOutOfRangeException($"{nameof(outputArray)}[{outputOffset}]");
}
// get a pointer to the i'th uint
uint* rgb888Pointer = (uint*)(outputLinePointer + outputOffset);
// write the bytes of the rgb888 to the output array
*rgb888Pointer = rgb888Value;
}
}
}
}
disclaimer: I wrote this on mobile

Defining a bit[] array in C#

currently im working on a solution for a prime-number calculator/checker. The algorythm is already working and verry efficient (0,359 seconds for the first 9012330 primes). Here is a part of the upper region where everything is declared:
const uint anz = 50000000;
uint a = 3, b = 4, c = 3, d = 13, e = 12, f = 13, g = 28, h = 32;
bool[,] prim = new bool[8, anz / 10];
uint max = 3 * (uint)(anz / (Math.Log(anz) - 1.08366));
uint[] p = new uint[max];
Now I wanted to go to the next level and use ulong's instead of uint's to cover a larger area (you can see that already), where i tapped into my problem: the bool-array.
Like everybody should know, bool's have the length of a byte what takes a lot of memory when creating the array... So I'm searching for a more resource-friendly way to do that.
My first idea was a bit-array -> not byte! <- to save the bool's, but haven't figured out how to do that by now. So if someone ever did something like this, I would appreciate any kind of tips and solutions. Thanks in advance :)

You can use BitArray collection:
http://msdn.microsoft.com/en-us/library/system.collections.bitarray(v=vs.110).aspx
MSDN Description:
Manages a compact array of bit values, which are represented as Booleans, where true indicates that the bit is on (1) and false indicates the bit is off (0).

You can (and should) use well tested and well known libraries.
But if you're looking to learn something (as it seems to be the case) you can do it yourself.
Another reason you may want to use a custom bit array is to use the hard drive to store the array, which comes in handy when calculating primes. To do this you'd need to further split addr, for example lowest 3 bits for the mask, next 28 bits for 256MB of in-memory storage, and from there on - a file name for a buffer file.
Yet another reason for custom bit array is to compress the memory use when specifically searching for primes. After all more than half of your bits will be 'false' because the numbers corresponding to them would be even, so in fact you can both speed up your calculation AND reduce memory requirements if you don't even store the even bits. You can do that by changing the way addr is interpreted. Further more you can also exclude numbers divisible by 3 (only 2 out of every 6 numbers has a chance of being prime) thus reducing memory requirements by 60% compared to plain bit array.
Notice the use of shift and logical operators to make the code a bit more efficient.
byte mask = (byte)(1 << (int)(addr & 7)); for example can be written as
byte mask = (byte)(1 << (int)(addr % 8));
and addr >> 3 can be written as addr / 8
Testing shift/logical operators vs division shows 2.6s vs 4.8s in favor of shift/logical for 200000000 operations.
Here's the code:
void Main()
{
var barr = new BitArray(10);
barr[4] = true;
Console.WriteLine("Is it "+barr[4]);
Console.WriteLine("Is it Not "+barr[5]);
}
public class BitArray{
private readonly byte[] _buffer;
public bool this[long addr]{
get{
byte mask = (byte)(1 << (int)(addr & 7));
byte val = _buffer[(int)(addr >> 3)];
bool bit = (val & mask) == mask;
return bit;
}
set{
byte mask = (byte) ((value ? 1:0) << (int)(addr & 7));
int offs = (int)addr >> 3;
_buffer[offs] = (byte)(_buffer[offs] | mask);
}
}
public BitArray(long size){
_buffer = new byte[size/8 + 1]; // define a byte buffer sized to hold 8 bools per byte. The spare +1 is to avoid dealing with rounding.
}
}

How do I properly loop through and print bits of an Int, Long, Float, or BigInteger?

I'm trying to debug some bit shifting operations and I need to visualize the bits as they exist before and after a Bit-Shifting operation.
I read from this answer that I may need to handle backfill from the shifting, but I'm not sure what that means.
I think that by asking this question (how do I print the bits in a int) I can figure out what the backfill is, and perhaps some other questions I have.
Here is my sample code so far.
static string GetBits(int num)
{
StringBuilder sb = new StringBuilder();
uint bits = (uint)num;
while (bits!=0)
{
bits >>= 1;
isBitSet = // somehow do an | operation on the first bit.
// I'm unsure if it's possible to handle different data types here
// or if unsafe code and a PTR is needed
if (isBitSet)
sb.Append("1");
else
sb.Append("0");
}
}

Convert.ToString(56,2).PadLeft(8,'0') returns "00111000"
This is for a byte, works for int also, just increase the numbers

To test if the last bit is set you could use:
isBitSet = ((bits & 1) == 1);
But you should do so before shifting right (not after), otherwise you's missing the first bit:
isBitSet = ((bits & 1) == 1);
bits = bits >> 1;
But a better option would be to use the static methods of the BitConverter class to get the actual bytes used to represent the number in memory into a byte array. The advantage (or disadvantage depending on your needs) of this method is that this reflects the endianness of the machine running the code.
byte[] bytes = BitConverter.GetBytes(num);
int bitPos = 0;
while(bitPos < 8 * bytes.Length)
{
int byteIndex = bitPos / 8;
int offset = bitPos % 8;
bool isSet = (bytes[byteIndex] & (1 << offset)) != 0;
// isSet = [True] if the bit at bitPos is set, false otherwise
bitPos++;
}

C to C# Bytearray + hex

I'm currently trying to get this C code converted into C#.
Since I'm not really familiar with C I'd really apprecheate your help!
static unsigned char byte_table[2080] = {0};
First of, some bytearray gets declared but never filled which I'm okay with
BYTE* packet = //bytes come in here from a file
int unknownVal = 0;
int unknown_field0 = *(DWORD *)(packet + 0x08);
do
{
*((BYTE *)packet + i) ^= byte_table[(i + unknownVal) & 0x7FF];
++i;
}
while (i <= packet[0]);
But down here.. I really have no idea how to translate this into C#
BYTE = byte[] right?
DWORD = double?
but how can (packet + 0x08) be translated? How can I add a hex to a bytearray? Oo
I'd be happy about anything that helps! :)

In C, setting any set of memory to {0} will set the entire memory area to zeroes, if I'm not mistaken.
That bottom loop can be rewritten in a simpler, C# friendly fashion.
byte[] packet = arrayofcharsfromfile;
int field = packet[8]+(packet[9]<<8)+(packet[10]<<16)+(packet[11]<<24); //Assuming 32 bit little endian integer
int unknownval = 0;
int i = 0;
do //Why waste the newline? I don't know. Conventions are silly!
{
packet[i] ^= byte_table[(i+unknownval) & 0x7FF];
} while( ++i <= packet[0] );
field is set by taking the four bytes including and following index 8 and generating a 32 bit int from them.
In C, you can cast pointers to other types, as is done in your provided snippet. What they're doing is taking an array of bytes (each one 1/4 the size of a DWORD) and adding 8 to the index which advances the pointer by 8 bytes (since each element is a byte wide) and then treating that pointer as a DWORD pointer. In simpler terms, they're turning the byte array in to a DWORD array, and then taking index 2, as 8/4=2.
You can simulate this behavior in a safe fashion by stringing the bytes together with bitshifting and addition, as I demonstrated above. It's not as efficient and isn't as pretty, but it accomplishes the same thing, and in a platform agnostic way too. Not all platforms are little endian.

Converting raw byte data to float[]

I have this code for converting a byte[] to float[].
public float[] ConvertByteToFloat(byte[] array)
{
float[] floatArr = new float[array.Length / sizeof(float)];
int index = 0;
for (int i = 0; i < floatArr.Length; i++)
{
floatArr[i] = BitConverter.ToSingle(array, index);
index += sizeof(float);
}
return floatArr;
}
Problem is, I usually get a NaN result! Why should this be? I checked if there is data in the byte[] and the data seems to be fine. If it helps, an example of the values are:
new byte[] {
231,
255,
235,
255,
}
But this returns NaN (Not a Number) after conversion to float. What could be the problem? Are there other better ways of converting byte[] to float[]? I am sure that the values read into the buffer are correct since I compared it with my other program (which performs amplification for a .wav file).

If endianness is the problem, you should check the value of BitConverter.IsLittleEndian to determine if the bytes have to be reversed:
public static float[] ConvertByteToFloat(byte[] array) {
float[] floatArr = new float[array.Length / 4];
for (int i = 0; i < floatArr.Length; i++) {
if (BitConverter.IsLittleEndian) {
Array.Reverse(array, i * 4, 4);
}
floatArr[i] = BitConverter.ToSingle(array, i * 4);
}
return floatArr;
}

Isn't the problem that the 255 in the exponent represents NaN (see Wikipedia in the exponent section), so you should get a NaN. Try changing the last 255 to something else...

If it's the endianness that is wrong (you are reading big endian numbers) try these (be aware that they are "unsafe" so you have to check the unsafe flag in your project properties)
public static unsafe int ToInt32(byte[] value, int startIndex)
{
fixed (byte* numRef = &value[startIndex])
{
var num = (uint)((numRef[0] << 0x18) | (numRef[1] << 0x10) | (numRef[2] << 0x8) | numRef[3]);
return (int)num;
}
}
public static unsafe float ToSingle(byte[] value, int startIndex)
{
int val = ToInt32(value, startIndex);
return *(float*)&val;
}

I assume that your byte[] doesn't contain the binary representation of floats, but either a sequence of Int8s or Int16s. The examples you posted don't look like audio samples based on float (neither NaN nor -2.41E+24 are in the range -1 to 1. While some new audio formats might support floats outside that range, traditionally audio data consists of signed 8 or 16 bit integer samples.
Another thing you need to be aware of is that often the different channels are interleaved. For example it could contain the first sample for the left, then for the right channel, and then the second sample for both... So you need to separate the channels while parsing.
It's also possible, but uncommon that the samples are unsigned. In which case you need to remove the offset from the conversion functions.
So you first need to parse current position in the byte array into an Int8/16. And then convert that integer to a float in the range -1 to 1.
If the format is little endian you can use BitConverter. Another possibility that works with both endiannesses is getting two bytes manually and combining them with a bit-shift. I don't remember if little or big endian is common. So you need to try that yourself.
This can be done with functions like the following(I didn't test them):
float Int8ToFloat(Int8 i)
{
return ((i-Int8.MinValue)*(1f/0xFF))-0.5f;
}
float Int16ToFloat(Int16 i)
{
return ((i-Int16.MinValue)*(1f/0xFFFF))-0.5f;
}

Depends on how you want to convert the bytes to float. Start out by trying to pin-point what is actually stored in the byte array. Perhaps there is a file format specification? Other transformations in your code?
In the case each byte should be converted to a float between 0 and 255:
public float[] ConvertByteToFloat(byte[] array)
{
return array.Select(b => (float)b).ToArray();
}
If the bytes array contains binary representation of floats, there are several representation and if the representation stored in your file does not match the c# language standard floating point representation (IEEE 754) weird things like this will happen.