We Keep Coding

Reverse Byte Array when converting an animated gif to a Unity Texture2D Array

I am currently trying to load animated gifs into a Unity application (during runtime), but I have run into a bit of a snag:
I am using System.Drawing to load a byte array from each gif-frame and then use Unity's LoadRawTextureData function to create the Texture. The problem I have now is that the order of the bytes in the array are not what Unity expects (even though I specify the Format as ARGB32 for both). Apparently, Unity either expects ABGR and System.Drawing gives me RGBA, or vice versa. Also, the image is flipped vertically (but that can be easily remedied).
This is my current code, which works, but I have to reverse the array which increases the time it takes by a factor of 10-20. It is still at least twice as fast as copying each pixel separately, but I would prefer if I could get closer to the performance I get without reversing.
GraphicsUnit graphicsUnit = GraphicsUnit.Pixel;
RectangleF rect1 = gifImage.GetBounds(ref graphicsUnit);
Rectangle rect2 = new Rectangle((int)rect1.X, (int)rect1.Y, (int)rect1.Width, (int)rect1.Height);
byte[] rgbValues = new byte[rect2.Width * rect2.Height * 4];
for (int i = 0; i < frameCount; i++) {
gifImage.SelectActiveFrame(dimension, i);
Bitmap frame = new Bitmap(gifImage.Width, gifImage.Height);
System.Drawing.Graphics.FromImage(frame).DrawImage(gifImage, Point.Empty);
frame.RotateFlip(RotateFlipType.RotateNoneFlipX);
Texture2D frameTexture = new Texture2D(frame.Width, frame.Height, TextureFormat.ARGB32, false);
BitmapData data = frame.LockBits(rect2, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
Marshal.Copy(data.Scan0, rgbValues, 0, rgbValues.Length);
frame.UnlockBits(data);
frameTexture.LoadRawTextureData(rgbValues.Reverse().ToArray());
frameTexture.Apply();
gifFrames[i] = frameTexture;
}
Is there something else I could try to get the bytes in the 'correct' order?

You can skip the .Reverse() if you construct your Texture2D using TextureFormat.BGRA32. however, this doesn't really address why its necessary

Memory efficient bitmap handling in mono for android

I got an application, which allows the user to take a picture. After the picture has been taken, the user can send it to my webserver. But before i do this, it needs to resize the bitmap because i like to have consistent sizes send to my webserver.
Anyway, the code i use to load the bitmap into memory and then manipulate it, does seem to occupy a lot of memory. This code is currently being used :
/*
* This method is used to calculate image size.
* And also resize/scale image down to 1600 x 1200
*/
private void ResizeBitmapAndSendToWebServer(string album_id) {
Bitmap bm = null;
// This line is taking up to much memory each time..
Bitmap bitmap = MediaStore.Images.Media.GetBitmap(Android.App.Application.Context.ApplicationContext.ContentResolver,fileUri);
/*
* My question is : Could i do the next image manipulation
* before i even load the bitmap into memory?
*/
int width = bitmap.Width;
int height = bitmap.Height;
if (width >= height) { // <-- Landscape picture
float scaledWidth = (float)height / width;
if (width > 1600) {
bm = Bitmap.CreateScaledBitmap (bitmap, 1600, (int)(1600 * scaledWidth), true);
} else {
bm = bitmap;
}
} else {
float scaledHeight = (float)width / height;
if (height > 1600) {
bm = Bitmap.CreateScaledBitmap (bitmap, (int)(1600 * scaledHeight), 1600 , true);
} else {
bm = bitmap;
}
}
// End of question code block.
MemoryStream stream = new MemoryStream ();
bitmap.Compress (Bitmap.CompressFormat.Jpeg, 80, stream);
byte[] bitmapData = stream.ToArray ();
bitmap.Dispose ();
app.api.SendPhoto (Base64.EncodeToString (bitmapData, Base64Flags.Default), album_id);
}
What would be a good and clean way for solving such memory problems?
EDIT 1 :
After reading other posts, it became clear to me that i am doing some inefficient things with my code. This is, in steps, what i have been doing :
Load full bitmap into memory.
Decide wether it is landscape or not.
Then create new bitmap with the right dimensions.
Then converting this bitmap into byte array
Disposing the initial bitmap. (But never remove the scaled bitmap out of memory).
What i really should be doing :
Determine real bitmap dimensions without loading it into memory with :
private void FancyMethodForDeterminingImageDimensions() {
BitmapFactory.Options options = new BitmapFactory.Options();
options.InJustDecodeBounds = true;
BitmapFactory.DecodeFile(fileUri.Path, options);
// Now the dimensions of the bitmap are known without loading
// the bitmap into memory.
// I am not further going to explain this, i think the purpose is
// explaining enough.
int outWidth = options.OutWidth;
int outHeight = options.OutHeight;
}
If set to true, the decoder will return null (no bitmap), but the
out... fields will still be set, allowing the caller to query the
bitmap without having to allocate the memory for its pixels.
Now i know the real dimenions. So i can downsample it before i load it into memory.
(in my case) Convert bitmap to base64 string and send it.
Dispose everything so the memory gets cleared.
I can't currently test this, because i am not on my development machine. Can anyone give me some feedback if this is the right way? It will be appreciated.

private void ResizeBitmapAndSendToWebServer(string album_id) {
BitmapFactory.Options options = new BitmapFactory.Options ();
options.InJustDecodeBounds = true; // <-- This makes sure bitmap is not loaded into memory.
// Then get the properties of the bitmap
BitmapFactory.DecodeFile (fileUri.Path, options);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Original width : {0}, and height : {1}", options.OutWidth, options.OutHeight) );
// CalculateInSampleSize calculates the right aspect ratio for the picture and then calculate
// the factor where it will be downsampled with.
options.InSampleSize = CalculateInSampleSize (options, 1600, 1200);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Downsampling factor : {0}", CalculateInSampleSize (options, 1600, 1200)) );
// Now that we know the downsampling factor, the right sized bitmap is loaded into memory.
// So we set the InJustDecodeBounds to false because we now know the exact dimensions.
options.InJustDecodeBounds = false;
// Now we are loading it with the correct options. And saving precious memory.
Bitmap bm = BitmapFactory.DecodeFile (fileUri.Path, options);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Downsampled width : {0}, and height : {1}", bm.Width, bm.Height) );
// Convert it to Base64 by first converting the bitmap to
// a byte array. Then convert the byte array to a Base64 String.
MemoryStream stream = new MemoryStream ();
bm.Compress (Bitmap.CompressFormat.Jpeg, 80, stream);
byte[] bitmapData = stream.ToArray ();
bm.Dispose ();
app.api.SendPhoto (Base64.EncodeToString (bitmapData, Base64Flags.Default), album_id);
}

Read and write directly to Unlocked Bitmap unmanaged memory (Scan0)

Is that ok to Write and Read directly from a unlocked Bitmap unmanaged memory?
Can I keep using the BitmapData after I UnlockBits of the Bitmap? I did a test app where I can read the pixel of the Bitmap of a PictureBox at mouse position while another thread is writing pixels to the same Bitmap.
EDIT 1: As Boing have pointed out in his answer: "Scan0 does not point to the actual pixel data of the Bitmap object; rather, it points to a temporary buffer that represents a portion of the pixel data in the Bitmap object." from MSDN.
But once I get the Scan0, I'm able to read/write to the Bitmap without the need of Lockbits or UnlockBits! I'm doing this a lot of times in a thread. Accordingly to MSDN, it should not happen, because Scan0 points to a COPY of the Bitmap data! Well, in C# all the test shows that it is not a copy. In C++ I don't know if it works as it should.
EDIT 2: Using the rotate method some times makes the OS to free the Bitmap pixel data copy. Conclusion, it is not safe to read/write an unlocked Bitmap Scan0. Thanks Boing for your answer and comments!
Below is how I get the BitmapData and read and write the pixel value.
/// <summary>
/// Locks and unlocks the Bitmap to get the BitmapData.
/// </summary>
/// <param name="bmp">Bitmap</param>
/// <returns>BitmapData</returns>
public static BitmapData GetBitmapData(Bitmap bmp)
{
BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadOnly, bmp.PixelFormat);
bmp.UnlockBits(bmpData);
return bmpData;
}
/// <summary>
/// Get pixel directly from unamanged pixel data based on the Scan0 pointer.
/// </summary>
/// <param name="bmpData">BitmapData of the Bitmap to get the pixel</param>
/// <param name="p">Pixel position</param>
/// <param name="channel">Channel</param>
/// <returns>Pixel value</returns>
public static byte GetPixel(BitmapData bmpData, Point p, int channel)
{
if ((p.X > bmpData.Width - 1) || (p.Y > bmpData.Height - 1))
throw new ArgumentException("GetPixel Point p is outside image bounds!");
int bitsPerPixel = ((int)bmpData.PixelFormat >> 8) & 0xFF;
int bpp = bitsPerPixel / 8;
byte data;
int id = p.Y * bmpData.Stride + p.X * bpp;
unsafe
{
byte* pData = (byte*)bmpData.Scan0;
data = pData[id + channel];
}
return data;
}
//Non UI Thread
private void DrawtoBitmapLoop()
{
while (_drawBitmap)
{
_drawPoint = new Point(_drawPoint.X + 10, _drawPoint.Y + 10);
if (_drawPoint.X > _backImageData.Width - 20)
_drawPoint.X = 0;
if (_drawPoint.Y > _backImageData.Height - 20)
_drawPoint.Y = 0;
DrawToScan0(_backImageData, _drawPoint, 1);
Thread.Sleep(10);
}
}
private static void DrawToScan0(BitmapData bmpData, Point start, int channel = 0)
{
int x = start.X;
int y = start.Y;
int bitsPerPixel = ((int)bmpData.PixelFormat >> 8) & 0xFF;
int bpp = bitsPerPixel / 8;
for (int i = 0; i < 10; i++)
{
unsafe
{
byte* p = (byte*)bmpData.Scan0;
int id = bmpData.Stride * y + channel + (x + i) * bpp;
p[id] = 255;
}
}
}

No, you cannot. The official explanation is clear about that.
Scan0 does not point to the actual pixel data of the Bitmap object; rather, it points to a temporary buffer that represents a portion of the pixel data in the Bitmap object. The code writes the value 0xff00ff00 (green) to 1500 locations in the temporary buffer. Later, the call to Bitmap::UnlockBits copies those values to the Bitmap object itself.
I would agree that there is a "bug" in UnLockBits(), because every non ImageLockModeUserInputBuf BitmapData should have its field reset (especially scan0) after the 'release/unlock'.
Scan0 GDI managed buffers may still be accessible after UnLockBits, but this is pure luck you do not get a invalid memory reference hard fault. The graphic subsystem may need this memory space to backup another Bitmap, or the same Bitmap but for another rectangle or in another pixelformat.
Scan0 don't represent the internal data of the bitmap, but a COPY, writen to by GDI while LockBits(...| ImageLockModeRead...) and read from by GDI while UnLockBits() (.. if LockBitswith(.. | ImageLockModeWrite ..)
That is what BitmapData abstraction is. Now maybe if you use a rectangle equals to the bitmap size AND a pixelmode matching the one of your display card, GDI may return the actual pixel storage address of the bitmap into scan0 (and not a copy), but you should never rely on that (or make a program that only work on your own computer).
EDIT 1: I allready explained above why you are lucky to be able to use scan0 outside the lock. Because you use the original bmp PixelFormat and that GDI is optimized in that case to give you the pointer and not a copy. This pointer is valid until the OS will decide to free it. The only time there is a garantee is between LockBits and UnLockBits. Period.
Here is the code to add to yours, put it in a Form to test it somewhat seriously. I can make it crash with a kind-of "neutral" call with Rotate180FlipX by hammering the button.
The bitmap internals are private. Period.
The OS may decide any moment to change its representation without even you making "action" on it (like minimizing the window, and zillions other possibilities).
EDIT 2: Your question:is there any practical difference locking a bitmap using ReadOnly or WriteOnly mode when no user buffer is given?
With or without user buffer, there IS a difference. One copy on LockBits(if readonly) AND/OR one copy on UnlockBits(if writeonly). Choose carefully to not do unwanted copies. Hint: stop thinking you are working in the same pixelformat, logically you do not. A write only buffer in 64bpp is received totally filled with noise (or untouched if it is also user buffer). You had better completely fill it before the unlock. (not just poking at some pixels). The naming of enum is misleading because WriteOnly | ReadOnly == ReadWrite
Accessing one pixel at a time using LockBits is BAD. Nobody wants to do that. What you do is to create/modify many*many pixel (using pointer/scan0) and commit them in quazy ATOMIC operation (Lock/Marhsal.Copy/UnLock) to the bitmap (and Invalidate()/redraw if you want to see something)
public MainForm()
{
InitializeComponent();
pictureBox.SizeMode = PictureBoxSizeMode.StretchImage;
// use a .gif for 8bpp
Bitmap bmp = (Bitmap)Bitmap.FromFile(#"C:\Users\Public\Pictures\Sample Pictures\Forest Flowers.jpg");
pictureBox.Image = bmp;
_backImageData = GetBitmapData(bmp);
_drawBitmap = true;
_thread= new Thread(DrawtoBitmapLoop);
_thread.IsBackground= true;
_thread.Start();
button.Text = "Let's get real";
button.Click += (object sender, EventArgs e) =>
{
// OK on my system, it does not rreallocate but ...
bmp.RotateFlip(RotateFlipType.Rotate180FlipX);
// ** FAIL with Rotate180FlipY on my system**
};
}
Thread _thread;
bool _drawBitmap;
BitmapData _backImageData;
//Non UI Thread
private void DrawtoBitmapLoop()
{
while (_drawBitmap)
{
ScrollColors(_backImageData);
this.Invoke((ThreadStart)(() =>
{
if (!this.IsDisposed)
this.pictureBox.Invalidate();
}));
Thread.Sleep(100);
}
}
private unsafe static void ScrollColors(BitmapData bmpData)
{
byte* ptr = (byte*)bmpData.Scan0;
ptr--;
byte* last = &ptr[(bmpData.Stride) * bmpData.Height];
while (++ptr <= last)
{
*ptr = (byte)((*ptr << 7) | (*ptr >> 1));
}
}

Fast Video Display WPF

I am working on a WPF application that needs to display several video streams at a fast frame rate (we would like 30 fps). The video streams are 1920x1080 raw (RGB24) frames (they are stored in a System.Drawing.Bitmap). Does anyone have any ideas on how to achieve this?
More details:
Our previous attempts have used a standard WPF Image control, changing its source for each frame. This worked well for a single stream but now that we have to render multiple streams, it is slowing down.
We have also tried using Direct2D to handle the drawing, using a D3D9 shared surface as the source for an Image control. While this was faster, we are still not able to get a stable 30fps out of it (it jumps between 24-32 fps as things back up).
The video stream is coming in on a background thread, then being marshaled (using the Dispatcher of the window) to the proper UI thread for drawing. All the drawing is then done on the UI thread. We have also tried giving each window its own thread.
I can provide code samples of what we have tried if anyone wants to see.
thanks!

Anyone looking for a solution, we wrote a custom winforms control using DirectX 11 with a highly optimized copy into graphics memory, then hosted the control in a WinformsHost, I can provide some code to anyone who is interested.
Optimized copy into gpu memory
// Using native memcpy for the fastest possible copy
[DllImport("msvcrt.dll", EntryPoint = "memcpy", CallingConvention = CallingConvention.Cdecl, SetLastError = false)]
private static extern IntPtr memcpy(IntPtr dest, IntPtr src, UIntPtr count);
/// <summary>
/// Copies a bitmap to gpu memory
/// </summary>
/// <param name="frame">The image to copy to the gpu</param>
/// <returns>A texture in gpu memory for the bitmap</returns>
public Texture2D CopyFrameToGpuMemory(Bitmap frame)
{
SampleDescription sampleDesc = new SampleDescription();
sampleDesc.Count = 1;
sampleDesc.Quality = 0;
Texture2DDescription texDesc = new Texture2DDescription()
{
ArraySize = 1,
MipLevels = 1,
SampleDescription = sampleDesc,
Format = Format.B8G8R8A8_UNorm,
CpuAccessFlags = CpuAccessFlags.Write,
BindFlags = BindFlags.ShaderResource,
Usage = ResourceUsage.Dynamic,
Height = frame.Height,
Width = frame.Width
};
Texture2D tex = new Texture2D(Device, texDesc);
Surface surface = tex.AsSurface();
DataRectangle mappedRect = surface.Map(SlimDX.DXGI.MapFlags.Write | SlimDX.DXGI.MapFlags.Discard);
BitmapData pixelData = frame.LockBits(new Rectangle(0, 0, frame.Width, frame.Height), ImageLockMode.ReadOnly, PixelFormat.Format32bppRgb);
unsafe //!!!
{
byte* pixelDataStart = (byte*)pixelData.Scan0.ToPointer();
byte* mappedRectDataStart = (byte*)mappedRect.Data.DataPointer.ToPointer();
for (int y = 0; y < texDesc.Height; y++)
{
byte* lineLocationDest = mappedRectDataStart + (y * mappedRect.Pitch);
byte* lineLocationSrc = pixelDataStart + (y * pixelData.Stride);
// Copy line by line for best performance
memcpy((IntPtr)lineLocationDest, (IntPtr)lineLocationSrc, (UIntPtr)(texDesc.Width * 4));
}
} //!!!
frame.UnlockBits(pixelData);
mappedRect.Data.Dispose();
surface.Unmap();
surface.Dispose();
return tex;
}

Can I copy this array any faster?

Is this the absolute fastest I could possibly copy a Bitmap to a Byte[] in C#?
If there is a speedier way I am dying to know!
const int WIDTH = /* width */;
const int HEIGHT = /* height */;
Bitmap bitmap = new Bitmap(WIDTH, HEIGHT, PixelFormat.Format32bppRgb);
Byte[] bytes = new byte[WIDTH * HEIGHT * 4];
BitmapToByteArray(bitmap, bytes);
private unsafe void BitmapToByteArray(Bitmap bitmap, Byte[] bytes)
{
BitmapData bitmapData = bitmap.LockBits(new Rectangle(0, 0, WIDTH, HEIGHT), ImageLockMode.ReadOnly, PixelFormat.Format32bppRgb);
fixed(byte* pBytes = &bytes[0])
{
MoveMemory(pBytes, bitmapData.Scan0.ToPointer(), WIDTH * HEIGHT * 4);
}
bitmap.UnlockBits(bitmapData);
}
[DllImport("Kernel32.dll", EntryPoint = "RtlMoveMemory", SetLastError = false)]
private static unsafe extern void MoveMemory(void* dest, void* src, int size);

Well, using Marshal.Copy() would be wiser here, that at least cuts out on the (one time) cost of looking up the DLL export. But that's it, they both use the C runtime memcpy() function. Speed is entirely throttled by the RAM bus bandwidth, only buying a more expensive machine can speed it up.
Beware that profiling is tricky, accessing the bitmap data the first time causes page faults to get the pixel data into memory. How long that takes is critically dependent on what your hard disk is doing and the state of the file system cache.

I'm reading into this that you're looking to do pixel manipulation of a bitmap.
So logically, you're wanting to access the pixels as an array in order to do this.
The problem your facing is fundamentally flawed, and the other guys didn't pick up on this - and maybe this is something they can take away too?
Basically, you've gotten so far to be messing with pointers from bitmaps, I'm going to give you one of my hard earned "secrets".
YOU DON'T NEED TO COPY THE BITMAP INTO AN ARRAY. Just use it as it is.
Unsafe pointers are your friend in this case. When you hit "bitmapData.Scan0.ToPointer()" you missed the trick.
So long as you have a pointer to the first pixel, and the Stride, and you're aware of the number of bytes per pixel, then you're on to a winner.
I define a bitmap specifically with 32 bits per pixel (memory efficient for UInt32 access) and I get a pointer to first pixel. I then treat the pointer as an array, and can both read and write pixel data as UInt32 values.
Code speaks louder than words. have a look.
I salute you for making it this far!
This is untested code, but much is copied from my source.
public delegate void BitmapWork(UInt32* ptr, int stride);
/// <summary>
/// you don't want to forget to unlock a bitmap do you? I've made that mistake too many times...
/// </summary>
unsafe private void UnlockBitmapAndDoWork(Bitmap bmp, BitmapWork work)
{
var s = new Rectangle (0, 0, bmp.Width, bmp.Height);
var locker = bmp.LockBits(new Rectangle(0, 0, 320, 200), ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
var ptr = (UInt32*)locker.Scan0.ToPointer();
// so many times I've forgotten the stride is expressed in bytes, but I'm accessing with UInt32's. So, divide by 4.
var stride = locker.Stride / 4;
work(ptr, stride);
bmp.UnlockBits(locker);
}
//using System.Drawing.Imaging;
unsafe private void randomPixels()
{
Random r = new Random(DateTime.Now.Millisecond);
// 32 bits per pixel. You might need to concern youself with the Alpha part depending on your use of the bitmap
Bitmap bmp = new Bitmap(300, 200, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
UnlockBitmapAndDoWork(bmp, (ptr, stride) =>
{
var calcLength = 300 * 200; // so we only have one loop, not two. It's quicker!
for (int i = 0; i < calcLength; i++)
{
// You can use the pointer like an array. But there's no bounds checking.
ptr[i] = (UInt32)r.Next();
}
});
}

I would also look at System.Buffer.BlockCopy. This function is also very fast and it might be competetive in this setup as you are copying from managed to managed in your case. Unfortunately I can not provide some performance tests.

I think this is faster (I actually used it):
// Bitmap bytes have to be created via a direct memory copy of the bitmap
private byte[] BmpToBytes_MemStream (Bitmap bmp)
{
MemoryStream ms = new MemoryStream();
// Save to memory using the Jpeg format
bmp.Save(ms, ImageFormat.Jpeg);
// read to end
byte[] bmpBytes = ms.GetBuffer();
bmp.Dispose();
ms.Close();
return bmpBytes;
}
Original Source