We Keep Coding

Why my Image updates from top to bottom row by row

I have a WPF application with Image control in it.
I'm using WriteableBitmap to update Image.source, but I can't understand why I see this strange behaviour: my image breaks in two parts and top part is slowly moving to bottom in cycle at pretty slow rate.
I dont understand why this is happens. Top and bottom parts are actually the same frame, because I can see real-time updates in both of them.
My code:
public MainWindow()
{
InitializeComponent();
InitVideo();
image.Source = frame;
}
private void InitVideo
{
/// .... some other init stuff ...
frame = new WriteableBitmap(width, height, 96, 96, PixelFormats.Rgb24, null);
rgbch = new byte[stride * height];
dataProc = new System.Threading.Thread(ReadData);
dataProc.Start();
}
// in separate thread
private void ReadData()
{
while (rxVideo)
{
for (int i = 0; i < rgbch.Length; i += stride)
{
pipe.Read(rgbch, i, stride);
}
Application.Current.Dispatcher.Invoke(() =>
{
frame.Lock();
frame.WritePixels(new Int32Rect(0, 0, width, height), rgbch, stride, 0);
frame.Unlock();
});
}
I tried to use frame.dispatcher.invoke -> same result.
Tried Marshal.Copy -> same result..

I've found source of a problem.
It was cause by my code inside thread
for (int i = 0; i < rgbch.Length; i += stride)
{
pipe.Read(rgbch, i, stride);
}
rgbch was set as a source for writablebitmap backbuffer, so when I wrote new data in it, update worked slow, so I got that strange top-bottom update.
I just did pipe.read(rgbch, 0, rgbch.Length) and it all worked faster without any borders in image.

It's almost surly not relevant to your code. It can be because of:
Very large image size(maybe 100s of MB)
Network low bandwidth
Weak graphic card
You should seek the reason of displaying image row by row in years ago. In those years the internet bandwidth was really low and this was a technique that let's user to see the image before loading completely. I know you know this. I just wrote it to make the answer complete!

How to create a list of the most recent 45 image frames of a kinect

Im trying to store the most recent 45 color frames from a kinect. My idea was that I could take the colorPixel array used by the kinect to store the pixel data in byte form, and save that into a jagged array I called Frames. The code below is what the event handler of a standard color viewer for the kinect would look like `
//These are global parameter
next index = 0;
byte[45][] Frames;
byte nextIndex = 0;
//This is the event handler within the code for a typical color viewer
void sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
{
//Opens the color frame sent from the event handler and declares it
//as 'colorFrame'.
using (ColorImageFrame colorFrame = e.OpenColorImageFrame())
{
/*Check to see if colorFrame is null. It will be null if this frame cannot
be processed before the event handler registers another frame. This
results in the null frame being dropped as there is no method for
colorFrame == null.*/
if (colorFrame != null)
{
//Copies a frame of data to 'colorPixels'.
colorFrame.CopyPixelDataTo(colorPixels);
//Assigns colorPixels to the array Frames. Also, this will only store
//the 45 most recent Frames
Frames[nextIndex] = colorPixels;
nextIndex++;
if (nextIndex == 45)
{
nextIndex = 0;
}
/*Updates the pixels in the specified region of the bitmap. Parameteres:
Int32Rect- Describes the width, height, and location of an integer rectangle
XAML value for X coordinate of top left corner
XAML value for Y coordinate of top left corner
Width of the rectangle
Heigth of the rectangle
Pixels- the pixel array used to update the bitmap
Stride- the stride of the update region in pixels
Offset- the input buffer offset */
colorBitmap.WritePixels(
new Int32Rect(0, 0, colorBitmap.PixelWidth, colorBitmap.PixelHeight),
colorPixels,
colorBitmap.PixelWidth * sizeof(int),
0);
}
}
}`
The issue with that code is that when it is run, it assigns the current array being processed (colorPixels) to every element of the Frames array. So the first array colorPixels from the first frame is stored as Frames[0], however when the event handler fires again and the second array of colorPixels is ready, Frames[0] and Frames[1] both are reassigned as the colorPixel array for frame 2. Thanks!

You have only one byte buffer and assign a reference to that single byte buffer to several array elements.
I would recommend to use a list of byte buffers instead of a multi-dimensional array. That would be easier to read.
First: Remove the definition of your byte buffer colorPixels and replace it with
const int frameCount = 45;
List<byte[]> frames;
Second: You have to initialize your buffers. Somewhere in your code you initialize your byte buffer with somthing like
this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength];
Initialize your byte buffers instead with something like:
frames = new List<byte[]>(frameCount);
for (int i = 0; i < frameCount; i++)
{
frames.Add(new byte[this.sensor.ColorStream.FramePixelDataLength]);
}
Third: Use the buffers in your eventhandler.
byte[] colorPixels = frames[nextIndex];
if (++nextIndex >= frameCount)
{
nextIndex = 0;
}
colorFrame.CopyPixelDataTo(colorPixels);
colorBitmap.WritePixels(...

Memory efficient bitmap handling in mono for android

I got an application, which allows the user to take a picture. After the picture has been taken, the user can send it to my webserver. But before i do this, it needs to resize the bitmap because i like to have consistent sizes send to my webserver.
Anyway, the code i use to load the bitmap into memory and then manipulate it, does seem to occupy a lot of memory. This code is currently being used :
/*
* This method is used to calculate image size.
* And also resize/scale image down to 1600 x 1200
*/
private void ResizeBitmapAndSendToWebServer(string album_id) {
Bitmap bm = null;
// This line is taking up to much memory each time..
Bitmap bitmap = MediaStore.Images.Media.GetBitmap(Android.App.Application.Context.ApplicationContext.ContentResolver,fileUri);
/*
* My question is : Could i do the next image manipulation
* before i even load the bitmap into memory?
*/
int width = bitmap.Width;
int height = bitmap.Height;
if (width >= height) { // <-- Landscape picture
float scaledWidth = (float)height / width;
if (width > 1600) {
bm = Bitmap.CreateScaledBitmap (bitmap, 1600, (int)(1600 * scaledWidth), true);
} else {
bm = bitmap;
}
} else {
float scaledHeight = (float)width / height;
if (height > 1600) {
bm = Bitmap.CreateScaledBitmap (bitmap, (int)(1600 * scaledHeight), 1600 , true);
} else {
bm = bitmap;
}
}
// End of question code block.
MemoryStream stream = new MemoryStream ();
bitmap.Compress (Bitmap.CompressFormat.Jpeg, 80, stream);
byte[] bitmapData = stream.ToArray ();
bitmap.Dispose ();
app.api.SendPhoto (Base64.EncodeToString (bitmapData, Base64Flags.Default), album_id);
}
What would be a good and clean way for solving such memory problems?
EDIT 1 :
After reading other posts, it became clear to me that i am doing some inefficient things with my code. This is, in steps, what i have been doing :
Load full bitmap into memory.
Decide wether it is landscape or not.
Then create new bitmap with the right dimensions.
Then converting this bitmap into byte array
Disposing the initial bitmap. (But never remove the scaled bitmap out of memory).
What i really should be doing :
Determine real bitmap dimensions without loading it into memory with :
private void FancyMethodForDeterminingImageDimensions() {
BitmapFactory.Options options = new BitmapFactory.Options();
options.InJustDecodeBounds = true;
BitmapFactory.DecodeFile(fileUri.Path, options);
// Now the dimensions of the bitmap are known without loading
// the bitmap into memory.
// I am not further going to explain this, i think the purpose is
// explaining enough.
int outWidth = options.OutWidth;
int outHeight = options.OutHeight;
}
If set to true, the decoder will return null (no bitmap), but the
out... fields will still be set, allowing the caller to query the
bitmap without having to allocate the memory for its pixels.
Now i know the real dimenions. So i can downsample it before i load it into memory.
(in my case) Convert bitmap to base64 string and send it.
Dispose everything so the memory gets cleared.
I can't currently test this, because i am not on my development machine. Can anyone give me some feedback if this is the right way? It will be appreciated.

private void ResizeBitmapAndSendToWebServer(string album_id) {
BitmapFactory.Options options = new BitmapFactory.Options ();
options.InJustDecodeBounds = true; // <-- This makes sure bitmap is not loaded into memory.
// Then get the properties of the bitmap
BitmapFactory.DecodeFile (fileUri.Path, options);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Original width : {0}, and height : {1}", options.OutWidth, options.OutHeight) );
// CalculateInSampleSize calculates the right aspect ratio for the picture and then calculate
// the factor where it will be downsampled with.
options.InSampleSize = CalculateInSampleSize (options, 1600, 1200);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Downsampling factor : {0}", CalculateInSampleSize (options, 1600, 1200)) );
// Now that we know the downsampling factor, the right sized bitmap is loaded into memory.
// So we set the InJustDecodeBounds to false because we now know the exact dimensions.
options.InJustDecodeBounds = false;
// Now we are loading it with the correct options. And saving precious memory.
Bitmap bm = BitmapFactory.DecodeFile (fileUri.Path, options);
Android.Util.Log.Debug ("[BITMAP]" , string.Format("Downsampled width : {0}, and height : {1}", bm.Width, bm.Height) );
// Convert it to Base64 by first converting the bitmap to
// a byte array. Then convert the byte array to a Base64 String.
MemoryStream stream = new MemoryStream ();
bm.Compress (Bitmap.CompressFormat.Jpeg, 80, stream);
byte[] bitmapData = stream.ToArray ();
bm.Dispose ();
app.api.SendPhoto (Base64.EncodeToString (bitmapData, Base64Flags.Default), album_id);
}

Read and write directly to Unlocked Bitmap unmanaged memory (Scan0)

Is that ok to Write and Read directly from a unlocked Bitmap unmanaged memory?
Can I keep using the BitmapData after I UnlockBits of the Bitmap? I did a test app where I can read the pixel of the Bitmap of a PictureBox at mouse position while another thread is writing pixels to the same Bitmap.
EDIT 1: As Boing have pointed out in his answer: "Scan0 does not point to the actual pixel data of the Bitmap object; rather, it points to a temporary buffer that represents a portion of the pixel data in the Bitmap object." from MSDN.
But once I get the Scan0, I'm able to read/write to the Bitmap without the need of Lockbits or UnlockBits! I'm doing this a lot of times in a thread. Accordingly to MSDN, it should not happen, because Scan0 points to a COPY of the Bitmap data! Well, in C# all the test shows that it is not a copy. In C++ I don't know if it works as it should.
EDIT 2: Using the rotate method some times makes the OS to free the Bitmap pixel data copy. Conclusion, it is not safe to read/write an unlocked Bitmap Scan0. Thanks Boing for your answer and comments!
Below is how I get the BitmapData and read and write the pixel value.
/// <summary>
/// Locks and unlocks the Bitmap to get the BitmapData.
/// </summary>
/// <param name="bmp">Bitmap</param>
/// <returns>BitmapData</returns>
public static BitmapData GetBitmapData(Bitmap bmp)
{
BitmapData bmpData = bmp.LockBits(new Rectangle(0, 0, bmp.Width, bmp.Height), ImageLockMode.ReadOnly, bmp.PixelFormat);
bmp.UnlockBits(bmpData);
return bmpData;
}
/// <summary>
/// Get pixel directly from unamanged pixel data based on the Scan0 pointer.
/// </summary>
/// <param name="bmpData">BitmapData of the Bitmap to get the pixel</param>
/// <param name="p">Pixel position</param>
/// <param name="channel">Channel</param>
/// <returns>Pixel value</returns>
public static byte GetPixel(BitmapData bmpData, Point p, int channel)
{
if ((p.X > bmpData.Width - 1) || (p.Y > bmpData.Height - 1))
throw new ArgumentException("GetPixel Point p is outside image bounds!");
int bitsPerPixel = ((int)bmpData.PixelFormat >> 8) & 0xFF;
int bpp = bitsPerPixel / 8;
byte data;
int id = p.Y * bmpData.Stride + p.X * bpp;
unsafe
{
byte* pData = (byte*)bmpData.Scan0;
data = pData[id + channel];
}
return data;
}
//Non UI Thread
private void DrawtoBitmapLoop()
{
while (_drawBitmap)
{
_drawPoint = new Point(_drawPoint.X + 10, _drawPoint.Y + 10);
if (_drawPoint.X > _backImageData.Width - 20)
_drawPoint.X = 0;
if (_drawPoint.Y > _backImageData.Height - 20)
_drawPoint.Y = 0;
DrawToScan0(_backImageData, _drawPoint, 1);
Thread.Sleep(10);
}
}
private static void DrawToScan0(BitmapData bmpData, Point start, int channel = 0)
{
int x = start.X;
int y = start.Y;
int bitsPerPixel = ((int)bmpData.PixelFormat >> 8) & 0xFF;
int bpp = bitsPerPixel / 8;
for (int i = 0; i < 10; i++)
{
unsafe
{
byte* p = (byte*)bmpData.Scan0;
int id = bmpData.Stride * y + channel + (x + i) * bpp;
p[id] = 255;
}
}
}

No, you cannot. The official explanation is clear about that.
Scan0 does not point to the actual pixel data of the Bitmap object; rather, it points to a temporary buffer that represents a portion of the pixel data in the Bitmap object. The code writes the value 0xff00ff00 (green) to 1500 locations in the temporary buffer. Later, the call to Bitmap::UnlockBits copies those values to the Bitmap object itself.
I would agree that there is a "bug" in UnLockBits(), because every non ImageLockModeUserInputBuf BitmapData should have its field reset (especially scan0) after the 'release/unlock'.
Scan0 GDI managed buffers may still be accessible after UnLockBits, but this is pure luck you do not get a invalid memory reference hard fault. The graphic subsystem may need this memory space to backup another Bitmap, or the same Bitmap but for another rectangle or in another pixelformat.
Scan0 don't represent the internal data of the bitmap, but a COPY, writen to by GDI while LockBits(...| ImageLockModeRead...) and read from by GDI while UnLockBits() (.. if LockBitswith(.. | ImageLockModeWrite ..)
That is what BitmapData abstraction is. Now maybe if you use a rectangle equals to the bitmap size AND a pixelmode matching the one of your display card, GDI may return the actual pixel storage address of the bitmap into scan0 (and not a copy), but you should never rely on that (or make a program that only work on your own computer).
EDIT 1: I allready explained above why you are lucky to be able to use scan0 outside the lock. Because you use the original bmp PixelFormat and that GDI is optimized in that case to give you the pointer and not a copy. This pointer is valid until the OS will decide to free it. The only time there is a garantee is between LockBits and UnLockBits. Period.
Here is the code to add to yours, put it in a Form to test it somewhat seriously. I can make it crash with a kind-of "neutral" call with Rotate180FlipX by hammering the button.
The bitmap internals are private. Period.
The OS may decide any moment to change its representation without even you making "action" on it (like minimizing the window, and zillions other possibilities).
EDIT 2: Your question:is there any practical difference locking a bitmap using ReadOnly or WriteOnly mode when no user buffer is given?
With or without user buffer, there IS a difference. One copy on LockBits(if readonly) AND/OR one copy on UnlockBits(if writeonly). Choose carefully to not do unwanted copies. Hint: stop thinking you are working in the same pixelformat, logically you do not. A write only buffer in 64bpp is received totally filled with noise (or untouched if it is also user buffer). You had better completely fill it before the unlock. (not just poking at some pixels). The naming of enum is misleading because WriteOnly | ReadOnly == ReadWrite
Accessing one pixel at a time using LockBits is BAD. Nobody wants to do that. What you do is to create/modify many*many pixel (using pointer/scan0) and commit them in quazy ATOMIC operation (Lock/Marhsal.Copy/UnLock) to the bitmap (and Invalidate()/redraw if you want to see something)
public MainForm()
{
InitializeComponent();
pictureBox.SizeMode = PictureBoxSizeMode.StretchImage;
// use a .gif for 8bpp
Bitmap bmp = (Bitmap)Bitmap.FromFile(#"C:\Users\Public\Pictures\Sample Pictures\Forest Flowers.jpg");
pictureBox.Image = bmp;
_backImageData = GetBitmapData(bmp);
_drawBitmap = true;
_thread= new Thread(DrawtoBitmapLoop);
_thread.IsBackground= true;
_thread.Start();
button.Text = "Let's get real";
button.Click += (object sender, EventArgs e) =>
{
// OK on my system, it does not rreallocate but ...
bmp.RotateFlip(RotateFlipType.Rotate180FlipX);
// ** FAIL with Rotate180FlipY on my system**
};
}
Thread _thread;
bool _drawBitmap;
BitmapData _backImageData;
//Non UI Thread
private void DrawtoBitmapLoop()
{
while (_drawBitmap)
{
ScrollColors(_backImageData);
this.Invoke((ThreadStart)(() =>
{
if (!this.IsDisposed)
this.pictureBox.Invalidate();
}));
Thread.Sleep(100);
}
}
private unsafe static void ScrollColors(BitmapData bmpData)
{
byte* ptr = (byte*)bmpData.Scan0;
ptr--;
byte* last = &ptr[(bmpData.Stride) * bmpData.Height];
while (++ptr <= last)
{
*ptr = (byte)((*ptr << 7) | (*ptr >> 1));
}
}

Raw bitmap data/scan lines (mirror driver raw data)?

I'm coding a live control/remote desktop solution using DFMirage's free mirror driver. There is a C# sample on how to interface and control the mirror driver here. You would need the mirror driver installed first, of course, here. So, the concept is, the client (helper) requests a screen update, and the server (victim) sends one, using raw pixel encoding. The concept of a mirror driver eliminates the need to expensively poll for screen changes, because a mirror driver is notified of all screen drawing operations in real-time. The mirror driver receives the location and size of the update rectangle, and can simply query memory for the new pixel bytes and send them.
Should be easy, except that I don't know how to do that part where we query memory for the new pixel bytes. The sample shows how to query memory to grab the pixels of the entire screen using something with raw bitmap data and scan lines and stride and all that good stuff:
Bitmap result = new Bitmap(_bitmapWidth, _bitmapHeight, format);
Rectangle rect = new Rectangle(0, 0, _bitmapWidth, _bitmapHeight);
BitmapData bmpData = result.LockBits(rect, ImageLockMode.WriteOnly, format);
// Get the address of the first line.
IntPtr ptr = bmpData.Scan0;
// Declare an array to hold the bytes of the bitmap.
int bytes = bmpData.Stride * _bitmapHeight;
var getChangesBuffer = (GetChangesBuffer)Marshal
.PtrToStructure(_getChangesBuffer, typeof (GetChangesBuffer));
var data = new byte[bytes];
Marshal.Copy(getChangesBuffer.UserBuffer, data, 0, bytes);
// Copy the RGB values into the bitmap.
Marshal.Copy(data, 0, ptr, bytes);
result.UnlockBits(bmpData);
return result;
This is great and works fine. The resulting Bitmap object now has the pixels of the entire screen. But if I wanted to just extract a rectangle of pixel data instead of getting the pixel data from the whole screen, how would I be able to do that? I guess this is more of a rawbitmap-scan-stride question, but I typed all of this so you might know where this is coming from. So any insight on how to get just a portion of pixel data instead of the entire screen's pixel data?
Update: Found something interesting (code portion only).

Here's a function to copy a rectangular area from some source image buffer to a Bitmap:
private static Bitmap ExtractImageRectangle(byte[] sourceBuffer, int sourceStride, PixelFormat sourcePixelFormat, Rectangle rectangle)
{
Bitmap result = new Bitmap(rectangle.Width, rectangle.Height, sourcePixelFormat);
BitmapData resultData = result.LockBits(new Rectangle(0, 0, result.Width, result.Height), ImageLockMode.WriteOnly, result.PixelFormat);
int bytesPerPixel = GetBytesPerPixel(sourcePixelFormat); // Left as an exercise for the reader
try
{
// Bounds checking omitted for brevity
for (int rowIndex = 0; rowIndex < rectangle.Height; ++rowIndex)
{
// The address of the start of this row in the destination image
IntPtr destinationLineStart = resultData.Scan0 + resultData.Stride * rowIndex;
// The index at which the current row of our rectangle starts in the source image
int sourceIndex = sourceStride * (rowIndex + rectangle.Top) + rectangle.Left * bytesPerPixel;
// Copy the row from the source to the destination
Marshal.Copy(sourceBuffer, sourceIndex, destinationLineStart, rectangle.Width * bytesPerPixel);
}
}
finally
{
result.UnlockBits(resultData);
}
return result;
}
You could then use it like this:
Rectangle roi = new Rectangle(100, 150, 200, 250);
Bitmap result = ExtractImageRectangle(getChangesBuffer.UserBuffer, getChangesBuffer.Stride, getChangesBuffer.PixelFormat, roi);
This assumes that GetChangesBuffer has properties for the stride and pixelformat of the source image buffer. It most likely hasn't, but you should have some means to determine the stride and pixel format of your input image. In your example you are assuming the stride of the input image is equal to the stride of your output image, which is a tricky assumption.