C# Fastest image resize routine for streaming video? - c#

I am trying to write a simple C# Windows App to display streaming video from a local USB webcam. I am using AForge.NET, FilterInfoCollection, VideoCaptureDevice etc to put video stream into a standard PictureBox. As is, it works fine. However, I need to downscale the videostream. When I try that, FPS drops significantly and becomes even lower as the app continues to run. My rescale routine is as follows (it aims to maintain aspect ratio and zoom in by preserving height):
public Image ResizeImage(Image img, int target_width, int height)
{
int width = (int)(height * img.Width / (float)img.Height);
Bitmap b = new Bitmap(width, height);
using (Graphics g = Graphics.FromImage(b))
{
g.DrawImage(img, (target_width - width)/2, 0, width, height);
}
return b;
}
And it is called by:
private void video_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
Invoke(new Action(() =>
{
picVideo.Image = ResizeImage((Bitmap)eventArgs.Frame.Clone(), picVideo.Width, picVideo.Height); // rescaled stream
//picVideo.Image = (Bitmap)eventArgs.Frame.Clone(); // unscaled stream
}));
}
Setup code is this:
filterInfoCollection = new FilterInfoCollection(FilterCategory.VideoInputDevice);
videoCaptureDevice = new VideoCaptureDevice(filterInfoCollection[frmMain.Camera].MonikerString);
videoCaptureDevice.NewFrame += video_NewFrame;
videoCaptureDevice.Start();
However, I have this old 3rd party EXE that runs fine and is able to rescale the same webcam videostream as I resize its window. So it looks like this rescaling routine is too slow.
So, is there a faster rescaling method? I basically need to achieve downscaling from FullHD to 1024*768. I don't really need this to be high-quality.

Problem completely solved by adding
g.InterpolationMode = InterpolationMode.NearestNeighbor;
before g.DrawImage()

Related

Properly disposing of Bitmap object

I am drawing images in a C# Winforms panel with:
private void DrawPanel_Paint(object sender, PaintEventArgs e)
{
DrawOnPanel(e.Graphics);
}
The called method takes an existing image from my resources (myImage), gives it to another method which resizes the image and returns the resized image so it can be drawn.
public static void DrawOnPanel(Graphics g)
{
var _resizedImage = ResizeImage(Resources.myImage);
g.DrawImage(_resizedImage, destX, destY);
// ... several other images resized & manipulated then drawn
}
The resize image function is:
public Bitmap ResizeImage(Bitmap image)
{
int scale = 3;
var destImage= new Bitmap(image.Width * scale, image.Height * scale);
destImage.SetResolution(image.HorizontalResolution, image.VerticalResolution);
using (var graphics = Graphics.FromImage(destImage))
{
graphics.CompositingMode = CompositingMode.SourceCopy;
graphics.CompositingQuality = CompositingQuality.HighSpeed;
graphics.InterpolationMode = InterpolationMode.NearestNeighbor;
graphics.SmoothingMode = SmoothingMode.HighSpeed;
graphics.PixelOffsetMode = PixelOffsetMode.None;
using var wrapMode = new ImageAttributes();
wrapMode.SetWrapMode(WrapMode.TileFlipXY);
graphics.DrawImage(image, destRect, 0, 0, image.Width, image.Height, GraphicsUnit.Pixel, wrapMode);
}
return destImage;
}
The program keeps calling DrawPanel.Invalidate() in its loop.
I am detecting a memory leak each time DrawPanel.Invalidate() is called. The memory consumption is rising steadily until the GC takes care of it. While this isn't a game breaking problem, I'm still wondering where and how should I dispose of my objects in the above code.
I tried using using var _resizedImage = ResizeImage(Resources.myImage); in the above DrawOnPanel method but the program returns an error System.ArgumentException: 'Parameter is not valid.'. If I remove using there is no error.
Every time that ResizeImage is called, you create a new Bitmap. This Bitmap should be Disposed as soon as you know that it is not needed anymore. It seems that you don't need the resized image after you've drawn it on the Graphics. Therefore I suggest the following change:
public static void DrawOnPanel(Graphics g)
{
using (Image_resizedImage = ResizeImage(Resources.myImage))
{
g.DrawImage(_resizedImage, destX, destY);
}
// resizedImage is Disposed
// ... several other images resized & manipulated then drawn
}
Room for improvement
Your current design will create a new Resized Bitmap every time DrawOnPanel is called. It seems to me that most of time that Resources.myImage is resized the resized Bitmap will be the same. Why don't you just remember it until the parameters that influence the resized Bitmap are changed?
If I look at the code it seems that you always create the same resized image from an original image. So you could even put all your resized images into one Dictionary for fast lookup:
// get all Image resources that you will resize and put them in a Dictionary
// Key original Bitmap, Value: resized Bitmap
private Dictionary<Bitmap, BitMap> images = this.FetchImages()
.ToDictionary(
// Parameter keySelector: key is original image
image => image,
// Parameter valueSelector: value is the resized image
imgage => ResizeImage(original));
Displaying the resized images will now be much faster: only a Lookup.
public static void DrawOnPanel(Graphics g)
{
var _resizedImage = this.Images[Resources.myImage];
g.DrawImage(_resizedImage, destX, destY);
// ... several other images resized & manipulated then drawn
}
Don't forget to Dispose the bitmaps when your Form is Closed, or at its latest when your Form is Disposed:
protected override void Dispose(bool disposing)
{
if (disposing)
{
foreach (var resizedImage in images.Values)
resizedImage.Dispose();
}
}

C# GDI+ ScaleTransform ok on picturebox but image saved is original

Hi I have the issue that when I use ScaleTransform(zoomFactor,zoomFactor) the image saved on disk is the original version always, while on screen in the picturebox the image is distorted in proportion to the zoomFactor.
Why this could be happening ? Shouldn't I have the final result as applied from e.Graphics on disk written image ?
My code is the following which is a version with matrix. but the instead of matrix I have used the ScaleTransform as well. Result is always the same:
g=e.Graphics;//inside picturebox_paint()
g.ScaleTransform(ratio * zoomFac, ratio * zoomFac);
e.Graphics.DrawImage((Bitmap)bmp, 0, 0);
int seed = Convert.ToInt32(Regex.Match(Guid.NewGuid().ToString(), #"\d+").Value);
String destinationFile = #"C:\tmp\photoid\" + new Random(seed).Next() + "_conv.jpg";
//Here I get always the original image back!!!!
bmp.Save(destinationFile);
I have used as well the following idiom but with same results:
//Matrix matrix = new Matrix();
//matrix.Scale(zoomFac, zoomFac);
//e.Graphics.Transform = matrix;
You need to make the PictureBox draw the things it shows on screen into a new Bitmap, which you then can save!
As it is the Image will be saved in the original form and nothing you did in the Paint event, which actually painst onto the surface of the PictureBox will be saved.
So to save everything, i.e. The Image, possibly a BackgroundImage and all you draw in the Paint event you would call DrawToBitmap somehwere.
Somewhere means somewhere else, not in the Paint event, as it will call the Paint event to create the new Bitmap, causing an endless loop..
To call it you would do something like this:
Bitmap bmpSave = new Bitmap(pictureBox1.ClientSize.Width, pictureBox1.ClientSize.Height);
pictureBox1.DrawToBitmap(bmpSave, pictureBox1.ClientRectangle);
But maybe this is not really what you want? Maybe you actually want to modify the Image? In that case do not use the Paint event at all!
Instead do something like this:
Bitmap bmpSave = new Bitmap(yourNewWidth, yourNewHeight);
using (Graphics g = Graphics.FromImage(bmpSave))
{
g.ScaleTransform(ratio * zoomFac, ratio * zoomFac);
g.DrawImage((Bitmap)pictureBox1.Image, 0, 0); //
pictureBox1.Image = bmpSave;
bmpSave.Save(...);
}
You could call this from somewhere where the scaling is being triggered from.
Note that doing the scaling repeatedly and each time from the previoulsy scaled version will degrade the quality rather fast. For this always scale from a saved version of the original!!
Btw: Using a Matrix for scaling doesn't really make a difference over ScaleTransform.
But if you want to do a direct scaling why not use the DrawImage overload which takes two Rectangles? This is the most common solution if all you want to to scale and maybe draw other stuff additionally..:
int newWidth = 100; int newHeight = 100; string yourFileName = "D:\\xyz123.jpg";
Bitmap bmpSave = new Bitmap(pictureBox1.ClientSize.Width, pictureBox1.ClientSize.Height);
Rectangle newRectangle = new Rectangle(0, 0, newWidth, newHeight);
Rectangle oldRectangle = new Rectangle(Point.Empty, pictureBox1.Image.Size);
using (Graphics g = Graphics.FromImage(bmpSave))
{
g.DrawImage((Bitmap)pictureBox1.Image, newRectangle, oldRectangle, GraphicsUnit.Pixel);
bmpSave.Save(yourFileName, ImageFormat.Jpeg);
}
And there there is the scaling Bitmap constructor:
Bitmap bmp = new Bitmap(pictureBox1.Image, newWidth, newHeight);
Which I would recommend if all you want is to scale the Image. As the other solutions it will not change the Image displayed until you assign it back into the PictureBox..:
pictureBox1.Image = bmp ;
Don't forget to dispose of the old Image..
Been a while since I messed with GDI but I think you need to copy back to the Bitmap here.
g.DrawImage(bmp, scaledwidth, scaledheight);
Try something like that before bmp.Save
Edit
Apologies for not seeing that you were copying back to the bitmap. Perhaps the overload which specifies the output rectangle is what you need. Try a DrawImage overload which has the destination Rect. https://msdn.microsoft.com/en-us/library/ms142040(v=vs.110).aspx

Draw directly to PictureBox

I'm developing a screen-sharing app which runs a loop constantly and receives small frames from a socket. The next step is to draw them into the PictureBox.
Of course, I use thread because I don't want to freeze the ui.
This is my code:
Bitmap frame = byteArrayToImage(buff) as Bitmap;//a praticular bitmap im getting from a socket.
Bitmap current = (Bitmap)pictureBox1.Image;
var graphics = Graphics.FromImage(current);
graphics.DrawImage(frame, left, top);//left and top are two int variables of course.
pictureBox1.Image = current;
But now I'm getting an error:
Object is already in use elsewhere.
in this line var graphics = Graphics.FromImage(current);
Tried to Clone it, Create a New Bitmap(current). still no success.
Invalidate() your PictureBox so it redraws itself:
Bitmap frame = byteArrayToImage(buff) as Bitmap;
using (var graphics = Graphics.FromImage(pictureBox1.Image))
{
graphics.DrawImage(frame, left, top);
}
pictureBox1.Invalidate();
If you need it to be thread safe, then:
pictureBox1.Invoke((MethodInvoker)delegate {
Bitmap frame = byteArrayToImage(buff) as Bitmap;
using (var graphics = Graphics.FromImage(pictureBox1.Image))
{
graphics.DrawImage(frame, left, top);
}
pictureBox1.Invalidate();
});

How to know what is the length of an edge in a resized image?

Lets say I want to click on an image and enlarge the part above where I clicked. I do that with the simplest code, but at the bottom of the image there is an annoying edge because of the enlargement.
There is no Graphics involved, so I can't use SetWrapMode(WrapMode.TileFlipXY); or graphic.CompositingMode = CompositingMode.SourceCopy;
I can crop the annoying edge part (its not important) but I don't know what is its height.
How can I know what is its height so I can crop it? Better yet - is there a better method to enlarge the image (I need a method that returns a bitmap so I can save it later)?
The simple code:
private void button1_Click(object sender, EventArgs e)
{
try
{
Bitmap img = new Bitmap(imageLoc);
Rectangle cropArea = new Rectangle(0, 0, img.Width, yLoc);
Bitmap bmpImage = new Bitmap(img);
Bitmap bmpCrop = bmpImage.Clone(cropArea, bmpImage.PixelFormat);
pictureBox1.Image = resizeImage(bmpCrop, new Size(bmpCrop.Width, newHeight));
}
catch
{
}
}
public static Bitmap resizeImage(Image imgToResize, Size size)
{
return new Bitmap(imgToResize, size);
}
private void pictureBox1_MouseClick(object sender, MouseEventArgs e)
{
xLoc = e.X;
yLoc = e.Y;
}
The annoying edge (look at the bottom):
I think this is casued by the resizing algorithm using a bicubic or linear interpolation. These alhgorithms calculate pixels value after resizing by sampling the values of the surrounding pixels and interpolating them. In the case of the original edge pixels, they dont have a value for the pixel below them, so tend towards transparency (i.e. get lighter) since the average of the surrounding pixels takes into account that their next pixel down is missing.
To eliminate sizing artefacts like this you probably need to investigate resizing using the Graphics object and the various quality and algorithm settings available through that.

Graphics.DrawImage speed

In my program, I'm coding a basic image editor. Part of this allows the user to draw a rectangular region and I pop up a display that shows that region zoomed by 3x or so (which they can adjust further with the mouse wheel). If they right click and drag this image, it will move the zoom region around on the original image, basically acting as a magnifying glass.
The problem is, I'm seeing some serious performance issues even on relatively small bitmaps. If the bitmap showing the zoomed region is around 400x400 it's still updating as fast as mouse can move and is perfectly smooth, but if I mouse wheel the zoom up to around 450x450, it immediately starts chunking, only down to around 2 updates per second, if that. I don't understand why such a small increase incurs such an enormous performance problem... it's like I've hit some internal memory limit or something. It doesn't seem to matter the size of the source bitmap that is being zoomed, just the size of the zoomed bitmap.
The problem is that I'm using Graphics.DrawImage and a PictureBox. Reading around this site, I see that the performance for both of these is typically not very good, but I don't know enough about the inner workings of GDI to improve my speed. I was hoping some of you might know where my bottlenecks are, as I'm likely just using these tools in poor ways or don't know of a better tool to use in its place.
Here are some snippets of my mouse events and related functions.
private void pictureBox_MouseDown(object sender, MouseEventArgs e)
{
else if (e.Button == System.Windows.Forms.MouseButtons.Right)
{
// slide the zoomed part to look at a different area of the original image
if (zoomFactor > 1)
{
isMovingZoom = true;
// try saving the graphics object?? are these settings helping at all??
zoomingGraphics = Graphics.FromImage(displayImage);
zoomingGraphics.CompositingQuality = System.Drawing.Drawing2D.CompositingQuality.HighSpeed;
zoomingGraphics.InterpolationMode = System.Drawing.Drawing2D.InterpolationMode.Low;
zoomingGraphics.SmoothingMode = System.Drawing.Drawing2D.SmoothingMode.HighSpeed;
zoomingGraphics.PixelOffsetMode = PixelOffsetMode.HighSpeed;
}
}
}
private void pictureBox_MouseMove(object sender, MouseEventArgs e)
{
if (isMovingZoom)
{
// some computation on where they moved mouse ommitted here
zoomRegion.X = originalZoomRegion.X + delta.X;
zoomRegion.Y = originalZoomRegion.Y + delta.Y;
zoomRegionEnlarged = scaleToOriginal(zoomRegion);
// overwrite the existing displayImage to prevent more Bitmaps being allocated
createZoomedImage(image.Bitmap, zoomRegionEnlarged, zoomFactor, displayImage, zoomingGraphics);
}
}
private void createZoomedImage(Bitmap source, Rectangle srcRegion, float zoom, Bitmap output, Graphics outputGraphics)
{
Rectangle destRect = new Rectangle(0, 0, (int)(srcRegion.Width * zoom), (int)(srcRegion.Height * zoom));
outputGraphics.DrawImage(source, destRect, srcRegion, GraphicsUnit.Pixel);
if (displayImage != originalDisplayImage && displayImage != output)
displayImage.Dispose();
setImageInBox(output);
}
// sets the picture box image, as well as resizes the window to fit
void setImageInBox(Bitmap bmp)
{
pictureBox.Image = bmp;
displayImage = bmp;
this.Width = pictureBox.Width + okButton.Width + SystemInformation.FrameBorderSize.Width * 2 + 25;
this.Height = Math.Max(450, pictureBox.Height) + SystemInformation.CaptionHeight + SystemInformation.FrameBorderSize.Height * 2 + 20;
}
private void pictureBox_MouseUp(object sender, MouseEventArgs e)
{
else if (e.Button == System.Windows.Forms.MouseButtons.Right)
{
if (isMovingZoom)
{
isMovingZoom = false;
zoomingGraphics.Dispose();
}
}
}
As you can see, I'm not declaring a new Bitmap every time I want to draw something, I'm reusing an old Bitmap (and the Bitmap's graphics object, though I don't know if there is much cost with calling Graphics.FromImage repeatedly). I tried adding Stopwatches around to benchmark my code, but I think DrawImage passes functionality to another thread so the function claims to be done relatively quickly. I'm trying to Dispose all my Bitmap and Graphics objects when I'm not using them, and avoid repeated calls to allocate/deallocate resources during the MouseMove event. I'm using a PictureBox but I don't think that's the problem here.
Any help to speed up this code or teach me what's happening in DrawImage is appreciated! I've trimmed some excess code to make it more presentable, but if I've accidentally trimmed something important, or don't show how I'm using something which may be causing problems, please let me know and I'll revise the post.
The way I handle issues like that is when receiving the Paint event, I draw the whole image to a memory bitmap, and then BLT it to the window.
That way, all visual flash is eliminated, and it looks fast, even if it actually is not.
To be more clear, I don't do any painting from within the mouse event handlers.
I just set up what's needed for the main Paint handler, and then do Invalidate.
So the painting happens after the mouse event completes.
ADDED: To answer Tom's question in a comment, here's how I do it. Remember, I don't claim it's fast, only that it looks fast, because the _e.Graphics.DrawImage(bmToDrawOn, new Point(0,0)); appears instantaneous. It just bips from one image to the next.
The user doesn't see the window being cleared and then repainted, thing by thing.
It gives the same effect as double-buffering.
Graphics grToDrawOn = null;
Bitmap bmToDrawOn = null;
private void DgmWin_Paint(object sender, PaintEventArgs _e){
int w = ClientRectangle.Width;
int h = ClientRectangle.Height;
Graphics gr = _e.Graphics;
// if the bitmap needs to be made, do so
if (bmToDrawOn == null) bmToDrawOn = new Bitmap(w, h, gr);
// if the bitmap needs to be changed in size, do so
if (bmToDrawOn.Width != w || bmToDrawOn.Height != h){
bmToDrawOn = new Bitmap(w, h, gr);
}
// hook the bitmap into the graphics object
grToDrawOn = Graphics.FromImage(bmToDrawOn);
// clear the graphics object before drawing
grToDrawOn.Clear(Color.White);
// paint everything
DoPainting();
// copy the bitmap onto the real screen
_e.Graphics.DrawImage(bmToDrawOn, new Point(0,0));
}
private void DoPainting(){
grToDrawOn.blahblah....
}

Categories