I capture images from a webcam, do some heavy processing on them, and then show the result. To keep the framerate high, i want to have the processing of different frames run in parallel.
So, I have a 'Producer', which captures the images and adds these to the 'inQueue'; also it takes an image from the 'outQueue' and displays it:
public class Producer
{
Capture capture;
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
Emgu.CV.UI.ImageBox screen;
public int frameCounter = 0;
public Producer(Emgu.CV.UI.ImageBox screen, Capture capture, Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject)
{
this.screen = screen;
this.capture = capture;
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
}
public void produce()
{
while (true)
{
lock (lockObject)
{
inQueue.Enqueue(capture.QueryFrame());
if (inQueue.Count == 1)
{
Monitor.PulseAll(lockObject);
}
if (outQueue.Count > 0)
{
screen.Image = outQueue.Dequeue();
}
}
frameCounter++;
}
}
}
There are different 'Consumers' who take an image from the inQueue, do some processing, and add them to the outQueue:
public class Consumer
{
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
string name;
Image<Bgr, Byte> image;
public Consumer(Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject, string name)
{
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
this.name = name;
}
public void consume()
{
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
}
}
Rest of the important code is this section:
private void Form1_Load(object sender, EventArgs e)
{
Consumer[] c = new Consumer[consumerCount];
Thread[] t = new Thread[consumerCount];
Object lockObj = new object();
Queue<Image<Bgr, Byte>> inQueue = new Queue<Image<Bgr, Byte>>();
Queue<Image<Bgr, Byte>> outQueue = new Queue<Image<Bgr, Byte>>();
p = new Producer(screen1, capture, inQueue, outQueue, lockObj);
for (int i = 0; i < consumerCount; i++)
{
c[i] = new Consumer(inQueue, outQueue, lockObj, "c_" + Convert.ToString(i));
}
for (int i = 0; i < consumerCount; i++)
{
t[i] = new Thread(c[i].consume);
t[i].Start();
}
Thread pt = new Thread(p.produce);
pt.Start();
}
The parallelisation actually works fine, I do get a linear speed increase with each added thread (up to a certain point of course). The problem is that I get artifacts in the output, even if running only one thread. The artifacts look like part of the picture is not in the right place.
Example of the artifact (this is without any processing to keep it clear, but the effect is the same)
Any ideas what causes this?
Thanks
Displaimer: This post isn't supposed to fully describe an answer, but instead give some hints on why the artifact is being shown.
A quick analysis show that the the actifact is, in fact, a partial, vertically mirrored snippet of a frame. I copied it, mirrored, and placed it back over the image, and added an awful marker to show its placement:
Two things immediately come to attention:
The artifact is roughly positioned on the 'correct' place it would be, only that the position is also vertically mirrored;
The image is slightly different, indicating that it may belong to a different frame.
It's been a while since I played around with raw capture and ran into a similar issue, but I remember that depending on how the driver is implemented (or set up - this particular issue happened when setting a specific imaging device for interlaced capture) it may fill its framebuffer alternating between 'top-down' and 'bottom-up' scans - as soon as the frame is full, the 'cursor' reverts direction.
It seems to me that you're running into a race condition/buffer underrun situation, where the transfer from the framebuffer to your application is happening before the full frame is transferred by the device.
In that case, you'd receive a partial image, and the area still not refreshed would show a bit of the previously transferred frame.
If I'd have to bet, I'd say that the artifact may appear on sequential order, not on the same position but 'fluctuating' on a specific direction (up or down), but always as a mirrored bit.
Well, I think the problem is here . The section of code is not guarantee that you will be access by one thread in here between two queue. The image is pop by inQueue is not actually received in order in outQueue
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
Similar to #OnoSendai, I'm not trying to solve the exact problem as stated. I would have to write an app and I just don't have the time. But, the two things that I would change right away would be to use the ConcurrentQueue class so that you have thread-safety. And, I would use the Task library functions in order to create parallel tasks on different processor cores. These are found in the System.Net and System.Net.Task namespaces.
Also, vertically flipping a chunk like that looks like more than an artifact to me. If it also happens when executing in a single thread as you mentioned, then I would definitely re-focus on the "heavy processing" part of the equation.
Good luck! Take care.
You may have two problems:
1) parallism doesn't ensure that images are added to the out queue in the right order. I imagine that displaying image 8 before image 6 and 7 can produce some artifacts. In consumer thread, you have to wait previous consumer have posted its image to the out queue to post next image. Tasks can help greatly for that because of their inherent synchronisation mecanism.
2) You may also have problems in the rendering code.
Related
private void RunEveryTenFrames(Color32[] pixels, int width, int height)
{
var thread = new Thread(() =>
{
Perform super = new HeavyOperation();
if (super != null)
{
Debug.Log("Result: " + super);
ResultHandler.handle(super);
}
});
thread.Start();
}
I'm running this function every 10 frames in Unity. Is this a bad idea. Also, when I try to add thread.Abort() inside the thread, it says thread is not defined and can't use local variable before it's defined error.
Is it a good idea to create a new thread every 10 frames in Unity?
No. 10 frames is too small for repeatedly creating new Thread.
Creating new Thread will cause overhead each time. It's not bad when done once in a while. It is when done every 10 frames. Remember this is not every 10 seconds. It is every 10 frames.
Use ThreadPool. By using ThreadPool with ThreadPool.QueueUserWorkItem, you are re-using Thread that already exist in the System in instead of creating new ones each time.
Your new RunEveryTenFrames function with ThreadPool should look something like this:
private void RunEveryTenFrames(Color32[] pixels, int width, int height)
{
//Prepare parameter to send to the ThreadPool
Data data = new Data();
data.pixels = pixels;
data.width = width;
data.height = height;
ThreadPool.QueueUserWorkItem(new WaitCallback(ExtractFile), data);
}
private void ExtractFile(object a)
{
//Retrive the parameters
Data data = (Data)a;
Perform super = new HeavyOperation();
if (super != null)
{
Debug.Log("Result: " + super);
ResultHandler.handle(super);
}
}
public struct Data
{
public Color32[] pixels;
public int width;
public int height;
}
I you ever need to call into Unity's API or use Unity's API from this Thread, see my other post or how to do that.
I'm trying to create a Screenshot of all Screens on my PC. In the past I've been using the GDI Method, but due to performance issues I'm trying the DirectX way.
I can take a Screenshot of a single Screen without issues, with a code like this:
using Microsoft.DirectX;
using Microsoft.DirectX.Direct3D;
using System.Windows.Forms;
using System.Drawing;
class Capture : Form
{
private Device device;
private Surface surface;
public Capture()
{
PresentParameters p = new PresentParameters();
p.Windowed = true;
p.SwapEffect = SwapEffect.Discard;
device = new Device(0, DeviceType.Hardware, this, CreateFlags.HardwareVertexProcessing, p);
surface = device.CreateOffscreenPlainSurface(Screen.PrimaryScreen.Bounds.Width, Screen.PrimaryScreen.Bounds.Height, Format.A8B8G8R8, Pool.Scratch);
}
public Bitmap Frame()
{
GraphicsStream gs = SurfaceLoader.SaveToStream(ImageFileFormat.Jpg, surface);
return new Bitmap(gs);
}
}
(Lets ignore deleting the Bitmap from memory for this question)
With that Code I can take a Screenshot of my Primary Screen. Changing the first parameter of the Device constructor to a different number corresponds to a different Screen. If I have 3 Screens and I pass 2 as a parameter, I get a Screenshot of my third Screen.
The issue I have is how to handle capturing all Screens. I came up with the following:
class CaptureScreen : Form
{
private int index;
private Screen screen;
private Device device;
private Surface surface;
public Rectangle ScreenBounds { get { return screen.Bounds; } }
public Device Device { get { return device; } }
public CaptureScreen(int index, Screen screen, PresentParameters p)
{
this.screen = screen; this.index = index;
device = new Device(index, DeviceType.Hardware, this, CreateFlags.HardwareVertexProcessing, p);
surface = device.CreateOffscreenPlainSurface(screen.Bounds.Width, screen.Bounds.Height, Format.A8R8G8B8, Pool.Scratch);
}
public Bitmap Frame()
{
device.GetFrontBufferData(0, surface);
GraphicsStream gs = SurfaceLoader.SaveToStream(ImageFileFormat.Jpg, surface);
return new Bitmap(gs);
}
}
class CaptureDirectX : Form
{
private CaptureScreen[] screens;
private int width = 0;
private int height = 0;
public CaptureDirectX()
{
PresentParameters p = new PresentParameters();
p.Windowed = true;
p.SwapEffect = SwapEffect.Discard;
screens = new CaptureScreen[Screen.AllScreens.Length];
for (int i = 0; i < Screen.AllScreens.Length; i++)
{
screens[i] = new CaptureScreen(i, Screen.AllScreens[i], p);
//reset previous devices
if (i > 0)
{
for(int j = 0; j < i; j++)
{
screens[j].Device.Reset(p);
}
}
width += Screen.AllScreens[i].Bounds.Width;
if (Screen.AllScreens[i].Bounds.Height > height)
{
height = Screen.AllScreens[i].Bounds.Height;
}
}
}
public Bitmap Frame()
{
Bitmap result = new Bitmap(width, height);
using (var g = Graphics.FromImage(result))
{
for (int i = 0; i < screens.Length; i++)
{
Bitmap frame = screens[i].Frame();
g.DrawImage(frame, screens[i].Bounds);
}
}
return result;
}
}
As you can see, I iterate though the available Screens and create multiple devices and surfaces in a seperate Class. But calling Frame() of the CaptureDirectX class throws the following error:
An unhandled exception of type 'Microsoft.DirectX.Direct3D.InvalidCallException' occurred in Microsoft.DirectX.Direct3D.dll
At the line
device.GetFrontBufferData(0, surface);
I've been researching this a bit but didn't have a whole lot of success. I'm not really sure what the issue is.
I've found a link that offers a solution that's talking about resetting the Device Objects. But as you can see in my code above, I've been trying to reset all previously created Device objects, sadly without success.
So my questions are:
Is what I'm trying to achieve even possible through this method (i.e. GetFrontBufferData) ?
What am I doing wrong? What am I missing?
Do you see any performance issues when capturing the Screen at a high rate, like say 30 fps? (Capturing a single screen with a target of 30fps gave me a rate of about 25 - 30fps, compared with the GDI methology which sinks to like 15fps sometimes)
FYI it's a WPF application, i.e. .NET 4.5
Edit: I should mention that I'm aware of IDXGI_DesktopDuplication but sadly it doesn't fit my requirements. As far as I know, that API is only available from Windows 8 onwards, but I'm trying to get a solution that works from Windows 7 onwards because of my clients.
Well, in the end the solution was something completely different. The System.Windows.Forms.Screen Class doesn't play nicely with the DirectX Classes. Why? Because the indexes don't match up. The first object in AllScreens does not necessarly have to be index 0 in the Device instatiation.
Now usually this isn't a problem, except when you have a "strange" monitor setup like mine. On the desk I have 3 screens, one vertical (1200,1920), one horizontal (1920, 1200) and another horizontal laptop screen (1920, 1080).
What happened in my case: The first object in AllScreens was the vertical monitor on the left. I try to create a device for index 0, 1200 width and 1920 height. Index 0 corresponds to my main monitor though, i.e. the horizontal monitor in the middle. So I'm essentially going out of the screen bounds with my instatiation. The instatiation doesn't throw an exception and at some point later I try to read the front buffer data. Bam, Exception because I'm trying to take a 1200x1920 screenshot of a monitor that's 1920x1200.
Sadly, even after I got this working, the performance was no good. A single frame of all 3 monitors takes about 300 to 500ms. Even with a single monitor, the execution time was something like 100ms. Not good enough for my usecase.
Didn't get the Backbuffer to work either, it just produces black images.
I went back to the GDI method and enhanced it by only updating specific chunks of the bitmap on each Frame() call. You want to capture a 1920x1200 region, which gets cut into 480x300 Rectangles.
I am developing C# WPF Auto Number Plate Recognition Using an OCR.
The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.
The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).
Can anyone suggest solution and better implementation.
//This is the queue where it will hold the frames
// produced from the video streaming(video_Newfram1)
private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();
//I am using AForg.Video to read the MJPEG Streaming
//this event will be triggered for every frame
private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
{
var frameDataAnpr = new Bitmap(eventArgs.Frame);
AnprCam1.Source = GetBitmapimage(frameDataAnpr);
//add current fram to the queue
_anpr1Produces.Enqueue(imgByteAnpr);
//this worker is the consumer that will
//take the frames from the queue to the OCR processing
if (!_workerAnpr1.IsBusy)
{
_workerAnpr1.RunWorkerAsync(imgByteAnpr);
}
}
//This is the consumer, it will take the frames from the queue to the OCR
private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (_anpr1Produces.Count <= 0) continue;
BgWorker1(_anpr1Produces.Dequeue());
}
}
//This method will process the frames that sent from the consumer
private void BgWorker1(byte[] imageByteAnpr)
{
var anpr = new cmAnpr("default");
var objgxImage = new gxImage("default");
if (imageByteAnpr != null)
{
objgxImage.LoadFromMem(imageByteAnpr, 1);
if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
{
var vehicleNumber = anpr.GetText();
var vehicleType = anpr.GetType().ToString();
if (vehicleType == "0") return;
var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
vehicleNumber + ".png";
//this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);
}
else
{
GC.Collect();
}
}
}
What you should do is this:
First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.
When it's small, not much changed (i.e: no car drove by).
That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.
That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.
Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.
The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.
Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.
Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.
My application is using WPF Control to display many video from Network Cameras.
The summary code is below
public void DisPlayVideoThreadProc()
{
while (isDisplayThreadRunning)
{
Global.mainWindow.Dispatcher.Invoke(new Action(delegate()
{
for (int i = 0; i < numOfCamera; i++)
{
BitmapSource img = bitmapQueue[i].Serve(); //Pop the frame from Queue
ControlDisplay[i].DrawImage(img); //Draw this frame on ControlDisplay[i]
}
}));
}
}
I encountered an problem when the amount of camera is large (> 15 camera), then the UI thread is very slow for user's interaction.
I know the UI thread works heavy when displaying many camera videos. But I don't know how to improve it. Someone can tell me, how to fix this issue.
Many thanks!
Don't draw all camera's in one invoke. This will block the gui thread too long. You'd better call invoke per camera draw. Or atleast in batches of maximum 4.
You might bring the Serve() method out of the invoke and store it in a dictionary and update it with a DispatcherTimer.
PSEUDO:
// hold the camera images.
public class CameraImage
{
public bool Updated {get; set; }
public BitmapSource Image {get; set; }
}
// cache
private Dictionary<int, CameraImage> _cameraCache = new Dictionary<int, CameraImage>();
// thread method to get the images.
while (isDisplayThreadRunning)
{
for (int i = 0; i < numOfCamera; i++)
{
BitmapSource img = bitmapQueue[i].Serve(); //Pop the frame from Queue
lock(_cameraCache)
{
CameraImage currentCameraImage;
if(!_cameraCache.TryGetValue(i, out currentCameraImage))
{
_cameraCache.Add(i, currentCameraImage = new CameraImage());
}
currentCameraImage.Image = img;
currentCameraImage.Updated = true;
}
}
}
// index cycler
private int _index;
// display timer.
public void DispatcherTimeMethod()
{
lock(_cameraCache)
{
CameraImage currentCameraImage;
if(_cameraCache.TryGetValue(_index, out currentCameraImage))
if(currentCameraImage.Updated)
{
ControlDisplay[_index].DrawImage(currentCameraImage.Image);
currentCameraImage.Updated = false;
}
}
_index++;
if(_index >= MAXCAMERAS)
_index = 0;
}
If the camera's (all together) will generate too many images, it will automatically skip images.
Currently you are updating all your cameras in a single thread, the UI thread. This makes the UI thread always freeze, even if you don't notice it.
What I'd recommend is using Parallel.For to update the camera feeds on (multiple) separate threads, then using the UI dispatcher to update the image on the UI.
Something like this:
while ( isDisplayThreadRunning ) {
//start a new parallel for loop
Parallel.For( 0, numOfCamera, num => {
BitmapSource img = bitmapQueue[i].Serve(); //Pop the frame from Queue
//draw the new image on the UI thread
Global.mainWindow.Dispatcher.Invoke(
new Action( delegate
{
ControlDisplay[i].DrawImage( img ); //Draw this frame on ControlDisplay[i]
} ) );
} );
Thread.Sleep( 50 );//sleep if desired, lowers CPU usage by limiting the max framerate
}
}
We have such a situation. We have a canvas, on which some ammount of figures are rendered. It may be 1 or many more (for example thousand) and we need to animate their translation to another location (on button click) using storyboard:
internal void someStoryBoard(figure someFigure, double coordMoveToValue)
{
string sbName = "StoryBoard_" + figure.ID;
string regName = "figure_" + figure.ID;
try
{
cnvsGame.Resources.Remove(sbName);
cnvsGame.UnregisterName(regName);
}
catch{ }
someCanvas.RegisterName(regName, someFigure.Geometry);
var moveFigureYAnimation = new PointAnimation();
moveFigureYAnimation.From = new Point(someFigure.Geometry.Center.X, someFigure.Geometry.Center.Y);
moveFigureYAnimation.To = new Point(someFigure.eGeometry.Center.X, coordMoveToValue);
moveFigureYAnimation.Duration = TimeSpan.FromSeconds(0.5);
var sbFigureMove = new Storyboard();
Storyboard.SetTargetName(sbFigureMove, regName);
Storyboard.SetTargetProperty(sbFigureMove, new PropertyPath(Geometry.CenterProperty));
sbFigureMove.Children.Add(moveFigureYAnimation);
cnvsGame.Resources.Add(sbName, sbFigureMove);
sbFigureMove.Begin();
}
Figures are stored in list. We are calling this StoryBoard using for loop:
for(int i = 0; i<listOfFigures.Count; i++)
{
someStoryBoard(listOfFigures[i], someCoord);
}
But here's the problem: if we have a little amount of figures - code completes quickly. But if ammount is big - there is a delay after a button is clicked and before the figures begin to move.
So, here's the question: is it possible to call someStoryBoard method asynchronously? Is next algorithm possible -> When someStoryBoard is called it begins to move figure instantly, not waiting for whole for loop to complete.?
You can add actions into Dispatcher queue by calling Dispatcher.InvokeAsync. You can also specify dispatcher priority, depending on your requirements.
Please note that moving thousands of items can't be reliably fast, so you may need to rethink the drawing logic. If even starting animation is slow, it's highly likely animating won't be fast enough too.
You can try use async/await modifier
async internal Task someStoryBoard(figure someFigure, double coordMoveToValue)