I am developing C# WPF Auto Number Plate Recognition Using an OCR.
The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.
The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).
Can anyone suggest solution and better implementation.
//This is the queue where it will hold the frames
// produced from the video streaming(video_Newfram1)
private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();
//I am using AForg.Video to read the MJPEG Streaming
//this event will be triggered for every frame
private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
{
var frameDataAnpr = new Bitmap(eventArgs.Frame);
AnprCam1.Source = GetBitmapimage(frameDataAnpr);
//add current fram to the queue
_anpr1Produces.Enqueue(imgByteAnpr);
//this worker is the consumer that will
//take the frames from the queue to the OCR processing
if (!_workerAnpr1.IsBusy)
{
_workerAnpr1.RunWorkerAsync(imgByteAnpr);
}
}
//This is the consumer, it will take the frames from the queue to the OCR
private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
{
while (true)
{
if (_anpr1Produces.Count <= 0) continue;
BgWorker1(_anpr1Produces.Dequeue());
}
}
//This method will process the frames that sent from the consumer
private void BgWorker1(byte[] imageByteAnpr)
{
var anpr = new cmAnpr("default");
var objgxImage = new gxImage("default");
if (imageByteAnpr != null)
{
objgxImage.LoadFromMem(imageByteAnpr, 1);
if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
{
var vehicleNumber = anpr.GetText();
var vehicleType = anpr.GetType().ToString();
if (vehicleType == "0") return;
var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
vehicleNumber + ".png";
//this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);
}
else
{
GC.Collect();
}
}
}
What you should do is this:
First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.
When it's small, not much changed (i.e: no car drove by).
That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.
That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.
Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.
The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.
Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.
Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.
Related
I am using AForge to capture video from a camera and display it in a WinForms PictureBox. AForge supplies an event that gets triggered on every new frame--so, for my camera, 30 times per second.
I'm finding that even when I am careful to dispose of the PictureBox's image, memory usage climbs rapidly (and in a weird pattern... see image below). My guess is that the event is being called more often than it can dispose of the bitmap, but I'm not sure.
Here is what I have inside the newframe event. I'm using Monitor.TryEnter per this answer, to avoid having events pile up waiting for the PictureBox to unlock. I had hoped that setting the timeout to 1 millisecond would effectively prevent any events from piling up.
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (Monitor.TryEnter(_pictureBox, millisecondsTimeout: 1))
{
try
{
_pictureBox.Image?.Dispose();
_pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(_pictureBox);
}
}
}
This is the memory usage pattern that results when I run the code.
Video capture started
Unknown event causes memory usage to increase
Unknown event causes memory usage to stabilize
Unknown event causes memory usage to increase
And that repeats. Note that if I change the TryEnter timeout to a higher value, the stable memory usage periods get shorter.
If anyone can offer a solution for this memory leak, I would be very grateful. I did see a similar question here, but it wasn't really answered: C# Picturebox memory leak
This is not the perfect way to do this, but I found a workaround that solves the problem of skyrocketing memory usage. I basically just limited the frequency with which the PictureBox image can update.
private const int MAX_UPDATE_TIME_MS = 70; // Effectively limits displayed video frame rate. Without this, events pile up faster than we can dispose of the PictureBox's current image, leading to a HUGE memory leak
private DateTime _lastUpdateTimeStamp = DateTime.Now;
private void Camera_NewFrame(object sender, NewFrameEventArgs e)
{
if (DateTime.Now < _lastUpdateTimeStamp.AddMilliseconds(MAX_UPDATE_TIME_MS))
{
return;
}
else
{
_lastUpdateTimeStamp = DateTime.Now;
try
{
UpdatePictureBoxImage(_pictureBox, e.Frame);
}
catch
{
// Do nothing, else PictureBox could freeze
}
}
}
private void UpdatePictureBoxImage(
PictureBox pictureBox,
Bitmap newImage)
{
if (Monitor.TryEnter(pictureBox, millisecondsTimeout: 1))
{
try
{
pictureBox.Image?.Dispose();
pictureBox.Image = AForge.Imaging.Image.Clone(newImage);
}
finally
{
Monitor.Exit(pictureBox);
}
}
}
Additional note specific to AForge
When stopping the video stream, AForge says you should do something like the StopVideoCapture() method I wrote below. I found that the memory AForge was using to control the camera, which was not insignificant, did not get deallocated in a timely way. So I added GC.Collect() to a new method called DisposeVideoSource() that I run when I'm done with the camera.
public void StopVideoCapture()
{
while (_videoCaptureDevice.IsRunning)
{
_videoCaptureDevice.SignalToStop();
_videoCaptureDevice.WaitForStop();
}
}
public void DisposeVideoSource()
{
StopVideoCapture();
GC.Collect();
}
Before adding GC.Collect():
After adding GC.Collect():
I am making a small application that shows a live webcam feed in a windows form, and also stores watermarked images to drive at a specified interval (Creating a timelapse video is the end goal).
I am using the AForge library for image and video processing.
I have problems were there seems to be a memory leak, even though i try to make sure to use "using" statements at every location where image processing occurs.
Below is the code were the image processing takes place (The NewFrame event)
private void Video_NewFrame(object sender, NewFrameEventArgs eventArgs)
{
try
{
if (ImageProcessing) // If the previous frame is not done processing, let this one go
return;
else
ImageProcessing = true;
using (Bitmap frame = (Bitmap)eventArgs.Frame)
{
// Update the GUI picturebox to show live webcam feed
Invoke((Action)(() =>
{
webcam_PictureBox.Image = (Bitmap)frame.Clone();
}));
// During tests, store images to drive at a certain interval
if (ImageStoreTimer.Elapsed.TotalSeconds > ImageStoreTime)
{
DateTime dt = DateTime.Now;
using (Graphics graphics = Graphics.FromImage(frame))
{
PointF firstLocation = new PointF(frame.Width / 2, frame.Height / 72);
PointF secondLocation = new PointF(frame.Width / 2, frame.Height / 15);
StringFormat drawFormat = new StringFormat();
drawFormat.Alignment = StringAlignment.Center;
using (Font arialFont = new Font("Arial", 15))
{
graphics.DrawString(dt.ToString(), arialFont, Brushes.Red, firstLocation, drawFormat);
graphics.DrawString(Pressure.ToString("F0") + " mbar", arialFont, Brushes.Red, secondLocation, drawFormat);
}
}
// Place images in a folder with the same name as the test
string filePath = Application.StartupPath + "\\" + TestName + "\\";
// Name images by number 1....N
string fileName = (Directory.GetFiles(filePath).Length + 1).ToString() + ".jpeg";
frame.Save(filePath + fileName, ImageFormat.Jpeg);
ImageStoreTimer.Restart();
}
}
//GC.Collect(); <----- I dont want this
}
catch
{
if (ProgramClosing == true){}
// Empty catch for exceptions caused by the program being closed incorrectly
else
throw;
}
finally
{
ImageProcessing = false;
}
}
Now, when running the program, i see memory usage going up and down, usually it gets to about 900MB before dropping. But occasionally it will rise to 2GB. Occasionally, i even get an out of memory exception at this line:
Graphics graphics = Graphics.FromImage(frame)
So after spending an hour or so to trying to reshape the code and looking for my memory leak, i at last tried the GC.Collect line that is commented out in the code (Shame). After that, my memory usage stays constant, at less than 60MB. And i can run the program for 24 hours without any problems.
So i read a bit about GC.Collect, and how bad it is, for example that it could take a lot of processing power to do it to often in a program. But when i compare the CPU power used by my program, it does not really change regardless if i comment the line out or leave it. But the memory problem is gone if i collect at the end of the new frame event.
I would like to find a solution to my problem that does not involve the GC.collect function, as i know it is bad programming practice and i should instead find the underlying problem source.
Thank you all in advance!
I'm not good with win forms but I think that this line:
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Will leave previous image undisposed, which leaks memory (unmanaged memory hold by Bitmap). Since Bitmap has finalizer - it will be reclaimed by GC at some future time (or when you call GC.Collect), but as you already understand - it's not a good practice to rely on GC in such case. So try to do it like this instead:
if (webcam_PictureBox.Image != null)
webcam_PictureBox.Image.Dispose();
webcam_PictureBox.Image = (Bitmap)frame.Clone();
Reasonable comment by Larse: it might be better to not dispose image while it's still being assigned to PictureBox.Image, because who knows, maybe PictureBox control does anything with old image when you are assigning a new one. So alternative is then:
var oldImage = webcam_PictureBox.Image;
webcam_PictureBox.Image = (Bitmap)frame.Clone();
if (oldImage != null)
oldImage.Dispose();
This worked well for us, https://stackoverflow.com/a/70914235/10876657
before that, we tried using statements and all sorts but one solution might work on one machine, but not on the other, but by referencing the image currently set in picturebox and disposing of it afterwords has worked well, not entirely sure why picture.image is not disposed of automatically when a new image is set (frustrating!) but hey, at least this simple workaround exists
I'm writing a simple optical measurement application using Xamarin for Android and the OpenCV C# bindings library.
In an effort to separate the frame grabber from the processing, I've created some blocking collections to pass around raw, and then processed imagery between different threads. I have an issue where over the period of about 30 seconds, the GUI shows beautifully smooth processed video (15s) down to choppy video (10s), then a crash.
The code below shows the definition of the collections. OnCameraFrame (bottom of the code) shoves each new frame into camframes collection. In OnCreate, I run a task called CamProcessor that takes the frame, does many things, and stuffs it into outframes collection. OnCameraFrame then takes that processed frame and shows it to the GUI. For the purposes of this post and testing, I've completely commented out all my processing, so this issue exists simply by passing raw data through the collections.
One other note is that my collections seem to be running very fast. At no point do I ever have more than 1 frame in there, so it's not an overflow issue (I think).
Can anyone point to why this strategy isn't working well?
BlockingCollection<Mat> camframes = new BlockingCollection<Mat>(10);
BlockingCollection<Mat> outframes = new BlockingCollection<Mat>(10);
public CameraBridgeViewBase mOpenCvCameraView { get; private set; }
protected override void OnCreate(Bundle savedInstanceState)
{
//LayoutStuff
mOpenCvCameraView = FindViewById<CameraBridgeViewBase>(Resource.Id.squish_cam);
Task.Run(() => camProcessor());
}
public void camProcessor()
{
while (!camframes.IsCompleted)
{
Mat frame = new Mat();
try
{
frame = camframes.Take();
}
catch (InvalidOperationException) { }
Mat frameT = frame.T();
Core.Flip(frame.T(), frameT, 1);
Imgproc.Resize(frameT, frameT, frame.Size());
outframes.Add(frameT);
}
}
public Mat OnCameraFrame(CameraBridgeViewBase.ICvCameraViewFrame inputFrame)
{
mRgba = inputFrame.Rgba();
Mat frame = new Mat();
Task.Run(() => camframes.Add(mRgba));
try
{
frame = outframes.Take();
}
catch (InvalidOperationException) { }
return frame;
}
After looking into the Android SDK monitor.bat output, I discovered this was a memory leak. Turns out it's common to the Java openCV wrapper, and is a result of OpenCV's mat heap being much larger than C# expects it to be, so it's not getting garbage collected.
The solution was to append these at every frame grab:
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
GC.WaitForPendingFinalizers();
I have a code to stream mp3 from url. I want to show the current time of the mp3 player in UI. I tried it using WaveOut.GetPosition, but it couldn't work for me.
How can I do that?
My code:
do
{
//..codes to get url stream,to create BufferedWaveProvider
int decompressed = decompressor.DecompressFrame(frame, buffer, 0);
provider.AddSamples(buffer, 0, decompressed);
if (provider.BufferedDuration.Seconds > 2 && waveOut == null)
{
waveOut = new WaveOut();
waveOut.Init(provider);
waveOut.Play();
}
if (waveOut != null)
{
currentTime = (int)(waveOut.GetPosition() * 1d / AvgBytesPerSec);
}
}
while (bytesRead > 0 || waveOut.PlaybackState == PlaybackState.Playing);
I think the only change you need to make to your code is to use waveOut.OutputWaveFormat.AverageBytesPerSecond instead of the AvgBytesPerSec property you are currently using. The IWavePosition interface (which is what you are actually using here) "thinks" in hardware terms, so if you are using a format that has to be converted before the hardware can use it, the hardware byte rate will be different than your source byte rate.
Note that the position returned by GetPosition() is only since playback was last started. If waveOut.Stop() is called, the position is reset to 0 when playback is started again. Mapping the position to the source is up to the caller (which is really simple; just track where you last started playback on the source and add it to the position returned. Buffering makes it more complicated, but still completely doable).
I wrote the original IWavePosition interface & implementations for NAudio. It works great in the project I built it for. :)
did you try the property current time and property position?
mp3Reader = new Mp3FileReader("example.mp3");
waveOut.Init(mp3Reader);
waveOut.Play();
// reposition to five seconds in
mp3Reader.CurrentTime = TimeSpan.FromSeconds(5.0);
so mp3Reader.CurrentTime should give you what you need I think
Hope this will help you
I capture images from a webcam, do some heavy processing on them, and then show the result. To keep the framerate high, i want to have the processing of different frames run in parallel.
So, I have a 'Producer', which captures the images and adds these to the 'inQueue'; also it takes an image from the 'outQueue' and displays it:
public class Producer
{
Capture capture;
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
Emgu.CV.UI.ImageBox screen;
public int frameCounter = 0;
public Producer(Emgu.CV.UI.ImageBox screen, Capture capture, Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject)
{
this.screen = screen;
this.capture = capture;
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
}
public void produce()
{
while (true)
{
lock (lockObject)
{
inQueue.Enqueue(capture.QueryFrame());
if (inQueue.Count == 1)
{
Monitor.PulseAll(lockObject);
}
if (outQueue.Count > 0)
{
screen.Image = outQueue.Dequeue();
}
}
frameCounter++;
}
}
}
There are different 'Consumers' who take an image from the inQueue, do some processing, and add them to the outQueue:
public class Consumer
{
Queue<Image<Bgr, Byte>> inQueue;
Queue<Image<Bgr, Byte>> outQueue;
Object lockObject;
string name;
Image<Bgr, Byte> image;
public Consumer(Queue<Image<Bgr, Byte>> inQueue, Queue<Image<Bgr, Byte>> outQueue, Object lockObject, string name)
{
this.inQueue = inQueue;
this.outQueue = outQueue;
this.lockObject = lockObject;
this.name = name;
}
public void consume()
{
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
}
}
Rest of the important code is this section:
private void Form1_Load(object sender, EventArgs e)
{
Consumer[] c = new Consumer[consumerCount];
Thread[] t = new Thread[consumerCount];
Object lockObj = new object();
Queue<Image<Bgr, Byte>> inQueue = new Queue<Image<Bgr, Byte>>();
Queue<Image<Bgr, Byte>> outQueue = new Queue<Image<Bgr, Byte>>();
p = new Producer(screen1, capture, inQueue, outQueue, lockObj);
for (int i = 0; i < consumerCount; i++)
{
c[i] = new Consumer(inQueue, outQueue, lockObj, "c_" + Convert.ToString(i));
}
for (int i = 0; i < consumerCount; i++)
{
t[i] = new Thread(c[i].consume);
t[i].Start();
}
Thread pt = new Thread(p.produce);
pt.Start();
}
The parallelisation actually works fine, I do get a linear speed increase with each added thread (up to a certain point of course). The problem is that I get artifacts in the output, even if running only one thread. The artifacts look like part of the picture is not in the right place.
Example of the artifact (this is without any processing to keep it clear, but the effect is the same)
Any ideas what causes this?
Thanks
Displaimer: This post isn't supposed to fully describe an answer, but instead give some hints on why the artifact is being shown.
A quick analysis show that the the actifact is, in fact, a partial, vertically mirrored snippet of a frame. I copied it, mirrored, and placed it back over the image, and added an awful marker to show its placement:
Two things immediately come to attention:
The artifact is roughly positioned on the 'correct' place it would be, only that the position is also vertically mirrored;
The image is slightly different, indicating that it may belong to a different frame.
It's been a while since I played around with raw capture and ran into a similar issue, but I remember that depending on how the driver is implemented (or set up - this particular issue happened when setting a specific imaging device for interlaced capture) it may fill its framebuffer alternating between 'top-down' and 'bottom-up' scans - as soon as the frame is full, the 'cursor' reverts direction.
It seems to me that you're running into a race condition/buffer underrun situation, where the transfer from the framebuffer to your application is happening before the full frame is transferred by the device.
In that case, you'd receive a partial image, and the area still not refreshed would show a bit of the previously transferred frame.
If I'd have to bet, I'd say that the artifact may appear on sequential order, not on the same position but 'fluctuating' on a specific direction (up or down), but always as a mirrored bit.
Well, I think the problem is here . The section of code is not guarantee that you will be access by one thread in here between two queue. The image is pop by inQueue is not actually received in order in outQueue
while (true)
{
lock (lockObject)
{
if (inQueue.Count == 0)
{
Monitor.Wait(lockObject);
continue;
}
image = inQueue.Dequeue();
}
// Do some heavy processing with the image
lock (lockObject)
{
outQueue.Enqueue(image);
}
}
Similar to #OnoSendai, I'm not trying to solve the exact problem as stated. I would have to write an app and I just don't have the time. But, the two things that I would change right away would be to use the ConcurrentQueue class so that you have thread-safety. And, I would use the Task library functions in order to create parallel tasks on different processor cores. These are found in the System.Net and System.Net.Task namespaces.
Also, vertically flipping a chunk like that looks like more than an artifact to me. If it also happens when executing in a single thread as you mentioned, then I would definitely re-focus on the "heavy processing" part of the equation.
Good luck! Take care.
You may have two problems:
1) parallism doesn't ensure that images are added to the out queue in the right order. I imagine that displaying image 8 before image 6 and 7 can produce some artifacts. In consumer thread, you have to wait previous consumer have posted its image to the out queue to post next image. Tasks can help greatly for that because of their inherent synchronisation mecanism.
2) You may also have problems in the rendering code.