I'm working on a project using EmguCV which requires capturing images from a camera and processing them, looking for certain things in the image. The processing takes up about ~.2 seconds, and is the biggest time-sink in the application so we're looking into making the capturing/processing different threads to speed up the overall process.
We've already tried the VideoCapture.ImageGrabbed event handler, and calling Retrieve, as well as setting up a threaded loop of our own calling QueryFrame and any other capturing method we can find.
Most of the threaded solutions end up causing empty images (not caught by the .IsEmpty property of the Mat, though, for some reason), which end up as images saved with 0 bytes.
After this, we tried simplifying but ran into an issue where the camera is always a few seconds behind, due to the buffer internal to the library. This leads to my question: is there a way to refresh the buffer, or clear it of memory? We cannot dispose of the capture object because of the time overhead for creating the object. Any tips about threading the capturing and processing are welcome as well, though the other suggestions I've found around this site about similar situations have not led to much success.
A brief example of the code we're using
_capture.ImageGrabbed += GrabFrames;
...
public void GrabFrames()
{
Mat image = new Mat();
_capture.Retrieve(image);
ProcessThread = new Thread(new ParameterizedThreadStart(StartProcess));
ProcessThread.Start(image);
}
...
public void StartProcess(object image)
{
Mat img = (Mat)image;
Process(image);
img.Dispose();
}
Related
Here is the basic idea of my code:
private void CaptureCameraFrame()
{
Capture = new VideoCapture();
CameraModel.Instance.CameraViewMat = Capture.QueryFrame();
// do stuff with queried matrix here
if(noAbortCondition)
{
CaptureCameraFrame();
}
}
The method should run in a separate thread updating my GUI with the current image after processing.
Only Problem is, that I get two different kinds of error:
Attempt to read/write protected memory: This happens on the second runthrough
of the method.
I get an null-reference error using `CameraModel.Instance.CameraViewMat right after querying the frame.
The two issues seem to be connected, seems like QueryFrame() runs asynchronously from the rest of the code and isn't done when the program jumps to the next step.
Question is: How can I make sure, that querying the image from the camera is finished, and I can use the information in the matrix as well as start a new query?
In all the examples I have found this is done by using time, but I would like to start with a new frame as soon as processing on the last frame is done.
I haven't really done much in C# when it comes to threading, but what I understand in such cases one would use the asyncand awaitkeywords to make sure a method in an asynchronous method is finished. However I wasn't able to make a working implementation in this case.
You are creating VideoCapture class instance repeatedly and even not disposing of it. Create your VideoCapture instance only once and use them for your task. At the end dispose it.
public YourConstructor()
{
Capture = new VideoCapture();
}
private void CaptureCameraFrame()
{
CameraModel.Instance.CameraViewMat = Capture.QueryFrame();
// do stuff with queried matrix here
if(noAbortCondition)
{
CaptureCameraFrame();
}
}
Hopefully, it will work for you!
I have an extensive image calculation task which uses about 1GB of memory (one calculation cycle takes about 4 seconds). I process those images automatically when they arrive in the folder using a FileSystemWatcher. When the FileSystemWatcher fires an event for a new file I queue the work in the eventhandler method with:
private void OnNewFileInDir(object source, FileSystemEventArgs evtArgs)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessTheNewImage), evtArgs.FullPath);
}
My problem is that the program crashes on a regular basis when the files arrive quickly. In the debug window I can see that neary 3GB memory are used in that moment. When I use smaller images in order to use less memory, there are no crashes (so far).
My question: What can I do to use less (maybe just 2) threads independent of cores of my computer?
Or is my approach of using a FileSystemWatcher to cue new files to a thread pool completely stupid? I am not at all experienced with thread races or similar things.
So, furthermore: Does that look threadsafe?
Thanks a lot upfront and all the best
Tim
For completeness here is the code executed by the threads (a bit simplified for ease of reading):
private void ProcessTheNewImage(object threadFilenameInfo)
{
String filename = (String)threadFilenameInfo;
// Load the image
Image currentImage = Image.FromFile(filename);
//Calculate the image in an external DLL
Image currentResultImage = ImageProcessing.getResultImage(currentImage);
//Create the filename with the result infos
string saveFileName = "blahblah";
//Save the image
currentResultImage.Save(saveFileName);
//dispose the images
currentImage.Dispose();
currentResultImage.Dispose();
}
The Threadpool has only a very limited form of resource management. It will slowly keep adding threads when the queue fills up. It is meant for relatively small (< 500 ms) jobs. There is no safety-valve to stop it from clogging up your application.
You could create a workflow for this: the watcher event pushes simple datapackets into a ConcurrentQueue and then you create 2 or more Threads (better: Tasks) to process the queue. This will allow you to tune the number of threads.
I have two high speed USB3 cameras (Ximea) and want to code an application for image recording. Framerates are up to 500fps at VGA resolution but I also want to use the 2Mpx resolution at 170fps.
Their .Net SDK tells me that I should simply "get" the images in a loop.
My problem is that I have no idea how to get the images and save them while still showing a live preview. Everytime I add some code to actually update the picturebox the frame rate drops drastically.
At the moment I utilize a recording function that is called with
Task.Run(() => Record());
and inside the Record() I have a loop getting the bitmaps
while(record == true)
{
Camera.GetImage(out myImage, timeout); //From Ximea .Net SDK
Info = Camera.GetLastImageParams();
Timestamp = Info.GetTimeStamp();
ThreadPool.QueueUserWorkItem(state => SaveImage(myImage, filepath, Timestamp));
}
with the SaveImage being
private void SaveImage(Bitmap myImage, string filepath, double Timestamp)
{
try
{
lock(myImage)
{
myImage.Save(filepath + Timestamp.ToString("0.00000") + ".tif");
}
}
catch{}
}
How can I show a live preview while recording and how can I make the entire code more stable (at the moment there are some dropped frames because of "object already in use"-errors or "generic error in GDI+" at the Image.Save() call, that I skip with the try/catch statement)?
I believe you can tell the Ximea API how many image buffers you want in the incoming queue... use XI_PRM_BUFFER_POLICY and XI_PRM_BUFFERS_QUEUE_SIZE appropriately to make that queue length somewhat long.
Then, have a thread that, when activated, copies an image out of a XI_IMG struct into your own buffer. Activate that thread every n frames (based on the size of the Ximea image buffer queue size)... but don't do any memory copies in the loop that actually calls xiGetImage. You should probably block in your thread to avoid tearing (because the Ximea code could get back around to using the same buffer again if you're not fast enough at copying the data out)... but you could then dynamically adjust the number of buffers so you can finish your copy within the time you have. Also, you may consider copying the image data to ANOTHER buffer if you're doing something that takes a long time...
pseudo-code (sorry, it's C-ish):
// sync objects and a global image buffer pointer
CRITICAL_SECTION cs;
void *buf;
HANDLE ev;
int CopyImageThreadProc(...)
{
while (true)
{
if (WaitOnSingleObject(ev) == WAIT_OBJ_0)
{
EnterCriticalSection(cs);
// copy the image data at buf where ever you want
LeaveCriticalSection(cs);
}
}
}
int main(...)
{
// set up ximea api with appropriate buffering
// create event and critsec, start thread
while (!done)
{
XI_IMG img;
xiGetImage(dev, 10, &img);
// every 15 frames, tell your thread to go...
// if you find that the critsec is causing a hiccup, you can adjust this
// but remember to adjust the queue length, too
// if you change this to TRY entercriticalsection, you can determine that
if ((img.acq_nframe % 15) == 0)
{
EnterCriticalSection(cs);
buf = img.bp;
SetEvent(ev);
LeaveCriticalSection(cs);
}
}
// clean up
}
Add each captured frame to a queue, then have a worker thread that takes those images, one at a time, and saves them. Trying to write multiple images to disk at the same time will most likely be slower. Also, always Dispose of any GDI objects or you will run into trouble really fast. I would think not doing that is what is giving you the exceptions.
As for showing the images, make sure you are not trying to display every image. Your monitor most likely runs at 60 Hz, so anything faster than that will be a waste. I also suspect that (with the performance of GDI), you won't necessarily be able to achieve even that. So I suggest you have a second queue with images to display, and if you see that queue getting too large, your program will need to slow down a bit and not push as many frames to the queue.
Edit: And of course, as #Franck mentions, if your disk can't keep up, your queue/buffer will fill up quickly. Compressing the images might help, assuming they have suitable content for compression and that your processor can keep up.
Edit: What you need is a producer-consumer pattern. There are many ways of doing this, but one might be something like:
// blocking collection
private BlockingCollection<Bitmap> m_Queue = ...
// camera thread
while( run )
{
var bitmap = GrabFrame();
m_Queue.Add( bitmap );
}
// worker thread
try
{
while( true )
{
// Take() will block if the queue is empty
var bitmap = m_Queue.Take();
bitmap.Save( ... );
bitmap.Dispose();
}
catch( InvalidOperationException )
{
// you'll end up here if you call `m_Queue.CompleteAdding()`
// (after the queue has been emptied, of course)
}
As for displaying the images, you could probably use something similar, with the addition of having some code that determines if it's time to push a new image or not.
I have a Windows Phone 8 App, which uses Background agent to:
Get data from internet;
Generate image based on a User Control that uses the data from step 1 as data source;
In the User Control I have Grid & StackPanel & some Text and Image controls;
When some of the Images use local resources from the installation folder (/Assets/images/...)
One of them which I used as a background is selected by user from the phone's photo library, so I have to set the source using C# code behind.
However, when it runs under the background, it get the OutOfMemoryException, some troubleshooting so far:
When I run the process in the "front", everything works fine;
If I comment out the update progress, and create the image directly, it also works fine;
If I don't set the background image, it also works fine;
The OutOfMemoryException was thrown out during var bmp = new WriteableBitmap(480, 800);
I already shrink the image size from 1280*768 to 800*480, I think it is the bottom line for a full screen background image, isn't it?
After some research, I found out this problem occurs because it exceeded the 11 MB limitation for a Periodic Task.
I tried use the DeviceStatus.ApplicationCurrentMemoryUsage to track the memory usageļ¼
-- the limitation is 11,534,336 (bit)
-- when background agent started, even without any task in it, the memory usage turns to be 4,648,960
-- When get update from internet, it grew up to 5,079,040
-- when finished, it dropped back to 4,648,960
-- When the invoke started (to generate image from the User Control), it grew up to 8,499,200
Well, I guess that's the problem, there is little memory available for it to render the image via WriteableBitmap.
Any idea how to work out this problem?
Is there a better method to generate an image from a User Control / or anything else?
Actually the original image might only be 100 kb or around, however, when rendering by WriteableBitmap, the file size (as well as the required memory size I guess) might grew up to 1-2MB.
Or can I release the memory from anywhere?
==============================================================
BTW, when this Code Project article says I can use only 11MB memory in a Periodic Task;
However, this MSDN article says that I can use up to 20 MB or 25MB with Windows Phone 8 Update 3;
Which is correct? And why am I in the first situation?
==============================================================
Edit:
Speak of the debugger, it also stated in the MSDN article:
When running under the debugger, memory and timeout restrictions are suspended.
But why would I still hit the limitation?
==============================================================
Edit:
Well, I found something seems to be helpful, I will check on them for now, suggestions are still welcome.
http://writeablebitmapex.codeplex.com/
http://suchan.cz/2012/07/pro-live-tiles-for-windows-phone/
http://notebookheavy.com/2011/12/06/microsoft-style-dynamic-tiles-for-windows-phone-mango/
==============================================================
The code to generate the image:
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
var customBG = new ImageUserControl();
customBG.Measure(new Size(480, 800));
var bmp = new WriteableBitmap(480, 800); //Thrown the **OutOfMemoryException**
bmp.Render(customBG, null);
bmp.Invalidate();
using (var isf = IsolatedStorageFile.GetUserStoreForApplication())
{
filename = "/Shared/NewBackGround.jpg";
using (var stream = isf.OpenFile(filename, System.IO.FileMode.OpenOrCreate))
{
bmp.SaveJpeg(stream, 480, 800, 0, 100);
}
}
}
The XAML code for the ImageUserControl:
<UserControl blabla... d:DesignHeight="800" d:DesignWidth="480">
<Grid x:Name="LayoutRoot">
<Image x:Name="nBackgroundSource" Stretch="UniformToFill"/>
//blabla...
</Grid>
</UserControl>
The C# code behind the ImageUserControl:
public ImageUserControl()
{
InitializeComponent();
LupdateUI();
}
public void LupdateUI()
{
DataInfo _dataInfo = new DataInfo();
LayoutRoot.DataContext = _dataInfo;
try
{
using (var isoStore = IsolatedStorageFile.GetUserStoreForApplication())
{
using (var isoFileStream = isoStore.OpenFile("/Shared/BackgroundImage.jpg", FileMode.Open, FileAccess.Read))
{
BitmapImage bi = new BitmapImage();
bi.SetSource(isoFileStream);
nBackgroundSource.Source = bi;
}
}
}
catch (Exception) { }
}
When DataInfo is another Class within the settings page that hold data get from the internet:
public class DataInfo
{
public string Wind1 { get { return GetValueOrDefault<string>("Wind1", "N/A"); } set { if (AddOrUpdateValue("Wind1", value)) { Save(); } } }
public string Wind2 { get { return GetValueOrDefault<string>("Wind2", "N/A"); } set { if (AddOrUpdateValue("Wind2", value)) { Save(); } } }
//blabla...
}
If I comment out the update progress, and create the image directly, it also works fine I think you should focus on that part. It seems to indicate some memory isn't freed after the update. Make sure all the references used during the update process go out of scope before rendering the picture. Forcing a garbage collection can help too:
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect(); // Frees the memory that was used by the finalizers
Another thing to consider is that the debugger is also using a lot of memory. Do a real test by compiling your project in "Release" mode and deploying on a phone to make sure you're running out of memory.
Still, I have already been in that situation so I know it may not be enough. The point is: some libraries in the .NET Framework are loaded lazily. For instance, if your update process involves downloading some data, then the background agent will load the network libraries. Those libraries can't be unloaded and will waste some of your agent's memory. That's why, even by freeing all the memory you used during the update process, you won't reach back the same amount of free memory you had when starting the background agent. Seeing this, what I did in one of my app was to span the workload of the background agent across two executions. Basically, when the agents executes:
Check in the isolated storage if there's pending data to be processed. If not, just execute the update process and store all needed data in the isolated storage
If there is pending data (that is, in the next execution), generate the picture and clear the data
It means the picture will be generated only once every hour instead of once every 30 minutes, so use this workaround only if everything else fail.
The larger memory limit is for background audio agents, it clearly states that in the documentation. You're stuck with 11 MB, which can really be a big pain when you're trying to do something smart with pictures in the background.
480x800 adds a MB to your memory because it takes 4 bytes for every pixels, so in the end it's around 1.22MB. When compressed in JPEG, then yes - it makes sense that it's only around 100KB. But whenever you use WriteableBitmap, it gets loaded into memory.
One of the things you could try before forcing the GC.Collect as mentioned in another answer is to null things out even before they go out of scope - whether it's a BitmapImage or a WriteableBitmap. Other than that you can try removing the Image object from the Grid programmatically when you're done and setting the Source of it also to be null.
Are there any other WriteableBitmap, BitmapImage or Image objects you're not showing us?
Also, try without the debugger. I've read somewhere that it adds another 1-2MB which is a lot when you have only 11 MB. Although, if it crashes so quickly with the debugger, I wouldn't risk it even if it suddenly seams OK without the debugger. But just for testing purposes you can give it a shot.
Do you need to use the ImageUserControl? Can you try doing step by step creating Image and all the other objects, without the XAML, so you can measure memory in each and every step to see at which point it goes through the roof?
Currently my program reads images and text in a record from an xml file, displays them on the screen, and then the click of the previous/next buttons moves to the next record. However, it seems to need a few seconds loading time between each photo and I'd like it to be instant, like how Windows Photo Gallery would...or Facebook photos (bear in mind this is not a web app).
I searched found a few similar situations to mine but none seemed to fit my situation. I tried making a class, based on my search, to deal with background loading and calling it in my program, but it's fraught with error and probably won't even do what I want it do:
//ImageManager.cs
class ImageManager
{
private Dictionary<string, Image> images = new Dictionary<string, Image>();
public Image get(string s)
{ // blocking call, returns the image
return load(s);
}
private Image load(string s)
{ // internal, thread-safe helper
lock (images)
{
if (!images.ContainsKey(s))
{
Image img = images.Add(s, img); //load the image s - ERROR cannot implicitly convert type void to image. Void??
return img;
}
return images[s];
}
}
public void preload(params string[] imgs)
{ // non-blocking preloading call
foreach (string img in imgs)
{
BackgroundWorker bw = new BackgroundWorker();
bw.DoWork += (s, e) => { load(img); }; // discard the actual image return
bw.RunWorkerAsync();
}
}
}
//MainWindow.cs
ImageManager im = new ImageManager();
im.preload("Data/Images"); // Errors - im is a field but used like a type/token '('
Many thanks in advance
Your ImageManager should work with ImageSources, not Images. Even if you get your current code to work you'll find that your UI still hangs because you have no choice but to perform the work on the UI thread. If you instead deal with ImageSources, you can load them on a background thread and then freeze them in order to use them from the UI thread. This frees you to pre-emptively load images, or to show a loading animation whilst they load.
BitmapFrame.Create is likely the method you want to be using to load the images.
Consider caching scaled down images - 1:1 of what you want to show, or even smaller. This way loading of preview will be much faster and if user looks at the image long enough you can load full image.
With modern photos original size of the image is usually way bigger than can be normally diaplayed. So if you always read original images you spend large amount of disk IO on something that will never be shown.
Usual note: it may not be case in your program. As with any performance issues measure, than optimize.