We Keep Coding

Emgu cv high resolution images stitching Issue

I am using EmguCV library to stitch images. Its working fine for small images but getting exception when processing high resolution images or images above 20MB size or even if I try to process more than 30 images it fails.
Libraries I am using
Emgu.CV.UI
Emgu.CV.UI.GL
Emgu.CV.World
opencv_core2410
opencv_imgproc2410
Code
List<Image<Bgr, Byte>> sourceImages = new List<Image<Bgr,byte>>();
foreach (string path in ImgPath)
sourceImages.Add(new Image<Bgr, Byte>(path));
using (Stitcher stitcher = new Stitcher(false))
{
using (VectorOfMat vm = new VectorOfMat())
{
Mat result = new Mat();
vm.Push(sourceImages.ToArray());
stitcher.Stitch(vm, result);
if (result.Bitmap != null)
{
result.Bitmap.Save(Application.StartupPath + "\\imgs\\StitchedImage.png");
}
else
{
MessageBox.Show("Some thing went wrong"); return null;
}
}
}
Exception
((Emgu.CV.MatDataAllocator)(result))._dataHandle.Target' threw an exception of type 'System.InvalidOperationException
Image

I was fairly certain that you are running into a memory issue and so I went ahead and made a simple console app targeting .Net 4.7.2, using the latest EmguCV package from NuGet, which is version 3.4.3.3016, and using the code below on sample data from Adobe, which can be downloaded here. If I compile as "AnyCPU" with "prefer 32 bit" and run this code against the rio image set (I loaded up the png's for the test purposes) and let it run the memory will slowly jump up until it hits close to 3 GB and then shortly thereafter crashes giving the exception about the refcount. Pretty clearly a memory issue. I then recompiled targeting 64 bit and was able to successfully run the code. The memory usage peaked out around 6 GB. So, with that in mind, I would be fairly certain that your issue is also memory related. You have yet to answer the question on whether you are building a 64 bit app or not, but based on what you are seeing, I would guess that you are not. So, the solution to your problem is to compile as 64 bit and then be sure you have enough memory. With the rio test set it jumped up to close to 6 GB. Without having your images I can't tell you how large it might grow, but these type of operations are pretty memory intensive - so more is better. This would explain both the issue with large image files and the issue with a large number of small image files. I was able to successfully process sets of images that were between 10 and 20 images using a 32 bit build, but as soon as I moved to the 50+ image sets it would only work with a 64 bit build due to the memory requirements.
var images = Directory.EnumerateFiles(#"C:\test\adobe\rio", "*.png", SearchOption.TopDirectoryOnly).Select(x => new Mat(x)).ToArray();
using(var stitcher = new Stitcher(false))
{
using (var vm = new VectorOfMat(images))
{
var result = new Mat();
stitcher.Stitch(vm, result);
result.Bitmap.Save(#"C:\test\adobe\rio_stitched.jpg", System.Drawing.Imaging.ImageFormat.Jpeg);
result.Dispose();
}
}
foreach (var image in images) { image.Dispose(); }

RDLC Memory Leak

In my application (.NET Framework 4.5) I'm rendering some RDLC reports (50-60) in order to export them to a single PDF.
Unfortunately there seems to be a big memory leak, basically every LocalReportnever gets disposed.
This is my code:
public void ProcessReport(ReportDataSource[] reportDS, string reportPath)
{
const string format = "PDF";
string deviceInfo = null;
string encoding = String.Empty;
string mimeType = String.Empty;
string extension = String.Empty;
Warning[] warnings = null;
string[] streamIDs = null;
Byte[] pdfArray = null;
using (var report = new LocalReport())
{
report.EnableExternalImages = true;
report.ReportEmbeddedResource = reportPath;
report.Refresh();
foreach (var rds in reportDS)
{
report.DataSources.Add(rds);
}
report.Refresh();
try
{
pdfArray = report.Render(format, deviceInfo, out mimeType, out encoding,
out extension, out streamIDs, out warnings);
}
catch (Exception ex)
{
Console.WriteLine(ex.InnerException.Message);
throw;
}
report.ReleaseSandboxAppDomain();
report.Dispose();
//Add pdfArray to MemoryStream and then to PDF - Doesn't leak
}
}
I found the memory leak just by looking to Visual Studio memory panel, every time report.Render get's called it add 20-30mb and they never go down until I close the application. I'm sure that using the MemoryStreamis not the issue because even if commented I still get 200mb-250mb in memory that never get released. This is bad because after running this application like 3-4 times it reaches >1GB until it doesn't even run anymore. I also tried to manually call the GarbageCollector but didn't work. The application is 32 bit.
What can I do to fix this ?

I have a real solution and can explain why!
It turns out that LocalReport here is using .NET Remoting to dynamically create a sub appdomain and run the report in order to avoid a leak internally somewhere. We then notice that, eventually, the report will release all the memory after 10 to 20 minutes. For people with a lot of PDFs being generated, this isn't going to work. However, the key here is that they are using .NET Remoting. One of the key parts to Remoting is something called "Leasing". Leasing means that it will keep that Marshal Object around for a while since Remoting is usually expensive to setup and its probably going to be used more than once. LocalReport RDLC is abusing this.
By default, the leasing time is... 10 minutes! Also, if something makes various calls into it, it adds another 2 minutes to the wait time! Thus, it can randomly be between 10 and 20 minutes depending how the calls line up. Luckily, you can change how long this timeout happens. Unluckily, you can only set this once per app domain... Thus, if you need remoting other than PDF generation, you will probably need to make another service running it so you can change the defaults. To do this, all you need to do is run these 4 lines of code at startup:
LifetimeServices.LeaseTime = TimeSpan.FromSeconds(5);
LifetimeServices.LeaseManagerPollTime = TimeSpan.FromSeconds(5);
LifetimeServices.RenewOnCallTime = TimeSpan.FromSeconds(1);
LifetimeServices.SponsorshipTimeout = TimeSpan.FromSeconds(5);
You'll see the memory use start to rise and then within a few seconds you should see the memory start coming back down. Took me days with a memory profiler to really track this down and realize what was happening.
You can't wrap ReportViewer in a using statement (Dispose crashes), but you should be able to if you use LocalReport directly. After that disposes, you can call GC.Collect() if you want to be doubly sure you are doing everything you can to free up that memory.
Hope this helps!
Edit
Apparently, you should call GC.Collect(0) after generating a PDF report or else it appears the memory use could still get high for some reason.

C# - How to tell if system has virtual memory / page file on?

I have an application that consumes large amounts of RAM that I deploy to users. Some of my users are running into out of memory exception when running it - and I am noticing this is because they have their system page file turned off (because who would use 16GB of memory these days? sigh...). I want to detect if user has set this to off (or maybe some other settings) so I can warn them, because we have a lot of users come to us for support and I want to automate out some of the users because they are eating up lots of our time.
I have googled around and I can't seem to find a way to get information about page file. Specifically, I am talking about information you can see in this page in windows:
I know this is our end users problem and has nothing to do with our application (our app is designed to use up a good chunk of memory and gets a significant speed benefit). I am unsure how to detect these kinds of settings - does anyone have an idea?

You'll need to add reference to System.Management beforehand.
AllocatedBaseSize will show the current page file size in MB
using (var query = new ManagementObjectSearcher("SELECT AllocatedBaseSize FROM Win32_PageFileUsage"))
{
foreach (ManagementBaseObject obj in query.Get())
{
uint used = (uint)obj.GetPropertyValue("AllocatedBaseSize");
Console.WriteLine(used);
}
}
While MaximumSize will show the maximum page file size in MB, if the user set the maximum size (if the system managed it, the query won't return anything).
using (var query = new ManagementObjectSearcher("SELECT MaximumSize FROM Win32_PageFileSetting"))
{
foreach (ManagementBaseObject obj in query.Get())
{
uint max = (uint)obj.GetPropertyValue("MaximumSize");
Console.WriteLine(max);
}
}
If the AllocatedBaseSize is less than what your app will use and the MaximumSize is large enough for your app (or it's system managed), you'll need to consider the edge case where the storage is not enough for Windows to grow the page file. Even if there is enough space in the beginning, user could be downloading a large file on other program or rendering a large video while running your app. Consider offering 'low storage' mode where your app may run slower but don't consume as much memory.

Whilst I don't have a complete working solution for you, I think the information you are after can be retrieved from the Win32_PageFileUsage WMI class. The AllocatedBaseSize property should contain the information you are after:
AllocatedBaseSize
Data type: uint32
Access type: Read-only
Qualifiers:
MappingStrings ("Win32API|MEMORYSTATUS|dwTotalPageFile"), units
("megabytes")
Actual amount of disk space allocated for use with this
page file. This value corresponds to the range established in
Win32_PageFileSetting under the InitialSize and MaximumSize
properties, set at system startup. Example: 178

public bool IsPagingEnabled
{
get
{
var pagingFileStrings = (string[])Registry.GetValue(#"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management", "PagingFiles", null);
if (pagingFileStrings == null)
return false;
foreach (var pagingFile in pagingFileStrings)
if (pagingFile != null && !string.IsNullOrEmpty(pagingFile))
return true;
return false;
}
}

LocalReport rendering causing OutOfMemory exceptions

I have a WindowsService that converts XML files to a PDF via LocalReports (RDLC reports). It works very well and is fast, but I'm struggling with memory leak problems similar to those here.
It appears the .Render("PDF") method is where the problems are appearing. A profiling session in VS2015 shows a huge about of memory used by ServerIdentity. This doesn't appear if I comment out .Render("PDF").
Specs
Application is .NET 4.5
Application uses ReportViewer library from VS2012 (I've tried 2015, too, and same behavior)
Built for 'Any CPU', runs as x86 presumably
Reports aren't huge, but we run quite a few so after a few weeks memory gets used up (1 page reports, maybe a dozen or more a day).
Code
public static byte[] GenerateReport(string name, IEnumerable<ReportDataSource> dataSources)
{
try
{
byte[] pdfBytes;
using (Stream reportStream = Assembly.GetExecutingAssembly().GetManifestResourceStream(name))
{
LocalReport localReport = new LocalReport();
localReport.LoadReportDefinition(reportStream);
foreach (var dataSource in dataSources)
{
localReport.DataSources.Add(dataSource);
}
pdfBytes = localReport.Render("PDF");
//Cleanup!
localReport.DataSources.Clear();
localReport.ReleaseSandboxAppDomain();
localReport.Dispose();
}
return pdfBytes;
}
catch (Exception ex)
{
throw new Exception("Error generating report at " + name, ex);
}
}
Other notes
If I never call .Render("PDF"), the memory never grows beyond 20MB or so.
Similarly, if i never render, there's no ServerIdentity or Microsoft.ReportingServices.OnDemanProcessing.*` objects in the heap. This is where 90%+ of the memory is consumed but I have no way to GC that memory.

Change the compilation target of your application to x64 and call
report.ReleaseSandboxAppDomain();
After Render method. Remember to Dispose the report object.

Recursive directory traversal/tree consumes extreme amounts of memory

I have written a recursive directory traversal method in C# (hosted from an asp.net page). The code works as I intended (I enumerate a list of shares on a target machine then recurse through the shares and add each file/directory to a TreeView). Unfortunately this consumes an extreme amount of memory and takes a very long time to run, opening the aspx page causes the Webdev.Webserver ram usage to spike to 800 megabytes, and the Chrome instance viewing the page consumes a whopping 1.5GB of RAM! (running the test code against SMB shares hosted on my local workstation) I can't even view the page source without chrome hanging.
foreach (TreeNode n in FileSelectList.Nodes)
{
Dir_Node_Recurse(n, hostName);
//break;
}
Uncommenting out the //break; statement results in only the first directory share being processed, and this consumes far less memory. FileSelectList is an Asp:TreeView.
public static void Dir_Node_Recurse(TreeNode node, string hostName)
{
DirectoryInfo dir = new DirectoryInfo(String.Format(#"\\{0}\{1}",
hostName,
node.ValuePath.ToString()
));
TreeNode tNode;
foreach (var i in dir.EnumerateDirectories())
{
tNode = new TreeNode(i.Name.ToString());
node.ChildNodes.Add(tNode);
Dir_Node_Recurse(tNode, hostName);
}
foreach (var i in dir.EnumerateFiles())
{
node.ChildNodes.Add(new TreeNode(i.Name.ToString()));
}
}
This appears to cause extreme resource usage because of the large number of TreeNode objects being created. Should I create my own node type to perhaps minimize memory usage, or is there another technique that would make this usable?

Is there a reason you need to get all the nodes? Can you use an on demand approach?
You can also profile the code. You can try pointing the code to a smaller directory and observe it's behavior.

What do you want to do?
You are creating a huge page and asking how to make it consume less memory? That's obvious – don't show all the tree in the page, it's never going to be useful to any user anyway.
You can limit the output to only several levels, for example.