I have written a recursive directory traversal method in C# (hosted from an asp.net page). The code works as I intended (I enumerate a list of shares on a target machine then recurse through the shares and add each file/directory to a TreeView). Unfortunately this consumes an extreme amount of memory and takes a very long time to run, opening the aspx page causes the Webdev.Webserver ram usage to spike to 800 megabytes, and the Chrome instance viewing the page consumes a whopping 1.5GB of RAM! (running the test code against SMB shares hosted on my local workstation) I can't even view the page source without chrome hanging.
foreach (TreeNode n in FileSelectList.Nodes)
{
Dir_Node_Recurse(n, hostName);
//break;
}
Uncommenting out the //break; statement results in only the first directory share being processed, and this consumes far less memory. FileSelectList is an Asp:TreeView.
public static void Dir_Node_Recurse(TreeNode node, string hostName)
{
DirectoryInfo dir = new DirectoryInfo(String.Format(#"\\{0}\{1}",
hostName,
node.ValuePath.ToString()
));
TreeNode tNode;
foreach (var i in dir.EnumerateDirectories())
{
tNode = new TreeNode(i.Name.ToString());
node.ChildNodes.Add(tNode);
Dir_Node_Recurse(tNode, hostName);
}
foreach (var i in dir.EnumerateFiles())
{
node.ChildNodes.Add(new TreeNode(i.Name.ToString()));
}
}
This appears to cause extreme resource usage because of the large number of TreeNode objects being created. Should I create my own node type to perhaps minimize memory usage, or is there another technique that would make this usable?
Is there a reason you need to get all the nodes? Can you use an on demand approach?
You can also profile the code. You can try pointing the code to a smaller directory and observe it's behavior.
What do you want to do?
You are creating a huge page and asking how to make it consume less memory? That's obvious – don't show all the tree in the page, it's never going to be useful to any user anyway.
You can limit the output to only several levels, for example.
Related
I am generating a large Report pdf file using PDFClown using data from a database.
The process takes a very long time and eventually runs out of memory when the number of pages nears the 150 mark taking up more than 1.5GB of ram and with the error:
A first chance exception of type 'System.OutOfMemoryException' occurred in PDFClown.dll
Since I will need to regularly generate reports of over 1500 pages this is a major problem.
Is there anything I can do to:
Not run out of memory (Necessary)
Speed up the file creation (Ideally)
Please note: the reports generated (with smaller data sets) are accurate, although the file size is rather large.
Here is a sample of my code:
protected void PopulateReport()
{
foreach (Page page in _lstPages)
{
if (page != _Titlepage)
{
PrimitiveComposer composer = new PrimitiveComposer(page);
BlockComposer blockComposer = new BlockComposer(composer);
DataRow drInspection;
if (_mapPage1Rows.TryGetValue(page, out dataRow))
{
GeneratePage1(page, composer, blockComposer, dataRow);
}
else if (_mapPage2Rows.TryGetValue(page, out dataRow))
{
GeneratePage2(page, composer, blockComposer, dataRow);
}
}
}
}
protected void GeneratePage1()
{
composer.BeginLocalState();
composer.SetFont(ReportFonts.GetFont(GetPDFDocument(), bIsFieldName, false), nFontSize);
blockComposer.Begin(GetRectangle(fMarginX, fMarginY, fWidth, nFontSize), AlignX, AlignY);
int nIndex = blockComposer.ShowText(strText, false);
blockComposer.End();
....
composer.End();
composer.Flush();
}
Screenshot Sample Report Page (redacted for client-privacy reasons):
The Function: ReportFonts.GetFont(...) was creating a new font every single time it was called.
This font was then stored in the dll's memory space and the final file which was what was taking up so much space.
Used a Dictionary<> to solve the issue, not only is the memory space clean and the file size acceptable, but the execution time is also much improved.
Moving to 64-bit also helped somewhat.
I have an application that consumes large amounts of RAM that I deploy to users. Some of my users are running into out of memory exception when running it - and I am noticing this is because they have their system page file turned off (because who would use 16GB of memory these days? sigh...). I want to detect if user has set this to off (or maybe some other settings) so I can warn them, because we have a lot of users come to us for support and I want to automate out some of the users because they are eating up lots of our time.
I have googled around and I can't seem to find a way to get information about page file. Specifically, I am talking about information you can see in this page in windows:
I know this is our end users problem and has nothing to do with our application (our app is designed to use up a good chunk of memory and gets a significant speed benefit). I am unsure how to detect these kinds of settings - does anyone have an idea?
You'll need to add reference to System.Management beforehand.
AllocatedBaseSize will show the current page file size in MB
using (var query = new ManagementObjectSearcher("SELECT AllocatedBaseSize FROM Win32_PageFileUsage"))
{
foreach (ManagementBaseObject obj in query.Get())
{
uint used = (uint)obj.GetPropertyValue("AllocatedBaseSize");
Console.WriteLine(used);
}
}
While MaximumSize will show the maximum page file size in MB, if the user set the maximum size (if the system managed it, the query won't return anything).
using (var query = new ManagementObjectSearcher("SELECT MaximumSize FROM Win32_PageFileSetting"))
{
foreach (ManagementBaseObject obj in query.Get())
{
uint max = (uint)obj.GetPropertyValue("MaximumSize");
Console.WriteLine(max);
}
}
If the AllocatedBaseSize is less than what your app will use and the MaximumSize is large enough for your app (or it's system managed), you'll need to consider the edge case where the storage is not enough for Windows to grow the page file. Even if there is enough space in the beginning, user could be downloading a large file on other program or rendering a large video while running your app. Consider offering 'low storage' mode where your app may run slower but don't consume as much memory.
Whilst I don't have a complete working solution for you, I think the information you are after can be retrieved from the Win32_PageFileUsage WMI class. The AllocatedBaseSize property should contain the information you are after:
AllocatedBaseSize
Data type: uint32
Access type: Read-only
Qualifiers:
MappingStrings ("Win32API|MEMORYSTATUS|dwTotalPageFile"), units
("megabytes")
Actual amount of disk space allocated for use with this
page file. This value corresponds to the range established in
Win32_PageFileSetting under the InitialSize and MaximumSize
properties, set at system startup. Example: 178
public bool IsPagingEnabled
{
get
{
var pagingFileStrings = (string[])Registry.GetValue(#"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management", "PagingFiles", null);
if (pagingFileStrings == null)
return false;
foreach (var pagingFile in pagingFileStrings)
if (pagingFile != null && !string.IsNullOrEmpty(pagingFile))
return true;
return false;
}
}
I am using named pipe to share some data between 2 processes in windows. One is a node process and other is a C# process. Here is a sample of code I use in my node process:
var net = require('net');
var PIPE_NAME = "mypipe";
var PIPE_PATH = "\\\\.\\pipe\\" + PIPE_NAME;
var L = console.log;
var server = net.createServer(function(stream) {
L('Server: on connection')
stream.on('data', function(c) {
L('Server: on data:', c.toString());
});
stream.on('end', function() {
L('Server: on end')
server.close();
});
stream.write('Take it easy!');
});
server.on('close',function(){
L('Server: on close');
})
server.listen(PIPE_PATH,function(){
L('Server: on listening');
})
I use a NamedPipeClientStream in c# to read the data. I do this in a loop on both the sides, such as my node process is a producer and C# process is a consumer.
This works fine.
But sometimes the C# loop hangs and at that point in my node process I want to overwrite the new data over the old data. I was wondering if I can specify some max size in my pipe (the one I create in nodejs) or a timeout for the data but couldn't find such things in standard documentation.
If it cannot be solved this way, there is a shared memory route to solve the problem but I couldn't find any stable shared memory library for nodejs which works nicely on windows (and I don't have much time to write one right now). I need some pointers to move in the right direction.
Any advice is appreciated. Thanks.
EDIT: I would really want to implement the above stuff using shared memory since I need to share large amount of data at a fast rate and I need to tweak for performance. Any pointers on how to implement it?
I figured out a way to use the drain event in writable stream of nodejs as per my requirement.
My goal is to load 1.48 million items from XML files into a in memory generic collections dictionary using C# ASP.NET MVC along with ability to render filter lists to views. Right now it works with about 1/2 million items in my dictionary as a single user in VS debug mode. Any more I get out of memory errors - right now I get: myDictionary.Count results in System.TypeInitializationException - 'The function evaluation was disabled because of an out of memory exception.'
In the past it has also been mscorlib.dll during the load from XML to Dict that has complained about out of memory.
I have plenty of system memory left, how can I give more to this MVC Web application? BTW, I tried XML to a Perl Dictionary and it works just fine with 1.5 million objects - no problem. I don't wish to write my app in Perl. If Perl can do it, C# has to be able to do it - I just have not been able to find a solution searching the web yet.
Example Code:
Dictionary<string, msg> masterDictionary = new Dictionary<string, mgs>();
foreach (string file in filePath)
{
XDocument xdoc = XDocument.Load(file);
Dictionary<string, msg> fileDictionary = xdoc
.Descendants("msg")
.ToDictionary(m => m.Element("msgId").Value,
m => new msg
{
msgId = m.Element("msgId").Value,
msgType = m.Element("msgType").Value,
name = m.Element("name").Value
});
//now insert your new values into the master
foreach(var newValue in fileDictionary)
masterDictionary.Add(newValue.Key, newValue.Value);
}
This DID FIX the problem. I'm still limited to 2 GB of memory from within VS 2013 in debug mode. However, when deploying to IIS7.5, 2008 R2 Server, and App. Pool of dot net 4.0x, I can use a lot more memory for my website. It took the following setting in my web.config. I now need to bump my physical from 8 to 16 GBs; But, that is a different story. Solution:
<gcAllowVeryLargeObjects enabled="true" />
https://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx
Between BeginUpdate() and EndUpdate(), Garbage collection fixed the problem for me in 32 and 64 bit.
GC.Collect();
GC.WaitForPendingFinalizers();
I'm currently working on an application which runs on Windows Mobile 6.1 (not WP). I built an application which synchronizes data from a remote server multiple times a day. But somehow it looks like this data is "remembered" after finishing. Task Manager shows that about 3MB is used at a regular start of the application, which increases with about 2MB everytime I run the synchronization. After multiple times I get a warning of the memory usage and I have to reset the device or restart the program.
What I'm looking for is some way to clear data after synchronization, a kind of garbage collector. In (regular) C# I've found Collect(), but I can't get this working in C# mobile.
Below is my code, which is working correctly, except at a certain point I get the message "Geheugentekort" ("Memory shortage").
Probably after the for{} code, I have to empty variables like doc, root, and the XmlNodeList, but the question is how...
My device: Pidion BIP-5000
OS: Windows Mobile 6.1
XmlDocument doc = new XmlDocument();
doc.Load(xmlUrl);
XmlElement root = doc.DocumentElement;
try
{
totaal = Int32.Parse(doc.GetElementsByTagName("Totaal")[0].InnerText.ToString());
// Create lists with values
XmlNodeList namen = doc.GetElementsByTagName("naam");
XmlNodeList ptypen = doc.GetElementsByTagName("ptype");
XmlNodeList ids = doc.GetElementsByTagName("id");
// Door het totaal heen itereren
for (int i = 0; i < totaal; i++)
{
// Create variables of it
int id = Int32.Parse(ids[i].InnerText.ToString());
int ptype = Int32.Parse(ptypen[i].InnerText.ToString());
string naam = namen[i].InnerText.ToString();
// Check if ID exists
int tot = this.tbl_klantTableAdapter.GetData(id).Count;
if (tot == 0)
{
// New item, add
this.tbl_klantTableAdapter.Insert(naam, ptype, id);
}
else
{
// Existing, update
this.tbl_klantTableAdapter.Update(naam, ptype, id);
}
}
}
catch
{
// Rest of code
Dispose Your nodelists after the loop may help
System.Xml.XmlNodeList tempNodelist = Your stuff;
IDisposable disposeMe = tempNodelist as IDisposable;
if (disposeMe != null)
{
disposeMe.Dispose();
}
XmlNodeList implements IDisposable, so you can call namen.Dispose() (also for the other XmlNodeList objects) to force the objects to be discarded and cleaned up.
Yes, you definitely should use the XML stuff locally and dispose after using the XML stuff. The xml stuff seems to ocupie large memory blocks.
You should use nameX.Dispose() and nameX=null to free up the memory used for these temporary xml objects.
You may use GC.Collect() to force memory collection: http://blogs.msdn.com/b/stevenpr/archive/2004/07/26/197254.aspx.
You may also use remote .Net performance viewer to get insides on memory usage: http://blogs.msdn.com/b/stevenpr/archive/2006/04/17/577636.aspx
If your app is consuming much memory before calling into the sync task, you may consider of creating a new application with a separate process for the sync stuff. You can also free up memory for your process when you move functions to a library. WM6.1 and higher have a new memory slot for compact Framework libraries, so the main process memory slot is not lowered: http://blogs.msdn.com/b/robtiffany/archive/2009/04/09/memmaker-for-the-net-compact-framework.aspx
If you need more help you should provide more details/code.