I have a windows service written in C#.Net. When the service is started, I spawn a new thread as shown below
new Thread(new ThreadStart(Function1)).Start();
This thread loops infinitely and performs the duties expected of my service. Once a day, I need to simultaneously perform a different operation for which my thread spawns a second thread as show below
new Thread(new ThreadStart(Function2)).Start();
This second thread performs a very simple function. It reads all the lines of a text file using FileReadAllLines , quickly processes this information and exits.
My problem is that the memory used by the second thread which reads the file is not getting collected. I let my service run for 3 hours hoping that the GC would be called but nothing happened and task manager still shows that my service is using 150mb of memory. The function to read and process the text file is a very simple one and I am sure there are no hidden references to the string array containing the text. Could someone shed some light on why this is happening? Could it be possible that a thread spawned by another spawned thread cannot cleanup after itself?
Thanks
Trust the garbage collector and stop worrying. 150 megs is nothing. You aren't even measuring the size of the file in that; most of that will be code.
If you are concerned about where memory is going, start by understanding how memory works in a modern operating system. You need to understand the difference between virtual and physical memory, the difference between committed and allocated memory, and all of that, before you start throwing around numbers like "150 megs of allocated memory". Remember, you have 2000 megs of virtual address space in a 32 bit process; I would not think that a 150 meg process is large by any means.
Like Jon says, what you want to be concerned about is a slow steady rise in private bytes. If that's not happening, then you don't have a memory leak. Let the garbage collector do its job and don't worry about it.
If you are still worried about it good heavens do not use task manager. Get a memory profiler and learn how to use it. Task manager is for inspecting processes by looking down on them from 30000 feet. You need to be using a microscope, not a telescope, to analyze how the process is freeing the bytes of a single file.
If you're using Windows task manager to try to work out the memory used, it's likely to be deceiving you. Memory used by the CLR isn't generally returned to the operating system as far as I'm aware... so you'll potentially still see a high working set, even though most of that memory is then still available to be reused within the process.
If you let the service run for a week, do you see the memory usage climb steadily through the week, or does it just increase in the first day, and then plateau? If so, do you definitely view this as a problem? If so, you may need to put your second task in a separate process.
Related
I have a C# program that runs on an Ubuntu VM as a server using mono 5.10. Every now and then, the program starts using lots of memory until it eventually crashes. It can often take days before such a memory leak occurs, and I haven't been able to reproduce the issue locally.
For the past few days, I've used the mono log profiler to be able to collect heapshots on demand. This has given me a 15Gb+ file. I managed to get a heapshot from when the memory leak occurs, which displays as follows in the Xamarin profiler:
Running mprof-report gives a similarly unhelpful report.
Neither really help debug the problem. Obviously there's something going on with the threadpool, but it seems like there is no way of figuring out what.
Being able to see where the objects are allocated might help, but that requires enabling alloc in the profiler, which is not possible because of the file size it will produce.
What is causing the massive amount of IThreadPoolWorkItems? The app is using 30-40% CPU, so it doesn't seem like tasks are being scheduled more quickly than the CPU can handle.
Is there a way to list the objects referenced by the ThreadPool jobs? That would at least allow me to identify what piece of code is being run so much. mprof-report only shows inverse references as far as I can see, which isn't very helpful as the IThreadPoolWorkItems are obviously owned by the ThreadPool itself.
Update: The tasks aren't (disk) IO bound. While leaking, the reads are in the 10s of kbs per second, which shouldn't be saturating the disk.
On second thought: if there were a lot of tasks scheduled, shouldn't I be seeing lots of individual IThreadPoolWorkItem objects too? Since IThreadPoolWorkItem isn't a struct, there should be a separate entry for the actual object and IThreadPoolWorkItem[] would just be an array of pointers. So this would suggest that those IThreadPoolWorkItem[]s are just arrays of (mostly) null pointers.
Update: ThreadPool.GetAvailableThreads shows 2 worker threads and 0 completion port threads are active, which seems to indicate that the ThreadPool is doing just fine. Custom logging also indicates that once again, I'm not scheduling too many tasks and the tasks that are scheduled are finished quickly.
I'm developing a C#, UWP 10 solution that communicates with a network device using a fast, continual read/write loop. The StreamSocket offered by the API seemed to work great, until I realized that there was a memory leak: there is an accumulation of Task<uint32> in the heap, in the order of hundreds per minute.
Whether I use a plain old while (true) loop inside an async Task, or using a self-posting ActionBlock<T> with TPL Dataflow (as per this answer), the result is the same.
I'm able to isolate the problem further if I eliminate reading from the socket and focus on writing:
Whether I use the DataWriter.StoreAsync approach or the more direct StreamSocket.OutputStream.WriteAsync(IBuffer buffer), the problem remains. Furthermore, adding the .AsTask() to these makes no difference.
Even when the garbage collector runs, these Task<uint32>'s are never removed from the heap. All of these tasks are complete (RanToCompletion), have no errors or any other property value that would indicate a "not quite ready to be reclaimed".
There seems to be a hint to my problem on this page (a byte array going from the managed to unmanaged world prevents release of memory), but the prescribed solution seems pretty stark: that the only way around this is to write all communications logic in C++/CX. I hope this is not true; surely other C# developers have successfully realized continual high-speed network communictions without memory leaks. And surely Microsoft wouldn't release an API that only works without memory leaks in C++/CX
EDIT
As requested, some sample code. My own code has too many layers, but a much simpler example can be observed with this Microsoft sample. I made a simple modification to send 1000 times in a loop to highlight the problem. This is the relevant code:
public sealed partial class Scenario3 : Page
{
// some code omitted
private async void SendHello_Click(object sender, RoutedEventArgs e)
{
// some code omitted
StreamSocket socket = //get global object; socket is already connected
DataWriter writer = new DataWriter(socket.OutputStream);
for (int i = 0; i < 1000; i++)
{
string stringToSend = "Hello";
writer.WriteUInt32(writer.MeasureString(stringToSend));
writer.WriteString(stringToSend);
await writer.StoreAsync();
}
}
}
Upon starting up the app and connecting the socket, there is only instance of Task<UInt32> on the heap. After clicking the "SendHello" button, there are 86 instances. After pressing it a 2nd time: 129 instances.
Edit #2
After running my app (with tight loop send/receive) for 3 hours, I can see that there definitely is a problem: 0.5 million Task instances, which never get GC'd, and the app's process memory rose from an initial 46 MB to 105 MB. Obviously this app can't run indefinitly.
However... this only applies to running in debug mode. If I compile my app in Release mode, deploy it and run it, there are no memory issues. I can leave it running all night and it is clear that memory is being managed properly.
Case closed.
there are 86 instances. After pressing it a 2nd time: 129 instances.
That's entirely normal. And a strong hint that the real problem here is that you don't know how to interpret the memory profiler report properly.
Task sounds like a very expensive object, it has a lot of bang for the buck and a thread is involved, the most expensive operating system object you could ever create. But it is not, a Task object is actually a puny object. It only takes 44 bytes in 32-bit mode, 80 bytes in 64-bit mode. The truly expensive resource is not owned by Task, the threadpool manager takes care of it.
That means you can create a lot of Task objects before you put enough pressure on the GC heap to trigger a collection. About 47 thousand of them to fill the gen #0 segment in 32-bit mode. Many more on a server, hundreds of thousands, its segments are much bigger.
In your code snippet, Task objects are the only objects you actually create. Your for(;;) loop does therefore not nearly loop often enough to ever see the number of Task objects decreasing or limiting.
So it is the usual story, accusations of the .NET Framework having leaks, especially on these kind of essential object types that are used heavily in server-style apps that run for months, are forever highly exaggerated. Double-guessing the garbage collector is always tricky, you typically only gain confidence by in fact having your app running for months and never failing on OOM.
I would create and close the DataWriter within the for .
We are designing a Stress Test Application which will send mass HTTP requests of size "1 MB" to a particular Web Service. To achieve stress, we are using multiple threads in the application. Structure is something like we have X EnqueueThreads which will create the HTTPRequest data and they will add it them to the queue. And the Y WorkerThreads will dequeue the requests and they will submit to web service.
All requests are aysnchronous.
Now the problem here is, Enqueue threads are working much faster than WorkerThreads so if there is no stop/wait condition, they will add the requests until the out of memory exception is thrown and thus making the injector (where this utility will be running) slow.
At present we are handling the OutOfMemory exceptions and making the enqueuethreads sleep for some time. Another way I could think of is limit the queue size.
However I would like to know the views on what should be the best approach to use the limited system resources (specially memory).?
Thanks in advance.
You can and probably should use the MemoryFailPoint class in a scenario like this.
If you get an OutOfMemoryException then the application state could be corrupt and you shouldn't try to recover from it. MemoryFailPoint is designed to avoid this by allowing you to determine how much to slow your application down so that you can avoid getting out of memory. You are letting the framework determine if you can perform the operation and not taking a guess on how much you "think" you can get away with based on how much memory your app is using.
You should also check for memory usage through the garbage collector not the process to get an accurate reading of how much managed memory is actually allocated. Using private memory size will give you a much lower reading and you could still land up in an out of memory situation although it appears you have plenty to spare.
The code sample on the MSDN page shows how to estimate memory usage for an operation and to use that information to wait until memory is available before trying to process more requests. If you can determine the areas of code that have large memory requirements this is a good option to constrain it and avoid running out of memory.
Well, according to the topic of the question, best way to avoid out of memory exception would be not to create objects that fill in that memory.
Handling exception is easiest solution, though may bring different difficulties and inconsistencies into the application with time. Another way would be getting available size of memory resources like this:
Process currentProcess = Process.GetCurrentProcess();
long memorySize = currentProcess.PrivateMemorySize64
Then you can calculate the length of your queue based on estimate of one object memory capacity.
Another way would be to check for memory size in each worker thread. Whenever there's no memory, the thread can just finish. This way many threads would be spawning and dying but the application would be at maximum available capacity.
recently I record several hours of .NET Memory counters of a WCF service. The service is hosted in IIS on a Win2k8, 8 core, x64 with 20GB ram.
I can see the GC being pretty healthy, performing a full collection only approx. every 2hrs!
I noticed that in the very same time period, the number of physical and logical threads increases. When the full collection occurs, the number of physical and logical threads drops back and continues to raise again to the same level.
Why does the GC full collection cycle and the drop in threads correlate?
Why is the number of threads continuously increasing?
This is pure ASP.NET/WCF threading model. No custom threads being spawned etc.
Thanks,
Alex
It looks like the answer was indeed pretty simple: Threads get created and die as the thread pool is grown and shrinked over time. Normally, those threads have a relatively long life-time which means they end up in generation 2. This is why they naturally only cleaned up on a full GC cycle.
The interesting bit is that a very well memory managed application can end up with loads of dead threads which I was able to observe in WinDbg. This is because a Full GC might only happen after several hours or even days! Some MS blog said that those dead threads are not a really nice thing to have as the corresponding unmanaged C++ thread object seems to be quite resource intensive (cannot find the damn blog anymore unfortunatly....). I am wondering if this is one of the rare cases where an induced Full GC would make sense in a certain time interval.
I have a windows service multithreaded application for indexing purpose which have six threads. It is working fine except memory leakage. Actually when the service is started, then the service is consuming 12,584kb memory, after some time it is taking memory of 61,584 kb. But after indexing process is complete it is not releasing memory.
I need it come back to its previous position after the indexing is complete, that is it should take the memory with which it started e.g. 12,584kb in this case.
I have used garbage collection but it is not doing what I want.
Can anyone please help me?
First advice would be to throw a memory profiler at it. I've always been happy with Red Gate's ANTS profiler, which will help identify which objects are leaking, if any.
Do bear in mind that there may be first time initialisation happening all over the place, so you probably want to track it over time
.NET doesn't release memory to satisfy people staring at Task Manager. Allocating memory is expensive. The CLR is designed to do so sparingly, and to hold onto any memory it allocates as long as possible. Think of it this way--what's the point of having 4gb of memory when you're not using half of it?
Unless you actually KNOW that you have a memory leak (as in, your app crashes after two days of uptime), let me give you a piece of advice... Close Task Manager. Don't optimize for memory before you know you need to. Relax, guy. Everything is fine.
61 MB does not sound out of the ordinary. You will need to let the service run for awhile and monitor its memory usage for a rising trend. If you see the application leveling out at a certain value then there is probably nothing to worry about.
I agree with Will completely - however I offer 2 small pieces of advice:
Don't use GC.Collect. Trust me you are not going to improve the CLR's Garbage Collection algorithm by doing this. In fact objects that are ready to be collected but can't be for some reason will be moved to Tier 2, where they will have to wait even longer before being collected (thereby potentially making any leak you may have worse!)
Check everywhere in your code that you have created some type of stream (usually MemoryStream) and verify that the streams are being closed properly (preferably in a "finally" block)
Here I am using "log4net-1.2.10" for handling exception, "Lucene.net-2.1.0.3" for indexing, here is one "IndexWriter" class for add document in index or delete documents from index.
lock (object)
{
indexWriter.AddDocument(document);// indexWriter object of "IndexWriter" class.
}.
Also we use microsoft message queue for retrieving messages.
I had the same problem until I changed the applcation from STA to MTA:
[MTAThread]
public static void Main()
instead of
[STAThread]
public static void Main()
(I've used Red Gate's ANTS profiler...)