To keep track of performance in our software we measure the duration of calls we are interested in.
for example:
using(var performanceTrack = new PerformanceTracker("pt-1"))
{
// do some stuff
CallAnotherMethod();
using(var anotherPerformanceTrack = new PerformanceTracker("pt-1a"))
{
// do stuff
// .. do something
}
using(var anotherPerformanceTrackb = new PerformanceTracker("pt-1b"))
{
// do stuff
// .. do something
}
// do more stuff
}
This will result in something like:
pt-1 [----------------------------] 28ms
[--] 2ms from another method
pt-1a [-----------] 11ms
pt-1b [-------------] 13ms
In the constructor of PerformanceTracker I start a stopwatch. (As far as I know it's the most reliable way to measure a duration.) In the dispose method I stop the stopwatch and save the results to application insights.
I have noticed a lot of fluctation between the results. To solve this I've already done the following:
Run in release built, outside of visual studio.
Warm up call first, not included in to the statistics.
Before every call (total 75 calls) I call the garbage collector.
After this the fluctation is less, but still not very accurate. For example I have run my test set twice. Both times
See here the results in milliseconds.
Avg: 782.946666666667 981.68
Min: 489 vs 513
Max: 2600 vs 4875
stdev: 305.854933523003 vs 652.343471128764
sampleSize: 75 vs 75
Why is the performance measurement with the stopwatch still giving a lot of variation in the results? I found on SO (https://stackoverflow.com/a/16157458/1408786) that I should maybe add the following to my code:
//prevent the JIT Compiler from optimizing Fkt calls away
long seed = Environment.TickCount;
//use the second Core/Processor for the test
Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2);
//prevent "Normal" Processes from interrupting Threads
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;
//prevent "Normal" Threads from interrupting this thread
Thread.CurrentThread.Priority = ThreadPriority.Highest;
But the problem is, we have a lot of async code. How can I get a reliable performance track in the code? My aim is to discover performance degradation when for example after a check in a method is 10ms slower than before...
I'm debugging a C# application that becomes almost unresponsive after a few days. The application calculates memory/CPU usage every second and displays it in the footer of the main UI.
The cause for the unresponsiveness is because of the time it takes to fetch the RawValue of a PerformanceCounter ("Working Set - Private"). After a couple of days, it takes almost a second to fetch the RawValue, freezing the main UI thread. If I restart my computer, everything is fast again for a few days until it slowly becomes less responsive. If I recompile this application without the PerformanceCounter code (it's open source), it runs normally immediately.
To rule out that it's the application, here's some sample code that does the exact same thing:
static void Main(string[] args)
{
using (var memoryWorkingSetCounter = new PerformanceCounter("Process", "Working Set - Private", Process.GetCurrentProcess().ProcessName, true))
{
while (!Console.KeyAvailable)
{
var memoryWorkingSetSw = new Stopwatch();
memoryWorkingSetSw.Start();
var memoryWorkingSetValue = memoryWorkingSetCounter.RawValue;
memoryWorkingSetSw.Stop();
Console.WriteLine(memoryWorkingSetValue.ToString());
Console.WriteLine(memoryWorkingSetSw.Elapsed.ToString());
Thread.Sleep(TimeSpan.FromSeconds(1));
}
}
Console.Read();
}
I left this running for ~2.5 days and graphed the Elapsed time in milliseconds:
What could cause a performance counter in Windows to become slow over time? Could another app be not cleaning it up? Is there a way to debug which apps are also looking at this performance counter? I'm on Windows 10.
why you declare Stopwatch inside a loop?
remove any new declaration possible inside loops.
when you do so the memory is increasing overtime and you count on garbage collector to do the work
I'm running into an issue when using my V8Engine instance, it appears to have a small memory leak, and disposing of it, as well as forcing the garbage collection doesn't seem to help much. It will eventually throw an AccessViolationException on V8Enging local_m_negine = new V8Engine() claiming a Fatal error in heap setup, Allocation failed - process out of memory and Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Monitoring the program's memory usage through Task manager whilst running confirms that it is leaking memory, around 1000 KB every couple of seconds I think. I suspect it is the variables being declared within the executed script not being collected, or something to do with the GlobalObject.SetProperty method. Calling V8Engine.ForceV8GarbageCollection(), V8Engine.Dispose() and even GC.WaitForPendingFinalizers() & GC.Collect() doesn't prevent this memory being leaked (Although it is worth noting that it seems to leak it slower with these commands in place, and I know I shouldn't use GC but it was there as a last resort to see if it would fix the issue.)
A tangential issue that could also provide a solution is the inability to clear the execution context for V8Engine. I am required to dispose and re-instantiate the engine for each script, which I believe is where the memory leak is happening, otherwise I run into issues where variables have already been declared, causing V8Engine.Execute() to throw an exception saying such.
I can definitely confirm that the memory leak is something to do with the V8Engine Implementation, as running the older version of this program that uses Microsoft.JScript has no such memory leak, and the memory used remains consistent.
The affected code is as follows;
//Create the V8Engine and dispose when done
using (V8Engine local_m_engine = new V8Engine())
{
//Set the Lookup instance as a global object so that the JS code in the V8.Net wrapper can access it
local_m_engine.GlobalObject.SetProperty("Lookup", m_lookup, null, true, ScriptMemberSecurity.ReadOnly);
//Execute the script
result = local_m_engine.Execute(script);
//Please just clear everything I can't cope.
local_m_engine.ForceV8GarbageCollection();
local_m_engine.GlobalObject.Dispose();
}
EDIT:
Not sure how useful this will be but I've been running some memory profiling tools on it and have learnt that after running an isolated version of the original code, My software ends up with a large amount of instances of IndexedObjectList's full of null values (see here: http://imgur.com/a/bll5K). It appears to have one instance of each class for each V8Engine instance that is made, but they aren't being disposed or freed. I cant help but feel like I'm missing a command or something here.
The code I'm using to test and recreate the memory leak that the above implementation causes is as follows:
using System;
using V8.Net;
namespace V8DotNetMemoryTest
{
class Program
{
static void Main(string[] args)
{
string script = #" var math1 = 5;
var math2 = 10;
result = 5 + 10;";
Handle result;
int i = 0;
V8Engine local_m_engine;
while (true)
{
//Create the V8Engine and dispose when done
local_m_engine = new V8Engine();
//Set the Lookup instance as a global object so that the JS code in the V8.Net wrapper can access it
//local_m_engine.GlobalObject.SetProperty("Lookup", m_lookup, null, true, ScriptMemberSecurity.ReadOnly);
//Execute the script
result = local_m_engine.Execute(script);
Console.WriteLine(i++);
result.ReleaseManagedObject();
result.Dispose();
local_m_engine.Dispose();
GC.WaitForPendingFinalizers();
GC.Collect();
local_m_engine = null;
}
}
}
}
Sorry, I had no idea this question existed. Make sure to use the v8.net tag.
Your problem is this line:
result = local_m_engine.Execute(script);
The result returned is never disposed. ;) You are responsible for returned handles. Those handles are struct values, not class objects.
You could also do using (result = local_m_engine.Execute(script)) { ... }
There is a new version released. I am finally resurrecting this project again as I will need it for the FlowScript VPL project - and it now supports .Net Standard as well for cross-platform support!
I've been doing some investigation to see how we can create a multithreaded application that runs through a tree.
To find how this can be implemented in the best way I've created a test application that runs through my C:\ disk and opens all directories.
class Program
{
static void Main(string[] args)
{
//var startDirectory = #"C:\The folder\RecursiveFolder";
var startDirectory = #"C:\";
var w = Stopwatch.StartNew();
ThisIsARecursiveFunction(startDirectory);
Console.WriteLine("Elapsed seconds: " + w.Elapsed.TotalSeconds);
Console.ReadKey();
}
public static void ThisIsARecursiveFunction(String currentDirectory)
{
var lastBit = Path.GetFileName(currentDirectory);
var depth = currentDirectory.Count(t => t == '\\');
//Console.WriteLine(depth + ": " + currentDirectory);
try
{
var children = Directory.GetDirectories(currentDirectory);
//Edit this mode to switch what way of parallelization it should use
int mode = 3;
switch (mode)
{
case 1:
foreach (var child in children)
{
ThisIsARecursiveFunction(child);
}
break;
case 2:
children.AsParallel().ForAll(t =>
{
ThisIsARecursiveFunction(t);
});
break;
case 3:
Parallel.ForEach(children, t =>
{
ThisIsARecursiveFunction(t);
});
break;
default:
break;
}
}
catch (Exception eee)
{
//Exception might occur for directories that can't be accessed.
}
}
}
What I have encountered however is that when running this in mode 3 (Parallel.ForEach) the code completes in around 2.5 seconds (yes I have an SSD ;) ). Running the code without parallelization it completes in around 8 seconds. And running the code in mode 2 (AsParalle.ForAll()) it takes a near infinite amount of time.
When checking in process explorer I also encounter a few strange facts:
Mode1 (No Parallelization):
Cpu: ~25%
Threads: 3
Time to complete: ~8 seconds
Mode2 (AsParallel().ForAll()):
Cpu: ~0%
Threads: Increasing by one per second (I find this strange since it seems to be waiting on the other threads to complete or a second timeout.)
Time to complete: 1 second per node so about 3 days???
Mode3 (Parallel.ForEach()):
Cpu: 100%
Threads: At most 29-30
Time to complete: ~2.5 seconds
What I find especially strange is that Parallel.ForEach seems to ignore any parent threads/tasks that are still running while AsParallel().ForAll() seems to wait for the previous Task to either complete (which won't soon since all parent Tasks are still waiting on their child tasks to complete).
Also what I read on MSDN was: "Prefer ForAll to ForEach When It Is Possible"
Source: http://msdn.microsoft.com/en-us/library/dd997403(v=vs.110).aspx
Does anyone have a clue why this could be?
Edit 1:
As requested by Matthew Watson I've first loaded the tree in memory before looping through it. Now the loading of the tree is done sequentially.
The results however are the same. Unparallelized and Parallel.ForEach now complete the whole tree in about 0.05 seconds while AsParallel().ForAll still only goes around 1 step per second.
Code:
class Program
{
private static DirWithSubDirs RootDir;
static void Main(string[] args)
{
//var startDirectory = #"C:\The folder\RecursiveFolder";
var startDirectory = #"C:\";
Console.WriteLine("Loading file system into memory...");
RootDir = new DirWithSubDirs(startDirectory);
Console.WriteLine("Done");
var w = Stopwatch.StartNew();
ThisIsARecursiveFunctionInMemory(RootDir);
Console.WriteLine("Elapsed seconds: " + w.Elapsed.TotalSeconds);
Console.ReadKey();
}
public static void ThisIsARecursiveFunctionInMemory(DirWithSubDirs currentDirectory)
{
var depth = currentDirectory.Path.Count(t => t == '\\');
Console.WriteLine(depth + ": " + currentDirectory.Path);
var children = currentDirectory.SubDirs;
//Edit this mode to switch what way of parallelization it should use
int mode = 2;
switch (mode)
{
case 1:
foreach (var child in children)
{
ThisIsARecursiveFunctionInMemory(child);
}
break;
case 2:
children.AsParallel().ForAll(t =>
{
ThisIsARecursiveFunctionInMemory(t);
});
break;
case 3:
Parallel.ForEach(children, t =>
{
ThisIsARecursiveFunctionInMemory(t);
});
break;
default:
break;
}
}
}
class DirWithSubDirs
{
public List<DirWithSubDirs> SubDirs = new List<DirWithSubDirs>();
public String Path { get; private set; }
public DirWithSubDirs(String path)
{
this.Path = path;
try
{
SubDirs = Directory.GetDirectories(path).Select(t => new DirWithSubDirs(t)).ToList();
}
catch (Exception eee)
{
//Ignore directories that can't be accessed
}
}
}
Edit 2:
After reading the update on Matthew's comment I've tried to add the following code to the program:
ThreadPool.SetMinThreads(4000, 16);
ThreadPool.SetMaxThreads(4000, 16);
This however does not change how the AsParallel peforms. Still the first 8 steps are being executed in an instant before slowing down to 1 step / second.
(Extra note, I'm currently ignoring the exceptions that occur when I can't access a Directory by the Try Catch block around the Directory.GetDirectories())
Edit 3:
Also what I'm mainly interested in is the difference between Parallel.ForEach and AsParallel.ForAll because to me it's just strange that for some reason the second one creates one Thread for every recursion it does while the first once handles everything in around 30 threads max. (And also why MSDN suggests to use the AsParallel even though it creates so much threads with a ~1 second timeout)
Edit 4:
Another strange thing I found out:
When I try to set the MinThreads on the Thread pool above 1023 it seems to ignore the value and scale back to around 8 or 16:
ThreadPool.SetMinThreads(1023, 16);
Still when I use 1023 it does the first 1023 elements very fast followed by going back to the slow pace I've been experiencing all the time.
Note: Also literally more then 1000 threads are now created (compared to 30 for the whole Parallel.ForEach one).
Does this mean Parallel.ForEach is just way smarter in handling tasks?
Some more info, this code prints twice 8 - 8 when you set the value above 1023: (When you set the values to 1023 or lower it prints the correct value)
int threadsMin;
int completionMin;
ThreadPool.GetMinThreads(out threadsMin, out completionMin);
Console.WriteLine("Cur min threads: " + threadsMin + " and the other thing: " + completionMin);
ThreadPool.SetMinThreads(1023, 16);
ThreadPool.SetMaxThreads(1023, 16);
ThreadPool.GetMinThreads(out threadsMin, out completionMin);
Console.WriteLine("Now min threads: " + threadsMin + " and the other thing: " + completionMin);
Edit 5:
As of Dean's request I've created another case to manually create tasks:
case 4:
var taskList = new List<Task>();
foreach (var todo in children)
{
var itemTodo = todo;
taskList.Add(Task.Run(() => ThisIsARecursiveFunctionInMemory(itemTodo)));
}
Task.WaitAll(taskList.ToArray());
break;
This is also as fast as the Parallel.ForEach() loop. So we still don't have the answer to why AsParallel().ForAll() is so much slower.
This problem is pretty debuggable, an uncommon luxury when you have problems with threads. Your basic tool here is the Debug > Windows > Threads debugger window. Shows you the active threads and gives you a peek at their stack trace. You'll easily see that, once it gets slow, that you'll have dozens of threads active that are all stuck. Their stack trace all look the same:
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) + 0x16 bytes
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) + 0x7 bytes
mscorlib.dll!System.Threading.ManualResetEventSlim.Wait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) + 0x182 bytes
mscorlib.dll!System.Threading.Tasks.Task.SpinThenBlockingWait(int millisecondsTimeout, System.Threading.CancellationToken cancellationToken) + 0x93 bytes
mscorlib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler scheduler, bool waitForCompletion) + 0xba bytes
mscorlib.dll!System.Threading.Tasks.Task.RunSynchronously(System.Threading.Tasks.TaskScheduler scheduler) + 0x13 bytes
System.Core.dll!System.Linq.Parallel.SpoolingTask.SpoolForAll<ConsoleApplication1.DirWithSubDirs,int>(System.Linq.Parallel.QueryTaskGroupState groupState, System.Linq.Parallel.PartitionedStream<ConsoleApplication1.DirWithSubDirs,int> partitions, System.Threading.Tasks.TaskScheduler taskScheduler) Line 172 C#
// etc..
Whenever you see something like this, you should immediately think fire-hose problem. Probably the third-most common bug with threads, after races and deadlocks.
Which you can reason out, now that you know the cause, the problem with the code is that every thread that completes adds N more threads. Where N is the average number of sub-directories in a directory. In effect, the number of threads grows exponentially, that's always bad. It will only stay in control if N = 1, that of course never happens on an typical disk.
Do beware that, like almost any threading problem, that this misbehavior tends to repeat poorly. The SSD in your machine tends to hide it. So does the RAM in your machine, the program might well complete quickly and trouble-free the second time you run it. Since you'll now read from the file system cache instead of the disk, very fast. Tinkering with ThreadPool.SetMinThreads() hides it as well, but it cannot fix it. It never fixes any problem, it only hides them. Because no matter what happens, the exponential number will always overwhelm the set minimum number of threads. You can only hope that it completes finishing iterating the drive before that happens. Idle hope for a user with a big drive.
The difference between ParallelEnumerable.ForAll() and Parallel.ForEach() is now perhaps also easily explained. You can tell from the stack trace that ForAll() does something naughty, the RunSynchronously() method blocks until all the threads are completed. Blocking is something threadpool threads should not do, it gums up the thread pool and won't allow it to schedule the processor for another job. And has the effect you observed, the thread pool is quickly overwhelmed with threads that are waiting on the N other threads to complete. Which isn't happening, they are waiting in the pool and are not getting scheduled because there are already so many of them active.
This is a deadlock scenario, a pretty common one, but the threadpool manager has a workaround for it. It watches the active threadpool threads and steps in when they don't complete in a timely manner. It then allows an extra thread to start, one more than the minimum set by SetMinThreads(). But not more then the maximum set by SetMaxThreads(), having too many active tp threads is risky and likely to trigger OOM. This does solve the deadlock, it gets one of the ForAll() calls to complete. But this happens at a very slow rate, the threadpool only does this twice a second. You'll run out of patience before it catches up.
Parallel.ForEach() doesn't have this problem, it doesn't block so doesn't gum up the pool.
Seems to be the solution, but do keep in mind that your program is still fire-hosing the memory of your machine, adding ever more waiting tp threads to the pool. This can crash your program as well, it just isn't as likely because you have a lot of memory and the threadpool doesn't use a lot of it to keep track of a request. Some programmers however accomplish that as well.
The solution is a very simple one, just don't use threading. It is harmful, there is no concurrency when you have only one disk. And it does not like being commandeered by multiple threads. Especially bad on a spindle drive, head seeks are very, very slow. SSDs do it a lot better, it however still takes an easy 50 microseconds, overhead that you just don't want or need. The ideal number of threads to access a disk that you can't otherwise expect to be cached well is always one.
The first thing to note is that you are trying to parallelise an IO-bound operation, which will distort the timings significantly.
The second thing to note is the nature of the parallelised tasks: You are recursively descending a directory tree. If you create multiple threads to do this, each thread is likely to be accessing a different part of the disk simultaneously - which will cause the disk read head to be jumping all over the place and slowing things down considerably.
Try changing your test to create an in-memory tree, and access that with multiple threads instead. Then you will be able to compare the timings properly without the results being distorted beyond all usefulness.
Additionally, you may be creating a great number of threads, and they will (by default) be threadpool threads. Having a great number of threads will actually slow things down when they exceed the number of processor cores.
Also note that when you exceed the thread pool minimum threads (defined by ThreadPool.GetMinThreads()), a delay is introduced by the thread pool manager between each new threadpool thread creation. (I think this is around 0.5s per new thread).
Also, if the number of threads exceeds the value returned by ThreadPool.GetMaxThreads(), the creating thread will block until one of the other threads has exited. I think this is likely to be happening.
You can test this hypothesis by calling ThreadPool.SetMaxThreads() and ThreadPool.SetMinThreads() to increase these values, and see if it makes any difference.
(Finally, note that if you are really trying to recursively descend from C:\, you will almost certainly get an IO exception when it reaches a protected OS folder.)
NOTE: Set the max/min threadpool threads like this:
ThreadPool.SetMinThreads(4000, 16);
ThreadPool.SetMaxThreads(4000, 16);
Follow Up
I have tried your test code with the threadpool thread counts set as described above, with the following results (not run on the whole of my C:\ drive, but on a smaller subset):
Mode 1 took 06.5 seconds.
Mode 2 took 15.7 seconds.
Mode 3 took 16.4 seconds.
This is in line with my expectations; adding a load of threading to do this actually makes it slower than single-threaded, and the two parallel approaches take roughly the same time.
In case anyone else wants to investigate this, here's some determinative test code (the OP's code is not reproducible because we don't know his directory structure).
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
namespace Demo
{
internal class Program
{
private static DirWithSubDirs RootDir;
private static void Main()
{
Console.WriteLine("Loading file system into memory...");
RootDir = new DirWithSubDirs("Root", 4, 4);
Console.WriteLine("Done");
//ThreadPool.SetMinThreads(4000, 16);
//ThreadPool.SetMaxThreads(4000, 16);
var w = Stopwatch.StartNew();
ThisIsARecursiveFunctionInMemory(RootDir);
Console.WriteLine("Elapsed seconds: " + w.Elapsed.TotalSeconds);
Console.ReadKey();
}
public static void ThisIsARecursiveFunctionInMemory(DirWithSubDirs currentDirectory)
{
var depth = currentDirectory.Path.Count(t => t == '\\');
Console.WriteLine(depth + ": " + currentDirectory.Path);
var children = currentDirectory.SubDirs;
//Edit this mode to switch what way of parallelization it should use
int mode = 3;
switch (mode)
{
case 1:
foreach (var child in children)
{
ThisIsARecursiveFunctionInMemory(child);
}
break;
case 2:
children.AsParallel().ForAll(t =>
{
ThisIsARecursiveFunctionInMemory(t);
});
break;
case 3:
Parallel.ForEach(children, t =>
{
ThisIsARecursiveFunctionInMemory(t);
});
break;
default:
break;
}
}
}
internal class DirWithSubDirs
{
public List<DirWithSubDirs> SubDirs = new List<DirWithSubDirs>();
public String Path { get; private set; }
public DirWithSubDirs(String path, int width, int depth)
{
this.Path = path;
if (depth > 0)
for (int i = 0; i < width; ++i)
SubDirs.Add(new DirWithSubDirs(path + "\\" + i, width, depth - 1));
}
}
}
The Parallel.For and .ForEach methods are implemented internally as equivalent to running iterations in Tasks, e.g. that a loop like:
Parallel.For(0, N, i =>
{
DoWork(i);
});
is equivalent to:
var tasks = new List<Task>(N);
for(int i=0; i<N; i++)
{
tasks.Add(Task.Factory.StartNew(state => DoWork((int)state), i));
}
Task.WaitAll(tasks.ToArray());
And from the perspective of every iteration potentially running in parallel with every other iteration, this is an ok mental model, but does not happen in reality. Parallel, in fact, does not necessarily use one Task per iteration, as that is significantly more overhead than is necessary. Parallel.ForEach tries to use the minimum number of tasks necessary to complete the loop as fast as possible. It spins up tasks as threads become available to process those tasks, and each of those tasks participates in a management scheme (I think its called chunking): A task asks for multiple iterations to be done, gets them, and then processes that work, and then goes back for more. The chunk sizes vary based the number of tasks participating, the load on the machine, etc.
PLINQ’s .AsParallel() has a different implementation, but it ‘can’ still similarly fetch multiple iterations into a temporary store, do the calculations in a thread (but not as a task), and put the query results into a small buffer. (You get something based on ParallelQuery, and then further .Whatever() functions bind to an alternative set of extension methods that provide parallel implementations).
So now that we have a small idea of how these two mechanisms work, I will try to provide an answer to your original question:
So why is .AsParallel() slower than Parallel.ForEach? The reason stems from the following. Tasks (or their equivalent implementation here) do NOT block on I/O-like calls. They ‘await’ and free up the CPU to do something else. But (quoting C# nutshell book): “PLINQ cannot perform I/O-bound work without blocking threads”. The calls are synchronous. They were written with the intention that you increase the degree of parallelism if (and ONLY if) you are doing such things as downloading web pages per task that do not hog CPU time.
And the reason why your function calls are exactly analogous to I/O bound calls is this: One of your threads (call it T) blocks and does nothing until all of its child threads have finished, which can be a slow process here. T itself is not CPU-intensive while it waits for the children to unblock, it is doing nothing but waiting. Hence it is identical to a typical I/O bound function call.
Based on the accepted answer to How exactly does AsParallel work?
.AsParallel.ForAll() casts back to IEnumerable before calling .ForAll()
so it creates 1 new thread + N recursive calls (each of which generates a new thread).
I have a program that runs some methods every 5 seconds in the background. However every 5 seconds its physical memory usage jumps by 16-20 Kb. Through commenting out segments of code, I've narrowed it down to this specific segment is what is causing the issue. What am I missing here to correctly release the allocated Memory?
Loop segment from main method:
while (true)
{
listMessages = FetchAllMessages();
//Commented out other segments. Not causing memory increase
System.Threading.Thread.Sleep(5000);
}
Method called:
public static List<Message> FetchAllMessages()
{
try
{
using (Pop3Client client = new Pop3Client())
{
client.Connect("pop.gmail.com", 995, true);
client.Authenticate("removed", "removed");
int messageCount = client.GetMessageCount();
List<Message> allMessages = new List<Message>(messageCount);
for (int i = messageCount; i > 0; i--)
{
if (verifiedEmail.Contains(client.GetMessage(i).Headers.From.Address) || verifiedSms.Contains(client.GetMessage(i).Headers.From.Address))
{
string tempMessage = client.GetMessage(i).ToMailMessage().Body.ToLower();
if (tempMessage.Contains("cmd") && tempMessage.Contains("fin"))
{
allMessages.Add(client.GetMessage(i));
}
}
client.DeleteMessage(i);
}
client.Disconnect();
return allMessages;
}
}
catch (Exception ex)
{
return null;
}
}
One of the things that could be causing a steady increase in memory usage is that you're calling GetMessage so many times. Depending on how your POP client is written, that could be allocating a new buffer every time so that it can download the message from the POP server. That memory will of course be collected eventually, but you're exercising the garbage collector needlessly. And you're also being highly inefficient.
You should consider changing your code to something like this:
for (int i = messageCount; i > 0; i--)
{
var msg = client.GetMessage(i);
if (verifiedEmail.Contains(msg.Headers.From.Address)
|| verifiedSms.Contains(msg.Headers.From.Address))
{
string tempMessage = msg.ToMailMessage().Body.ToLower();
if (tempMessage.Contains("cmd") && tempMessage.Contains("fin"))
{
allMessages.Add(msg);
}
}
client.DeleteMessage(i);
}
So instead of calling client.GetMessage(i) four times, you call it only once.
It also makes the code easier to read.
That said, I think it's likely that your "memory leak" is just the GC taking its own sweet time in collecting memory.
One other thing. You have a sleep loop:
while (true)
{
listMessages = FetchAllMessages();
Thread.Sleep(5000);
}
You're tying up a thread that spends most of its time doing nothing. You'd be better off creating a timer with a 5 second interval, like this:
System.Threading.Timer MailTimer; // declare at class scope
// Do this in your initialization
MailTimer = new Timer(MessageFetcher, null, 5000, -1);
And your MessageFetcher method is:
void MessageFetcher(object state)
{
listMessages = FetchAllMessages();
// do that other stuff that you didn't show
// reset the timer so that it fires 5 seconds from now
MailTimer.Change(5000, -1);
}
The initialization creates a one-shot timer that expires in five seconds and calls MessageFetcher. When MessageFetcher is done, it sets a timer so that mail will be checked in another five seconds. You want to do it this way rather than setting a periodic interval, because you don't want the timer to call MessageFetcher again if the previous tick isn't done processing.
The MessageFetcher method is executed on a pool thread. Using the timer prevents you from having to keep a thread around all the time, occupying memory while it's doing essentially nothing.
As Paddy said, eventually garbage collection will get around to disposing the objects and releasing memory, but you CAN manually force it, although it's generally better to allow it to happen automatically.
But to test that garbage collection will reduce the memory, exit the While loop after several calls and call GC.Collect();. The memory should go down.
Calling GC.Collect(); is expensive and that's why you're better off to let the OS choose the optimal time to call garbage collection automatically.
This is a great answer to a similar question regarding your concerns: C# Garbage collection
Who knows? It's not deterministic. Think of it like this: on a system
with infinite memory, the garbage collector doesn't have to do
anything. And you might think that's a bad example, but that's what
the garbage collector is simulating for you: a system with infinite
memory. Because on a system with sufficiently more memory available
than required by your program, the garbage collector never has to run.
Consequently, your program can not make any assumptions about when
memory will (if ever) be collected.
So, the answer to your question is: we don't know.
I would recommend setting up a memory profiler to log the memory consumption of your application and run it for a while to test. You should see that the Garbage Collector will keep things under control automatically without you having to change anything with your code.