We Keep Coding

c# ThreadPool using less threads than cores because of memory restrictions with FileSystemWatcher

I have an extensive image calculation task which uses about 1GB of memory (one calculation cycle takes about 4 seconds). I process those images automatically when they arrive in the folder using a FileSystemWatcher. When the FileSystemWatcher fires an event for a new file I queue the work in the eventhandler method with:
private void OnNewFileInDir(object source, FileSystemEventArgs evtArgs)
{
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessTheNewImage), evtArgs.FullPath);
}
My problem is that the program crashes on a regular basis when the files arrive quickly. In the debug window I can see that neary 3GB memory are used in that moment. When I use smaller images in order to use less memory, there are no crashes (so far).
My question: What can I do to use less (maybe just 2) threads independent of cores of my computer?
Or is my approach of using a FileSystemWatcher to cue new files to a thread pool completely stupid? I am not at all experienced with thread races or similar things.
So, furthermore: Does that look threadsafe?
Thanks a lot upfront and all the best
Tim
For completeness here is the code executed by the threads (a bit simplified for ease of reading):
private void ProcessTheNewImage(object threadFilenameInfo)
{
String filename = (String)threadFilenameInfo;
// Load the image
Image currentImage = Image.FromFile(filename);
//Calculate the image in an external DLL
Image currentResultImage = ImageProcessing.getResultImage(currentImage);
//Create the filename with the result infos
string saveFileName = "blahblah";
//Save the image
currentResultImage.Save(saveFileName);
//dispose the images
currentImage.Dispose();
currentResultImage.Dispose();
}

The Threadpool has only a very limited form of resource management. It will slowly keep adding threads when the queue fills up. It is meant for relatively small (< 500 ms) jobs. There is no safety-valve to stop it from clogging up your application.
You could create a workflow for this: the watcher event pushes simple datapackets into a ConcurrentQueue and then you create 2 or more Threads (better: Tasks) to process the queue. This will allow you to tune the number of threads.

How do I read a file in parts with NAudio?

I am fairly new to C# and Mark's library NAudio. So I've tried learning by myself and I've come up with an basic audio player. But I have a problem.
When trying to load big files in the player the app freezes for 2-10 seconds while loading the entire file (I suppose). This is my code for reading the file:
if (target.EndsWith("mp3") || target.EndsWith("Mp3") || target.EndsWith("MP3"))
{
NAudio.Wave.WaveStream pcm = NAudio.Wave.WaveFormatConversionStream.CreatePcmStream(new NAudio.Wave.Mp3FileReader(target));
stream = new NAudio.Wave.BlockAlignReductionStream(pcm);
}
All I really want is to read the file in parts. Like a buffer. Read 10 seconds from the HDD to RAM memory, then after those 10 seconds run out, read the next 10 seconds, and so on. I think this should resolve the freeze issue I have with large files.

The cause of the delay is that Mp3FileReader creates a table of contents to allow it to determine the file length and to enable quicker repositioning. You could try using MediaFoundationReader instead which would be quicker, but won't work on Windows XP.

all programs have delay to load big files. this is depended to client computer speed.
but you can use backgroundWorker in your program and show a loading animation on your application Form during the file loading.
add backgroundWorker tool on your form
use this code on open button click:
backgroundWorker_name.RunWorkerAsync();
and put your code to the DoWork event
private void backgroundWorker_name_DoWork(object sender, DoWorkEventArgs e)
{
}

Reading StandardInput stops code execution without breaking application

Im using Stockfish game engine to power Human Vs Computer games.
Here is first part of the code:
Process _proc= new Process();
_proc.StartInfo = new ProcessStartInfo(path);
_proc.StartInfo.RedirectStandardInput = true;
_proc.StartInfo.RedirectStandardOutput = true;
_proc.StartInfo.UseShellExecute = false;
_proc.StartInfo.CreateNoWindow = true;
_proc.Start();
_proc.StandardInput.WriteLine("uci");
_proc.StandardInput.WriteLine("ucinewgame");
At this point everything is ok, but when I try to read StandardOutput something weird happens.
string result = _proc.StandardOutput.ReadToEnd();
Stockfish.exe program pops-up my application is running but code after that line is not executing. When I press pause, it points at this line:
If I use:
while (!_proc.StandardOutput.EndOfStream)
{
result += _proc.StandardOutput.ReadLine();
}
Same thing happens only at while statement. result has its full value there, all the text is written into it.
Is there any way to overcome this without async reading?
Side problem:
Since this is all part of singleton class that is used over whole ASP.NET application, i dont feel like using async reading since Im not sure how can I protect (with locking) multiple threads writing into it. Also, I dont know how to stop current thread since the processing of command can last up to 10 sec.
I don't feel like using Thread.Sleep() to constantly check for end of reading output, not elegant.
Considering side problem, how could i avoid multithread problems if async is only solution?
My threading knowledge is weak, so please have that in mind when giving thread related answers. Thank you.

The call to StandardOutput.ReadToEnd will block until this process ends. Is the goal here to read, process, and respond to various text commands from the process you spawn as you receive them?
You must approach this via asynchronous reading.
For example, you could setup a listener to Process.OutputDataReceived. Then call Process.BeginOutputReadLine to start reading. Your code will continue execution. Meanwhile, the .NET Framework will handle incoming text messages on a separate thread.

ASP.NET: Firing batch jobs

My application could have up to roughly 100 requests for a batch job within a few milliseconds but in actuality, these job requests are being masked as one job request.
To fix this issue so that only one job request is just not feasible at the moment.
A workaround that I have thought is to program my application to fulfill only 1 batch job every x milliseconds, in this case I was thinking of 200 milliseconds, and ignore any other batch job that may come in within those 200 milliseconds or when my batch job have completed. After those 200 milliseconds are up or when the batch job is completed, my application will wait and accept 1 job request from that time on and it will not process any requests that may have been ignored before. Once my application accepts another job requests, it will repeat the cycle above.
What's the best way of doing this using .Net 4.0? Are there any boiler plate code that I can simply follow as a guide?
Update
Sorry for being unclear. I have added more details about my scenario. Also I just realized that my proposed workaround above will not work. Sorry guys, lol. Here's some background information.
I have an application that builds an index using files in a specified directory. When a file is added, deleted or modified in this directory, my application listens for these events using a FileSystemWatcher and re-indexes these files. The problem is that around 100 files can be added, deleted or modified by an external process and they occur very quickly, ie: within a few milliseconds. My end goal is to re-index these files after the last file change have occurred by the external process. The best solution is to modify the external process to signal my application when it has finished modifying the files I'm listening to but that's not feasible at the moment. Thus, I have to create a workaround.
A workaround that may solve my problem is to wait for the first file change. When the first file change have occurred, wait 200 milliseconds for any other subsequent file changes. Why 200 milliseconds? Because I'm hoping and confident that the external process can perform its file changes within 200 milliseconds. Once my application have waited for 200 milliseconds, I would like it to start a task that will re-index the files and go through another cycle of listening to a file change.
What's the best way of doing this?
Again, sorry for the confusion.

This question is a bit too high level to guess at.
My guess is your application is run as a service, you have your requests come into your application and arrive in a queue to be processed. And every 200 ms, you wake the queue and pop and item off for processing.
I'm confused about the "masked as one job request". Since you mentioned you will "ignore any other batch job", my guess is you haven't arranged your code to accept the incoming requests in a queue.
Regardless, you will generally always have one application process running (your service) and if you choose you could spawn a new thread for each item you process in the queue. You can monitor how much cpu/memory utilization this required and adjust the firing time (200ms) accordingly.

I may not be accurately understanding the problem, but my recommendation is to use the singleton pattern to work around this issue.
With the singleton approach, you can implement a lock on an object (the access method could potentially be something along the lines of BatchProcessor::GetBatchResults) that would then lock all requests to the batch job results object. Once the batch has finished, the lock will be released, and the underlying object, will have the results of the batch job available.
Please keep in mind that this is a "work around". There may be a better solution that involves looking into and changing the underlying business logic that causes multiple requests to come in for a job that's processing on demand.
Update:
Here is a link for information regarding Singleton (includes code examples): http://msdn.microsoft.com/en-us/library/ff650316.aspx

It is my understanding that the poster has some sort of an application that sits and waits for incoming requests to perform a batch job. The problem that he is receiving multiple requests within a short period of time that should actually have come in as just a single request. And, unfortunately, he is not able to solve this problem.
So, his solution is to assume that all requests received within a 200 ms timespan are the same, and to only process these once. My concern with this would be whether this assumption is correct or not? This entirely depends on the sending systems and the environment in which this is being used. The general idea to be able to do this would be to update a lastReceived date/time when a request is processed. Then when a new request comes in, compare the current date/time to the lastReceived date/time and only process it if the difference is greater than 200 ms.
Other possible solutions:
You said you could not modify the sending application so only one job request was sent, but could you add additional information to it, for instance a unique identifier?
Could you store the parameters from the last job request and compare it with the next job request and only process them if they are different?
Based on your Update
Here is an example how you could wait 200ms using a Timer:
static Timer timer;
static int waitTime = 200; //in ms
static void Main(string[] args)
{
FileSystemWatcher fsw = new FileSystemWatcher();
fsw.Path = #"C:\temp\";
fsw.Created += new FileSystemEventHandler(fsw_Created);
fsw.EnableRaisingEvents = true;
Console.ReadLine();
}
static void fsw_Created(object sender, FileSystemEventArgs e)
{
DateTime currTime = DateTime.Now;
if (timer == null)
{
Console.WriteLine("Started # " + currTime);
timer = new Timer();
timer.Interval = waitTime;
timer.Elapsed += new ElapsedEventHandler(timer_Elapsed);
timer.Start();
}
else
{
Console.WriteLine("Ignored # " + currTime);
}
}
static void timer_Elapsed(object sender, ElapsedEventArgs e)
{
//Start task here
Console.WriteLine("Elapsed # " + DateTime.Now);
timer = null;
}

Implementing stop and restart in file stream transfer - how to? C# .NET

I'm looking for texts or advice on implementing stop and restart in file stream transfer.
The goal is for the application to use a single read source, and output to multiple write sources, and be able to restart from a recorded position if a transfer fails.
The application is being written in C# .NET.
Psuedo code:
while (reader.Read())
{
foreach(writer in writers)
{
writer.WriteToStream();
}
}
I need to be able to implement stop or pause. Which could work like so. To stop, continue is marked false:
while (reader.Read() && Continue)
{
foreach(writer in writers)
{
writer.WriteToStream();
}
}
Clearly at this stage I need to record the number of bytes read, and the number of bytes written to each write source.
My questions are:
If I were to only record the read bytes, and use this for restarts, one or more writers could have written while others have not. Simply restarting using a measure of read progress might corrupt the written data. So I need to use a 'written bytes per writer' record as my new start position. How can I be sure that the bytes were written (I may not have the ability to read the file from the write source to read the file length)?
Can anyone adviser or point me in the right direction of a text on this kind of issue?

Use a thread synchronization event.
(pseudocode):
ManualResetEvent _canReadEvent = new ManualResetEvent(true);
public void WriterThreadFunc()
{
while (_canReadEvent.Wait() && reader.Read())
{
foreach(writer in writers)
{
writer.WriteToStream();
}
}
}
public void Pause()
{
_canReadEvent.Reset();
}
public void Continue()
{
_canReadEvent.Set();
}
The good thing is that the writer thread won't consume any CPU when it's paused and it will continue directly it's signaled (as opposed to using a flag and Thread.Sleep())
The other note is that any check should be the first argument in the while since reader.Read() will read from the stream otherwise (but the content will be ignored since the flag will prevent the while block from being executed).