We Keep Coding

Executing multiple threads

I am developing Windows Form C# program which reads Excel data from shared drive every 20 minutes (I'm using "Timer") - function "inserting". I want to read multiple Excel files at once because of the performance. For that reason I'm using threads.
Each thread is calling a function (LoadExcelData) which reads data from Excel to ArrayList. I want to know when all threads are finished (when all excel files were loaded to ArrayList) in order to insert this ArrayList to internal database.
I tried with thread[i].Join() but this freezes GUI. I also do not know what would happen if I have 100+ files and for this reason 100+ threads. Would that cause memory exception or some other exception?
//Execute every 20 minutes (Timer). Do not Execute in case previouse run is not finished
void inserting(List<String> excels){
int numOfThreads=excels.length;
Thread[] threads = new Thread[numOfThreads];
for (int index = 0; index < numOfThreads; index++)
{
int i = index;
threads[index] = new Thread(() =>
{
LoadExcelData(excels[i].File_name); //function loads excel data to global array "Weather" which is used later on
});
}
for (int i = 0; i < threads.Length; i++)
{
threads[i].Start(); //start thread
}
for (int i = 0; i < threads.Length; i++)
{
// threads[i].Join(); //this freezes GUI!
}
InsertToDB(object of ArrayList<ClassName>); //insert data which was read from Excels
isRunning=false;//Data was successefully inserted to DB
}
I want to run this every 20 minutes. I'm using Timer:
timer = new System.Windows.Forms.Timer();
timer.Tick += new EventHandler(timerEventHanlder);
timer.Interval = 20 * 60000; // in miliseconds
timer.Start();
private void timerEventHanlder(object sender, EventArgs e)
{
List<String> excels = getExcels();
if (!isRunning){ //in case previous timer even is not finished wait another 20 minutes...
isRunning=true; //flag to true
inserting(excels);
}
}
Is there any better wait to solve above problem?

The UI thread is freezing because you're using a System.Windows.Forms.Timer which fires the timer ticked event on the UI thread; this is useful in that you don't have to Invoke anything on the tick event. Calling Join blocks the calling thread and in your case this is the UI thread.
To avoid this (and since you're not needing to Invoke any UI elements), you can change your System.Windows.Forms.Timer to a System.Timers.Timer, which runs in a thread separate from the UI thread. If you switch to a System.Timers.Timer, you'll need to change some of the syntax in your code (e.g. the Tick event is the Elapsed event instead, etc.).
There's also the System.Thread.Timer and the System.Web.UI.Timer, additionally, you could also spawn a second thread from within the timer tick event to avoid it waiting on the threads within the UI thread, example:
private void timerEventHanlder(object sender, EventArgs e)
{
(new System.Threading.Thread(() => {
List<String> excels = getExcels();
if (!isRunning){ //in case previous timer even is not finished wait another 20 minutes...
isRunning=true; //flag to true
inserting(excels);
}
})).Start();
}
Starting a new thread avoids changing any of your current code and allows you to change it back if you do ever need to invoke anything in the UI.
Answering you're other question though:
I also do not know what would happen if I have 100+ files and for this reason 100+ threads. Would that cause memory exception or some other exception?
Spawning 100+ threads won't cause any exceptions unless your code has a specific exception (like a null delegate passed as the ThreadStart), or if the OS can't create a thread, which if the OS can't create a thread you have bigger problems. It is possible that memory exhaustion could happen since the Thread is a managed object and thus takes up memory (along with an ArrayList, but the amount of memory for 100+ threads (even 1000+) is negligible on any system that is capable of running the .NET framework (even on most embedded systems), so the number of threads won't necessarily be an issue.
Looking at your code, you might want to consider instead of spawning 100+ threads, utilizing the System.Threading.ThreadPool and a System.Threading.CountDownEvent, example:
CountdownEvent Countdown;
void LoadExcelData(object data)
{
// loads excel data to global array "Weather" which is used later on
Countdown.Signal();
}
//Execute every 20 minutes (Timer). Do not Execute in case previouse run is not finished
void inserting(List<object> excels)
{
Countdown = new CountdownEvent(excels.Count);
int i = 0;
while (i < excels.Count) {
ThreadPool.QueueUserWorkItem(LoadExcelData, excels[i++].File_name);
}
Countdown.Wait();
InsertToDB(WeatherList); //insert data which was read from Excels
isRunning = false; //Data was successefully inserted to DB
}
This will utilize the system thread pool to execute your functions and allows .NET to handle the scheduling of the threads to avoid massive resource contention if the number of threads is a lot. You could use other methods to block, like a Mutex or Semaphore, but the CountDownEvent pretty much encapsulates what you'd need to do with other wait objects and joining on the threads from the thread pool.
To be honest though, since you're reading data from Excel files in multiple threads, unless each thread reads the entire contents of the file into RAM then executes the operations that way, you might not see a huge increase in performance. Multi-threaded applications that have heavy I/O usually don't see a huge performance increase unless said I/O is on performance minded equipment or the initial input of the entire file is read into RAM. Just a side note as you're multi-threading with files.
It should also be noted too that utilizing the System.Threading.ThreadPool is ideally for threads you expect to only run for a few seconds or so; if you anticipate that a thread could take longer, you should stick with spawning the threads as you have now. You can still use the CountDownEvent and you don't need an array of threads like you have (you could just just use the (new Thread(function)).Start() syntax).
Hope that can help

The parent thread is going to reach the for loop that joins all the worker threads and wait there until all the threads have finished (and can be joined). If the GUI is running in that same parent thread, execution is not going to return to the GUI until all threads have finished, which is going to be a long time as you've set up timers. Try running the GUI in a different thread.
Edit:
Also on a side note, I'd set your timer lengths to something much shorter while you're debugging to see if it's actually waiting as you expect it to. Then once you have it functioning correctly you can set it back to 20 minutes.

C# multithreading debugging

i am trying to build a multi threaded server that is supposed to spawn new threads for every incoming connection, BUT for all my effort, it spawns new threads only when it feels like it, Can anybody help me debug this code? am i missing something obvious?
while (true)
{
if (txtAddress.Text.Trim() == "Any" || txtAddress.Text.Trim() == "any")
ipEndP = new IPEndPoint(IPAddress.Any, 778);
else
ipEndP = new IPEndPoint(IPAddress.Parse(txtAddress.Text.Trim()), 778);
tcpL = new TcpListener(ipEndP);
tcpL.Server.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, true);
tcpL.Start();
tempSock = tcpL.AcceptSocket();
//t = new Thread(ConnectionHandler); //When new client is connected, new thread //is created to handle the connection
//t.Priority = ThreadPriority.Highest;
//t.Start(tempSock);
ThreadPool.QueueUserWorkItem(ConnectionHandler, tempSock);
}

Check out the MSDN docs for QueueUserWorkItem
Queues a method for execution. The method executes when a thread pool thread becomes available.
Placing a thread in the user work item queue does not guarantee that it will begin executing right away. That's actually a good thing. If you got so many connections that you needed hundreds or thousands of threads, that could easily bring your server to it's knees (and certainly would be very wasteful due to excessive context switches).
Your commented out code should kick off a new thread for every connection. Is that not working? If so, what exactly is not working? Note that creating a new thread for every connection is much more expensive than using the thread pool.
UPDATE
Based on your remark that the commented out code is also failing to create too many threads, I would add...
You are creating WAY too many threads if that happens.
Often I see people asking why they can't create more than around 2000 threads in a process. The reason is not that there is any particular limit inherent in Windows. Rather, the programmer failed to take into account the amount of address space each thread uses.
A thread consists of some memory in kernel mode (kernel stacks and object management), some memory in user mode (the thread environment block, thread-local storage, that sort of thing), plus its stack. (Or stacks if you're on an Itanium system.)
Usually, the limiting factor is the stack size.
http://blogs.msdn.com/b/oldnewthing/archive/2005/07/29/444912.aspx

Limiting the number of threadpool threads

I am using ThreadPool in my application. I have first set the limit of the thread pool by using the following:
ThreadPool.SetMaxThreads(m_iThreadPoolLimit,m_iThreadPoolLimit);
m_Events = new ManualResetEvent(false);
and then I have queued up the jobs using the following
WaitCallback objWcb = new WaitCallback(abc);
ThreadPool.QueueUserWorkItem(objWcb, m_objThreadData);
Here abc is the name of the function that I am calling.
After this I am doing the following so that all my threads come to 1 point and the main thread takes over and continues further
m_Events.WaitOne();
My thread limit is 3. The problem that I am facing is, inspite of the thread pool limit set to 3, my application is processing more than 3 files at the same time, whereas it was supposed to process only 3 files at a time. Please help me solve this issue.

What kind of computer are you using?
From MSDN
You cannot set the number of worker
threads or the number of I/O
completion threads to a number smaller
than the number of processors in the
computer.
If you have 4 cores, then the smallest you can have is 4.
Also note:
If the common language runtime is
hosted, for example by Internet
Information Services (IIS) or SQL
Server, the host can limit or prevent
changes to the thread pool size.
If this is a web site hosted by IIS then you cannot change the thread pool size either.

A better solution involves the use of a Semaphore which can throttle the concurrent access to a resource1. In your case the resource would simply be a block of code that processes work items.
var finished = new CountdownEvent(1); // Used to wait for the completion of all work items.
var throttle = new Semaphore(3, 3); // Used to throttle the processing of work items.
foreach (WorkItem item in workitems)
{
finished.AddCount();
WorkItem capture = item; // Needed to safely capture the loop variable.
ThreadPool.QueueUserWorkItem(
(state) =>
{
throttle.WaitOne();
try
{
ProcessWorkItem(capture);
}
finally
{
throttle.Release();
finished.Signal();
}
}, null);
}
finished.Signal();
finished.Wait();
In the code above WorkItem is a hypothetical class that encapsulates the specific parameters needed to process your tasks.
The Task Parallel Library makes this pattern a lot easier. Just use the Parallel.ForEach method and specify a ParallelOptions.MaxDegreesOfParallelism that throttles the concurrency.
var options = new ParallelOptions();
options.MaxDegreeOfParallelism = 3;
Parallel.ForEach(workitems, options,
(item) =>
{
ProcessWorkItem(item);
});
1I should point out that I do not like blocking ThreadPool threads using a Semaphore or any blocking device. It basically wastes the threads. You might want to rethink your design entirely.

You should use Semaphore object to limit concurent threads.

You say the files are open: are they actually being actively processed, or just left open?
If you're leaving them open: Been there, done that! Relying on connections and resources (it was a DB connection in my case) to close at end of scope should work, but it can take for the dispose / garbage collection to kick in.

Why Thread.Sleep() is so CPU intensive?

I have an ASP.NET page with this pseduo code:
while (read)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
}
Any client who requests this page will start to download a binary file. Everything is OK at this point but clients had no limit in download speed so changed the above code to this:
while (read)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
Thread.Sleep(500);
}
Speed problem is solved now, but under test with 100 concurrent clients who connect one after another (3 seconds lag between each new connection) the CPU usage increases when the number of clients increases and when there are 70 ~ 80 concurrent clients CPU reaches 100% and any new connection is refused. Numbers may be different on other machines but the question is why Thread.Sleep() is so CPU intensive and is there any way to speed done the client without CPU rising ?
I can do it at IIS level but I need more control from inside of my application.

Let's take a look at whether Michael's answer seems reasonable.
Now, Michael wisely points out that Thread.Sleep(500) shouldn't cost much in the way of CPU. That's all well and good in theory, but let's see if that pans out in practice.
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
Thread.Sleep(500);
}
}
Running this, the CPU use of the application hovers around the 0% mark.
Michael also points out that since all the threads that ASP.NET has to use are sleeping, it will have to spawn new threads, and offers that this is expensive. Let's try not sleeping, but doing lots of spawning:
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
new Thread(o => {}).Start();
}
}
We create lots of threads, but they just execute a null operation. That uses a lot of CPU, even though the threads aren't doing anything.
The total number of threads never gets very high though, because each lives for such a short time. Lets combine the two:
static void Main(string[] args) {
for(int i = 0; i != 10000; ++i)
{
new Thread(o => {Thread.Sleep(500);}).Start();
}
}
Adding this operation that we have shown to be low in CPU use to each thread increases CPU use even more, as the threads mount up. If I run it in a debugger it pushes up to near 100% CPU. If I run it outside of a debugger, it performs a bit better, but only because it throws an out of memory exception before it gets a chance to hit 100%.
So, it isn't Thread.Sleep itself that is the problem, but the side-effect that having all available threads sleep forces more and more threads to be created to handle other work, just as Michael said.

Just a guess:
I don't think it's Thread.Sleep() that's tying up the CPU - it's the fact that you're causing threads to be tied up responding to a request for so long, and the system needs to spin up new threads (and other resources) to respond to new requests since those sleeping threads are no longer available in the thread pool.

Rather than an ASP.NET page you should implement an IHttpAsyncHandler. ASP.NET page code puts many things between your code and the browser that would not be appropriate for transferring binary files. Also, since you're attempting to perform rate limitation, you should use asynchronous code to limit resource usage, which would be difficult in an ASP.NET page.
Creating an IHttpAsyncHandler is fairly simple. Just trigger some asynchronous operations in the BeginProcessRequest method, and don't forget to properly close the context to show you have reached the end of the file. IIS won't be able to close it for you here.
The following is my really bad example of how to perform an an asynchronous operation consisting of a series of steps, counting from 0 to 10, each performed at a 500ms interval.
using System;
using System.Threading;
namespace ConsoleApplication1 {
class Program {
static void Main() {
// Create IO instances
EventWaitHandle WaitHandle = new EventWaitHandle(false, EventResetMode.AutoReset); // We don't actually fire this event, just need a ref
EventWaitHandle StopWaitHandle = new EventWaitHandle(false, EventResetMode.AutoReset);
int Counter = 0;
WaitOrTimerCallback AsyncIOMethod = (s, t) => { };
AsyncIOMethod = (s, t) => {
// Handle IO step
Counter++;
Console.WriteLine(Counter);
if (Counter >= 10)
// Counter has reaced 10 so we stop
StopWaitHandle.Set();
else
// Register the next step in the thread pool
ThreadPool.RegisterWaitForSingleObject(WaitHandle, AsyncIOMethod, null, 500, true);
};
// Do initial IO
Console.WriteLine(Counter);
// Register the first step in the thread pool
ThreadPool.RegisterWaitForSingleObject(WaitHandle, AsyncIOMethod, null, 500, true);
// We force the main thread to wait here so that the demo doesn't close instantly
StopWaitHandle.WaitOne();
}
}
}
You'll also need to register your IHttpAsyncHandler implementation with IIS in whichever way is appropriate for your situation.

Its because the thread gets a priority boost every time it yields its time slice. Avoid calling sleep often ( particularly with low values ).

Multiple Threads

I post a lot here regarding multithreading, and the great stackoverflow community have helped me alot in understand multithreading.
All the examples I have seen online only deal with one thread.
My application is a scraper for an insurance company (family company ... all free of charge). Anyway, the user is able to select how many threads they want to run. So lets say for example the user wants the application to scrape 5 sites at one time, and then later in the day he choses 20 threads because his computer isn't doing anything else so it has the resources to spare.
Basically the application builds a list of say 1000 sites to scrape. A thread goes off and does that and updates the UI and builds the list.
When thats finished another thread is called to start the scraping. Depending on the number of threads the user has set to use it will create x number of threads.
Whats the best way to create these threads? Should I create 1000 threads in a list. And loop through them? If the user has set 5 threads to run, it will loop through 5 at a time.
I understand threading, but it's the application logic which is catching me out.
Any ideas or resources on the web that can help me out?

You could consider using a thread pool for that:
using System;
using System.Threading;
public class Example
{
public static void Main()
{
ThreadPool.SetMaxThreads(100, 10);
// Queue the task.
ThreadPool.QueueUserWorkItem(new WaitCallback(ThreadProc));
Console.WriteLine("Main thread does some work, then sleeps.");
Thread.Sleep(1000);
Console.WriteLine("Main thread exits.");
}
// This thread procedure performs the task.
static void ThreadProc(Object stateInfo)
{
Console.WriteLine("Hello from the thread pool.");
}
}

This scraper, does it use a lot of CPU when its running?
If it does a lot of communication with these 1000 remote sites, downloading their pages, that may be taking more time than the actual analysis of the pages.
And how many CPU cores does your user have? If they have 2 (which is common these days) then beyond two simultaneous threads performing analysis, they aren't going to see any speed up.
So you probably need to "parallelize" the downloading of the pages. I doubt you need to do the same for the analysis of the pages.
Take a look into asynchronous IO, instead of explicit multi-threading. It lets you launch a bunch of downloads in parallel and then get called back when each one completes.

If you really just want the application, use something someone else already spent time developing and perfecting:
http://arachnode.net/
arachnode.net is a complete and comprehensive .NET web crawler for
downloading, indexing and storing
Internet content including e-mail
addresses, files, hyperlinks, images,
and Web pages.
Whether interested or involved in
screen scraping, data mining, text
mining, research or any other
application where a high-performance
crawling application is key to the
success of your endeavors,
arachnode.net provides the solution
you need for success.
If you also want to write one yourself because it's a fun thing to write (I wrote one not long ago, and yes, it is alot of fun ) then you can refer to this pdf provided by arachnode.net which really explains in detail the theory behind a good web crawler:
http://arachnode.net/media/Default.aspx?Sort=Downloads&PageIndex=1
Download the pdf entitled: "Crawling the Web" (second link from top). Scroll to Section 2.6 entitled: "2.6 Multi-threaded Crawlers". That's what I used to build my crawler, and I must say, I think it works quite well.

I think this example is basically what you need.
public class WebScraper
{
private readonly int totalThreads;
private readonly List<System.Threading.Thread> threads;
private readonly List<Exception> exceptions;
private readonly object locker = new object();
private volatile bool stop;
public WebScraper(int totalThreads)
{
this.totalThreads = totalThreads;
threads = new List<System.Threading.Thread>(totalThreads);
exceptions = new List<Exception>();
for (int i = 0; i < totalThreads; i++)
{
var thread = new System.Threading.Thread(Execute);
thread.IsBackground = true;
threads.Add(thread);
}
}
public void Start()
{
foreach (var thread in threads)
{
thread.Start();
}
}
public void Stop()
{
stop = true;
foreach (var thread in threads)
{
if (thread.IsAlive)
{
thread.Join();
}
}
}
private void Execute()
{
try
{
while (!stop)
{
// Scrap away!
}
}
catch (Exception ex)
{
lock (locker)
{
// You could have a thread checking this collection and
// reporting it as you see fit.
exceptions.Add(ex);
}
}
}
}

The basic logic is:
You have a single queue in which you put the URLs to scrape then you create your threads and use a queue object to which every thread has access. Let the threads start a loop:
lock the queue
check if there are items in the queue, if not, unlock queue and end thread
dequeue first item in the queue
unlock queue
process item
invoke an event that updates the UI (Remember to lock the UI Controller)
return to step 1
Just let the Threads do the "get stuff from the queue" part (pulling the jobs) instead of giving them the urls (pushing the jobs), that way you just say
YourThreadManager.StartThreads(numberOfThreadsTheUserWants);
and everything else happens automagically. See the other replies to find out how to create and manage the threads .

I solved a similar problem by creating a worker class that uses a callback to signal the main app that a worker is done. Then I create a queue of 1000 threads and then call a method that launches threads until the running thread limit is reached, keeping track of the active threads with a dictionary keyed by the thread's ManagedThreadId. As each thread completes, the callback removes its thread from the dictionary and calls the thread launcher.
If a connection is dropped or times out, the callback reinserts the thread back into the queue. Lock around the queue and the dictionary. I create threads vs using the thread pool because the overhead of creating a thread is insignificant compared to the connection time, and it allows me to have a lot more threads in flight. The callback also provides a convenient place with which to update the user interface, even allowing you to change the thread limit while it's running. I've had over 50 open connections at one time. Remember to increase your MacConnections property in your app.config (default is two).

I would use a queue and a condition variable and mutex, and start just the requested number of threads, for example, 5 or 20 (and not start 1,000).
Each thread blocks on the condition variable. When woken up, it dequeues the first item, unlocks the queue, works with the item, locks the queue and checks for more items. If the queue is empty, sleep on the condition variable. If not, unlock, work, repeat.
While the mutex is locked, it can also check if the user has requested the count of threads to be reduced. Just check if count > max_count, and if so, the thread terminates itself.
Any time you have more sites to queue, just lock the mutex and add them to the queue, then broadcast on the condition variable. Any threads that are not already working will wake up and take new work.
Any time the user increases the requested thread count, just start them up and they will lock the queue, check for work, and either sleep on the condition variable or get going.
Each thread will be continually pulling more work from the queue, or sleeping. You don't need more than 5 or 20.

Consider using the event-based asynchronous pattern (AsyncOperation and AsyncOperationManager Classes)

You might want to take a look at the ProcessQueue article on CodeProject.
Essentially, you'll want to create (and start) the number of threads that are appropriate, in your case that number comes from the user. Each of these threads should process a site, then find the next site needed to process. Even if you don't use the object itself (though it sounds like it would suit your purposes pretty well, though I'm obviously biased!) it should give you some good insight into how this sort of thing would be done.