We Keep Coding

VB.Net (or C#), how to have about 100 threads do 10.000 jobs?

I have about 10.000 jobs that I want to be handled by approx 100 threads. Once a thread finished, the free 'slot' should get a new job untill there are no more jobs available.
Side note: processor load is not an issue, these jobs are mostly waiting for results or (socket) timeouts. And the amount of 100 is something that I am going to play with to find an optimum. Each job will take between 2 seconds and 5 minutes. So I want to assign new jobs to free threads and not pre-assign all jobs to threads.
My problem is that I am not sure how to do this. Im primarily using Visual Basic .Net (but C# is also ok).
I tried to make an array of threads but since each job/thread also returns a value (it also takes 2 input vars), I used 'withevents' and found out that you cannot do that on an array... maybe a collection would work? But I also need a way to manage the threads and feed them new jobs... And all results should go back to the main-form (thread)...
I have it all running in one thread, but now I want to speed up.
And then I though: Actually this is a rather common problem. There is a bunch of work to be done that needs to be distributed over an amount of worker threads.... So thats why I am asking. Whats the most common solution here?
I tried to make it question as generic as possible, so lots of people with the same kind of problem can be helped with your reply. Thanks!
Edit:
What I want to do in more detail is the following. I currently have about 1200 connected sensors that I want to read from via sockets. First thing I want to know is if the device is online (can connect on ip:port) or not. After it connects it will be depending on the device type. The device type is known after connect and Some devices I just read back a sensor value. Other devices need calibration to be performed, taking up to 5 minutes with mostly wait times and some reading/setting of values. All via the socket. Some even have FTP that I need to download a file from, but that I do via socket to.
My problem: Lot's of waiting time, so lot's of possibility to do things paralel and speed it up hugely.
My starting point is a list of ip:port addresses and I want to end up with a file with that shows the results and the results are also shown on a textbox on the main form (next to a start/pause/stop button)

This was very helpfull:
Multi Threading with Return value : vb.net
It explains the concept of a BackgroundWorker which takes away a lot of the hassle. I am now trying to see where it will bring me.

C# - Get processes sorted by processor load usage

What's the lighest way to get a running processes list sorted by processor load? (kind of same as task manager but only sorted by processor usage load)
Lighest is the key word, because i want to make an updated and refreshed list every 2 seconds or something like that.
EDIT : using windows 7

I would recommend reading into the Process class
- System.Diagnostics.Process
Documentation
I would recommend looking at the GetProcesses() method within here.

Need help optimizing this query

I have a big problem with first run query form my SQL Server CE database.
I already applied this optimization Performance and the Entity Framework, but still first query take about 15 sec to run.
Something that I noticed when I run my application for first time first query take about 15 sec to run.
If I close my application and run again, the first query runs immediately. So if I restart my PC and run application again first query take 15 sec to run.
Overall, after two week research on internet I could not find a good way to solve my problem.
I used ANTS Performance Profiler and I noticed my first query takes about 11 sec and form initialization for each page take 4 sec for first time.
I have some questions:
I want to know which resource loaded to Ram when my application start?
why my application is fast in second time run?
how can I load those resource into Ram before application start?
why these resource Remains in Ram until windows restart?
Maybe 15 sec is good but when I run my application from DVD it take 45 sec to run first query.
Edited
I used multiple database for each section of my application.
for example This query Take 11 sec to run form first time:
public void GetContent(short SubjectID)
{
new QuranOtherEntities(CDataBase.CSQuranOtherEntities))
{
CHtmlDesign.HtmlFile = QODB.AdabTbls.First(data => data.ID == SubjectID).Content;
}
}
Table Structure

First(data => data.ID == SubjectID)
Ok, forget entity framework. Grab the generated SQL code, run it through profiler.
This SMELLS like "I have no clue what an index is". Check "Use-the-index-luke.com".
Also make sure that
Maybe 15 sec is good but when I run my application from DVD it take 45 sec to run first
query
Actually IS the sql statement. It could easily be loading and initializing your application, in which case you ask a totally irrelevant question here. In this case there is no a lot you CAN do - there still are optimizations etc., but youhave to ask specific questions here, i.e. do your homework and this question here is then totally irrelevant. This includes, btw, the time it takes entity framework to initialize - which is not related to the query per se, but can happen on the first query you run at all.

How to get cpu usage % in C# for each running process

I use System.Diagnostics.Process.GetProcesses() to get process list.
I found TotalProcessorTime property for each process - it is TimeSpan.
But how to get relative values of CPU usage, i. e. i need % of total CPU usage for each running process.

use WQL (queries WMI like SQL)
see attached link for few samples:WQL
Win32_PerfFormattedData_PerfProc_Process is your class for getting CPU data.

I found a workaround decision see this question
The plan is as follows: take counters data in stndard way, then process log files.
Realtime monitoring is not critical.

Can a C# program measure its own CPU usage somehow?

I am working on a background program that will be running for a long time, and I have a external logging program (SmartInspect) that I want to feed with some values periodically, to monitor it in realtime when debugging.
I know I can simply fire up multiple programs, like the Task Manager, or IARSN TaskInfo, but I'd like to keep everything in my own program for this, as I also wants to add some simple rules like if the program uses more than X% CPU, flag this in the log.
I have a background thread that periodically feeds some statistics to SmartInspect, like memory consumption, working set, etc.
Is it possible for this thread to get a reasonably accurate measure of how much of the computer's CPU resources it consumes? The main program is a single-threaded application (apart from the watchdog thread that logs statistics) so if a technique is limited to how much does a single thread use then that would be good too.
I found some entries related to something called rusage for Linux and C. Is there something similar I can use for this?
Edit: Ok, I tried the performance counter way, but it added quite a lot of GC-data each time called, so the graph for memory usage and garbage collection skyrocketed. I guess I'll just leave this part out for now.

You can also use System.Diagnostics.Process.TotalProcessorTime and System.Diagnostics.ProcessThread.TotalProcessorTime properties to calculate your processor usage as this article describes.

Have a look at System.Diagnostics.PerformanceCounter. If you run up perfmon.exe, you'll see the range of performance counters available to you (set the 'performance object' to 'Process'), one of which is '% Processor Time'.

You can through the System.Diagnostic.PerformanceCounter class. Here's an example of somebody monitoring CPU usage:
http://blogs.msdn.com/dotnetinterop/archive/2007/02/02/system-diagnostics-performancecounter-and-processor-time-on-multi-core-or-multi-cpu.aspx
Note that this does require elevated privileges. And there may be a performance hit using it.

It is good that you are logging to monitors like smartinspect. But windows itself gathers the data for each resource in this case your program (or process). WMI is the standard for Application monitoring. We can view the data captured by WMI. Many application management, health monitoring or applicaiton monitoring tools support WMI out of the box.
So I would not recommend you to log your CPU usage within the application to a log file.
If you think availablity and performance is critical then go for solutions like Microsoft Operations manager solution.
To get an idea about WMI and to get the list of process see below:
- Win32_PerfFormattedData_PerfProc_Process to get Cpu time, filter is processID
See this article
- You can get the processID from Win32_process class.
WMI Made Easy For C# by Kevin Matthew Goss
oConn.Username = "JohnDoe";
oConn.Password = "JohnsPass";
System.Management.ManagementScope oMs = new System.Management.ManagementScope("\\MachineX", oConn);
//get Fixed disk stats
System.Management.ObjectQuery oQuery = new System.Management.ObjectQuery("select FreeSpace,Size,Name from Win32_LogicalDisk where DriveType=3");
//Execute the query
ManagementObjectSearcher oSearcher = new ManagementObjectSearcher(oMs,oQuery);
//Get the results
ManagementObjectCollection oReturnCollection = oSearcher.Get();
//loop through found drives and write out info
foreach( ManagementObject oReturn in oReturnCollection )
{
// Disk name
Console.WriteLine("Name : " + oReturn["Name"].ToString());
// Free Space in bytes
Console.WriteLine("FreeSpace: " + oReturn["FreeSpace"].ToString());
// Size in bytes
Console.WriteLine("Size: " + oReturn["Size"].ToString());
}
You can monitor the process from Remote system as well.

This code project article describes how to use the high performance timer:
http://www.codeproject.com/KB/cs/highperformancetimercshar.aspx
You can use it to time the execution of your code.
Here you can find a number of open source C# profilers:
http://csharp-source.net/open-source/profile