We Keep Coding

Threads can run on different processors or cores for both Task.Factory.StartNew and Parallel.Invoke

I am looking to clarify my understanding of .NET multithreading, and in particular, which .NET methods create threads which may potentially execute at the same time on different processors or cores in a multi-processor/core system.
In the .NET TPL framework you can use the methods Parallel.Invoke, or Task.Factory.StartNew to achieve some kind of parallelism.
My understanding is that in both cases .NET creates new Tasks (behind the scenes for Parallel.Invoke), which the .NET environment then allocates to managed threads behind the scenes, which are then assigned to threads, which the CPU may allocate to different cores or processors depending on the workload. The main difference between the two methods is semantics - Parallel.Invoke executes multiple tasks and waits for them to complete; Task.Factory.StartNew starts a new task in the background. In both cases, the actual work may be done on different cores or processors. As per Task Parallel Library (TPL).
I have a colleague who is convinced that only the Parallel.Invoke method allows the threads to be executed on different cores/processors, and that Task.Factory.StartNew starts a new thread but that thread will only be scheduled on one core/processor - so doesn't actually give parallelism.
I can't find any documentation or articles which explicitly state whether this is the case or not. My colleague refers me to the same articles that I am looking at, such as Task-based Asynchronous Programming, which I think validate my understanding but my colleague thinks validates his.
The documentation sometimes uses the term "parallel processing" with reference to Parallel.Invoke and "asynchronous tasks" with reference to "Task.Factory.StartNew", but as far as I understand the same thing happens in the background with regards to allocation to multiple processors/cores.
Can anyone please help to clarify the situation, if possible with links to documentation/articles.
I know this sounds like seeking resolution to an argument with a colleague, but I am genuinely looking to clarify whether or not I understand this correctly.

It's actually pretty easy to answer.
Task.Run()
Queues the specified work to run on the ThreadPool ....
Task Parallel Library
... In addition, the TPL handles the partitioning of the work, the scheduling of threads on the ThreadPool, ....
Using the same ThreadPool how is it possible for the ThreadPool to determine the type of task in order to limit the CPU? Either they both run on all Processors or the they all run one a Single Processor.
Extra Credit:
This begs the question, Is the ThreadPool Multi-Core aware?
The answer is surprisingly, it doesn't care. The ThreadPool asks the operating system (just like any c# application that uses new Thread()) for a Thread, it actually the responsibility of the OS. I think it would be pretty clear by now that with all the abstraction that even suggesting that C# can by default limit how threads are used is a pretty ridiculous assertion. (Yes you can run a thread on whatever core you want etc etc, but that is not how the ThreadPool works by default).
I highly recommend reading StartNew is Dangerous... TLDR? Use Task.Run().

Although operating systems sometimes provide for "processor affinity," this is an edge-case and its use (or availability) is quite rare. So far as I am aware, .NET does not make any use of such things.
Your foundation assumption must always be: "a runnable thread/process will run where it damn well pleases," and it might switch from one CPU resource to another at any time. The .NET framework makes things a whole lot "nicer" for you in a lot of ways, but the underlying scheduling decisions are still being made – exclusively – by the host operating system.

What is a multithreading program and how does it work?

What is a multithreading program and how does it work exactly? I read some documents but I'm confused. I know that code is executed line by line, but I can't understand how the program manages this.
A simple answer would be appreciated.c# example please (only animation!)

What is a multi-threading program and how does it work exactly?
Interesting part about this question is complete books are written on the topic, but still it is elusive to lot of people. I will try to explain in the order detailed underneath.
Please note this is just to provide a gist, an answer like this can never do justice to the depth and detail required. Regarding videos, best that I have come across are part of paid subscriptions (Wintellect and Pluralsight), check out if you can listen to them on trial basis, assuming you don't already have the subscription:
Wintellect by Jeffery Ritcher (from his Book, CLR via C#, has same chapter on Thread Fundamentals)
CLR Threading by Mike Woodring
Explanation Order
What is a thread ?
Why were threads introduced, main purpose ?
Pitfalls and how to avoid them, using Synchronization constructs ?
Thread Vs ThreadPool ?
Evolution of Multi threaded programming API, like Parallel API, Task API
Concurrent Collections, usage ?
Async-Await, thread but no thread, why they are best for IO
What is a thread ?
It is software implementation, which is purely a Windows OS concept (multi-threaded architecture), it is bare minimum unit of work. Every process on windows OS has at least one thread, every method call is done on the thread. Each process can have multiple threads, to do multiple things in parallel (provided hardware support).
Other Unix based OS are multi process architecture, in fact in Windows, even the most complex piece of software like Oracle.exe have single process with multiple threads for different critical background operations.
Why were threads introduced, main purpose ?
Contrary to the perception that concurrency is the main purpose, it was robustness that lead to the introduction of threads, imagine every process on Windows is running using same thread (in the initial 16 bit version) and out of them one process crash, that simply means system restart to recover in most of the cases. Usage of threads for concurrent operations, as multiple of them can be invoked in each process, came in picture down the line. In fact it is even important to utilize the processor with multiple cores to its full ability.
Pitfalls and how to avoid using Synchronization constructs ?
More threads means, more work completed concurrently, but issue comes, when same memory is accessed, especially for Write, as that's when it can lead to:
Memory corruption
Race condition
Also, another issue is thread is a very costly resource, each thread has a thread environment block, Kernel memory allocation. Also for scheduling each thread on a processor core, time is spent for context switching. It is quite possible that misuse can cause huge performance penalty, instead of improvement.
To avoid Thread related corruption issues, its important to use the Synchronization constructs, like lock, mutex, semaphore, based on requirement. Read is always thread safe, but Write needs appropriate Synchronization.
Thread Vs ThreadPool ?
Real threads are not the ones, we use in C#.Net, that's just the managed wrapper to invoke Win32 threads. Challenge remain in user's ability to grossly misuse, like invoking lot more than required number of threads, assigning the processor affinity, so isn't it better that we request a standard pool to queue the work item and its windows which decide when the new thread is required, when an already existing thread can schedule the work item. Thread is a costly resource, which needs to be optimized in usage, else it can be bane not boon.
Evolution of Multi threaded programming, like Parallel API, Task API
From .Net 4.0 onward, variety of new APIs Parallel.For, Parallel.ForEach for data paralellization and Task Parallelization, have made it very simple to introduce concurrency in the system. These APIs again work using a Thread pool internally. Task is more like scheduling a work for sometime in the future. Now introducing concurrency is like a breeze, though still synchronization constructs are required to avoid memory corruption, race condition or thread safe collections can be used.
Concurrent Collections, usage ?
Implementations like ConcurrentBag, ConcurrentQueue, ConcurrentDictionary, part of System.Collections.Concurrent are inherent thread safe, using spin-wait and much easier and quicker than explicit Synchronization. Also much easier to manage and work. There's another set API like ImmutableList System.Collections.Immutable, available via nuget, which are thread safe by virtue of creating another copy of data structure internally.
Async-Await, thread but no thread, why they are best for IO
This is an important aspect of concurrency meant for IO calls (disk, network), other APIs discussed till now, are meant for compute based concurrency so threads are important and make it faster, but for IO calls thread has no use except waiting for the call to return, IO calls are processed on hardware based queue IO Completion ports

A simple analogy might be found in the kitchen.
You've probably cooked using a recipe before -- start with the specified ingredients, follow the steps indicated in the recipe, and at the end you (hopefully) have a delicious dish ready to eat. If you do that, then you have executed a traditional (non-multithreaded) program.
But what if you have to cook a full meal, which includes a number of different dishes? The simple way to do it would be to start with the first recipe, do everything the recipe says, and when it's done, put the finished dish (and the first recipe) aside, then start on the second recipe, do everything it says, put the second dish (and second recipe) aside, and so on until you've gone through all of the recipes one after another. That will work, but you might end up spending 10 hours in the kitchen, and of course by the time the last dish is ready to eat, the first dish might be cold and unappetizing.
So instead you'd probably do what most chefs do, which is to start working on several recipes at the same time. For example, you might put the roast in the oven for 45 minutes, but instead of sitting in front of the oven waiting 45 minutes for the roast to cook, you'd spend the 45 minutes chopping the vegetables. When the oven timer rings, you put down your vegetable knife, pull the cooked roast out of the oven and let it cool, then go back to chopping vegetables, and so on. If you can do that, then you are successfully multitasking several recipes/programs. That is, you aren't literally working on multiple recipes at once (you still have only two hands!), but you are jumping back and forth from following one recipe to following another whenever necessary, and thereby making progress on several tasks rather than twiddling your thumbs a lot. Do this well and you can have the whole meal ready to eat in a much shorter amount of time, and everything will be hot and fresh at about the same time too. If you do this, you are executing a simple multithreaded program.
Then if you wanted to get really fancy, you might hire a few other chefs to work in the kitchen at the same time as you, so that you can get even more food prepared in a given amount of time. If you do this, your team is doing multiprocessing, with each chef taking one part of the total work and all of them working simultaneously. Note that each chef may well be working on multiple recipes (i.e. multitasking) as described in the previous paragraph.
As for how a computer does this sort of thing (no more analogies about chefs), it usually implements it using a list of ready-to-run threads and a timer. When the timer goes off (or when the thread that is currently executing has nothing to do for a while, because e.g. it is waiting to load data from a slow hard drive or something), the operating system does a context switch, in which pauses the current thread (by putting it into a list somewhere and no longer executing instructions from that thread's code anymore), then pulls another ready-to-run thread from the list of ready-to-run threads and starts executing instructions from that thread's code instead. This repeats for as long as necessary, often with context switches happening every few milliseconds, giving the illusion that multiple programs are running "at the same time" even on a single-core CPU. (On a multi-core CPU it does this same thing on each core, and in that case it's no longer just an illusion; multiple programs really are running at the same time)

Why don't you refer to Microsoft's very own documentation of the .net class System.Threading.Thread?
It has a handfull of simple example programs written in C# (at the bottom of the page) just as you asked for:
Thread Examples

actually multi thread is do multiple process at the same time together . and you can complete process parallel .

it's actually multi thread is do multiple process at the same time together . and you can complete process parallel . you can take task from your main thread then execute some other way and done .

Are there any cases when it's preferable to use a plain old Thread object instead of one of the newer constructs?

I see a lot of people in blog posts and here on SO either avoiding or advising against the usage of the Thread class in recent versions of C# (and I mean of course 4.0+, with the addition of Task & friends). Even before, there were debates about the fact that a plain old thread's functionality can be replaced in many cases by the ThreadPool class.
Also, other specialized mechanisms are further rendering the Thread class less appealing, such as Timers replacing the ugly Thread + Sleep combo, while for GUIs we have BackgroundWorker, etc.
Still, the Thread seems to remain a very familiar concept for some people (myself included), people that, when confronted with a task that involves some kind of parallel execution, jump directly to using the good old Thread class. I've been wondering lately if it's time to amend my ways.
So my question is, are there any cases when it's necessary or useful to use a plain old Thread object instead of one of the above constructs?

The Thread class cannot be made obsolete because obviously it is an implementation detail of all those other patterns you mention.
But that's not really your question; your question is
are there any cases when it's necessary or useful to use a plain old Thread object instead of one of the above constructs?
Sure. In precisely those cases where one of the higher-level constructs does not meet your needs.
My advice is that if you find yourself in a situation where existing higher-abstraction tools do not meet your needs, and you wish to implement a solution using threads, then you should identify the missing abstraction that you really need, and then implement that abstraction using threads, and then use the abstraction.

Threads are a basic building block for certain things (namely parallelism and asynchrony) and thus should not be taken away. However, for most people and most use cases there are more appropriate things to use which you mentioned, such as thread pools (which provide a nice way of handling many small jobs in parallel without overloading the machine by spawning 2000 threads at once), BackgroundWorker (which encapsulates useful events for a single shortlived piece of work).
But just because in many cases those are more appropriate as they shield the programmer from needlessly reinventing the wheel, doing stupid mistakes and the like, that does not mean that the Thread class is obsolete. It is still used by the abstractions named above and you would still need it if you need fine-grained control over threads that is not covered by the more special classes.
In a similar vein, .NET doesn't forbid the use of arrays, despite List<T> being a better fit for many cases where people use arrays. Simply because you may still want to build things that are not covered by the standard lib.

Task and Thread are different abstractions. If you want to model a thread, the Thread class is still the most appropriate choice. E.g. if you need to interact with the current thread, I don't see any better types for this.
However, as you point out .NET has added several dedicated abstractions which are preferable over Thread in many cases.

The Thread class is not obsolete, it is still useful in special circumstances.
Where I work we wrote a 'background processor' as part of a content management system: a Windows service that monitors directories, e-mail addresses and RSS feeds, and every time something new shows up execute a task on it - typically to import the data.
Attempts to use the thread pool for this did not work: it tries to execute too much stuff at the same time and trash the disks, so we implemented our own polling and execution system using directly the Thread class.

The new options make direct use and management of the (expensive) threads less frequent.
people that, when confronted with a task that involves some kind of parallel execution, jump directly to using the good old Thread class.
Which is a very expensive and relatively complex way of doing stuff in parallel.
Note that the expense matters most: You cannot use a full thread to do a small job, it would be counterproductive. The ThreadPool combats the costs, the Task class the complexities (exceptions, waiting and canceling).

To answer the question of "are there any cases when it's necessary or useful to use a plain old Thread object", I'd say a plain old Thread is useful (but not necessary) when you have a long running process that you won't ever interact with from a different thread.
For example, if you're writing an application that subscribes to receive messages from some sort of message queue and you're application is going to do more than just process those messages then it would be useful to use a Thread because the thread will be self-contained (i.e. you aren't waiting on it to get done), and it isn't short-lived. Using the ThreadPool class is more for queuing up a bunch of short-lived work items and allowing the ThreadPool class manage efficiently processing each one as a new Thread is available. Tasks can be used where you would use Thread directly, but in the above scenario I don't think they would buy you much. They help you interact with the thread more easily (which the above scenario doesn't need) and they help determine how many Threads actually should be used for the given set of tasks based on the number of processors you have (which isn't what you want, so you'd tell the Task your thing is LongRunning in which case in the current 4.0 implementation it would simply create a separate non-pooled Thread).

Probably not the answer you were expecting, but I use Thread all the time when coding against the .NET Micro Framework. MF is quite cut down and doesn't include higher level abstractions and the Thread class is super flexible when you need to get the last bit of performance out of a low MHz CPU.

You could compare the Thread class to ADO.NET. It's not the recommended tool for getting the job done, but its not obsolete. Other tools build on top of it to ease the job.
Its not wrong to use the Thread class over other things, especially if those things don't provide a functionality that you need.

It's not definitely obsolete.
The problem with multithreaded apps is that they are very hard to get right (often indeterministic behavior, input, output and also internal state is important), so a programmer should push as much work as possible to framework/tools. Abstract it away. But, the mortal enemy of abstraction is performance.
So my question is, are there any cases when it's necessary or useful
to use a plain old Thread object instead of one of the above
constructs?
I'd go with Threads and locks only if there will be serious performance problems, high performance goals.

I've always used the Thread class when I need to keep count and control over the threads I've spun up. I realize I could use the threadpool to hold all of my outstanding work, but I've never found a good way to keep track of how much work is currently being done or what the status is.
Instead, I create a collection and place the threads in them after I spin them up - the very last thing a thread does is remove itself from the collection. That way, I can always tell how many threads are running, and I can use the collection to ask each what it's doing. If there's a case when I need to kill them all, normally you'd have to set some kind of "Abort" flag in your application, wait for every thread to notice that on its own and self-terminate - in my case, I can walk the collection and issue a Thread.Abort to each one in turn.
In that case, I haven't found a better way that working directly with the Thread class. As Eric Lippert mentioned, the others are just higher-level abstractions, and it's appropriate to work with the lower-level classes when the available high-level implementations don't meet your need. Just as you sometimes need to do Win32 API calls when .NET doesn't address your exact needs, there will always be cases where the Thread class is the best choice despite recent "advancements."

Is it possible to use thread-concurrency and parallelism together?

For one of my projects thats kind of a content-aggregator i'd like to introduce concurrency and if possible parallelism. At first hand this may seem pointless because concurrency and parallelism take different approaches. (Concurrency via threads introduces immediate concurrency, where as parallelism provides a potential).
So to better explain my problem, let me summarize my problem set.
As my project is a content-aggregator (that aggregates feeds,podcasts and similar stuff) it basically reads the data from web, parses them to return the meaningful data.
So as of right now i took a very simplistic sequential approach. Let's say that we've some amount of feeds we have to parse.
foreach(feed in feeds)
{
read_from_web(feed)
parse(feed)
}
So with sequential approach time taken parse all feeds and process them greatly depends on not only the parser code but time needed to get the xml source from web. We all know that it may take variable time to get read the source from web (because of the network conditions and similar issues).
So to speed up the code i can take an approach of worker threads which will introduce an immediate concurrency;
So a defined number of worker threads can take a feed & parse concurrently (which will for sure speed up the whole the process - as we'll see lesser impact of waiting for data over the net).
This is all okay until the point that, my target audience of the project mostly runs multi-core cpus -- because of the fact that they're gamers --.
I want to also utilize these cores while processing the content so started reading on the potential parallelism http://oreilly.com/catalog/0790145310262. I've still not finished reading it yet and don't know if this is already discusses but i'm quite a bit obsessed with this and wanted to ask over stackoverflow to get an overall idea.
So as the book describes potential parallelism: Potential Parallelism means that your program is written so that it runs faster when parallel hardware is available and roughly the same as an equivalent sequential program when it's not.
So the real question is, while i'm using worker threads for concurrency, can i still use possible parallelism? (running my feed parsers on worker threads and still distributing them to cpu cores -- if the cpu supports multi-cores of course)

I think it's more useful to think about IO-bound work and CPU-bound work; threads can help with both.
For IO-bound work you are presumably waiting for external resources (in your case, feeds to be read). If you must wait on multiple external resources then it only makes sense to wait on them in parallel rather than wait on them one after the other. This is best done by spinning up threads which block on the IO.
For CPU-bound work you want to use all of your cores to maximize the throughput of completing that work. To do that, you should create a pool of worker threads roughly the same size as your number of cores and break up and distribute the work across them. [How you break up and distribute the work is itself an interesting problem.]
In practice, I find that most applications have both of these problems and it makes sense to use threads to solve both kinds of problems.

Okay it seems i was greatly mistaken by the books description on possible parallelism. Thanks to answers i was able to figure out things;
From msdn: http://msdn.microsoft.com/en-us/library/dd460717(VS.100).aspx
The Task Parallel Library (TPL) is a
set of public types and APIs in the
System.Threading and
System.Threading.Tasks namespaces in
the .NET Framework version 4. The
purpose of the TPL is to make
developers more productive by
simplifying the process of adding
parallelism and concurrency to
applications. The TPL scales the
degree of concurrency dynamically to
most efficiently use all the
processors that are available. In
addition, the TPL handles the
partitioning of the work, the
scheduling of threads on the
ThreadPool, cancellation support,
state management, and other low-level
details. By using TPL, you can
maximize the performance of your code
while focusing on the work that your
program is designed to accomplish.
So basically it means TPL can handle all the details of concurrency via threading and also supports parallelism on multi-cores.

C# Multi-Threading - What method to use?

My goal is to write a program that handles an arbitrary number of tasks based on given user input.
Let's say the # of tasks are 1000 in this case.
Now, I'd like to be able to have a dynamic number of threads that are spawned and start handling the tasks one by one.
I would assume I need to use a "synchronous" method, as opposed to a "asynchronous" one, so that in case one tasks has a problem, I wouldn't want it to slow down the completion of the rest.
What method would I use to accomplish the above? Semaphores? ThreadPools? And how do I make sure that a thread does not try to start a task that is already being handled by another thread? Would a "lock" handle this?
Code examples and/or links to sites that will point me in the right direction will be appreciated.
edit: The problem with the MSDN Fibonacci example is that the waitall method can only handle up to 64 waits. I need more than that due to the 1000 tasks. How to fix that situation without creating deadlocks?

Are these tasks independent? If so, you basically want a producer/consumer queue or a custom threadpool, which are effectively different views on the same thing. You need to be able to place tasks in a queue, and have multiple threads be able to read from that queue.
I have a custom threadpool in MiscUtil or there's a simple (nongeneric due to age) producer/consumer queue in my threading tutorial (about half way down this page).
If these tasks are reasonably long-running, I wouldn't use the system threadpool for this - it will spawn more threads than you probably want. If you're using .NET 4.0 beta 1 you could use Parallel Extensions though.
I'm not quite sure about your comment on WaitAll... are you trying to work out when everything's finished? In the producer/consumer queue case, that would probably involve having some sort of "stop" entry in the queue (e.g. null references which the consuming threads would understand to mean "quit") and then add a "WaitUntilEmpty" method (which should be fairly easy to implement). Note that you wouldn't need to wait until the last items had been processed, as they'd all be stop signals... by the time the queue has emptied, all the real work items will definitely have been processed anyway.

You'll probably want to use the ThreadPool to manage this.
I recommend reading up on MSDN on How to use the ThreadPool in C#. It covers many aspects of this, including firing tasks, and simple synchronization.
Using Threading in C# is the main section, and will cover other options.
If you happen to be using VS 2010 beta, and targetting .NET 4, the Task Parallel Library is a very good option for this - it simplifies some of these patterns.

You can't use it (yet) but the new Task class in .NET 4 would be ideal for this kind of situation.
Until then, the ThreadPool is your best bet. It has a (very) limited form of load-balancing. Note that if you try to start 1000 Threads you will probably get an Out Of Memory exception. The ThreadPool will handle that with ease.
Your sync problem can be handled with a simple (Interlocked) counter, if the timing is such that you can tolerate a Sleep(1) loop in the main thread. The ThreadPool is missing a more convenient way to do this.

A simple strategy to avoid a task is get by two or more thread is a syncronized (with a mutext for example) vector.
See this http://msdn.microsoft.com/en-us/library/yy12yx1f.aspx

Perhaps you can use the BackgroundWorker class. It creates a nice abstraction on top of the thread pool. You can even subclass it if you want to setup many similar jobs.

As has been mentioned, .NET 4 features the excellent Task Parallel Library. But you can use the June 2008 CTP of it in .NET 3.5 just fine. I've been doing this for some hobby projects myself, but if this is a commercial project, you should check out if there are legal issues.