We Keep Coding

Join statement in multiple thread

I have a program that starts 2 threads and use Join.My understanding says that joins blocks the calling operation till it is finished executing .So,the below program should give 2 Million as answer since both the threads blocks till execution is completed but I am always getting the different value.This might be because first thread is completed but second thread is not run completely.
Can someone please explain the output.
Reference -Multithreading: When would I use a Join?
namespace ThreadSample
{
class Program
{
static int Total = 0;
public static void Main()
{
Thread thread1 = new Thread(Program.AddOneMillion);
Thread thread2 = new Thread(Program.AddOneMillion);
thread1.Start();
thread2.Start();
thread1.Join();
thread2.Join();
Console.WriteLine("Total = " + Total);
Console.ReadLine();
}
public static void AddOneMillion()
{
for (int i = 1; i <= 1000000; i++)
{
Total++;
}
}
}
}

When you call start method of thread, it starts immediately. hence by the time u call join on the thread1, thread2 would also have started. As a result variable 'Total' will be accessed by both threads simultaneously. Hence you will not get correct result as one thread operation is overwriting the value of 'Total' value causing data lose.
public static void Main()
{
Thread thread1 = new Thread(Program.AddOneMillion);
Thread thread2 = new Thread(Program.AddOneMillion);
thread1.Start(); //starts immediately
thread2.Start();//starts immediately
thread1.Join(); //By the time this line executes, both threads have accessed the Total varaible causing data loss or corruption.
thread2.Join();
Console.WriteLine("Total = " + Total);
Console.ReadLine();
}
Inorder to correct results either u can lock the Total variable as follows
static object _l = new object();
public static void AddOneMillion()
{
for (int i = 0; i < 1000000; i++)
{
lock(_l)
ii++;
}
}
U can use Interlocked.Increment which atomically updates the variable.
Please refer the link posted by #Emanuel Vintilă in the comment for more insight.
public static void AddOneMillion()
{
for (int i = 0; i < 1000000; i++)
{
Interlocked.Increment(ref Total);
}
}

It's because the increment operation is not done atomically. That means that each thread may hold a copy of Total and increment it. To avoid that you can use a lock or Interlock.Increment that is specific to incrementing a variable.
Clarification:
thread 1: read copy of Total
thread 2: read copy of Total
thread 1: increment and store Total
thread 2: increment and store Total (overwriting previous value)
I leave you with all possible scenarios where things could go wrong.
I would suggest avoiding explicit threading when possible and use map reduce operations that are less error prone.
You need to read about multi-threading programming and functional programming constructs available in mainstream languages. Most languages have added libraries to leverage the multicore capabilities of modern CPUs.

Concurrent Tasks and different cursor positions for each of them

I am starting to practice with Tasks and I tried the following code:
static void Main()
{
Task.Factory.StartNew(() =>
{
Write('a', 0);
});
var t = new Task(() =>
{
Write('b', 10);
});
t.Start();
Write('c', 20);
Console.ReadLine();
}
static void Write(char c, int x)
{
int yCounter = 0;
for (int i = 0; i < 1000; i++)
{
Console.WriteLine(c);
Console.SetCursorPosition(x, yCounter);
yCounter++;
Thread.Sleep(100);
}
}
My idea was to see how the console will go between the three different columns to output the different characters. It does swap the columns, but it does not output the correct characters. For example, in the first column it needs to output only 'a', but it also outputs 'b' and 'c', same goes for the other 2 columns.

This might be a particularly bad example of using tasks - or an example of how to use tasks badly.
Within your tasks you are setting a global state (SetCursorPosition), which will of course affect the other tasks (Console is static after all). It's possible that
Console.WriteLine('b')
is called after the cursor has been set to 0, to 10 or to 20, vice versa for the other values. Tasks should not rely on any global (or class level) state that might have changed (except if it's okay for the task that the value might have changed). With regards to your example, you would have to assure that none of the other tasks call SetCursorPosition before you have written your output. The easiest way to achieve this is locking the task
private static object lockObject = new object(); // you need an object of a reference type for locking
static void Write(char c, int x)
{
int yCounter = 0;
for (int i = 0; i < 1000; i++)
{
lock(lockObject)
{
Console.SetCursorPosition(x, yCounter);
Console.Write(c);
}
yCounter++;
Thread.Sleep(100);
}
}
The lock assures that no two tasks enter the block at a time (given that the lock object is the very same), hence each task can set the cursor to the position it wants to write at and write its char without any other tasks setting the cursor to any other position. (Plus, I've swapped Write and SetCursorPosition, since we'll have to call SetCursorPosition before writing to the output - the lock would be useless without swappinng those two lines, anyway.)

In addition to Paul's answer.
If you're dealing with tasks and async/await, don't mix Task and Thread in any way.
Executing your Write method using Task.Run/Task.Start is called "async-over-sync". This is a bad practice, and should be avoided.
Here's your code, rewritten in async manner, with async synchronization:
class Program
{
static void Main(string[] args)
{
var asyncLock = new AsyncLock();
// we need ToList here, since IEnumerable is lazy,
// and must be enumerated to produce values (tasks in this case);
// WriteAsync call inside Select produces a "hot" task - task, that is already scheduled;
// there's no need to start hot tasks explicitly - they are already started
new[] { ('a', 0), ('b', 10), ('c', 20) }
.Select(_ => WriteAsync(_.Item1, _.Item2, asyncLock))
.ToList();
Console.ReadLine();
}
static async Task WriteAsync(char c, int x, AsyncLock asyncLock)
{
for (var i = 0; i < 1000; i++)
{
using (await asyncLock.LockAsync())
{
Console.SetCursorPosition(x, i);
Console.Write(c);
}
await Task.Delay(100);
}
}
}
AsyncLock lives in Nito.AsyncEx package.

Thread Local Storage working principle

This is an example about Thread Local Storage (TLS) from Apress parallel programming book. I know that if we have 4 cores computer 4 thread can run parallel in same time. In this example we create 10 task and we suppose that have 4 cores computer. Each Thread local storage live in on thread so when start 10 task parallel only 4 thread perform. And We have 4 TLS so 10 task try to change 4 Thread local storage object. i want to ask how Tls prevent data race problem when thread count < Task count ??
using System;
using System.Threading;
using System.Threading.Tasks;
namespace Listing_04
{
class BankAccount
{
public int Balance
{
get;
set;
}
}
class Listing_04
{
static void Main(string[] args)
{
// create the bank account instance
BankAccount account = new BankAccount();
// create an array of tasks
Task<int>[] tasks = new Task<int>[10];
// create the thread local storage
ThreadLocal<int> tls = new ThreadLocal<int>();
for (int i = 0; i < 10; i++)
{
// create a new task
tasks[i] = new Task<int>((stateObject) =>
{
// get the state object and use it
// to set the TLS data
tls.Value = (int)stateObject;
// enter a loop for 1000 balance updates
for (int j = 0; j < 1000; j++)
{
// update the TLS balance
tls.Value++;
}
// return the updated balance
return tls.Value;
}, account.Balance);
// start the new task
tasks[i].Start();
}
// get the result from each task and add it to
// the balance
for (int i = 0; i < 10; i++)
{
account.Balance += tasks[i].Result;
}
// write out the counter value
Console.WriteLine("Expected value {0}, Balance: {1}",
10000, account.Balance);
// wait for input before exiting
Console.WriteLine("Press enter to finish");
Console.ReadLine();
}
}
}

We have 4 TLS so 10 task try to change 4 Thread local storage object
In your example, you could have anywhere between 1 and 10 TLS slots. This is because a) you are not managing your threads explicitly and so the tasks are executed using the thread pool, and b) the thread pool creates and destroys threads over time according to demand.
A loop of only 1000 iterations will completely almost instantaneously. So it's likely all ten of your tasks will get through the thread pool before the thread pool decides a work item has been waiting long enough to justify adding any new threads. But there is no guarantee of this.
Some important parts of the documentation include these statements:
By default, the minimum number of threads is set to the number of processors on a system
and
When demand is low, the actual number of thread pool threads can fall below the minimum values.
In other words, on your four-core system, the default minimum number of threads is four, but the actual number of threads active in the thread pool could in fact be less than that. And if the tasks take long enough to execute, the number of active threads could rise above that.
The biggest thing to keep in mind here is that using TLS in the context of a thread pool is almost certainly the wrong thing to do.
You use TLS when you have control over the threads, and you want a thread to be able to maintain some data private or unique to that thread. That's the opposite of what happens when you are using the thread pool. Even in the simplest case, multiple tasks can use the same thread, and so would wind up sharing TLS. And in more complicated scenarios, such as when using await, a single task could wind up executed in different threads, and so that one task could wind up using different TLS values depending on what thread is assigned to that task at that moment.
how Tls prevent data race problem when thread count < Task count ??
That depends on what "data race problem" you're talking about.
The fact is, the code you posted is filled with problems that are at the very least odd, if not outright wrong. For example, you are passing account.Balance as the initial value for each task. But why? This value is evaluated when you create the task, before it could ever be modified later, so what's the point of passing it?
And if you thought you were passing whatever the current value is when the task starts, that seems like that would be wrong too. Why would it be valid to make the starting value for a given task vary according to how many tasks had already completed and been accounted for in your later loop? (To be clear: that's not what's happening…but even if it were, it'd be a strange thing to do.)
Beyond all that, it's not clear what you thought using TLS here would accomplish anyway. When each task starts, you reinitialize the TLS value to 0 (i.e. the value of account.Balance that you've passed to the Task<int> constructor). So no thread involved ever sees a value other than 0 during the context of executing any given task. A local variable would accomplish exactly the same thing, without the overhead of TLS and without confusing anyone who reads the code and tries to figure out why TLS was used when it adds no value to the code.
So, does TLS solve some sort of "data race problem"? Not in this example, it doesn't appear to. So asking how it does that is impossible to answer. It doesn't do that, so there is no "how".
For what it's worth, I modified your example slightly so that it would report the individual threads that were assigned to the tasks. I found that on my machine, the number of threads used varied between two and eight. This is consistent with my eight-core machine, with the variation due to how much the first thread in the pool can get done before the pool has initialized additional threads and assigned tasks to them. Most commonly, I would see the first thread completing between three and five of the tasks, with the remaining tasks handled by remaining individual threads.
In each case, the thread pool created eight threads as soon as the tasks were started. But most of the time, at least one of those threads wound up unused, because the other threads were able to complete the tasks before the pool was saturated. That is, there is overhead in the thread pool just managing the tasks, and in your example the tasks are so inexpensive that this overhead allows one or more thread pool threads to finish one task before the thread pool needs that thread for another.
I've copied that version below. Note that I also added a delay between trial iterations, to allow the thread pool to terminate the threads it created (on my machine, this took 20 seconds, hence the delay time hard-coded…you can see the threads being terminated in the debugger output).
static void Main(string[] args)
{
while (_PromptContinue())
{
// create the bank account instance
BankAccount account = new BankAccount();
// create an array of tasks
Task<int>[] tasks = new Task<int>[10];
// create the thread local storage
ThreadLocal<int> tlsBalance = new ThreadLocal<int>();
ThreadLocal<(int Id, int Count)> tlsIds = new ThreadLocal<(int, int)>(
() => (Thread.CurrentThread.ManagedThreadId, 0), true);
for (int i = 0; i < 10; i++)
{
int k = i;
// create a new task
tasks[i] = new Task<int>((stateObject) =>
{
// get the state object and use it
// to set the TLS data
tlsBalance.Value = (int)stateObject;
(int id, int count) = tlsIds.Value;
tlsIds.Value = (id, count + 1);
Console.WriteLine($"task {k}: thread {id}, initial value {tlsBalance.Value}");
// enter a loop for 1000 balance updates
for (int j = 0; j < 1000; j++)
{
// update the TLS balance
tlsBalance.Value++;
}
// return the updated balance
return tlsBalance.Value;
}, account.Balance);
// start the new task
tasks[i].Start();
}
// Make sure this thread isn't busy at all while the thread pool threads are working
Task.WaitAll(tasks);
// get the result from each task and add it to
// the balance
for (int i = 0; i < 10; i++)
{
account.Balance += tasks[i].Result;
}
// write out the counter value
Console.WriteLine("Expected value {0}, Balance: {1}", 10000, account.Balance);
Console.WriteLine("{0} thread ids used: {1}",
tlsIds.Values.Count,
string.Join(", ", tlsIds.Values.Select(t => $"{t.Id} ({t.Count})")));
System.Diagnostics.Debug.WriteLine("done!");
_Countdown(TimeSpan.FromSeconds(20));
}
}
private static void _Countdown(TimeSpan delay)
{
System.Diagnostics.Stopwatch sw = System.Diagnostics.Stopwatch.StartNew();
TimeSpan remaining = delay - sw.Elapsed,
sleepMax = TimeSpan.FromMilliseconds(250);
int cchMax = $"{delay.TotalSeconds,2:0}".Length;
string format = $"\r{{0,{cchMax}:0}}", previousText = null;
while (remaining > TimeSpan.Zero)
{
string nextText = string.Format(format, remaining.TotalSeconds);
if (previousText != nextText)
{
Console.Write(format, remaining.TotalSeconds);
previousText = nextText;
}
Thread.Sleep(remaining > sleepMax ? sleepMax : remaining);
remaining = delay - sw.Elapsed;
}
Console.Write(new string(' ', cchMax));
Console.Write('\r');
}
private static bool _PromptContinue()
{
Console.Write("Press Esc to exit, any other key to proceed: ");
try
{
return Console.ReadKey(true).Key != ConsoleKey.Escape;
}
finally
{
Console.WriteLine();
}
}

Multi threading in C# (Parallel.For(), the Threading class or async and await)

Introduction
Just out of curiosity, how can I make a multi threading loop where the order is guaranteed from low to high and Console.WriteLine("Done"); has been executed after the loop has done. I've tried three cases explained below.
Case one: Parallel.For(...)
If you will make a for loop that runs parallel in C#, you can use this code:
ParallelLoopResult result = Parallel.For(0, 10, i =>
{
Console.WriteLine($"{i}, task: {Task.CurrentId}, Thread: {Thread.CurrentThread.ManagedThreadId}";
Thread.Sleep(10);
}
Console.WriteLine($"Is completed: {result.IsCompleted}";
This will give next output where the order is not guaranteed. You will see different results if you run the code once more. The run of the program has the order 0-2-4-6-8-... with five tasks and six threads.
0, task: 1, thread: 1
2, task: 2, thread: 3
4, task: 3, thread: 4
6, task: 4, thread: 5
8, task: 5, thread: 6
5, task: 3, thread: 4
7, task: 4, thread: 5
9, task: 5, thread: 6
3, task: 2, thread: 3
1, task: 1, thread: 1
Is completed: True
Case two: the Thread class
I've also tried to use threading instead of the Parallel class. Here is the code:
public static void Main(string[] args)
{
for (int i = 0; i < number; i++)
{
Thread tr = new Thread(() => Print(i));
tr.IsBackground = true;
tr.Start();
}
Console.WriteLine("Done");
}
private static void Print(int i)
{
Console.WriteLine(i);
Thread.Sleep(10);
}
Following result of this code is now that the order isn't also guaranteed form low to high and the string Done has been written after the for loop end but not always guaranteed, however I've place thread.Start(); after Console.WriteLine("Done");.
1
2
3
5
7
7
8
9
9
Done
Like in my second run:
2
3
3
4
5
6
7
8
Done
10
Case tree: async and await
I'll also try this way, but I didn't know how I can implement this way of threading for a loop.
Question
In all the cases I haven't the result I needed. In the first case was it not guaranteed to be ordered and in the second case has Done been written before the loop has run. So my question is how can I run a for loop where the order is always guaranteed and Done written after the loop has run?

You're asking how to make a multithreaded loop where the order of execution is guaranteed. As has been mentioned you can't. However, if you want to order the output, that can be done. Here you've two options
Save the results until they are all complete (see pseudocode below):
Create a dictionary of inputs to outputs
Create a multithreaded loop to process all the inputs and record the outputs
Once they are all complete, write the outputs (in order of inputs) to the screen
Order the outputs as they are complete. This is more difficult than #1, but allows you to show results faster.
Create a multithreaded loop to process all the inputs
Once each thread is complete, show its output, but don't just write it to the screen, insert it into the appropriate spot among all the sorted outputs.

Although you shouldn't use Threads for such sequential uses. What you're asking for is possible.
Here's my example of what you're trying to achieve. It uses multiple thread objects and is guaranteed order. However, you'd probably get better performance just doing it sequentially.
public static void PrintDone()
{
Console.WriteLine("Done");
}
public static void Print(int i)
{
Console.WriteLine(i);
Thread.Sleep(10);
}
static void Main(string[] args)
{
List<Thread> threads = new List<Thread>();
for(int i = 0; i < 10; ++i)
{
int numCopy = i;
Thread th = new Thread(() => Print(numCopy));
threads.Add(th);
}
for (int i = 0; i < 10; ++i)
{
threads[i].Start();
threads[i].Join();
}
PrintDone();
Console.ReadLine();
}
So the gist is to create a bunch of threads, then start them all sequentially (there's no way around that part really), and join the main thread directly after. What'll happen is the main thread will start a "subthread" and then wait for it to finish, once it's finished it'll start the next thread, and repeat until it's out of threads in the list.
However, if you're trying to parallelize the printing of the numbers and guarantee the output's order, you're out of luck. Still possible, but the overhead you'll get for trying to maintain the order will defeat the purpose of the parallelization.
Here's an truly parallel version of the code:
public static void PrintDone()
{
Console.WriteLine("Done");
}
public static void Print(int i, int[] threadStatus)
{
if(i != 0)
{
while(threadStatus[i-1] < 0)
{
Thread.Sleep(10);
}
}
Thread.Sleep(100);
threadStatus[i] = i;
Console.WriteLine(i);
}
static void Main(string[] args)
{
int[] threadStatus = Enumerable.Repeat(-1, 10).ToArray();
List<Thread> threads = new List<Thread>();
for(int i = 0; i < 10; ++i)
{
int numCopy = i;
Thread th = new Thread(() => Print(numCopy, threadStatus));
threads.Add(th);
}
for (int i = 0; i < 10; ++i)
{
threads[i].Start();
}
threads[9].Join();
PrintDone();
Console.ReadLine();
}
Since each thread can only write to one part of the int array and reading is atomic, we can use the int array to keep status on all the threads currently running.
This basically means that in any particular "print" thread you can check to see if they are ready to print within the thread, if not you tell them to sleep in a loop so they periodically check their status.

Why volatile and MemoryBarrier do not prevent operations reordering?

If I understand meaning of volatile and MemoryBarrier correctly than the program below has never to be able to show any result.
It catches reordering of write operations every time I run it. It does not matter if I run it in Debug or Release. It also does not matter if I run it as 32bit or 64bit application.
Why does it happen?
using System;
using System.Threading;
using System.Threading.Tasks;
namespace FlipFlop
{
class Program
{
//Declaring these variables as volatile should instruct compiler to
//flush all caches from registers into the memory.
static volatile int a;
static volatile int b;
//Track a number of iteration that it took to detect operation reordering.
static long iterations = 0;
static object locker = new object();
//Indicates that operation reordering is not found yet.
static volatile bool continueTrying = true;
//Indicates that Check method should continue.
static volatile bool continueChecking = true;
static void Main(string[] args)
{
//Restarting test until able to catch reordering.
while (continueTrying)
{
iterations++;
var checker = new Task(Check);
var writter = new Task(Write);
lock (locker)
{
continueChecking = true;
checker.Start();
}
writter.Start();
checker.Wait();
writter.Wait();
}
Console.ReadKey();
}
static void Write()
{
//Writing is locked until Main will start Check() method.
lock (locker)
{
//Using memory barrier should prevent opration reordering.
a = 1;
Thread.MemoryBarrier();
b = 10;
Thread.MemoryBarrier();
b = 20;
Thread.MemoryBarrier();
a = 2;
//Stops spinning in the Check method.
continueChecking = false;
}
}
static void Check()
{
//Spins until finds operation reordering or stopped by Write method.
while (continueChecking)
{
int tempA = a;
int tempB = b;
if (tempB == 10 && tempA == 2)
{
continueTrying = false;
Console.WriteLine("Caught when a = {0} and b = {1}", tempA, tempB);
Console.WriteLine("In " + iterations + " iterations.");
break;
}
}
}
}
}

You aren't cleaning the variables between tests, so (for all but the first) initially a is 2 and b is 20 - before Write has done anything.
Check can get that initial value of a (so tempA is 2), and then Write can get in, get as far as changing b to 10.
Now Check reads the b (so tempB is 10).
Et voila. No re-order necessary to repro.
Reset a and b to 0 between runs and I expect it will go away.
edit: confirmed; "as is" I get the issue almost immediately (<2000 iterations); but by adding:
while (continueTrying)
{
a = b = 0; // reset <======= added this
it then loops for any amount of time without any issue.
Or as a flow:
Write A= B= Check
(except first run) 2 20
int tempA = a;
a = 1; 1 20
Thread.MemoryBarrier();
b = 10; 1 10
int tempB = b;

I don't think this is re-ordering.
This piece of code is simply not thread-safe:
while (continueChecking)
{
int tempA = a;
int tempB = b;
...
I think this scenario is possible:
int tempA = a; executes with the values of the last loop (a == 2)
There is a context switch to the Write thread
b = 10 and the loop stops
There is a context switch to the Check thread
int tempB = b; executes with b == 10
I notice that the calls to MemoryBarrier() enhance the chances of this scenario. Probably because they cause more context-switching.

The result has nothing to do with reordering, with memory barries, or with volatile. All these constructs are needed to avoid effects of compiler or CPU reordering of the instructions.
But this program would produce the same result even assuming fully consistent single-CPU memory model and no compiler optimization.
First of all, notice that there will be multiple Write() tasks started in parallel. They are running sequentially due to lock() inside Write(), but a signle Check() method can read a and b produced by different instances of Write() tasks.
Because Check() function has no synchronization with Write function - it can read a and b at two arbitrary and different moments. There is nothing in your code that prevents Check() from reading a produced by previous Write() at one moment and then reading b produced by following Write() at another moment. First of all you need synchronization (lock) in Check() and then you might (but probably not in this case) need memory barriers and volatile to fight with memory model problems.
This is all you need:
int tempA, tempB;
lock (locker)
{
tempA = a;
tempB = b;
}

If you use MemoryBarrier in writer, why don't you do that in checker? Put Thread.MemoryBarrier(); before int tempA = a;.
Calling Thread.MemoryBarrier(); so many times blocks all of the advantages of the method. Call it only once before or after a = 1;.