We Keep Coding

Update console text from multiple threads not working

I am executing/processing very big files in multi threaded mode in a console app.
When I don't update/write to the console from threads, for testing the whole process take about 1 minute.
But when I try to update/write to console from threads to show the progress, the process stuck and it never finishes (waited several minutes even hours). And also console text/window does not updated as it should.
Update-1: As requested by few kind responder, i added minimal code that can reproduce the same error/problem
Here is the code from the thread function/method:
using System;
using System.Collections;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
namespace Large_Text_To_Small_Text
{
class Program
{
static string sAppPath;
static ArrayList objThreadList;
private struct ThreadFileInfo
{
public string sBaseDir, sRFile;
public int iCurFile, iTFile;
public bool bIncludesExtension;
}
static void Main(string[] args)
{
string sFileDir;
DateTime dtStart;
Console.Clear();
sAppPath = Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location);
sFileDir = #"d:\Test";
dtStart = DateTime.Now;
///process in multi threaded mode
List<string> lFiles;
lFiles = new List<string>();
lFiles.AddRange(Directory.GetFiles(sFileDir, "*.*", SearchOption.AllDirectories));
if (Directory.Exists(sFileDir + "-Processed") == true)
{
Directory.Delete(sFileDir + "-Processed", true);
}
Directory.CreateDirectory(sFileDir + "-Processed");
sPrepareThreading();
for (int iFLoop = 0; iFLoop < lFiles.Count; iFLoop++)
{
//Console.WriteLine(string.Format("{0}/{1}", (iFLoop + 1), lFiles.Count));
sThreadProcessFile(sFileDir + "-Processed", lFiles[iFLoop], (iFLoop + 1), lFiles.Count, Convert.ToBoolean(args[3]));
}
sFinishThreading();
Console.WriteLine(DateTime.Now.Subtract(dtStart).ToString());
Console.ReadKey();
return;
}
private static void sProcSO(object oThreadInfo)
{
var inputLines = new BlockingCollection<string>();
long lACounter, lCCounter;
ThreadFileInfo oProcInfo;
lACounter = 0;
lCCounter = 0;
oProcInfo = (ThreadFileInfo)oThreadInfo;
var readLines = Task.Factory.StartNew(() =>
{
foreach (var line in File.ReadLines(oProcInfo.sRFile))
{
inputLines.Add(line);
lACounter++;
}
inputLines.CompleteAdding();
});
var processLines = Task.Factory.StartNew(() =>
{
Parallel.ForEach(inputLines.GetConsumingEnumerable(), line =>
{
lCCounter++;
/*
some process goes here
*/
/*If i Comment out these lines program get stuck!*/
//Console.SetCursorPosition(0, oProcInfo.iCurFile);
//Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
});
});
Task.WaitAll(readLines, processLines);
}
private static void sPrepareThreading()
{
objThreadList = new ArrayList();
for (var iTLoop = 0; iTLoop < 5; iTLoop++)
{
objThreadList.Add(null);
}
}
private static void sThreadProcessFile(string sBaseDir, string sRFile, int iCurFile, int iTFile, bool bIncludesExtension)
{
Boolean bMatched;
Thread oCurThread;
ThreadFileInfo oProcInfo;
Salma_RecheckThread:
bMatched = false;
for (int iTLoop = 0; iTLoop < 5; iTLoop++)
{
if (objThreadList[iTLoop] == null || ((System.Threading.Thread)(objThreadList[iTLoop])).IsAlive == false)
{
oProcInfo = new ThreadFileInfo()
{
sBaseDir = sBaseDir,
sRFile = sRFile,
iCurFile = iCurFile,
iTFile = iTFile,
bIncludesExtension = bIncludesExtension
};
oCurThread = new Thread(sProcSO);
oCurThread.IsBackground = true;
oCurThread.Start(oProcInfo);
objThreadList[iTLoop] = oCurThread;
bMatched = true;
break;
}
}
if (bMatched == false)
{
System.Threading.Thread.Sleep(250);
goto Salma_RecheckThread;
}
}
private static void sFinishThreading()
{
Boolean bRunning;
Salma_RecheckThread:
bRunning = false;
for (int iTLoop = 0; iTLoop < 5; iTLoop++)
{
if (objThreadList[iTLoop] != null && ((System.Threading.Thread)(objThreadList[iTLoop])).IsAlive == true)
{
bRunning = true;
}
}
if (bRunning == true)
{
System.Threading.Thread.Sleep(250);
goto Salma_RecheckThread;
}
}
}
}
And here is the screenshot, if I try to update console window:
You see? Nor the line number (oProcInfo.iCurFile) or the whole line is correct!
It should be like this:
1 = xxxxx
2 = xxxxx
3 = xxxxx
4 = xxxxx
5 = xxxxx
Update-1: To test just change the sFileDir to any folder that has some big text file or if you like you can download some big text files from following link:
https://wetransfer.com/downloads/8aecfe05bb44e35582fc338f623ad43b20210602005845/bcdbb5
Am I missing any function/method to update console text from threads?

I can't reproduce it. In my tests the process always runs to completion, without getting stuck. The output is all over the place though, because the two lines below are not synchronized:
Console.SetCursorPosition(0, oProcInfo.iCurFile);
Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
Each thread of the many threads involved in the computation invokes these two statements concurrently with the other threads. This makes it possible for one thread to preempt another, and move the cursor before the first thread has the chance to write in the console. To solve this problem you must add proper synchronization, and the easiest way to do it is to use the lock statement:
class Program
{
static object _locker = new object();
And in the sProcSO method:
lock (_locker)
{
Console.SetCursorPosition(0, oProcInfo.iCurFile);
Console.Write(oProcInfo.iCurFile + " = " + lCCounter.ToString());
}
If you want to know more about thread synchronization, I recommend this online resource: Threading in C# - Part 2: Basic Synchronization
If you would like to hear my opinion about the code in the question, and you don't mind receiving criticism, my opinion is that honestly the code is so much riddled with problems that the best course of action would be to throw it away and restart from scratch. Use of archaic data structures (ArrayList???), liberal use of casting from object to specific types, liberal use of the goto statement, use of hungarian notation in public type members, all make the code difficult to follow, and easy for bugs to creep in. I found particularly problematic that each file is processed concurrently with all other files using a dedicated thread, and then each dedicated thread uses a ThreadPool thread (Task.Factory.StartNew) to starts a parallel loop (Parallel.ForEach) with unconfigured MaxDegreeOfParallelism. This setup ensures that the ThreadPool will be saturated so badly, that there is no hope that the availability of threads will ever match the demand. Most probably it will also result to a highly inefficient use of the storage device, especially if the hardware is a classic hard disk.

Your freezing problem may not be C# or code related
on the top left of your console window, on the icon .. right click
select Properties
remove the option of Quick Edit Mode and Insert Mode
you can google that feature, but essentially manifests in the problem you describe above
The formatting problem on the other hand does seem to be, here you need to create a class that serializes writes to the console window from a singe thread. a consumer/producer pattern would work (you could use a BlockingCollection to implement this quite easily)

Checking if HttpStatusCode represents success or failure

Let's suppose I have the following variable:
System.Net.HttpStatusCode status = System.Net.HttpStatusCode.OK;
How can I check if this is a success status code or a failure one?
For instance, I can do the following:
int code = (int)status;
if(code >= 200 && code < 300) {
//Success
}
I can also have some kind of white list:
HttpStatusCode[] successStatus = new HttpStatusCode[] {
HttpStatusCode.OK,
HttpStatusCode.Created,
HttpStatusCode.Accepted,
HttpStatusCode.NonAuthoritativeInformation,
HttpStatusCode.NoContent,
HttpStatusCode.ResetContent,
HttpStatusCode.PartialContent
};
if(successStatus.Contains(status)) //LINQ
{
//Success
}
None of these alternatives convinces me, and I was hoping for a .NET class or method that can do this work for me, such as:
bool isSuccess = HttpUtilities.IsSuccess(status);

If you're using the HttpClient class, then you'll get a HttpResponseMessage back.
This class has a useful property called IsSuccessStatusCode that will do the check for you.
using (var client = new HttpClient())
{
var response = await client.PostAsync(uri, content);
if (response.IsSuccessStatusCode)
{
//...
}
}
In case you're curious, this property is implemented as:
public bool IsSuccessStatusCode
{
get { return ((int)statusCode >= 200) && ((int)statusCode <= 299); }
}
So you can just reuse this algorithm if you're not using HttpClient directly.
You can also use EnsureSuccessStatusCode to throw an exception in case the response was not successful.

The accepted answer bothers me a bit as it contains magic numbers, (although they are in standard) in its second part. And first part is not generic to plain integer status codes, although it is close to my answer.
You could achieve exactly the same result by instantiating HttpResponseMessage with your status code and checking for success. It does throw an argument exception if the value is smaller than zero or greater than 999.
if (new HttpResponseMessage((HttpStatusCode)statusCode).IsSuccessStatusCode)
{
// ...
}
This is not exactly concise, but you could make it an extension.

I am partial to the discoverability of extension methods.
public static class HttpStatusCodeExtensions
{
public static bool IsSuccessStatusCode(this HttpStatusCode statusCode)
{
var asInt = (int)statusCode;
return asInt >= 200 && asInt <= 299;
}
}
As long as your namespace is in scope, usage would be statusCode.IsSuccessStatusCode().

The HttpResponseMessage class has a IsSuccessStatusCode property, looking at the source code it is like this so as usr has already suggested 200-299 is probably the best you can do.
public bool IsSuccessStatusCode
{
get { return ((int)statusCode >= 200) && ((int)statusCode <= 299); }
}

Adding to #TomDoesCode answer If you are using HttpWebResponse
you can add this extension method:
public static bool IsSuccessStatusCode(this HttpWebResponse httpWebResponse)
{
return ((int)httpWebResponse.StatusCode >= 200) && ((int)httpWebResponse.StatusCode <= 299);
}

It depends on what HTTP resource you are calling. Usually, the 2xx range is defined as the range of success status codes. That's clearly a convention that not every HTTP server will adhere to.
For example, submitting a form on a website will often return a 302 redirect.
If you want to devise a general method then the code >= 200 && code < 300 idea is probably your best shot.
If you are calling your own server then you probably should make sure that you standardize on 200.

This is an extension of the previous answer, that avoids the creation and subsequent garbage collection of a new object for each invocation.
public static class StatusCodeExtensions
{
private static readonly ConcurrentDictionary<HttpStatusCode, bool> IsSuccessStatusCode = new ConcurrentDictionary<HttpStatusCode, bool>();
public static bool IsSuccess(this HttpStatusCode statusCode) => IsSuccessStatusCode.GetOrAdd(statusCode, c => new HttpResponseMessage(c).IsSuccessStatusCode);
}

Writing a unit test for concurrent C# code?

I've been trying to solve this issue for quite some time now. I've written some example code showcasing the usage of lock in C#. Running my code manually I can see that it works the way it should, but of course I would like to write a unit test that confirms my code.
I have the following ObjectStack.cs class:
enum ExitCode
{
Success = 0,
Error = 1
}
public class ObjectStack
{
private readonly Stack<Object> _objects = new Stack<object>();
private readonly Object _lockObject = new Object();
private const int NumOfPopIterations = 1000;
public ObjectStack(IEnumerable<object> objects)
{
foreach (var anObject in objects) {
Push(anObject);
}
}
public void Push(object anObject)
{
_objects.Push(anObject);
}
public void Pop()
{
_objects.Pop();
}
public void ThreadSafeMultiPop()
{
for (var i = 0; i < NumOfPopIterations; i++) {
lock (_lockObject) {
try {
Pop();
}
//Because of lock, the stack will be emptied safely and no exception is ever caught
catch (InvalidOperationException) {
Environment.Exit((int)ExitCode.Error);
}
if (_objects.Count == 0) {
Environment.Exit((int)ExitCode.Success);
}
}
}
}
public void ThreadUnsafeMultiPop()
{
for (var i = 0; i < NumOfPopIterations; i++) {
try {
Pop();
}
//Because there is no lock, an exception is caught when popping an already empty stack
catch (InvalidOperationException) {
Environment.Exit((int)ExitCode.Error);
}
if (_objects.Count == 0) {
Environment.Exit((int)ExitCode.Success);
}
}
}
}
And Program.cs:
public class Program
{
private const int NumOfObjects = 100;
private const int NumOfThreads = 10000;
public static void Main(string[] args)
{
var objects = new List<Object>();
for (var i = 0; i < NumOfObjects; i++) {
objects.Add(new object());
}
var objectStack = new ObjectStack(objects);
Parallel.For(0, NumOfThreads, x => objectStack.ThreadUnsafeMultiPop());
}
}
I'm trying to write a unit that tests the thread unsafe method, by checking the exit code value (0 = success, 1 = error) of the executable.
I tried to start and run the application executable as a process in my test, a couple of 100 times, and checked the exit code value each time in the test. Unfortunately, it was 0 every single time.
Any ideas are greatly appreciated!

Logically, there is one, very small, piece of code where this problem can happen. Once one of the threads enters the block of code that pops a single element, then either the pop will work in which case the next line of code in that thread will Exit with success OR the pop will fail in which case the next line of code will catch the exception and Exit.
This means that no matter how much parallelization you put into the program, there is still only one single point in the whole program execution stack where the issue can occur and that is directly before the program exits.
The code is genuinely unsafe, but the probability of an issue happening in any single execution of the code is extremely low as it requires the scheduler to decide not to execute the line of code that will exit the environment cleanly and instead let one of the other Threads raise an exception and exit with an error.
It is extremely difficult to "prove" that a concurrency bug exists, except for really obvious ones, because you are completely dependent on what the scheduler decides to do.
Looking up some other posts I see this post which is written related to Java but references C#: How should I unit test threaded code?
It includes a link to this which might be useful to you: http://research.microsoft.com/en-us/projects/chess/
Hope this is useful and apologies if it is not. Testing concurrency is inherently unpredictable as is writing example code to cause it.

Thanks for all the input! Although I do agree that this is a concurrency issue quite hard to detect due to the scheduler execution among other things, I seem to have found an acceptable solution to my problem.
I wrote the following unit test:
[TestMethod]
public void Executable_Process_Is_Thread_Safe()
{
const string executablePath = "Thread.Locking.exe";
for (var i = 0; i < 1000; i++) {
var process = new Process() {StartInfo = {FileName = executablePath}};
process.Start();
process.WaitForExit();
if (process.ExitCode == 1) {
Assert.Fail();
}
}
}
When I ran the unit test, it seemed that the Parallel.For execution in Program.cs threw strange exceptions at times, so I had to change that to traditional for-loops:
public class Program
{
private const int NumOfObjects = 100;
private const int NumOfThreads = 10000;
public static void Main(string[] args)
{
var objects = new List<Object>();
for (var i = 0; i < NumOfObjects; i++) {
objects.Add(new object());
}
var tasks = new Task[NumOfThreads];
var objectStack = new ObjectStack(objects);
for (var i = 0; i < NumOfThreads; i++)
{
var task = new Task(objectStack.ThreadUnsafeMultiPop);
tasks[i] = task;
}
for (var i = 0; i < NumOfThreads; i++)
{
tasks[i].Start();
}
//Using this seems to throw exceptions from unit test context
//Parallel.For(0, NumOfThreads, x => objectStack.ThreadUnsafeMultiPop());
}
Of course, the unit test is quite dependent on the machine you're running it on (a fast processor may be able to empty the stack and exit safely before reaching the critical section in all cases).

1.) You could inject IL Inject Context switches on a post build of your code in the form of Thread.Sleep(0) using ILGenerator which would most likely help these issues to arise.
2.) I would recommend you take a look at the CHESS project by Microsoft research team.

Why System.Net.ServicePoint.ConnectionLimit uses '7FFFFFFF' (Int32.MaxValue/2147483647) when a client connects to a service on 'localhost'?

Why System.Net.ServicePoint.ConnectionLimit uses '7FFFFFFF' (Int32.MaxValue/2147483647) when a client connects to a service on 'localhost', whereas it decide to use '2' as default if the service is running on remote machine?
Initially I thought it will be ServicePointManager.DefaultConnectionLimit if servicepoint.connectionlimit is not set. However, I just realized (once I got an issue from customer), that its Int32.MaxValue/2147483647.
I have done some research (for details please refer to below links), however I couldn't find out why it uses to int32.maxvalue. I can kind of conjecture its probably for better performance as the input requests and response messages are not going across boundary.
My Question(s):
Why Int32.MaxValue if the service is running on 'localhost'?
(any explanation in English ;) of the code snippet I copied from reflector is also great - as I kind of conjectured the intentions - but didn't understand the code totally :))
I understand its for perf - but from '2' (default) to 'int32.maxvalue' sounds stretch to me. In other words why is it ok to open as many TCP connections as long as the requests are not going across network.
(in other words - why default to int32.maxvalue - doesn't it have side affects)
Some useful links related to this:
How and where the TCP connection has been created in httpwebrequest, and how is it related to servicepoint?
http://blogs.microsoft.co.il/idof/2011/06/20/servicepointmanagerdefaultconnectionlimit-2-depends/
http://msdn.microsoft.com/en-us/library/system.net.servicepoint.connectionlimit(v=vs.110).aspx
http://arnosoftwaredev.blogspot.com/2006/09/net-20-httpwebrequestkeepalive-and.html
Code Snippet from Reflector
public int ConnectionLimit
{
get
{
if ((!this.m_UserChangedLimit && (this.m_IPAddressInfoList == null)) && (this.m_HostLoopbackGuess == TriState.Unspecified))
{
lock (this)
{
if ((!this.m_UserChangedLimit && (this.m_IPAddressInfoList == null)) && (this.m_HostLoopbackGuess == TriState.Unspecified))
{
IPAddress address = null;
if (IPAddress.TryParse(this.m_Host, out address))
{
this.m_HostLoopbackGuess = IsAddressListLoopback(new IPAddress[] { address }) ? TriState.True : TriState.False;
}
else
{
this.m_HostLoopbackGuess = NclUtilities.GuessWhetherHostIsLoopback(this.m_Host) ? TriState.True : TriState.False;
}
}
}
}
if (!this.m_UserChangedLimit && !((this.m_IPAddressInfoList == null) ? (this.m_HostLoopbackGuess != TriState.True) : !this.m_IPAddressesAreLoopback))
{
return 0x7fffffff;
}
return this.m_ConnectionLimit;
}
set
{
if (value <= 0)
{
throw new ArgumentOutOfRangeException("value");
}
if (!this.m_UserChangedLimit || (this.m_ConnectionLimit != value))
{
lock (this)
{
if (!this.m_UserChangedLimit || (this.m_ConnectionLimit != value))
{
this.m_ConnectionLimit = value;
this.m_UserChangedLimit = true;
this.ResolveConnectionLimit();
}
}
}
}
}
Regards,

Int32.maxvalue is just a placeholder for no limit. You should be able to create as many connections to yourself as you need to.
The code you pasted basically just checks whether you are connecting to the loopback address or not, and if you are, returns maxint, if you are not, returns the value of servicepoint.connectionlimit (2 by default, but you can change it)

IsConnectedToNetwork for Windows8 WinRT Fails

Anyone know a good way to see if the user is online/offline? When I use public static bool IsConnectedToNetwork(); and see if it's false/true it seems to always be true, even when I shut down my internet to test it...
Am I missing something?
public static bool nets()
{
bool go =
System.Net.NetworkInformation.NetworkInterface.GetIsNetworkAvailable();
if (go == false)
{
return false;
}
return true;
}
Now, in my start-up I run:
var ba = nets();
if (ba == false)
{
txtHeader.Text = "err";
}
if (ba != false)
{
// Code
}
I also tried:
public static bool IsConnectedToNetwork();

I use
public static bool IsConnectedToInternet()
{
ConnectionProfile connectionProfile = NetworkInformation.GetInternetConnectionProfile();
return (connectionProfile!=null && connectionProfile.GetNetworkConnectivityLevel() == NetworkConnectivityLevel.InternetAccess);
}

You could ping one of the few well known URLs to see if you get responses.

I have a way that seems to work pretty decent, especially noticing sometimes Ethernet could appear to be in use even when its not in Windows 8 RTM... That's why nothing seems to work!
However, doing the following is a good choice for me to test...
var profile = Windows.Networking.Connectivity.NetworkInformation.GetInternetConnectionProfile();
var interfaceType = profile.NetworkAdapter.IanaInterfaceType;
// 71 is WiFi & 6 is Ethernet
if (interfaceType == 71 | interfaceType == 6)
{
// Run Code
}
/* 3G/Mobile Detect
else if (interfaceType == 243 | interfaceType == 244)
{
// Run Code if you need to use a less quality feature.
}*/
else
{
txtHeader.Text = "Error, Check connection or Try connecting to the Internet...";
}