We Keep Coding

Calculating Estimated Time Left in Loop (When iteration number is very big and time required for each iteration is very low)

I'm trying to calculate total time remaining of a for loop. Total number of iterations can be more than ~1010000000. Total time required for doing "Real job done here" is much lower than a second and it doesnt change much. When i use my current solution time remaining increases for a long time and than starts to decrease.
I'm looking for a better solution. What can i do?
long totalTiles = ((((rightBottom.X) - (topLeft.X)) + 1) * (((rightBottom.Y) - (topLeft.Y)) + 1));
long currentTileProccessed = 0;
DateTime startTime = DateTime.Now;
for (long x = (topLeft.X); x <= (rightBottom.X); x++)
{
for (long y = (topLeft.Y); y <= (rightBottom.Y); y++)
{
**//Real job done here//**
TimeSpan timeRemaining = TimeSpan.FromTicks(DateTime.Now.Subtract(startTime).Ticks * (totalTiles - (currentTileProccessed + 1)) / (currentTileProccessed + 1));
this.Dispatcher.Invoke((Action)delegate () {
EstimatedTimeLeft_TextBlock.Text = "Days : " + timeRemaining.Days.ToString("D2") + ", Hours : " + timeRemaining.Hours.ToString("D2") + ", Minutes :" + timeRemaining.Minutes.ToString("D2") + ", Seconds :" + timeRemaining.Seconds.ToString("D2");
});
currentTileProccessed++;
}
}

Here I am using Stopwatch from System.Diagnostics to determine the average time for the last 100 tasks, which I multiply against the total expected number of tasks minus the number already run.
NOTE: Remaining time can still increase, if suddenly the last 100 tasks start running slower than the previous 10k, this will readjust your remaining time to match the recent runs.
long totalTiles = ((((rightBottom.X) - (topLeft.X)) + 1) * (((rightBottom.Y) - (topLeft.Y)) + 1));
long currentTileProccessed = 0;
Queue<long> taskTimes = new Queue<long>();
int taskTimeHistoryLimit = 100;
long taskTotal = (rightBottom.X - topLeft.X) * (rightBottom.Y - topLeft.Y);
Stopwatch watch = new Stopwatch();
long index = 0;
for (long x = (topLeft.X); x <= (rightBottom.X); x++)
{
for (long y = (topLeft.Y); y <= (rightBottom.Y); y++)
{
index++;
watch.Start();
//Real job done here//
watch.Stop();
taskTimes.Enqueue(watch.ElapsedTicks);
watch.Reset();
while (taskTimes.Count > taskTimeHistoryLimit)
{
taskTimes.Dequeue();
}
TimeSpan timeRemaining = new TimeSpan((taskTotal - index) * (long)taskTimes.Average());
this.Dispatcher.Invoke((Action)delegate () {
EstimatedTimeLeft_TextBlock.Text = "Days : " + timeRemaining.Days.ToString("D2") + ", Hours : " + timeRemaining.Hours.ToString("D2") + ", Minutes :" + timeRemaining.Minutes.ToString("D2") + ", Seconds :" + timeRemaining.Seconds.ToString("D2");
});
}
}

C# Starting all thread in the same time while parsing

I'm currently learning C# and I've been working on a XML parser for the last two days. It's actually working fine my issue is the amount of time it take to parse more than 10k pages. this is my code.
public static void startParse(int id_min, int id_max, int numberofthreads)
{
int start;
int end;
int part;
int threadnbrs;
threadnbrs = numberofthreads;
List<Thread> workerThreads;
List<string> results;
part = (id_max - id_min) / threadnbrs;
start = id_min;
end = 0;
workerThreads = new List<Thread>();
results = new List<string>();
for (int i = 0; i < threadnbrs; i++)
{
if (i != 0)
start = end + 1;
end = start + (part);
if (i == (threadnbrs - 1))
end = id_max;
int _i = i;
int _start = start;
int _end = end;
Thread t = new Thread(() =>
{
Console.WriteLine("i = " + _i);
Console.WriteLine("start =" + _start);
Console.WriteLine("end =" + _end + "\r\n");
string parse = new ParseWH().parse(_start, _end);
lock (results)
{
results.Add(parse);
}
});
workerThreads.Add(t);
t.Start();
}
foreach (Thread thread in workerThreads)
thread.Join();
File.WriteAllText(".\\result.txt", String.Join("", results));
Console.Beep();
}
what i'm actually doing is splitting in different thread a range of element that need to be parsed so each thread handle X elements.
for each 100 elements it take approx 20 seconds.
however it took me 17 minutes to parse 10 0000 Elements.
what i need is each thread working simultaneously on 100 of those 10 000 Elements so it can be done in 20 seconds. is there is a solution for that ?
Parse Code :
public string parse(int id_min, int id_max)
{
XmlDocument xml;
WebClient user;
XmlElement element;
XmlNodeList nodes;
string result;
string address;
int i;
//Console.WriteLine(id_min);
//Console.WriteLine(id_max);
i = id_min;
result = "";
xml = new XmlDocument();
while (i <= id_max)
{
user = new WebClient();
// user.Headers.Add("User-Agent", "Mozilla/5.0 (Linux; U; Android 4.0.3; ko-kr; LG-L160L Build/IML74K) AppleWebkit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30");
user.Encoding = UTF8Encoding.UTF8;
address = "http://fr.wowhead.com/item=" + i + "?xml";
if (address != null)
xml.LoadXml(user.DownloadString(new Uri(address)));
element = xml.DocumentElement;
nodes = element.SelectNodes("/wowhead");
if (xml.SelectSingleNode("/wowhead/error") != null)
{
Console.WriteLine("error " + i);
i++;
continue;
}
result += "INSERT INTO item_wh (entry, class, subclass, displayId, ,quality, name, level) VALUES (";
foreach (XmlNode node in nodes)
{
// entry
result += node["item"].Attributes["id"].InnerText;
result += ", ";
// class
result += node["item"]["class"].Attributes["id"].InnerText;
result += ", ";
// subclass
result += node["item"]["subclass"].Attributes["id"].InnerText;
result += ", ";
// displayId
result += node["item"]["icon"].Attributes["displayId"].InnerText;
result += ", ";
// quality
result += node["item"]["quality"].Attributes["id"].InnerText;
result += ", \"";
// name
result += node["item"]["name"].InnerText;
result += "\", ";
// level
result += node["item"]["level"].InnerText;
result += ");";
// bakcline
result += "\r\n";
}
i++;
}
return (result);
}

Best solution for CPU bound work (such as parsing) is to launch as many threads as the numbers of the cores in your machine, less than that and you are not taking advantage of all of your cores, more than that and excessive context-switching might kick in and hinder performance.
So essentially, threadnbrs should be set to Environment.ProcessorCount
Also, consider using the Parallel class instead of creating threads yourself:
Parallel.ForEach(thingsToParse, (somethingToParse) =>
{
var parsed = Parse(somethingToParse);
results.Add(parsed);
});
You must agree that it looks much cleaner and much easier to maintain.
Also, you'll be better off using ConcurrentBag instead of a regular List + lock as ConcurrentBag is more built for concurrent loads and could give you better performance.

Finally ! Got it working by launching multiple process of my application Simultaneously.
Witch means if i have 10 k elements i run in 10 process of 1000 Elements. increase the numbers of process to decrease the number of elements and it goes faster and faster !
(I'm currently running on a very fast Internet Speed) and have a Samsung M.2 960 as Storage as well as core I7 Skylake 6 cores

Okay so i found "Trying to run multiple HTTP requests in parallel, but being limited by Windows (registry)" it's called "Thread Pool" I finally decided to download directly the XML file then parse the document directly offline, instead of parsing the website directly to get an SQL format. the new method work, i can download and write up to 10 000 K XML in only 9 seconds. I tried to push it to 150 K (All Websites Pages) but now i got a strange bug i got duplicates items... I'm going to try to rewrite the full code using the correct method for pools, multi Task/Thread, dictionary and IEnumerable Containers cross finger to work on 150 k Item without losing data in process and post back the full code.

How does semaphore.CurrentCount work in this context?

I just started learning SemaphoreSlim but how does the semaphore.CurrentCount increments and decrements in this program? From my understanding, when semaphore.Wait() is called the release counter is decremented by 1 and when semaphore.Release(), two threads are allowed to execute but how does the semaphore.CurrentCount get incremented? does it start from 0 or 1 ?
var semaphore = new SemaphoreSlim(2, 10);
for (int i = 0; i < 20; i++)
{
Task.Factory.StartNew(() =>
{
Console.WriteLine("Entering task " + Task.CurrentId);
semaphore.Wait(); //releasecount--
Console.WriteLine("Processing task " + Task.CurrentId);
});
}
while (semaphore.CurrentCount <= 2)
{
Console.WriteLine("Semaphore count: " + semaphore.CurrentCount);
Console.ReadKey();
semaphore.Release(2);
}
Console.ReadKey();

The semaphore is like a room with a certain capacity. With SemaphoreSlim you are specifying the initial capacity and the maximum. After it reach the maximum, no one can enter anymore to the room. Per every item that leaves the room, only one is allowed to enter.
The CurrentCount gets the number of remaining threads that can enter the room.
for (int i = 0; i < 20; i++)
{
Task.Factory.StartNew(() =>
{
Console.WriteLine("Entering task " + Task.CurrentId);
semaphore.Wait(); //only from 2 - 10 threads can be at the time
Console.WriteLine("Processing task " + Task.CurrentId);
});
}
And here
while (semaphore.CurrentCount <= 2)
if at that moment the number of remaining threads is less than 2 then you are releasing two spaces on the room

Wait decrements it and Release increments it.

Estimate loop time for progress bar

I'm currently using a loop in C# to generate files, but the program takes a few seconds to actually do this and I feel the program user would benefit from a progress bar that updates to tell them how far the loop is through so they can estimate when the loop is going to finish and all of their files are generated.
I was wondering if there was a way to calculate the time it's going to take a loop to complete or update a progress bar with the loop to show how much progress the loop has left.
Here's my loop.
String GeneratedLevelName;
int levelx = 0;
int levely = 0;
for (int i = 0; i < GmapLevelArea; i ++) {
if (levelx >= (GmapWidth - 1)) {
levelx = 0;
levely ++;
GeneratedLevelName = (GmapPrefix + "_" + levelx + "_" + levely + ".nw");
File.Copy(ApplicationDirectory + TemplateFileName, GmapDirectory + GeneratedLevelName);
GmapFile.AppendLine('"' + GeneratedLevelName + '"' + ",");
} else {
levelx ++;
GeneratedLevelName = (GmapPrefix + "_" + levelx + "_" + levely + ".nw");
File.Copy(ApplicationDirectory + TemplateFileName, GmapDirectory + GeneratedLevelName);
GmapFile.Append('"' + GeneratedLevelName + '"' + ",");
}
}
Any help is greatly appreciated.

Since it would be difficult to time the File.Copy, I think you should just base your progress bar on total files worked on.
So if you have a progress bar: pb
pb.Minimum = 0;
pb.Maximum = GMapLevelArea;
then set the value of the progress bar to you i value
pb.Value = i;
Now when I usually use progress bars like this, I usually don't update ever iteration. I usually update after a few iterations.
So to do that I usually check for the iteration count:
if(i % 10 == 0) //only true when i is a multiple of 10
pb.Value = i;
Then at the end of the loop, if it doesn't happen to be a multiple of your update, you can either set the progress bar to its maximum value or reset it to zero to let them no its done.
pb.Value = GMapLevelArea; or pb.Value = 0;

It depends on how consistent your files are generated (do they all take a second, or take some longer than others?) and if you know how many you are going to generate. In the easiest case, you can estimate the time by taking the number of files to generate. For example if you create 50 files, you can increment the progress bar by two percent after a file is generated.

Where is the race condition in this code snippet? (from C# 5.0 in a Nutshell)

This is from Joseph Albahari's excellent C# 5.0 in a Nutshell book
In one of his chapters, he mentions a race-condition in this code block ..my guess is it's meant to be pretty self-evident, as he didn't bother to specify where it was but running the code multiple times I was unable to produce the said-race condition
_button.Click += (sender, args) =>
{
_button.IsEnabled = false;
Task.Run (() => Go());
};
void Go()
{
for (int i = 1; i < 5; i++)
{
int result = GetPrimesCount (i * 1000000, 1000000);
Dispatcher.BeginInvoke (new Action (() =>
_results.Text += result + " primes between " + (i*1000000) + " and " +
((i+1)*1000000-1) + Environment.NewLine));
}
Dispatcher.BeginInvoke (new Action (() => _button.IsEnabled = true));
}

I don't agree with #Serge's answer. You don't even need multiple threads to see the problem. Try to run your code in its original form and notice the output. For me it's the following and is sometimes random (I fixed the first value):
1000000 primes between 5000000 and 5999999
1000000 primes between 5000000 and 5999999
1000000 primes between 5000000 and 5999999
1000000 primes between 5000000 and 5999999
Notice the last 2 values. They're all the same, but they should depend on i. The problem is not that the operation is not atomic, because the GUI thread will execute the actions sequentially anyway.
The reason for this occurring is that the lambda function passed to BeginInvoke takes the value of i at the moment of execution, not at the moment of initialization, so they will all see the last value of i by the time they get executed. The solution is to explicitly pass i as a parameter to the lambda like so:
for (int i = 1; i < 5; i++)
{
int result = 1000000;
Dispatcher.BeginInvoke(new Action<int>(j =>
results.Text += result + " primes between " + (j * 1000000) + " and " +
((j + 1) * 1000000 - 1) + Environment.NewLine), i);
}

This line will not be calculated atomically:
_results.Text += result + " primes between " + (i*1000000) + " and " + ((i+1)*1000000-1) + Environment.NewLine));
So being executed from 5 concurrently running threads it may produce different funny results.