Reducing runtime of method from 24 hours to 1 hour [closed] - c#

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
I have a 'For' loop I need to run from 0 to 2,147,483,647 (int.MaxValue). Inside the loop I'm calling a method that calculates a very complex equation and it returns the value for each parameter. I wish to get the highest return value.
The total runtime of my for loop takes about full 24 hours. How and can I reduce the runtime of my code to only one hour? Maybe 4 hours if I will use distributed computing?
static void Main(string[] args)
{
CheckValuesZeroToMaxInt();
Console.ReadLine();
}
private static void CheckValuesZeroToMaxInt()
{
for (int i = 0; i < int.MaxValue; i++)
{
ComplexMath(i);
}
}
I know how to use 'Parallel.For', but still this offers very little help. How can I take my c# code and run it on a cluster of computers to get the return value from my method in no time? Is there a site that offers this service for a low or free cost?

The easiest way to do it on one machine would be with the Parallel.For-loop
private static void CheckValuesZeroToMaxInt()
{
object maxValueLock = new object();
int maxValue = int.MinValue;
Parallel.For(0, int.MaxValue, i =>
{
int tmpMaxValue = ComplexMath(i);
if (tmpMaxValue > maxValue)
{
lock(maxValueLock)
{
if (tmpMaxValue > maxValue)
maxValue = tmpMaxValue;
}
}
}
Console.WriteLine(maxValue);
}
However, since your ComplexMath function only takes ~0.04ms ((24 / (2^31)) * 3600 * 1000) to execute, you'd maybe get stuck with the constant locking and wouldn't get almost any improvement. To prevent this from happening, we can change the above function to execute more steps within a thread.
const long million = 1000*1000;
private static void CheckValuesZeroToMaxInt()
{
object maxValueLock = new object();
int maxValue = int.MinValue;
Parallel.For(0, (int.MaxValue / million) + 1, i =>
{
int privateMaxValue = int.MinValue;
for (long j = i * million; j < (i+1) * million && j < int.MaxValue; j++)
{
int tmpMaxValue = ComplexMath(j);
if (tmpMaxValue > privateMaxValue)
{
privateMaxValue = tmpMaxValue;
}
}
lock(maxValueLock)
{
if (privateMaxValue > maxValue)
maxValue = privateMaxValue;
}
}
Console.WriteLine(maxValue);
}
The second version should scale pretty much linearly with the number of processors, so if you run it on a machine with 24 cores, it will finish within 1h. Of course instead of Parallel.For you can use any kind of communication to distribute the computation between multiple machines.

Related

Best practice to refactor if staments in C# [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
what's the best practice to refactor the following c# code: Can replace this with Strategy pattern?
private int GetAccountType(int balance)
{
if (balance <= 5000)
{
return 1;
}
else if (balance > 5000 && balance <= 10000)
{
return 2;
}
else if (balance >= 10001)
{
return 3;
}
_logger.LogError("Invalid AccountType");
// TO DO : Throw custom exception
throw new Exception("Invalid Balance");
}
As others have already mentioned Strategy would definitely be over-engineering in this case. Since your logging statement is never reached, I'd simply do:
private int GetAccountType(int balance)
{
return balance switch
{
<= 5000 => 1,
> 5000 and <= 10000 => 2,
>= 10001 => 3
};
}
But normally I would not refactor anything in this case - perhaps just remove the "else".
One easy simplification would be:
private int GetAccountType(int balance)
{
if (balance <= 5000)
{
return 1;
}
else if (balance <= 10000) // balance is certain to be > 5000
{
return 2;
}
else // balance is certain to be > 10000
{
return 3;
}
_logger.LogError("Invalid AccountType");
// TO DO : Throw custom exception
throw new Exception("Invalid Balance");
}
If you already checked that balance <= 5000 then in the else-if you don't need to check that balance > 5000 - you wouldn't get there otherwise.
For the rest your logic is clear this way - readability is also an important aspect!
In case you have many balances, I suggest extracting a model:
// Note, now you can add easily as many balances as you want
private static readonly IReadOnlyDictionary<int, Func<int>> s_Balances =
new Dictionary<int, int>() {
{ int.MinValue, 1}, // 1 starting from int.MinValue + 1
{ 5000, 2}, // 2 starting from 5001
{ 10000, 3}, // 3 starting from 10001
};
then you can easily query this model with a help of Linq:
using System.Linq;
...
private int GetAccountType(int balance) {
int key = s_Balances
.Keys
.Where(k < balance)
.Max();
return s_Balances[key];
}
Here you can refactor in another way :
int result = balance > 5000 && balance <= 10000 ? 2 : balance > 10000 ? 3 : 1;
Few lines
There are no fundamental difference between ternary and if/else.
Ternary is faster then if/else as long as no additional computation is required to convert the logic to us ternary. When it is a simply ternary operation, it has better readability as well.
If only statement is faster than if/else, so if the logic doesn’t require an else statement, do use it.
Performance Comparison : LINK
Strategy Design Pattern
It is behavioral design pattern and you can implement when you have such issues. This code doesn't require patterns as it will be over engineering.
Identification: Strategy pattern can be recognized by a method that lets nested object do the actual work, as well as the setter that allows replacing that object with a different one
Usage examples: It’s often used in various frameworks to provide users a way to change the behavior of a class without extending it.
Here is another option, but I think the if/else statement is more readable. I added it because it can be a useful technique when you need to look up values in an array for instance.
using static System.Math;
...
private int GetAccountType(int balance) => 1 + Min(Max(balance / 5000, 0), 2);

Why Seconds show small value when actual elapsed time is in minutes? [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
I have a parallel loop in a console application with a StopWatch.
If I input 10000 loops it says it finishes in 13 seconds but in real world the console doesn't stop printing until 15 minutes later. So how can I properly time this loop so the stopwatch stops after the last line has finished writing to the console?
My expected result is "Process took 903 Seconds"
int i = 0;
int N = 10000;
Stopwatch sw = new StopWatch();
sw.Start
Parallel.For(0, N, i =>
{
//Do some logic task here then write to the console
console.Writeline("Thread " + i + " has finished");
});
sw.Stop()
console.WriteLine("Process took" + sw.Elapsed.Seconds + " Seconds");
Is is similar to C# Timespan Milliseconds vs TotalMilliseconds where one used Milliseconds instead of TotalMilliseconds?
From the documentation of the TimeSpan.Seconds property:
public int Seconds { get; }
Gets the seconds component of the time interval represented by the current TimeSpan structure.
So the Seconds return a value between 0 - 59. To get the total duration of the TimeSpan in seconds you need to query the TimeSpan.TotalSeconds property:
public double TotalSeconds { get; }
Gets the value of the current TimeSpan structure expressed in whole and fractional seconds.
You need to use sw.Elapsed.TotalSeconds which returns a double instead of .Seconds which returns the integer seconds part of (hours,minutes,seconds,..)
The rest of the code is correct.
example code:
static class Program
{
static void Main(string[] args)
{
int N = 10000;
Stopwatch sw = new Stopwatch();
sw.Start();
Parallel.For(0, N, (i) =>
{
//Do some logic task here then write to the console
LongTask();
Console.WriteLine($"Iteration {i} has finished in thread {Thread.CurrentThread.ManagedThreadId}");
});
sw.Stop();
Console.WriteLine($"Process took {sw.Elapsed.TotalSeconds} Seconds");
}
static double LongTask()
{
double x = 1;
for (int i = 1; i <= 2<<55; i++)
{
x += Math.Pow(1+1/i,i);
}
return x;
}
}
NOTE: Always run timing on Release builds, and if run from VS, use the Start without Debugging command.

Parallel.ForEach slower than normal foreach

I'm playing around with the Parallel.ForEach in a C# console application, but can't seem to get it right. I'm creating an array with random numbers and i have a sequential foreach and a Parallel.ForEach that finds the largest value in the array. With approximately the same code in c++ i started to see a tradeoff to using several threads at 3M values in the array. But the Parallel.ForEach is twice as slow even at 100M values. What am i doing wrong?
class Program
{
static void Main(string[] args)
{
dostuff();
}
static void dostuff() {
Console.WriteLine("How large do you want the array to be?");
int size = int.Parse(Console.ReadLine());
int[] arr = new int[size];
Random rand = new Random();
for (int i = 0; i < size; i++)
{
arr[i] = rand.Next(0, int.MaxValue);
}
var watchSeq = System.Diagnostics.Stopwatch.StartNew();
var largestSeq = FindLargestSequentially(arr);
watchSeq.Stop();
var elapsedSeq = watchSeq.ElapsedMilliseconds;
Console.WriteLine("Finished sequential in: " + elapsedSeq + "ms. Largest = " + largestSeq);
var watchPar = System.Diagnostics.Stopwatch.StartNew();
var largestPar = FindLargestParallel(arr);
watchPar.Stop();
var elapsedPar = watchPar.ElapsedMilliseconds;
Console.WriteLine("Finished parallel in: " + elapsedPar + "ms Largest = " + largestPar);
dostuff();
}
static int FindLargestSequentially(int[] arr) {
int largest = arr[0];
foreach (int i in arr) {
if (largest < i) {
largest = i;
}
}
return largest;
}
static int FindLargestParallel(int[] arr) {
int largest = arr[0];
Parallel.ForEach<int, int>(arr, () => 0, (i, loop, subtotal) =>
{
if (i > subtotal)
subtotal = i;
return subtotal;
},
(finalResult) => {
Console.WriteLine("Thread finished with result: " + finalResult);
if (largest < finalResult) largest = finalResult;
}
);
return largest;
}
}
It's performance ramifications of having a very small delegate body.
We can achieve better performance using the partitioning. In this case the body delegate performs work with a high data volume.
static int FindLargestParallelRange(int[] arr)
{
object locker = new object();
int largest = arr[0];
Parallel.ForEach(Partitioner.Create(0, arr.Length), () => arr[0], (range, loop, subtotal) =>
{
for (int i = range.Item1; i < range.Item2; i++)
if (arr[i] > subtotal)
subtotal = arr[i];
return subtotal;
},
(finalResult) =>
{
lock (locker)
if (largest < finalResult)
largest = finalResult;
});
return largest;
}
Pay attention to synchronize the localFinally delegate. Also note the need for proper initialization of the localInit: () => arr[0] instead of () => 0.
Partitioning with PLINQ:
static int FindLargestPlinqRange(int[] arr)
{
return Partitioner.Create(0, arr.Length)
.AsParallel()
.Select(range =>
{
int largest = arr[0];
for (int i = range.Item1; i < range.Item2; i++)
if (arr[i] > largest)
largest = arr[i];
return largest;
})
.Max();
}
I highly recommend free book Patterns of Parallel Programming by Stephen Toub.
As the other answerers have mentioned, the action you're trying to perform against each item here is so insignificant that there are a variety of other factors which end up carrying more weight than the actual work you're doing. These may include:
JIT optimizations
CPU branch prediction
I/O (outputting thread results while the timer is running)
the cost of invoking delegates
the cost of task management
the system incorrectly guessing what thread strategy will be optimal
memory/cpu caching
memory pressure
environment (debugging)
etc.
Running each approach a single time is not an adequate way to test, because it enables a number of the above factors to weigh more heavily on one iteration than on another. You should start with a more robust benchmarking strategy.
Furthermore, your implementation is actually dangerously incorrect. The documentation specifically says:
The localFinally delegate is invoked once per task to perform a final action on each task’s local state. This delegate might be invoked concurrently on multiple tasks; therefore, you must synchronize access to any shared variables.
You have not synchronized your final delegate, so your function is prone to race conditions that would make it produce incorrect results.
As in most cases, the best approach to this one is to take advantage of work done by people smarter than we are. In my testing, the following approach appears to be the fastest overall:
return arr.AsParallel().Max();
The Parallel Foreach loop should be running slower because the algorithm used is not parallel and a lot more work is being done to run this algorithm.
In the single thread, to find the max value, we can take the first number as our max value and compare it to every other number in the array. If one of the numbers larger than our first number, we swap and continue. This way we access each number in the array once, for a total of N comparisons.
In the Parallel loop above, the algorithm creates overhead because each operation is wrapped inside a function call with a return value. So in addition to doing the comparisons, it is running overhead of adding and removing these calls onto the call stack. In addition, since each call is dependent on the value of the function call before, it needs to run in sequence.
In the Parallel For Loop below, the array is divided into an explicit number of threads determined by the variable threadNumber. This limits the overhead of function calls to a low number.
Note, for low values, the parallel loops performs slower. However, for 100M, there is a decrease in time elapsed.
static int FindLargestParallel(int[] arr)
{
var answers = new ConcurrentBag<int>();
int threadNumber = 4;
int partitionSize = arr.Length/threadNumber;
Parallel.For(0, /* starting number */
threadNumber+1, /* Adding 1 to threadNumber in case array.Length not evenly divisible by threadNumber */
i =>
{
if (i*partitionSize < arr.Length) /* check in case # in array is divisible by # threads */
{
var max = arr[i*partitionSize];
for (var x = i*partitionSize;
x < (i + 1)*partitionSize && x < arr.Length;
++x)
{
if (arr[x] > max)
max = arr[x];
}
answers.Add(max);
}
});
/* note the shortcut in finding max in the bag */
return answers.Max(i=>i);
}
Some thoughts here: In the parallel case, there is thread management logic involved that determines how many threads it wants to use. This thread management logic presumably possibly runs on your main thread. Every time a thread returns with the new maximum value, the management logic kicks in and determines the next work item (the next number to process in your array). I'm pretty sure that this requires some kind of locking. In any case, determining the next item may even cost more than performing the comparison operation itself.
That sounds like a magnitude more work (overhead) to me than a single thread that processes one number after the other. In the single-threaded case there are a number of optimization at play: No boundary checks, CPU can load data into the first level cache within the CPU, etc. Not sure, which of these optimizations apply for the parallel case.
Keep in mind that on a typical desktop machine there are only 2 to 4 physical CPU cores available so you will never have more than that actually doing work. So if the parallel processing overhead is more than 2-4 times of a single-threaded operation, the parallel version will inevitably be slower, which you are observing.
Have you attempted to run this on a 32 core machine? ;-)
A better solution would be determine non-overlapping ranges (start + stop index) covering the entire array and let each parallel task process one range. This way, each parallel task can internally do a tight single-threaded loop and only return once the entire range has been processed. You could probably even determine a near optimal number of ranges based on the number of logical cores of the machine. I haven't tried this but I'm pretty sure you will see an improvement over the single-threaded case.
Try splitting the set into batches and running the batches in parallel, where the number of batches corresponds to your number of CPU cores.
I ran some equations 1K, 10K and 1M times using the following methods:
A "for" loop.
A "Parallel.For" from the System.Threading.Tasks lib, across the entire set.
A "Parallel.For" across 4 batches.
A "Parallel.ForEach" from the System.Threading.Tasks lib, across the entire set.
A "Parallel.ForEach" across 4 batches.
Results: (Measured in seconds)
Conclusion:
Processing batches in parallel using the "Parallel.ForEach" has the best outcome in cases above 10K records. I believe the batching helps because it utilizes all CPU cores (4 in this example), but also minimizes the amount of threading overhead associated with parallelization.
Here is my code:
public void ParallelSpeedTest()
{
var rnd = new Random(56);
int range = 1000000;
int numberOfCores = 4;
int batchSize = range / numberOfCores;
int[] rangeIndexes = Enumerable.Range(0, range).ToArray();
double[] inputs = rangeIndexes.Select(n => rnd.NextDouble()).ToArray();
double[] weights = rangeIndexes.Select(n => rnd.NextDouble()).ToArray();
double[] outputs = new double[rangeIndexes.Length];
/// Series "for"...
var startTimeSeries = DateTime.Now;
for (var i = 0; i < range; i++)
{
outputs[i] = Math.Sqrt(Math.Pow(inputs[i] * weights[i], 2));
}
var durationSeries = DateTime.Now - startTimeSeries;
/// "Parallel.For"...
var startTimeParallel = DateTime.Now;
Parallel.For(0, range, (i) => {
outputs[i] = Math.Sqrt(Math.Pow(inputs[i] * weights[i], 2));
});
var durationParallelFor = DateTime.Now - startTimeParallel;
/// "Parallel.For" in Batches...
var startTimeParallel2 = DateTime.Now;
Parallel.For(0, numberOfCores, (c) => {
var endValue = (c == numberOfCores - 1) ? range : (c + 1) * batchSize;
var startValue = c * batchSize;
for (var i = startValue; i < endValue; i++)
{
outputs[i] = Math.Sqrt(Math.Pow(inputs[i] * weights[i], 2));
}
});
var durationParallelForBatches = DateTime.Now - startTimeParallel2;
/// "Parallel.ForEach"...
var startTimeParallelForEach = DateTime.Now;
Parallel.ForEach(rangeIndexes, (i) => {
outputs[i] = Math.Sqrt(Math.Pow(inputs[i] * weights[i], 2));
});
var durationParallelForEach = DateTime.Now - startTimeParallelForEach;
/// Parallel.ForEach in Batches...
List<Tuple<int,int>> ranges = new List<Tuple<int, int>>();
for (var i = 0; i < numberOfCores; i++)
{
int start = i * batchSize;
int end = (i == numberOfCores - 1) ? range : (i + 1) * batchSize;
ranges.Add(new Tuple<int,int>(start, end));
}
var startTimeParallelBatches = DateTime.Now;
Parallel.ForEach(ranges, (range) => {
for(var i = range.Item1; i < range.Item1; i++) {
outputs[i] = Math.Sqrt(Math.Pow(inputs[i] * weights[i], 2));
}
});
var durationParallelForEachBatches = DateTime.Now - startTimeParallelBatches;
Debug.Print($"=================================================================");
Debug.Print($"Given: Set-size: {range}, number-of-batches: {numberOfCores}, batch-size: {batchSize}");
Debug.Print($".................................................................");
Debug.Print($"Series For: {durationSeries}");
Debug.Print($"Parallel For: {durationParallelFor}");
Debug.Print($"Parallel For Batches: {durationParallelForBatches}");
Debug.Print($"Parallel ForEach: {durationParallelForEach}");
Debug.Print($"Parallel ForEach Batches: {durationParallelForEachBatches}");
Debug.Print($"");
}

Quickly returning all numbers less than or divisible by Seven

So i had an interview question: Write a function that takes a number and returns all numbers less than or divisible by 7
private List<int> GetLessThanOrDivisbleBySeven(int num)
{
List<int> ReturnList = new List<int>();
for(int i = 0; i <= num; i++)
{
if(i <7 || i % 7 == 0)
{
ReturnList.Add(i);
}
}
return ReturnList;
}
So far so good. The follow up question was: Let's say that call was being made 10s of thousands of times an hour. How could you speed it up?
I said if you knew what your queue was you could break up your queue and thread it. That got me some points i feel. However, he wanted to know if there was anything in the function i could do.
I came up with the idea to test if the num was greater than 7. if so initialize the list with 1 - 7 and start the loop int i = 8 which i think was ok but is there another way i am missing?
If you want to speed it up without caching, you can just increment i by 7 to get all numbers divisible by 7, it will be something like this:
static private List<int> GetLessThanOrDivisbleBySeven(int num) {
List<int> ReturnList;
int i;
if (num <= 7) {
ReturnList = new List<int>();
for (i = 0; i <= num; i++) {
ReturnList.Add(i);
}
return ReturnList;
}
ReturnList = new List<int> { 0, 1, 2, 3, 4, 5, 6 };
i = 7;
while (i <= num) {
ReturnList.Add(i);
i += 7;
}
return ReturnList;
}
You can cache the results. Each time your function is being called, check what numbers are in the cache, and calculate the rest.
If the current number is smaller, return the appropriate cached results.
use the previous results when calculating new list
int oldMax = 0;
List<int> ReturnList = new List<int>();
private List<int> GetLessThanOrDivisbleBySeven(int num)
{
if (num > oldMax )
{
oldMax = num;
for(int i = oldMax ; i <= num; i++)
{
if(i <7 || i % 7 == 0)
{
ReturnList.Add(i);
}
}
return ReturnList;
}
else
{
// create a copy of ReturnList and Remove from the copy numbers bigger than num
}
}
Interview questions are usually more about how you approach problems in general and not so much the technical implementation. In your case you could do a lot of small things, like caching the list outside. Caching different versions of the list in a dictionary, if space was not a problem. Maybe somebody can come up with some smarter math, to save on calculations, but usually it's more about asking the right questions and considering the right options. Say, if you ask "does this program run on a web server? maybe I can store all data in a table and use it as a quick lookup instead of recalculating every time." There might not even be a correct or best answer, they probably just want to hear, that you can think of special situations.
You can find all the numbers that are divisible by 7 and smaller than num by calculating res = num/7 and then create a loop from 1 to res and multiply each number by 7.
private List<int> GetLessThanOrDivisbleBySeven(int num)
{
List<int> ReturnList = new List<int>();
// Add all the numbers that are less than 7 first
int i = 0;
for(i = 0; i < 7; i++)
ReturnList.Add(i);
int res = num / 7;// num = res*7+rem
for(i = 1; i <= res; i++)
{
ReturnList.Add(i*7);
}
return ReturnList;
}
Think about memory management and how the List class works.
Unless you tell it the capacity it will need, it allocates a new array whenever it runs out of space, however it is easy to work out the size it will need to be.
Returning an array would save one object allocation compared to using a List, so discuss the tradeoff between the two.
What about using "yeild return" to advoid allocating memory, or does it have other costs to consider?
Is the same number requested often, if so consider cacheing.
Would LINQ, maybe using Enumerable.Range help?
An experienced C# programmer would be expected to know at least a little about all the above and that memory management is often an hidden issue.

How to calculate daily total? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 8 years ago.
Improve this question
i have this code and its working but i want it to return dailyTotals list with four values, at the moment it is being returned with three values ... i cant seem to figure out how to make it return dailyTotals with four values. BTW the billTotals and dateList are of the same lengh which is currently 7 and they will grow. billTotals and dateList will always remain the same length
List<double> dailyTotals = new List<double>();
string dateValue = "";
double oneDayTotal = 0;
int n = 0;
for (int i = 0; i < billTotals.Count; i += n)
{
dateValue = dateList[i];
oneDayTotal = 0;
while (dateValue == dateList[n])
{
oneDayTotal += billTotals[n];
n++;
}
dailyTotals.Add(oneDayTotal);
}
return dailyTotals;
[EDIT]: Im sorry i should have written this before :/
in the database i have billTotals and the date for each bill stored. so, one date can have multiple bills associated with it. what im trying to do is grab one months data and sum the totals for each day. so the logic i used in the while loop is supposed to sum up the totals while the date is the same ... i hope this makes the scenario more clear. :)
You never reset n, so you are increasing i by increasingly large amount, skipping over some numbers.
You need to set n = 0 inside the loop.
int n = 0;
for (int i = 0; i < billTotals.Count; i += n)
{
dateValue = dateList[i];
oneDayTotal = 0;
n = 0; // This counts the number of equal dates.
while (i + n < dateList.Length && dateValue == dateList[i + n])
{
oneDayTotal += billTotals[i + n];
n++;
}
dailyTotals.Add(oneDayTotal);
}
return dailyTotals;
You could also rewrite the code entirely to simplify it and avoid this curious way of incrementing your loop variable. I would suggest creating an array of objects that hold both the date and the total, then you can use LINQ to solve your problem without any complex loops:
var dailyTotals = datesAndTotals
.GroupBy(x => x.Date)
.Select(g => g.Sum(x => x.BillTotal))
.ToList();
You can also use Zip instead of creating a separate class, though it will be more readable with the class.

Categories