We Keep Coding

Nested Threads (Tasks) Timing Out Prematurely

I have the following code, what it does I don't believe is important, but I'm getting strange behavior.
When I run just the months on separate threads, it runs fine(how it is below), but when I multi-thread the years(uncomment the tasks), it will timeout every time. The timeout is set for 5 minutes for months/20 minutes for years and it will timeout within a minute.
Is there a known reason for this behavior? Am I missing something simple?
public List<PotentialBillingYearItem> GeneratePotentialBillingByYear()
{
var years = new List<PotentialBillingYearItem>();
//var tasks = new List<Task>();
var startYear = new DateTime(DateTime.Today.Year - 10, 1, 1);
var range = new DateRange(startYear, DateTime.Today.LastDayOfMonth());
for (var i = range.Start; i <= range.End; i = i.AddYears(1))
{
var yearDate = i;
//tasks.Add(Task.Run(() =>
//{
years.Add(new PotentialBillingYearItem
{
Total = GeneratePotentialBillingMonths(new PotentialBillingParameters { Year = yearDate.Year }).Average(s => s.Total),
Date = yearDate
});
//}));
}
//Task.WaitAll(tasks.ToArray(), TimeSpan.FromMinutes(20));
return years;
}
public List<PotentialBillingItem> GeneratePotentialBillingMonths(PotentialBillingParameters Parameters)
{
var items = new List<PotentialBillingItem>();
var tasks = new List<Task>();
var year = new DateTime(Parameters.Year, 1, 1);
var range = new DateRange(year, year.LastDayOfYear());
range.Start = range.Start == range.End ? DateTime.Now.FirstDayOfYear() : range.Start.FirstDayOfMonth();
if (range.End > DateTime.Today) range.End = DateTime.Today.LastDayOfMonth();
for (var i = range.Start; i <= range.End; i = i.AddMonths(1))
{
var firstDayOfMonth = i;
var lastDayOfMonth = i.LastDayOfMonth();
var monthRange = new DateRange(firstDayOfMonth, lastDayOfMonth);
tasks.Add(Task.Run(() =>
{
using (var db = new AlbionConnection())
{
var invoices = GetInvoices(lastDayOfMonth);
var timeslipSets = GetTimeslipSets();
var item = new PotentialBillingItem
{
Date = firstDayOfMonth,
PostedInvoices = CalculateInvoiceTotals(invoices.Where(w => w.post_date <= lastDayOfMonth), monthRange),
UnpostedInvoices = CalculateInvoiceTotals(invoices.Where(w => w.post_date == null || w.post_date > lastDayOfMonth), monthRange),
OutstandingDrafts = CalculateOutstandingDraftTotals(timeslipSets)
};
items.Add(item);
}
}));
}
Task.WaitAll(tasks.ToArray(), TimeSpan.FromMinutes(5));
return items;
}

You might consider pre-allocating a bigger number of threadpool threads. The threadpool is very slow to allocate new threads. The code below task only 10 seconds (the theoretical minimum) to run setting the minimum number of threadpool threads to 2.5k, but commenting out the SetMinThreads makes it take over 1:30 seconds.
static void Main(string[] args)
{
ThreadPool.SetMinThreads(2500, 10);
Stopwatch sw = Stopwatch.StartNew();
RunTasksOutter(10);
sw.Stop();
Console.WriteLine($"Finished in {sw.Elapsed}");
}
public static void RunTasksOutter(int num) => Task.WaitAll(Enumerable.Range(0, num).Select(x => Task.Run(() => RunTasksInner(10))).ToArray());
public static void RunTasksInner(int num) => Task.WaitAll(Enumerable.Range(0, num).Select(x => Task.Run(() => Thread.Sleep(10000))).ToArray());
You could also be running out of threadpool threads. Per: https://msdn.microsoft.com/en-us/library/0ka9477y(v=vs.110).aspx one of the times to not use the threadpool (which is used by tasks) is:
You have tasks that cause the thread to block for long periods of time. The thread pool has a maximum number of threads, so a large number of blocked thread pool threads might prevent tasks from starting.
Since IO is being done on these threads maybe consider replacing them with async code or starting them with the LongRunning option? https://msdn.microsoft.com/en-us/library/system.threading.tasks.taskcreationoptions(v=vs.110).aspx

Display CPU usage of all running processes, programmatically?

Question
How do I get the cpu usage of each process into PopulateApplications()?
What's happening
getCPUUsage() gives me the same value for each process. It's like it's getting the cpu usage for only one process.
The rest of the code seems to work fine.
getCPUUsage() from class Core:
public static double getCPUUsage()
{
ManagementObject processor = new ManagementObject("Win32_PerfFormattedData_PerfOS_Processor.Name='_Total'");
processor.Get();
return double.Parse(processor.Properties["PercentProcessorTime"].Value.ToString());
}
What I've tried
In form1, I have a method by which I display information about processes like icons, name, and statuses (i.e. running/not running).
void PopulateApplications()
{
DoubleBufferedd(dataGridView1, true);
int rcount = dataGridView1.Rows.Count;
int rcurIndex = 0;
foreach (Process p in Process.GetProcesses())
{
try
{
if (File.Exists(p.MainModule.FileName))
{
var icon = Icon.ExtractAssociatedIcon(p.MainModule.FileName);
Image ima = icon.ToBitmap();
ima = resizeImage(ima, new Size(25, 25));
ima = (Image)(new Bitmap(ima, new Size(25, 25)));
String status = p.Responding ? "Running" : "Not Responding";
if (rcurIndex < rcount - 1)
{
var currentRow = dataGridView1.Rows[rcurIndex];
currentRow.Cells[0].Value = ima;
currentRow.Cells[1].Value = p.ProcessName;
currentRow.Cells[2].Value = cpuusage;
currentRow.Cells[3].Value = status;
}
else
{
dataGridView1.Rows.Add(
ima, p.ProcessName,cpuusage, status);//false, ima, p.ProcessName, status);
}
rcurIndex++;
}
}
catch ( Exception e)
{
string t = "error";
}
}
if (rcurIndex < rcount - 1)
{
for (int i = rcurIndex; i < rcount - 1; i++)
{
dataGridView1.Rows.RemoveAt(rcurIndex);
}
}
}
I added this line:
currentRow.Cells[2].Value = cpuusage;
cpuusage is double-type variable.
I changed this line, also, to include addition of cpuusage:
dataGridView1.Rows.Add(
ima, p.ProcessName,cpuusage, status);
Now I have a background worker event, dowork, whereby I use cpuusage to get the cpu usage values:
this.Invoke(new Action(() => cpuusage = Core.getCPUUsage()));
Maybe I don't need to call the method getCPUUsage() through backgroundworker.
This is what i see when im running the program:
All the processes have the same cpu usage ? Not logic.
Then when there is an update i see:
Again all the cells have the same cpu usage value. But on the left there are many processes each should have it's own cpu usage.

Parallel.ForEach and DbContext

I'm using Parallel.ForEach and it's hugely improving the performance of my code, but I'm curious about DbContext with multiple threads. I know it's not thread safe so I'm using locks where I need to.
The loop iterates over a dictionary and calculates statistics:
Dictionary<string, List<decimal>> decimalStats = new Dictionary<string, List<decimal>>(); // this gets populated in another irrelevant loop
List<ComparativeStatistic> comparativeStats = db.ComparativeStatistics.ToList();
var statLock = new object();
Parallel.ForEach(decimalStats, entry =>
{
List<decimal> vals = ((List<decimal>)entry.Value).ToList();
if (vals.Count > 0)
{
string[] ids = entry.Key.Split('#');
int questionId = int.Parse(ids[0]);
int yearId = int.Parse(ids[1]);
int adjacentYearId = int.Parse(ids[2]);
var stat = comparativeStats.Where(l => l.QuestionID == questionId && l.YearID == yearId && l.AdjacentYearID == adjacentYearId).FirstOrDefault();
if (stat == null)
{
stat = new ComparativeStatistic();
stat.QuestionnaireQuestionID = questionId;
stat.FinancialYearID = yearId;
stat.AdjacentFinancialYearID = adjacentYearId;
stat.CurrencyID = currencyId;
stat.IndustryID = industryId;
lock (statLock) { db.ComparativeStatistics.Add(stat); }
}
stat.TimeStamp = DateTime.Now;
decimal total = 0M;
decimal? mean = null;
foreach (var val in vals)
{
total += val;
}
mean = Decimal.Round((total / vals.Count), 2, MidpointRounding.AwayFromZero);
stat.Mean = mean;
}
});
db.SaveChanges();
My question: Why do I only need the lock when I'm adding something to the database? If stat is never null - if there's always already a database entry for it - I can run this loop without a lock with no problems, and the database gets updated as intended. If stat is null for a particular loop and I don't have the lock there, a System.AggregateException gets thrown.
edit1: I've tried opening a new connection to the database each time instead of using lock, which also works when adding to the database (identical to the loop above, I've added comments where it differs):
Parallel.ForEach(decimalStats, entry =>
{
List<decimal> vals = ((List<decimal>)entry.Value).ToList();
if (vals.Count > 0)
{
using (var dbThread = new PDBContext()) // new db connection
{
string[] ids = entry.Key.Split('#');
int questionId = int.Parse(ids[0]);
int yearId = int.Parse(ids[1]);
int adjacentYearId = int.Parse(ids[2]);
var stat = comparativeStats.Where(l => l.QuestionID == questionId && l.YearID == yearId && l.AdjacentYearID == adjacentYearId).FirstOrDefault();
if (stat == null)
{
stat = new ComparativeStatistic();
stat.QuestionnaireQuestionID = questionId;
stat.FinancialYearID = yearId;
stat.AdjacentFinancialYearID = adjacentYearId;
stat.CurrencyID = currencyId;
stat.IndustryID = industryId;
dbThread.ComparativeStatistics.Add(stat); // no need for a lock
}
stat.TimeStamp = DateTime.Now;
decimal total = 0M;
decimal? mean = null;
foreach (var val in vals)
{
total += val;
}
mean = Decimal.Round((total / vals.Count), 2, MidpointRounding.AwayFromZero);
stat.Mean = mean;
dbThread.SaveChanges(); // save
}
}
});
Is this safe to do? I'm sure Entity Framework's connection pooling is smart enough but I'm wondering if I should add any parameters to limit the number of threads/connections.

c# RavenDB embedded optimize

I have a database (RavenDB) which needs to be able to handle 300 queries (Full text search) every 10 seconds. To increase peformance I splitted up the database so I have multiple documentStores
my Code:
var watch = Stopwatch.StartNew();
int taskcnt = 0;
int sum = 0;
for (int i = 0; i < 11; i++)
{
Parallel.For(0, 7, new Action<int>((x) =>
{
for(int docomentStore = 0;docomentStore < 5; docomentStore++)
{
var stopWatch = Stopwatch.StartNew();
Task<IList<eBayItem>> task = new Task<IList<eBayItem>>(Database.ExecuteQuery, new Filter()
{
Store = "test" + docomentStore,
MaxPrice = 600,
MinPrice = 200,
BIN = true,
Keywords = new List<string>() { "Canon", "MP", "Black" },
ExcludedKeywords = new List<string>() { "G1", "T3" }
});
task.ContinueWith((list) => {
stopWatch.Stop();
sum += stopWatch.Elapsed.Milliseconds;
taskcnt++;
if (taskcnt == 300)
{
watch.Stop();
Console.WriteLine("Average time: " + (sum / (float)300).ToString());
Console.WriteLine("Total time: " + watch.Elapsed.ToString() + "ms");
}
});
task.Start();
}
}));
Thread.Sleep(1000);
}
Average query time: 514,13 ms
Total time: 00:01:29.9108016
The code where I query ravenDB:
public static IList<eBayItem> ExecuteQuery(object Filter)
{
IList<eBayItem> items;
Filter filter = (Filter)Filter;
if (int.Parse(filter.Store.ToCharArray().Last().ToString()) > 4)
{
Console.WriteLine(filter.Store); return null;
}
using (var session = Shards[filter.Store].OpenSession())
{
var query = session.Query<eBayItem, eBayItemIndexer>().Where(y => y.Price <= filter.MaxPrice && y.Price >= filter.MinPrice);
query = filter.Keywords.ToArray()
.Aggregate(query, (q, term) =>
q.Search(xx => xx.Title, term, options: SearchOptions.And));
if (filter.ExcludedKeywords.Count > 0)
{
query = filter.ExcludedKeywords.ToArray().Aggregate(query, (q, exterm) =>
q.Search(it => it.Title, exterm, options: SearchOptions.Not));
}
items = query.ToList<eBayItem>();
}
return items;
}
And the initialization of RavenDB:
static Dictionary<string, EmbeddableDocumentStore> Shards = new Dictionary<string, EmbeddableDocumentStore>();
public static void Connect()
{
Shards.Add("test0", new EmbeddableDocumentStore() { DataDirectory = "test.db" });
Shards.Add("test1", new EmbeddableDocumentStore() { DataDirectory = "test1.db" });
Shards.Add("test2", new EmbeddableDocumentStore() { DataDirectory = "test2.db" });
Shards.Add("test3", new EmbeddableDocumentStore() { DataDirectory = "test3.db" });
Shards.Add("test4", new EmbeddableDocumentStore() { DataDirectory = "test4.db" });
foreach (string key in Shards.Keys)
{
EmbeddableDocumentStore store = Shards[key];
store.Initialize();
IndexCreation.CreateIndexes(typeof(eBayItemIndexer).Assembly, store);
}
}
How can I optimize my code so my total time is lower ? Is it good to divide my database up in 5 different ones ?
EDIT: The program has only 1 documentStore instead of 5. (As sugested by Ayende Rahien)
Also this is the Query on its own:
Price_Range:[* TO Dx600] AND Price_Range:[Dx200 TO NULL] AND Title:(Canon) AND Title:(MP) AND Title:(Black) -Title:(G1) -Title:(T3)

No, this isn't good.
Use a single embedded RavenDB. If you need sharding, this involved multiple machines.
In general, RavenDB queries are in the few ms each. You need to show what your queries looks like (you can call ToString() on them to see that).
Having shards of RavenDB in this manner means that all of them are fighting for CPU and IO

I know this is an old post but this was the top search result I got.
I had the same problem that my queries were taking 500ms. It now takes 100ms by applying the following search practices: http://ravendb.net/docs/article-page/2.5/csharp/client-api/querying/static-indexes/searching

Task Parallel Library - I don't understand what I'm doing wrong

This is a two part question.
I have a class that gets all processes asynchronously and polls them for CPU usage. Yesterday I had a bug with it and it was solved here.
The first part of the question is why the solution helped. I didn't understand the explanation.
The second part of the question is that I still get an "Object reference not set to an instance of object" exception occasionally when I try to print the result at the end of the process. This is because item.Key is indeed null. I don't understand why that is because I put a breakpoint checking for (process == null) and it was never hit. What am I doing wrong?
Code is below.
class ProcessCpuUsageGetter
{
private IDictionary<Process, int> _usage;
public IDictionary<Process, int> Usage { get { return _usage; } }
public ProcessCpuUsageGetter()
{
while (true)
{
Process[] processes = Process.GetProcesses();
int processCount = processes.Count();
Task[] tasks = new Task[processCount];
_usage = new Dictionary<Process, int>();
for (int i = 0; i < processCount; i++)
{
var localI = i;
var localProcess = processes[localI];
tasks[localI] = Task.Factory.StartNew(() => DoWork(localProcess));
}
Task.WaitAll(tasks);
foreach (var item in Usage)
{
Console.WriteLine("{0} - {1}%", item.Key.ProcessName, item.Value);
}
}
}
private void DoWork(object o)
{
Process process = (Process)o;
PerformanceCounter pc = new PerformanceCounter("Process", "% Processor Time", process.ProcessName, true);
pc.NextValue();
Thread.Sleep(1000);
int cpuPercent = (int)pc.NextValue() / Environment.ProcessorCount;
if (process == null)
{
var x = 5;
}
if (_usage == null)
{
var t = 6;
}
_usage.Add(process, cpuPercent);
}
}

The line
_usage.Add(process, cpuPercent);
is accessing a not-threadsafe collection from a thread.
Use a ConcurrentDictionary<K,V> instead of the normal dictionary.
The 'null reference' error is just a random symptom, you could get other errors too.