I have written some test code to retrieve 1000 strings from my Redis cache. Obviously it is getting the same string in this test but it was written to see how long it would take to get these 1000 items.
The test completes in 23 seconds, so that is only around 43 strings per second that seems quite slow.
I am running this locally against the Redis instance that is in Azure, so I’m assuming there will be some latency. Have I missed out something or is there a way to reduce the time to get these 1000 items?
In my production environment, there could be several thousand items that need to be retrieved.
class Program
{
static async Task Main(string[] args)
{
var connectionString = #"testserver-rc.redis.cache.windows.net:6380,password=password,ssl=True,abortConnect=False,defaultDatabase=2";
var redisClient = new StackExchangeRedisCacheClient(new NewtonsoftSerializer(), connectionString, 2);
await TestGets(redisClient);
Console.ReadLine();
}
private static async Task TestGets(StackExchangeRedisCacheClient redisClient)
{
Console.WriteLine("Running...");
var sw = new Stopwatch();
sw.Start();
for (var i = 0; i < 1000; i++)
{
await redisClient.Database.StringGetAsync("test_T-0004");
}
Console.WriteLine($"{sw.Elapsed.Seconds} seconds");
}
}
43 per second? That sounds pretty fast. That means including overhead and latency you are spending 23ms per query.
I think you want to parallelize the query.
Try replacing your query line with
await Task.WhenAll(Enumerable.Range(0, 1000).Select(I => redisClient.Database.StringGetAsync("test_T-0004"));
The problem is that you are latency bound. You are waiting for each request to complete before firing the next one off.
Related
I'm new in MongoDB ecosystem and faced with a problem using MongoDB on my ASP.NET MVC application. If I call Find method for a collection just once it works very fast, just a few milliseconds. But if I call the Find method several times in the same moment using Task.Run() it gets extremely slow and even could exceed 30 sec timeout.
Performance degradation numbers are like the following:
a single query takes ~0.002 second to finish
4 parallel queries take ~3 seconds to finish (average duration is ~2.5 seconds)
40 parallel queries take 30+ seconds to finish and some of them are failed because of timeout (~18 seconds to finish in average).
If I run the same 40 queries one after each other without using tasks, all of them will complete less than in a second (~0.7 sec), i.e. the same ~0.002 sec per query.
What I have done already and it did not help:
No matter whether I create connection object inside the task code or create it once in the beginning and store in static field.
No matter which way I query the data:
collection.AsQueriable().Where(...).Take(1)
collection.Find(Builders<T>.Filter.Eq(...)).FirstOrDefault()
collection.Find(m => ...).FirstOrDefault()
Increasing timeout helps to avoid timeout exceptions but performance is still expectedly bad.
My environment:
I have very few data in my MongoDB collection (21 object, 1995 bytes total, 95 bytes per object average). The MongoDB server is at localhost environment, so network could not be an issue here. Server version is 3.2.4, works as Windows service. I get the same result using MongoDB hosted by linux VM in Azure. .NET MongoDB Driver is the last one and installed from nuget, the version is 2.3.0.157.
I cannot believe that such a mature system cannot handle few queries in the same time. So probably I have missed something. Could someone help and point me out the direction?
EDIT. The code sample I used for testing:
// IMongoDatabase database;
int iterations = 40;
var tasks = new Task<TimeSpan>[iterations];
for (int i = 0; i < iterations; i++) {
var tempI = i;
tasks[i] = new Task<TimeSpan>(() => {
var stopwatch = new Stopwatch();
stopwatch.Start();
var integrationId = INTEGRATION_IDS[tempI];
try {
var metadataCollection = database.GetCollection<CacheMetadata>(METADATA_COLLECTION_NAME);
var query = metadataCollection.AsQueryable()
.Where(m => m.IntegrationId == integrationId)
.Take(1);
var metadata = query.FirstOrDefault();
stopwatch.Stop();
// Write elapsed time to the console
}
catch (Exception ex) {
// Write the exception details to the console
}
return stopwatch.Elapsed;
});
}
for (int i = 0; i < iterations; i++) {
tasks[i].Start();
}
Task.WaitAll(tasks);
// Write summary to the console
I'm implementing a Redis caching layer using the Stackexchange Redis client and the performance right now is bordering on unusable.
I have a local environment where the web application and the redis server are running on the same machine. I ran the Redis benchmark test against my Redis server and the results were actually really good (I'm just including set and get operations in my write up):
C:\Program Files\Redis>redis-benchmark -n 100000
====== PING_INLINE ======
100000 requests completed in 0.88 seconds
50 parallel clients
3 bytes payload
keep alive: 1
====== SET ======
100000 requests completed in 0.89 seconds
50 parallel clients
3 bytes payload
keep alive: 1
99.70% <= 1 milliseconds
99.90% <= 2 milliseconds
100.00% <= 3 milliseconds
111982.08 requests per second
====== GET ======
100000 requests completed in 0.81 seconds
50 parallel clients
3 bytes payload
keep alive: 1
99.87% <= 1 milliseconds
99.98% <= 2 milliseconds
100.00% <= 2 milliseconds
124069.48 requests per second
So according to the benchmarks I am looking at over 100,000 sets and 100,000 gets, per second. I wrote a unit test to do 300,000 set/gets:
private string redisCacheConn = "localhost:6379,allowAdmin=true,abortConnect=false,ssl=false";
[Fact]
public void PerfTestWriteShortString()
{
CacheManager cm = new CacheManager(redisCacheConn);
string svalue = "t";
string skey = "testtesttest";
for (int i = 0; i < 300000; i++)
{
cm.SaveCache(skey + i, svalue);
string valRead = cm.ObtainItemFromCacheString(skey + i);
}
}
This uses the following class to perform the Redis operations via the Stackexchange client:
using StackExchange.Redis;
namespace Caching
{
public class CacheManager:ICacheManager, ICacheManagerReports
{
private static string cs;
private static ConfigurationOptions options;
private int pageSize = 5000;
public ICacheSerializer serializer { get; set; }
public CacheManager(string connectionString)
{
serializer = new SerializeJSON();
cs = connectionString;
options = ConfigurationOptions.Parse(connectionString);
options.SyncTimeout = 60000;
}
private static readonly Lazy<ConnectionMultiplexer> lazyConnection = new Lazy<ConnectionMultiplexer>(() => ConnectionMultiplexer.Connect(options));
private static ConnectionMultiplexer Connection => lazyConnection.Value;
private static IDatabase cache => Connection.GetDatabase();
public string ObtainItemFromCacheString(string cacheId)
{
return cache.StringGet(cacheId);
}
public void SaveCache<T>(string cacheId, T cacheEntry, TimeSpan? expiry = null)
{
if (IsValueType<T>())
{
cache.StringSet(cacheId, cacheEntry.ToString(), expiry);
}
else
{
cache.StringSet(cacheId, serializer.SerializeObject(cacheEntry), expiry);
}
}
public bool IsValueType<T>()
{
return typeof(T).IsValueType || typeof(T) == typeof(string);
}
}
}
My JSON serializer is just using Newtonsoft.JSON:
using System.Collections.Generic;
using Newtonsoft.Json;
namespace Caching
{
public class SerializeJSON:ICacheSerializer
{
public string SerializeObject<T>(T cacheEntry)
{
return JsonConvert.SerializeObject(cacheEntry, Formatting.None,
new JsonSerializerSettings()
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore
});
}
public T DeserializeObject<T>(string data)
{
return JsonConvert.DeserializeObject<T>(data, new JsonSerializerSettings()
{
ReferenceLoopHandling = ReferenceLoopHandling.Ignore
});
}
}
}
My test times are around 21 seconds (for 300,000 sets and 300,000 gets). This gives me around 28,500 operations per second (at least 3 times slower than I would expect using the benchmarks). The application I am converting to use Redis is pretty chatty and certain heavy requests can approximate 200,000 total operations against Redis. Obviously I wasn't expecting anything like the same times I was getting when using the system runtime cache, but the delays after this change are significant. Am I doing something wrong with my implementation and does anyone know why my benchmarked figures are so much faster than my Stackechange test figures?
Thanks,
Paul
My results from the code below:
Connecting to server...
Connected
PING (sync per op)
1709ms for 1000000 ops on 50 threads took 1.709594 seconds
585137 ops/s
SET (sync per op)
759ms for 500000 ops on 50 threads took 0.7592914 seconds
658761 ops/s
GET (sync per op)
780ms for 500000 ops on 50 threads took 0.7806102 seconds
641025 ops/s
PING (pipelined per thread)
3751ms for 1000000 ops on 50 threads took 3.7510956 seconds
266595 ops/s
SET (pipelined per thread)
1781ms for 500000 ops on 50 threads took 1.7819831 seconds
280741 ops/s
GET (pipelined per thread)
1977ms for 500000 ops on 50 threads took 1.9772623 seconds
252908 ops/s
===
Server configuration: make sure persistence is disabled, etc
The first thing you should do in a benchmark is: benchmark one thing. At the moment you're including a lot of serialization overhead, which won't help get a clear picture. Ideally, for a like-for-like benchmark, you should be using a 3-byte fixed payload, because:
3 bytes payload
Next, you'd need to look at parallelism:
50 parallel clients
It isn't clear whether your test is parallel, but if it isn't we should absolutely expect to see less raw throughput. Conveniently, SE.Redis is designed to be easy to parallelize: you can just spin up multiple threads talking to the same connection (this actually also has the advantage of avoiding packet fragmentation, as you can end up with multiple messages per packet, where-as a single-thread sync approach is guaranteed to use at most one message per packet).
Finally, we need to understand what the listed benchmark is doing. Is it doing:
(send, receive) x n
or is it doing
send x n, receive separately until all n are received
? Both options are possible. Your sync API usage is the first one, but the second test is equally well-defined, and for all I know: that's what it is measuring. There are two ways of simulating this second setup:
send the first (n-1) messages with the "fire and forget" flag, so you only actually wait for the last one
use the *Async API for all messages, and only Wait() or await the last Task
Here's a benchmark that I used in the above, that shows both "sync per op" (via the sync API) and "pipeline per thread" (using the *Async API and just waiting for the last task per thread), both using 50 threads:
using StackExchange.Redis;
using System;
using System.Diagnostics;
using System.Threading;
using System.Threading.Tasks;
static class P
{
static void Main()
{
Console.WriteLine("Connecting to server...");
using (var muxer = ConnectionMultiplexer.Connect("127.0.0.1"))
{
Console.WriteLine("Connected");
var db = muxer.GetDatabase();
RedisKey key = "some key";
byte[] payload = new byte[3];
new Random(12345).NextBytes(payload);
RedisValue value = payload;
DoWork("PING (sync per op)", db, 1000000, 50, x => { x.Ping(); return null; });
DoWork("SET (sync per op)", db, 500000, 50, x => { x.StringSet(key, value); return null; });
DoWork("GET (sync per op)", db, 500000, 50, x => { x.StringGet(key); return null; });
DoWork("PING (pipelined per thread)", db, 1000000, 50, x => x.PingAsync());
DoWork("SET (pipelined per thread)", db, 500000, 50, x => x.StringSetAsync(key, value));
DoWork("GET (pipelined per thread)", db, 500000, 50, x => x.StringGetAsync(key));
}
}
static void DoWork(string action, IDatabase db, int count, int threads, Func<IDatabase, Task> op)
{
object startup = new object(), shutdown = new object();
int activeThreads = 0, outstandingOps = count;
Stopwatch sw = default(Stopwatch);
var threadStart = new ThreadStart(() =>
{
lock(startup)
{
if(++activeThreads == threads)
{
sw = Stopwatch.StartNew();
Monitor.PulseAll(startup);
}
else
{
Monitor.Wait(startup);
}
}
Task final = null;
while (Interlocked.Decrement(ref outstandingOps) >= 0)
{
final = op(db);
}
if (final != null) final.Wait();
lock(shutdown)
{
if (--activeThreads == 0)
{
sw.Stop();
Monitor.PulseAll(shutdown);
}
}
});
lock (shutdown)
{
for (int i = 0; i < threads; i++)
{
new Thread(threadStart).Start();
}
Monitor.Wait(shutdown);
Console.WriteLine($#"{action}
{sw.ElapsedMilliseconds}ms for {count} ops on {threads} threads took {sw.Elapsed.TotalSeconds} seconds
{(count * 1000) / sw.ElapsedMilliseconds} ops/s");
}
}
}
You are fetching data in synchronous way (50 clients in parallel but each client's requests are made synchronously instead of asynchronously)
One option would be to use the async/await methods (StackExchange.Redis support that).
If you need to get multiple keys at once (for example to build a daily graph of visitors to your website assuming you save visitors counter per day keys) then you should try fetching data from redis in asynchronous manner using redis pipelining, this should give you much better performance.
StackExchange redis client old versions have performance issues.
Upgrade to the newest version. Read more here:
https://www.gitmemory.com/issue/mgravell/Pipelines.Sockets.Unofficial/28/479932064
and in this article:
https://blog.marcgravell.com/2019/02/fun-with-spiral-of-death.html
this is the issue in the repo:
https://github.com/StackExchange/StackExchange.Redis/issues/1003
I need to process a large number of files overnight, with a defined start and end time to avoid disrupting users. I've been investigating but there are so many ways of handling threading now that I'm not sure which way to go. The files come into an Exchange inbox as attachments.
My current attempt, based on some examples from here and a bit of experimentation, is:
while (DateTime.Now < dtEndTime.Value)
{
var finished = new CountdownEvent(1);
for (int i = 0; i < numThreads; i++)
{
object state = offset;
finished.AddCount();
ThreadPool.QueueUserWorkItem(delegate
{
try
{
StartProcessing(state);
}
finally
{
finished.Signal();
}
});
offset += numberOfFilesPerPoll;
}
finished.Signal();
finished.Wait();
}
It's running in a winforms app at the moment for ease, but the core processing is in a dll so I can spawn the class I need from a windows service, from a console running under a scheduler, however is easiest. I do have a Windows Service set up with a Timer object that kicks off the processing at a time set in the config file.
So my question is - in the above code, I initialise a bunch of threads (currently 10), then wait for them all to process. My ideal would be a static number of threads, where as one finishes I fire off another, and then when I get to the end time I just wait for all threads to complete.
The reason for this is that the files I'm processing are variable sizes - some might take seconds to process and some might take hours, so I don't want the whole application to wait while one thread completes if I can have it ticking along in the background.
(edit)As it stands, each thread instantiates a class and passes it an offset. The class then gets the next x emails from the inbox, starting at the offset (using the Exchange Web Services paging functionality). As each file is processed, it's moved to a separate folder. From some of the replies so far, I'm wondering if actually I should grab the e-mails in the outer loop, and spawn threads as needed.
To cloud the issue, I currently have a backlog of e-mails that I'm trying to process through. Once the backlog has been cleared, it's likely that the nightly run will have a significantly lower load.
On average there are around 1000 files to process each night.
Update
I've rewritten large chunks of my code so that I can use the Parallel.Foreach and I've come up against an issue with thread safety. The calling code now looks like this:
public bool StartProcessing()
{
FindItemsResults<Item> emails = GetEmails();
var source = new CancellationTokenSource(TimeSpan.FromHours(10));
// Process files in parallel, with a maximum thread count.
var opts = new ParallelOptions { MaxDegreeOfParallelism = 8, CancellationToken = source.Token };
try
{
Parallel.ForEach(emails, opts, processAttachment);
}
catch (OperationCanceledException)
{
Console.WriteLine("Loop was cancelled.");
}
catch (Exception err)
{
WriteToLogFile(err.Message + "\r\n");
WriteToLogFile(err.StackTrace + "r\n");
}
return true;
}
So far so good (excuse temporary error handling). I have a new issue now with the fact that the properties of the "Item" object, which is an email, not being threadsafe. So for example when I start processing an e-mail, I move it to a "processing" folder so that another process can't grab it - but it turns out that several of the threads might be trying to process the same e-mail at a time. How do I guarantee that this doesn't happen? I know I need to add a lock, can I add this in the ForEach or should it be in the processAttachments method?
Use the TPL:
Parallel.ForEach( EnumerateFiles(),
new ParallelOptions { MaxDegreeOfParallelism = 10 },
file => ProcessFile( file ) );
Make EnumerateFiles stop enumerating when your end time is reached, trivially like this:
IEnumerable<string> EnumerateFiles()
{
foreach (var file in Directory.EnumerateFiles( "*.txt" ))
if (DateTime.Now < _endTime)
yield return file;
else
yield break;
}
You can use a combination of Parallel.ForEach() along with a cancellation token source which will cancel the operation after a set time:
using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
static class Program
{
static Random rng = new Random();
static void Main()
{
// Simulate having a list of files.
var fileList = Enumerable.Range(1, 100000).Select(i => i.ToString());
// For demo purposes, cancel after a few seconds.
var source = new CancellationTokenSource(TimeSpan.FromSeconds(10));
// Process files in parallel, with a maximum thread count.
var opts = new ParallelOptions {MaxDegreeOfParallelism = 8, CancellationToken = source .Token};
try
{
Parallel.ForEach(fileList, opts, processFile);
}
catch (OperationCanceledException)
{
Console.WriteLine("Loop was cancelled.");
}
}
static void processFile(string file)
{
Console.WriteLine("Processing file: " + file);
// Simulate taking a varying amount of time per file.
int delay;
lock (rng)
{
delay = rng.Next(200, 2000);
}
Thread.Sleep(delay);
Console.WriteLine("Processed file: " + file);
}
}
}
As an alternative to using a cancellation token, you can write a method that returns IEnumerable<string> which returns the list of filenames, and stop returning them when time is up, for example:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
namespace Demo
{
static class Program
{
static Random rng = new Random();
static void Main()
{
// Process files in parallel, with a maximum thread count.
var opts = new ParallelOptions {MaxDegreeOfParallelism = 8};
Parallel.ForEach(fileList(), opts, processFile);
}
static IEnumerable<string> fileList()
{
// Simulate having a list of files.
var fileList = Enumerable.Range(1, 100000).Select(x => x.ToString()).ToArray();
// Simulate finishing after a few seconds.
DateTime endTime = DateTime.Now + TimeSpan.FromSeconds(10);
int i = 0;
while (DateTime.Now <= endTime)
yield return fileList[i++];
}
static void processFile(string file)
{
Console.WriteLine("Processing file: " + file);
// Simulate taking a varying amount of time per file.
int delay;
lock (rng)
{
delay = rng.Next(200, 2000);
}
Thread.Sleep(delay);
Console.WriteLine("Processed file: " + file);
}
}
}
Note that you don't need the try/catch with this approach.
You should consider using Microsoft's Reactive Framework. It lets you use LINQ queries to process multithreaded asynchronous processing in a very simple way.
Something like this:
var query =
from file in filesToProcess.ToObservable()
where DateTime.Now < stopTime
from result in Observable.Start(() => StartProcessing(file))
select new { file, result };
var subscription =
query.Subscribe(x =>
{
/* handle result */
});
Truly, that's all the code you need if StartProcessing is already defined.
Just NuGet "Rx-Main".
Oh, and to stop processing at any time just call subscription.Dispose().
This was a truly fascinating task, and it took me a while to get the code to a level that I was happy with it.
I ended up with a combination of the above.
The first thing worth noting is that I added the following lines to my web service call, as the operation timeout I was experiencing, and which I thought was because I'd exceeded some limit set on the endpoint, was actually due to a limit set by microsoft way back in .Net 2.0:
ServicePointManager.DefaultConnectionLimit = int.MaxValue;
ServicePointManager.Expect100Continue = false;
See here for more information:
What to set ServicePointManager.DefaultConnectionLimit to
As soon as I added those lines of code, my processing increased from 10/minute to around 100/minute.
But I still wasn't happy with the looping, and partitioning etc. My service moved onto a physical server to minimise CPU contention, and I wanted to allow the operating system to dictate how fast it ran, rather than my code throttling it.
After some research, this is what I ended up with - arguably not the most elegant code I've written, but it's extremely fast and reliable.
List<XElement> elements = new List<XElement>();
while (XMLDoc.ReadToFollowing("ElementName"))
{
using (XmlReader r = XMLDoc.ReadSubtree())
{
r.Read();
XElement node = XElement.Load(r);
//do some processing of the node here...
elements.Add(node);
}
}
//And now pass the list of elements through PLinQ to the actual web service call, allowing the OS/framework to handle the parallelism
int failCount=0; //the method call below sets this per request; we log and continue
failCount = elements.AsParallel()
.Sum(element => IntegrationClass.DoRequest(element.ToString()));
It ended up fiendishly simple and lightning fast.
I hope this helps someone else trying to do the same thing!
I am experimenting / learning the new Task library and I have written a very simple html downloader using WebClient and Task.Run. However I can never reach anything more than 5% on my network usage. I would like to understand why and how I can improve my code to reach 100% network usage / throughput (probably not possible but it has to be a lot more than 5%).
I would also like to be able to limit the number of thread however it seems it's not as easy as I thought (i.e. custom task scheduler). Is there a way to just do something like this to set the max thread count: something.SetMaxThread(2)?
internal static class Program
{
private static void Main()
{
for (var i = 0; i < 1000000; i++)
{
Go(i, Thread.CurrentThread.ManagedThreadId);
}
Console.Read();
}
private static readonly Action<int, int> Go = (counter, threadId) => Task.Run(() =>
{
var stopwatch = new Stopwatch();
stopwatch.Start();
var webClient = new WebClient();
webClient.DownloadString(new Uri("http://stackoverflow.com"));
stopwatch.Stop();
Console.Write("{0} == {1} | ", threadId.ToString("D3"), Thread.CurrentThread.ManagedThreadId.ToString("D3"));
Console.WriteLine("{0}: {1}ms ", counter.ToString("D3"), stopwatch.ElapsedMilliseconds.ToString("D4"));
});
}
This is the async version according to #spender. However my understanding is that await will "remember" the point in time and hand off the download to OS level and skip (the 2 console.write) and return to main immediately and continue scheduling the remaining Go method in the for loop. Am I understanding it correctly? So there's no blocking on the UI.
private static async void Go(int counter, int threadId)
{
using (var webClient = new WebClient())
{
var stopWatch = new Stopwatch();
stopWatch.Start();
await webClient.DownloadStringTaskAsync(new Uri("http://ftp.iinet.net.au/test500MB.dat"));
stopWatch.Stop();
Console.Write("{0} == {1} | ", threadId.ToString("D3"), Thread.CurrentThread.ManagedThreadId.ToString("D3"));
Console.WriteLine("{0}: {1}ms ", counter.ToString("D3"), stopWatch.ElapsedMilliseconds.ToString("D4"));
}
}
What I noticed was that when I am downloading large files there's no that much difference in terms of download speed / network usage. They (threading version and the async version) both peaked at about 12.5% network usage and about 12MByte download /sec. I also tried to run multiple instances (multiple .exe running) and again there's no huge difference between the two. And when I am trying to download large files from 2 URLs concurrently (20 instances) I get similar network usage (12.5%) and download speed (10-12MByte /sec). I guess I am reaching the peak?
As it stands, your code is suboptimal because, although you are using Task.Run to create asynchronous code that runs in the ThreadPool, the code that is being run in the ThreadPool is still blocking on the line:
webClient.DownloadString(...
This amounts to an abuse of the ThreadPool because it is not designed to run blocking tasks, and is slow to spin up additional threads to deal with peaks in workload. This in turn will have a seriously degrading effect on the smooth running of any API that uses the ThreadPool (timers, async callbacks, they're everywhere), because they'll schedule work that goes to the back of the (saturated) queue for the ThreadPool (which is tied up reluctantly and slowly spinning up hundreds of threads that are going to spend 99.9% of their time doing nothing).
Stop blocking the ThreadPool and switch to proper async methods that do not block.
So now you can literally break your router and seriously upset the SO site admins with the following simple mod:
private static void Main()
{
for (var i = 0; i < 1000000; i++)
{
Go(i, Thread.CurrentThread.ManagedThreadId);
}
Console.Read();
}
private static async Task Go(int counter, int threadId)
{
var stopwatch = new Stopwatch();
stopwatch.Start();
using (var webClient = new WebClient())
{
await webClient.DownloadStringTaskAsync(
new Uri("http://stackoverflow.com"));
}
//...
}
HttpWebRequest (and therefore WebClient) are also constrained by a number of limits.
I am having this issue when I ran something like this:
Parallel.ForEach(dataTable.AsEnumerable(), row =>
{
//do processing
}
Assuming that there are 500+ records say 870. Once the Parallel.ForEach completes 850, it seems to be running sequentially i.e. only 1 operation at a time. It completed 850 operations very fast but when it comes close to the end of the iteration it becomes very slow and seems to be performing like a regular for each. I even tried for 2000 records.
Is something wrong in my code? Please give suggestions.
Below is the code I am using
Sorry I just posted the wrong example. This is the correct code:
Task newTask = Task.Factory.StartNew(() =>
{
Parallel.ForEach(dtResult.AsEnumerable(), dr =>
{
string extractQuery = "";
string downLoadFileFullName = "";
lock (foreachObject)
{
string fileName = extractorConfig.EncodeFileName(dr);
extractQuery = extractorConfig.GetExtractQuery(dr);
if (string.IsNullOrEmpty(extractQuery)) throw new Exception("Extract Query not found. Please check the configuration");
string newDownLoadPath = CommonUtil.GetFormalizedDataPath(sDownLoadPath, uKey.CobDate);
//create folder if it doesn't exist
if (!Directory.Exists(newDownLoadPath)) Directory.CreateDirectory(newDownLoadPath);
downLoadFileFullName = Path.Combine(newDownLoadPath, fileName);
}
Interlocked.Increment(ref index);
ExtractorClass util = new ExtractorClass(SourceDbConnStr);
util.LoadToFile(extractQuery, downLoadFileFullName);
Interlocked.Increment(ref uiTimerIndex);
});
});
My guess:
This looks to have a high degree of potential IO from:
Database+Disk
Network communication to DB and back
Writing results to disk
Therefore a lot of time is going to be spent waiting for IO. My guess is that the waiting is only getting worse as more threads are being added to the mix and IO is being further stressed. For instance a disk only has one set of heads, so you cannot write to it concurrently. If you have a large number of threads trying to write concurrently, performance degrades.
Try limiting the maximum number of threads you are using:
var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };
Parallel.ForEach(dtResult.AsEnumerable(), options, dr =>
{
//Do stuff
});
Update
After your code edit, I would suggest the following which has a couple of changes:
Reduce maximum number of threads - this can be experimented with.
Only perform directory check and creation once.
Code:
private static bool isDirectoryCreated;
//...
var options = new ParallelOptions { MaxDegreeOfParallelism = 2 };
Parallel.ForEach(dtResult.AsEnumerable(), options, dr =>
{
string fileName, extractQuery, newDownLoadPath;
lock (foreachObject)
{
fileName = extractorConfig.EncodeFileName(dr);
extractQuery = extractorConfig.GetExtractQuery(dr);
if (string.IsNullOrEmpty(extractQuery))
throw new Exception("Extract Query not found. Please check the configuration");
newDownLoadPath = CommonUtil.GetFormalizedDataPath(sDownLoadPath, uKey.CobDate);
if (!isDirectoryCreated)
{
if (!Directory.Exists(newDownLoadPath))
Directory.CreateDirectory(newDownLoadPath);
isDirectoryCreated = true;
}
}
string downLoadFileFullName = Path.Combine(newDownLoadPath, fileName);
Interlocked.Increment(ref index);
ExtractorClass util = new ExtractorClass(SourceDbConnStr);
util.LoadToFile(extractQuery, downLoadFileFullName);
Interlocked.Increment(ref uiTimerIndex);
});
It’s hard to give details without the relevant code but in general this is the expected behaviour. .NET tries to schedule the tasks such that every processor is evenly busy.
But this can only ever be approximated sind not all of the tasks take the same amount of time. At the end some processors will be done working and some won’t, and re-distributing the work is costly and not always beneficial.
I don’t know details about the load balancing used by PLinq but the bottom line is that this behaviour can never be fully prevented.