We Keep Coding

Parallel.ForEach blocking calling method

I am having a problem with Parallel.ForEach. I have written simple application that adds file names to be downloaded to the queue, then using while loop it iterates through the queue, downloads file one at a time, then when file has been downloaded, another async method is called to create object from downloaded memoryStream. Returned result of this method is not awaited, it is discarded, so the next download starts immediately. Everything works fine if I use simple foreach in object creation - objects are being created while download is continuing. But if I would like to speed up the object creation process and use Parallel.ForEach it stops download process until the object is created. UI is fully responsive, but it just won't download the next object. I don't understand why is this happening - Parallel.ForEach is inside await Task.Run() and to my limited knowledge about asynchronous programming this should do the trick. Can anyone help me understand why is it blocking first method and how to avoid it?
Here is a small sample:
public async Task DownloadFromCloud(List<string> constructNames)
{
_downloadDataQueue = new Queue<string>();
var _gcsClient = StorageClient.Create();
foreach (var item in constructNames)
{
_downloadDataQueue.Enqueue(item);
}
while (_downloadDataQueue.Count > 0)
{
var memoryStream = new MemoryStream();
await _gcsClient.DownloadObjectAsync("companyprojects",
_downloadDataQueue.Peek(), memoryStream);
memoryStream.Position = 0;
_ = ReadFileXml(memoryStream);
_downloadDataQueue.Dequeue();
}
}
private async Task ReadFileXml(MemoryStream memoryStream)
{
var reader = new XmlReader();
var properties = reader.ReadXmlTest(memoryStream);
await Task.Run(() =>
{
var entityList = new List<Entity>();
foreach (var item in properties)
{
entityList.Add(CreateObjectsFromDownloadedProperties(item));
}
//Parallel.ForEach(properties item =>
//{
// entityList.Add(CreateObjectsFromDownloadedProperties(item));
//});
});
}
EDIT
This is simplified object creation method:
public Entity CreateObjectsFromDownloadedProperties(RebarProperties properties)
{
var path = new LinearPath(properties.Path);
var section = new Region(properties.Region);
var sweep = section.SweepAsMesh(path, 1);
return sweep;
}

Returned result of this method is not awaited, it is discarded, so the next download starts immediately.
This is also dangerous. "Fire and forget" means "I don't care when this operation completes, or if it completes. Just discard all exceptions because I don't care." So fire-and-forget should be extremely rare in practice. It's not appropriate here.
UI is fully responsive, but it just won't download the next object.
I have no idea why it would block the downloads, but there's a definite problem in switching to Parallel.ForEach: List<T>.Add is not threadsafe.
private async Task ReadFileXml(MemoryStream memoryStream)
{
var reader = new XmlReader();
var properties = reader.ReadXmlTest(memoryStream);
await Task.Run(() =>
{
var entityList = new List<Entity>();
Parallel.ForEach(properties, item =>
{
var itemToAdd = CreateObjectsFromDownloadedProperties(item);
lock (entityList) { entityList.Add(itemToAdd); }
});
});
}
One tip: if you have a result value, PLINQ is often cleaner than Parallel:
private async Task ReadFileXml(MemoryStream memoryStream)
{
var reader = new XmlReader();
var properties = reader.ReadXmlTest(memoryStream);
await Task.Run(() =>
{
var entityList = proeprties
.AsParallel()
.Select(CreateObjectsFromDownloadedProperties)
.ToList();
});
}
However, the code still suffers from the fire-and-forget problem.
For a better fix, I'd recommend taking a step back and using something more suited to "pipeline"-style processing. E.g., TPL Dataflow:
public async Task DownloadFromCloud(List<string> constructNames)
{
// Set up the pipeline.
var gcsClient = StorageClient.Create();
var downloadBlock = new TransformBlock<string, MemoryStream>(async constructName =>
{
var memoryStream = new MemoryStream();
await gcsClient.DownloadObjectAsync("companyprojects", constructName, memoryStream);
memoryStream.Position = 0;
return memoryStream;
});
var processBlock = new TransformBlock<MemoryStream, List<Entity>>(memoryStream =>
{
var reader = new XmlReader();
var properties = reader.ReadXmlTest(memoryStream);
return proeprties
.AsParallel()
.Select(CreateObjectsFromDownloadedProperties)
.ToList();
});
var resultsBlock = new ActionBlock<List<Entity>>(entities => { /* TODO */ });
downloadBlock.LinkTo(processBlock, new DataflowLinkOptions { PropagateCompletion = true });
processBlock.LinkTo(resultsBlock, new DataflowLinkOptions { PropagateCompletion = true });
// Push data into the pipeline.
foreach (var constructName in constructNames)
await downloadBlock.SendAsync(constructName);
downlodBlock.Complete();
// Wait for pipeline to complete.
await resultsBlock.Completion;
}

Why are these tasks not completing?

So I have a block of code like
EventLog.WriteEntry("About to run the task");
// run the dequeued task
var task = PageRetrievals.Dequeue();
PageRetrieval retrieval = new PageRetrieval();
var continuation = task.ContinueWith(t => retrieval = t.Result);
task.Wait();
EventLog.WriteEntry("html = " + retrieval.Html.DocumentNode.OuterHtml);
where the WriteEntrys are just my sanity-check that this is working. But the 2nd isn't getting called and I'm trying to figure out why my code isn't reaching that point.
The above block of code is inside a method that is
MyTimer.Elapsed += new ElapsedEventHandler(MethodThatInvokesTheAboveCode);
and the type of task is like
PageRetrievals.Enqueue(new Task<PageRetrieval>(() =>
new PageRetrieval()
{
Html = PageOpener.GetBoardPage(pagenum),
Page = PageType.Board,
Number = pagenum
}
));
where PageOpener.GetBoardPage simply gets the HTML from a URL, like
private static HtmlDocument GetDocumentFromUrl(string url)
{
var webget = new HtmlWeb();
var doc = webget.Load(url);
return webget.StatusCode == System.Net.HttpStatusCode.OK ? doc : null;
}
public static HtmlDocument GetBoardPage(int pageNumber)
{
return GetDocumentFromUrl(string.Format(BoardPageUrlFormat, pageNumber));
}
Is there anything about this that looks obviously wrong?

How to use HttpClient PostAsync() with threadpool in C#?

I'm using the following code to post an image to a server.
var image= Image.FromFile(#"C:\Image.jpg");
Task<string> upload = Upload(image);
upload.Wait();
public static async Task<string> Upload(Image image)
{
var uriBuilder = new UriBuilder
{
Host = "somewhere.net",
Path = "/path/",
Port = 443,
Scheme = "https",
Query = "process=false"
};
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("locale", "en_US");
client.DefaultRequestHeaders.Add("country", "US");
var content = ConvertToHttpContent(image);
content.Headers.ContentType = MediaTypeHeaderValue.Parse("image/jpeg");
using (var mpcontent = new MultipartFormDataContent("--myFakeDividerText--")
{
{content, "fakeImage", "myFakeImageName.jpg"}
}
)
{
using (
var message = await client.PostAsync(uriBuilder.Uri, mpcontent))
{
var input = await message.Content.ReadAsStringAsync();
return "nothing for now";
}
}
}
}
I'd like to modify this code to run multiple threads. I've used "ThreadPool.QueueUserWorkItem" before and started to modify the code to leverage it.
private void UseThreadPool()
{
int minWorker, minIOC;
ThreadPool.GetMinThreads(out minWorker, out minIOC);
ThreadPool.SetMinThreads(1, minIOC);
int maxWorker, maxIOC;
ThreadPool.GetMaxThreads(out maxWorker, out maxIOC);
ThreadPool.SetMinThreads(4, maxIOC);
var events = new List<ManualResetEvent>();
foreach (var image in ImageCollection)
{
var resetEvent = new ManualResetEvent(false);
ThreadPool.QueueUserWorkItem(
arg =>
{
var img = Image.FromFile(image.getPath());
Task<string> upload = Upload(img);
upload.Wait();
resetEvent.Set();
});
events.Add(resetEvent);
if (events.Count <= 0) continue;
foreach (ManualResetEvent e in events) e.WaitOne();
}
}
The problem is that only one thread executes at a time due to the call to "upload.Wait()". So I'm still executing each thread in sequence. It's not clear to me how I can use PostAsync with a thread-pool.
How can I post images to a server using multiple threads by tweaking the code above? Is HttpClient PostAsync the best way to do this?

I'd like to modify this code to run multiple threads.
Why? The thread pool should only be used for CPU-bound work (and I/O completions, of course).
You can do concurrency just fine with async:
var tasks = ImageCollection.Select(image =>
{
var img = Image.FromFile(image.getPath());
return Upload(img);
});
await Task.WhenAll(tasks);
Note that I removed your Wait. You should avoid using Wait or Result with async tasks; use await instead. Yes, this will cause async to grow through you code, and you should use async "all the way".

C# await lambda function

I'll start off by publishing the code that is troubled:
public async Task main()
{
Task t = func();
await t;
list.ItemsSource = jlist; //jlist previously defined
}
public async Task func()
{
TwitterService service = new TwitterService(_consumerKey, _consumerSecret);
service.AuthenticateWith(_accessToken, _accessTokenSecret);
TwitterGeoLocationSearch g = new TwitterGeoLocationSearch(40.758367, -73.982706, 25, 0);
SearchOptions s = new SearchOptions();
s.Geocode = g;
s.Q = "";
s.Count = 1;
service.Search(s, (statuses, response) => get_tweets(statuses, response));
void get_tweets(TwitterSearchResult statuses, TwitterResponse response)
{
//unimportant code
jlist.Add(info);
System.Diagnostics.Debug.WriteLine("done with get_tweets, jlist created");
}
I am having issues with the get_tweets(..) function running (on what I believe a different thread) and the Task t is not awaited like I have in the main function. Basically, my issue is that the list.Itemsource = jlist is ran before the get_tweets function is finished. Does anyone have a solution or the right direction to point me in?

First, create a TAP wrapper for TwitterService.Search, using TaskCompletionSource. So something like:
public static Task<Tuple<TwitterSearchResult, TwitterResponse>> SearchAsync(this TwitterService service, SearchOptions options)
{
var tcs = new TaskCompletionSource<Tuple<TwitterSearchResult, TwitterResponse>>();
service.Search(options, (status, response) => tcs.SetResult(Tuple.Create(status, response)));
return tcs.Task;
}
Then you can consume it using await:
SearchOptions s = new SearchOptions();
s.Geocode = g;
s.Q = "";
s.Count = 1;
var result = await service.SearchAsync(s);
get_tweets(result.Item1, result.Item2);

How to wait for an asynchronization method in silverlight

I have made two functions in a WCF service and call them in silverlight using asynchronization. I call one method after the other, but before completion of the first method, silverlight executes the second method. I want the first method to completely finish executing before the second method call.
thanks for your reply i m pasting my code please suggest in code how i implement.
private GDOperations.GDDoneOperationsClient _gdDoneOperation;
private ImageOperationsClient proxy = null;
foreach (var file in _filesCollection)
{
clsImageTransactionEntity _clsImageEntity = new clsImageTransactionEntity();
_clsImageEntity.ImageByte = GetFileData(file.OpenRead());
_clsImageEntity.ImageExtension = file.Extension;
_clsImageEntity.ImageName = file.Name;
_clsImageEntity.ImageType = 2;
_clsImageEntity.ImagePath = "~/CMSImages/FinalImages/" + lblSelectedBarcode.Content.ToString() + "/" + file.Name;
_clsImageEntity.JabongBarcode = lblSelectedBarcode.Content.ToString();
GDOperations.clsImageTransactionEntity _clsImageGDEntity = new GDOperations.clsImageTransactionEntity();
_clsImageGDEntity.ImageExtension = file.Extension;
_clsImageGDEntity.ImageName = file.Name;
_clsImageGDEntity.ImageType = 2;
_clsImageGDEntity.ImagePath = "~/CMSImages/FinalImages/" + lblSelectedBarcode.Content.ToString() + "/" + file.Name;
_clsImageGDEntity.JabongBarcode = lblSelectedBarcode.Content.ToString();
_clsImageGDEntity.RoleId = roleID;
_clsImageGDEntity.TaskID = taskID;
_clsImageGDEntity.UserID = UserId;
_clsImageGDEntity.SystemIP = systemIP;
_clsGdAllotment.clsImageTransactionEntity.Add(_clsImageGDEntity);
----- first method calling-----
proxy.UploadFinalImageCompleted += (s, e) =>
{
if (e.Error == null)
{
}
};
proxy.UploadFinalImageAsync(_clsImageEntity);
countfile = countfile + 1;
pbUploadFiles.Value = countfile;
}
_clsGdAllotment.GdID = int.Parse(lblUserID.Content.ToString());
_clsGdAllotment.JabongBarcode = lblSelectedBarcode.Content.ToString();
_clsGdAllotment.TaskID = taskID;
--- after for loop completion calling second method -----
_gdDoneOperation.InsertGDDoneInformationCompleted += _gdDoneOperation_InsertGDDoneInformationCompleted;
_gdDoneOperation.InsertGDDoneInformationAsync(_clsGdAllotment);`
Please help its urgent.

If you're using Task-Based Async Pattern:
var task1 = CallFirstAsyncMethod();
task1.Wait(); // waiting task to finish
var task2 = CallAnotherAsyncMethod();
// or subscribe to the task continuation to call second
// method when first operation will finished
task1.ContinueWith(t =>
{
// add error handling
var task2 = CallAnotherAsyncMethod();
});
If you're using Classical Async Pattern (a.k.a. APM):
IAsyncResult ar1 = CallFirstAsyncMethod();
ar1.WaitHandle.Wait();
IAsyncResult ar2 = CallSecondAsyncMethod();
// or use the same technique asynchronously
CallFirstAsyncMethod(ar => // Suppose we should provide appropriate callback
{
// Call to appropriate EndInvoke method
IAsyncResult ar2 = CallSecondAsyncMethod();
}, state);

You can call the second in the callback of the first, no ?
Or is Visual Studio < 2012
You can use AutoResetEvent :
MyServiceClient clientService;
AutoResetEvent autoResetEvent = new AutoResetEvent(false);
public void Execute()
{
InvokeCompletedEventArgs data = null;
clientService.InvokeCompleted += (e, f) =>
{
data = f;
autoResetEvent.Set();
};
m_proxy.MyCallAync();
autoResetEvent.WaitOne(); // Wait the set of autoResetEvent
m_proxy.MySecondCallAync(data); // Data return by the first call
}
Antoine