We Keep Coding

Using Parallel ForEach to POST in API and single Handle each Response

I would like to know if the code I produced is good practice and does not produce leaks, I have more than 7000 objects Participant which I will push individually and Handle the Response to save the "external" id in our database. First I use the Parallel ForEach on the list pPartcipant:
Parallel.ForEach(pParticipant, participant =>
{
try
{
//Create
if (participant.id == null)
{
ExecuteRequestCreate(res, participant);
}
else
{//Update
ExecuteRequestUpdate(res, participant);
}
}
catch (Exception ex)
{
LogHelper.Log("Fail Parallel ", ex);
}
});
Then I do a classic (not async request), but after I need to "handle" the response (print in the console, save in a text file in async mode, and Update in my database)
private async void ExecuteRequestCreate(Uri pRes, ParticipantDo pParticipant)
{
try
{
var request = SetRequest(pParticipant);
//lTaskAll.Add(Task.Run(() => { ExecuteAll(request, pRes, pParticipant); }));
//Task.Run(() => ExecuteAll(request, pRes, pParticipant));
var result = RestExecute(request, pRes);
await HandleResult(result, pParticipant);
//lTaskHandle.Add(Task.Run(() => { HandleResult(result, pParticipant); }));
}
catch (Exception e)
{
lTaskLog.Add(LogHelper.Log(e.Message + " " + e.InnerException));
}
}
Should I run a new task for handeling the result (as commented) ? Will it improve the performance ?
In comment you can see that I created a list of tasks so I can wait all at the end (tasklog is all my task to write in a textfile) :
int nbtask = lTaskHandle.Count;
try
{
Task.WhenAll(lTaskHandle).Wait();
Task.WhenAll(lTaskLog).Wait();
}
catch (Exception ex)
{
LogHelper.Log("Fail on API calls tasks", ex);
}
I don't have any interface it is a console program.

I would like to know if the code I produced is good practice
No; you should avoid async void and also avoid Parallel for async work.
Here's a similar top-level method that uses asynchronous concurrency instead of Parallel:
var tasks = pParticipant
.Select(participant =>
{
try
{
//Create
if (participant.id == null)
{
await ExecuteRequestCreateAsync(res, participant);
}
else
{//Update
await ExecuteRequestUpdateAsync(res, participant);
}
}
catch (Exception ex)
{
LogHelper.Log("Fail Parallel ", ex);
}
})
.ToList();
await Task.WhenAll(tasks);
And your work methods should be async Task instead of async void:
private async Task ExecuteRequestCreateAsync(Uri pRes, ParticipantDo pParticipant)
{
try
{
var request = SetRequest(pParticipant);
var result = await RestExecuteAsync(request, pRes);
await HandleResult(result, pParticipant);
}
catch (Exception e)
{
LogHelper.Log(e.Message + " " + e.InnerException);
}
}

using ping in infinite loop

I want to check the instrument connectivity, continuously. These instruments are connected by LAN.
I ping them in an infinite loop. After a few seconds, I'm met with a blue screen and the system restarts.
private async void checkingDevice()
{
await Task.Run(() =>
{
do
{
Panel selectedPanel;
multicastPing.send();
foreach (var device in (Device[])Enum.GetValues(typeof(Device)))
{
selectedPanel = (Panel)this.Controls.Find((device).ToString(), true)[0];
if (multicastPing.check(device.ToString()))
selectedPanel.BackgroundImage = Image.FromFile(Configs.imagesUrl + "enable\\" + selectedPanel.AccessibleName + ".png");
else
selectedPanel.BackgroundImage = Image.FromFile(Configs.imagesUrl + "disable\\" + selectedPanel.AccessibleName + ".png");
}
} while (!flag);
// TODO
// delete instrument object after using in this snippet code
});
}
public bool send()
{
foreach (var device in (Device[])Enum.GetValues(typeof(Device)))
replyDictionary[device.ToString()] = sendDictionary[device.ToString()].Send(Configs.instrumentSpecification[device.ToString()].ip).Status;
return true;
}

My career was in Test and Measurement and doing this kind of thing seems quite familiar. So I'm offering this as one approach to see if I can assist you in pinging your instrument stack.
Getting a blue screen is pretty rare and just from personal experience it's likely that a very low-level IO kernel driver (ethernet port?) is having a bad time of something. Potentially, it appears that the loop could run very fast to the point of overrunning a buffer perhaps? Or, pathologically, you may inadvertently be binding the UI thread to that of a kernel driver one. Ouch.
I agree with comments about safely calling the UI thread from a task using MethodInvoker. But I think perhaps the main issue might be coming from the fact that this loop "runs as fast as it can" which might be pretty fast. I modeled something like this that throttles the pings so that they occur some limited number of times per second and it seems to work fine.
A safely-threaded task that performs the look without ever blocking on the UI thread looks something like this:
CancellationTokenSource _cts = null;
SemaphoreSlim ssBusy = new SemaphoreSlim(1);
private void ExecMulticastPing()
{
ssBusy.Wait();
Task.Run(() =>
{
try
{
_cts = new CancellationTokenSource();
do
{
List<Task<PingReply>> asyncPings = new List<Task<PingReply>>();
// Sends out the pings in rapid succession to execute asynchronously in parallel
foreach (var device in (Device[])Enum.GetValues(typeof(Device)))
{
asyncPings.Add(Task.Run(() => SinglePingAsync(device, _cts.Token)));
}
// Waits for all the async pings to complete.
Task.WaitAll(asyncPings.ToArray());
// See if flag is already cancelled
if (_cts.IsCancellationRequested) break;
foreach (var device in (Device[])Enum.GetValues(typeof(Device)))
{
SetPanelImage(device, asyncPings[(int)device].Result);
}
// I believe that it's very important to throttle this to
// a reasonable number of repeats per second.
Task.Delay(1000).Wait();
BeginInvoke((MethodInvoker)delegate
{
WriteLine(); // Newline
});
} while (!_cts.IsCancellationRequested); // Check if it's cancelled now
}
finally
{
ssBusy.Release();
}
BeginInvoke((MethodInvoker)delegate
{
WriteLine("CANCELLED");
});
});
}
... where ...
const string URL_FOR_TEST = #"www.ivsoftware.com";
private PingReply SinglePingAsync(Device device, CancellationToken token)
{
if(token.IsCancellationRequested)
{
return null;
}
Ping pingSender = new Ping();
PingOptions options = new PingOptions()
{
DontFragment = true
};
PingReply reply = pingSender.Send(URL_FOR_TEST);
BeginInvoke((MethodInvoker)delegate
{
if (reply.Status == IPStatus.Success)
{
WriteLine("Address: " + reply.Address.ToString());
WriteLine("RoundTrip time: " + reply.RoundtripTime);
WriteLine("Time to live: " + reply.Options.Ttl);
WriteLine("Don't fragment: " + reply.Options.DontFragment);
WriteLine("Buffer size: " + reply.Buffer.Length);
WriteLine();
}
else
{
WriteLine("REQUEST TIMEOUT");
}
});
return reply;
}
... and ...
private void SetPanelImage(Device device, PingReply reply)
{
BeginInvoke((MethodInvoker)delegate
{
WriteLine("Setting panel image for " + device.ToString() + " " + reply.Status.ToString() );
Panel selectedPanel = (
from Control unk in Controls
where
(unk is Panel) &&
(unk.Name == device.ToString()) // ... or however you go about finding the panel...
select unk as Panel
).FirstOrDefault();
if (selectedPanel != null)
{
switch (reply.Status)
{
case IPStatus.Success:
// Set image for enabled
break;
case IPStatus.TimedOut:
// Set image as disabled
break;
default:
// Set image as disabled
break;
}
}
});
}
This 10-second screen capture demonstrates the success of this approach and you can browse the complete example code on our GitHub repo if you think this would be helpful. I also answered a different-but-related question here.

C# TaskCanceledException and my CPU shoots to 100% (process is slowed down incredible)

I have this code to download data from multiple websites.
I need to run this code for about 50.000 times.
However after running the code for 2 minutes (about 4000 times) I get the TaskCanceledException and my CPU goes to 100% and my process slows down incredibly.
This is the code:
public async Task<string[]> GetDataAsync(string address, string postalCode)
{
var path = $"{address} {postalCode}/"; // build proper path for request
var textFrom1 = "";
string textFrom3 = "";
string textFrom2 = "";
while (true)
{
try
{
textFrom1 = await client.GetStringAsync("http://website1.com/" + path);
break;
}
catch (Exception e) //404
{
await Task.Delay(10000); // try every 10 seconds (so I do not bomb the server with requests).
}
}
if (textFrom1.Split(':', ',')[1] == "0")
{
return new string[] { "null", "null", "null", "null", "null" };
}
while (true)
{
try
{
textFrom2 = await client.GetStringAsync("http://website2.com/" + textFrom1.Split('"', '"')[11]);
break;
}
catch (Exception e)
{
await Task.Delay(10000);
}
}
while (true)
{
try
{
textFrom3 = await client.GetStringAsync("http://website3.com/" + textFrom2.Split('"', '"')[3]);
break;
}
catch (Exception e)
{
await Task.Delay(10000);
}
}
var allData = await Task.Run(() => JsonConvert.DeserializeObject<List<RootObject>>(textFrom3));
var item = allData.First();
return new string[] { item.item1, item.item2, item.item3, item.item4, item.item5 };
}
And this is the way I create my tasks:
public string[][] GetAll(
IEnumerable<string> adresses,
IEnumerable<string> postalCodes)
{
// Start all tasks one by one without waiting for responses
var tasks = adresses.Zip(postalCodes, (addr, code) => { return GetDataAsync(addr, code); });
return Task.WhenAll(tasks).Result;
}
Is there any way I can optimize my code, so that I do not have this exception and therefor not slowing down my process?
I hope someone can help me, thank you!

I think you are reaching the maximum number of tasks windows can handle at once. You could split your 50.000 requests to smaller groups or write some code to handle your running async-tasks to make sure that you don't start too much tasks at once.
Furthermore you probably reach the maximum number of ports. Everytime starting a new connection your application will open a new dynamic port.

Program throwing OperationCanceledException despite catching it

I'm trying to use Tokens to cancel Task started by Task.Run. I took pattern from microsoft site: https://msdn.microsoft.com/pl-pl/library/hh160373(v=vs.110).aspx
This is my code:
public static class Sender
{
public static async Task sendData(NetworkController nc) {
await Task.Run(() => {
IPEndPoint endPoint = new IPEndPoint(nc.serverIp, nc.dataPort);
byte[] end = Encoding.ASCII.GetBytes("end");
while (true) {
if (Painting.pointsQueue.Count > 0 && !nc.paintingSenderToken.IsCancellationRequested) {
byte[] sendbuf = Encoding.ASCII.GetBytes(Painting.color.ToString());
nc.socket.SendTo(sendbuf, endPoint);
do {
sendbuf = Painting.pointsQueue.Take();
nc.socket.SendTo(sendbuf, endPoint);
} while (sendbuf != end && !nc.paintingSenderToken.IsCancellationRequested);
}
else if (nc.paintingSenderToken.IsCancellationRequested) {
nc.paintingSenderToken.ThrowIfCancellationRequested();
return;
}
}
}, nc.paintingSenderToken);
}
}
And here I start this task:
public void stopController() {
try {
paintingSenderTokenSource.Cancel();
senderTask.Wait();
} catch(AggregateException e) {
string message = "";
foreach (var ie in e.InnerExceptions)
message += ie.GetType().Name + ": " + ie.Message + "\n";
MessageBox.Show(message, "Przerwano wysylanie");
}
finally {
paintingSenderTokenSource.Dispose();
byte[] message = Encoding.ASCII.GetBytes("disconnect");
IPEndPoint endPoint = new IPEndPoint(serverIp, serverPort);
socket.SendTo(message, endPoint);
socket.Close();
mw.setStatus("disconnected");
}
}
public async void initialize() {
Task t = Reciver.waitForRespond(this);
sendMessage("connect");
mw.setStatus("connecting");
if (await Task.WhenAny(t, Task.Delay(5000)) == t) {
mw.setStatus("connected");
Painting.pointsQueue = new System.Collections.Concurrent.BlockingCollection<byte[]>();
senderTask = Sender.sendData(this);
}
else {
mw.setStatus("failed");
}
}
}
In initialize() method I'm waiting for the response from the server and if I get it I start new thread in this sendData() method. It is in static class to make code cleaner. If I want to stop this thread I call stopController() method. In microsoft site we can read:
The CancellationToken.ThrowIfCancellationRequested method throws an OperationCanceledException exception that is handled in a catch block when the calling thread calls the Task.Wait method.
But my program breaks onnc.paintingSenderToken.ThrowIfCancellationRequested(); which is in 'sendData()' method and the error says that OperationCanceledException was not handled. I started program from microsoft site and it works perfectly. I think I'm doing everything like they did but unfortunately it doesnt't work like it should.

You may have "Enable Just My Code" enabled.
To find the settings go to:
Tools => Options => Debugging => General => Enable Just My Code
If this checkbox is checked could you un-check it and then run your application again.

.Net DownloadFileTaskAsync robust WPF code

The WPF code below hangs forever when network connection is lost for 3 or more minutes. When connection is restored it neither throws nor continues downloading nor timeouts. If network connection is lost for a shorter period say half a minute, it throws after connection is restored. How can i make it more robust to survive network outage?
using System;
using System.Net;
using System.Net.NetworkInformation;
using System.Windows;
namespace WebClientAsync
{
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
NetworkChange.NetworkAvailabilityChanged +=
(sender, e) => Dispatcher.Invoke(delegate()
{
this.Title = "Network is " + (e.IsAvailable ? " available" : "down");
});
}
const string SRC = "http://ovh.net/files/10Mio.dat";
const string TARGET = #"d:\stuff\10Mio.dat";
private async void btnDownload_Click(object sender, RoutedEventArgs e)
{
btnDownload.IsEnabled = false;
btnDownload.Content = "Downloading " + SRC;
try {
using (var wcl = new WebClient())
{
wcl.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
await wcl.DownloadFileTaskAsync(new Uri(SRC), TARGET);
btnDownload.Content = "Downloaded";
}
}
catch (Exception ex)
{
btnDownload.Content = ex.Message + Environment.NewLine
+ ((ex.InnerException != null) ? ex.InnerException.Message : String.Empty);
}
btnDownload.IsEnabled = true;
}
}
}
UPDATE
Current solution is based on restarting Timer in DownloadProgressChangedEventHandler, so the timer fires only if no DownloadProgressChanged events occur within the timeout. Looks like an ugly hack, still looking for a better solution.
using System;
using System.Net;
using System.Threading;
using System.Threading.Tasks;
using System.Windows;
namespace WebClientAsync
{
public partial class MainWindow : Window
{
const string SRC = "http://ovh.net/files/10Mio.dat";
const string TARGET = #"d:\stuff\10Mio.dat";
// Time needed to restore network connection
const int TIMEOUT = 30 * 1000;
public MainWindow()
{
InitializeComponent();
}
private async void btnDownload_Click(object sender, RoutedEventArgs e)
{
btnDownload.IsEnabled = false;
btnDownload.Content = "Downloading " + SRC;
CancellationTokenSource cts = new CancellationTokenSource();
CancellationToken token = cts.Token;
Timer timer = new Timer((o) =>
{
// Force async cancellation
cts.Cancel();
}
, null //state
, TIMEOUT
, Timeout.Infinite // once
);
DownloadProgressChangedEventHandler handler = (sa, ea) =>
{
// Restart timer
if (ea.BytesReceived < ea.TotalBytesToReceive && timer != null)
{
timer.Change(TIMEOUT, Timeout.Infinite);
}
};
btnDownload.Content = await DownloadFileTA(token, handler);
// Note ProgressCallback will fire once again after awaited.
timer.Dispose();
btnDownload.IsEnabled = true;
}
private async Task<string> DownloadFileTA(CancellationToken token, DownloadProgressChangedEventHandler handler)
{
string res = null;
WebClient wcl = new WebClient();
wcl.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
wcl.DownloadProgressChanged += handler;
try
{
using (token.Register(() => wcl.CancelAsync()))
{
await wcl.DownloadFileTaskAsync(new Uri(SRC), TARGET);
}
res = "Downloaded";
}
catch (Exception ex)
{
res = ex.Message + Environment.NewLine
+ ((ex.InnerException != null) ? ex.InnerException.Message : String.Empty);
}
wcl.Dispose();
return res;
}
}
}

You need to implement proper timeout for that download. But you don't need to use timer, just use Task.Delay and Task.WaitAny. For example:
static async Task DownloadFile(string url, string output, TimeSpan timeout) {
using (var wcl = new WebClient())
{
wcl.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
var download = wcl.DownloadFileTaskAsync(url, output);
// await two tasks - download and delay, whichever completes first
await Task.WhenAny(Task.Delay(timeout), download);
var exception = download.Exception; // need to observe exception, if any
bool cancelled = !download.IsCompleted && exception == null;
// download is not completed yet, nor it is failed - cancel
if (cancelled) {
wcl.CancelAsync();
}
if (cancelled || exception != null) {
// delete partially downloaded file if any (note - need to do with retry, might not work with a first try, because CancelAsync is not immediate)
int fails = 0;
while (true) {
try {
File.Delete(output);
break;
}
catch {
fails++;
if (fails >= 10)
break;
await Task.Delay(1000);
}
}
}
if (exception != null) {
throw new Exception("Failed to download file", exception);
}
if (cancelled) {
throw new Exception($"Failed to download file (timeout reached: {timeout})");
}
}
}
Usage:
const string SRC = "http://ovh.net/files/10Mio.dat";
const string TARGET = #"d:\stuff\10Mio.dat";
// Time needed to restore network connection
TimeSpam TIMEOUT = TimeSpan.FromSeconds(30);
DownloadFile(SRC,TARGET, TIMEOUT); // might want to await this to handle exceptions
Update in response to comment. If you want timeout based on received data, not on whole operation time, it's also possible with Task.Delay. For example:
static async Task DownloadFile(string url, string output, TimeSpan timeout)
{
using (var wcl = new WebClient())
{
wcl.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
DateTime? lastReceived = null;
wcl.DownloadProgressChanged += (o, e) =>
{
lastReceived = DateTime.Now;
};
var download = wcl.DownloadFileTaskAsync(url, output);
// await two tasks - download and delay, whichever completes first
// do that until download fails, completes, or timeout expires
while (lastReceived == null || DateTime.Now - lastReceived < timeout) {
await Task.WhenAny(Task.Delay(1000), download); // you can replace 1 second with more reasonable value
if (download.IsCompleted || download.IsCanceled || download.Exception != null)
break;
}
var exception = download.Exception; // need to observe exception, if any
bool cancelled = !download.IsCompleted && exception == null;
// download is not completed yet, nor it is failed - cancel
if (cancelled)
{
wcl.CancelAsync();
}
if (cancelled || exception != null)
{
// delete partially downloaded file if any (note - need to do with retry, might not work with a first try, because CancelAsync is not immediate)
int fails = 0;
while (true)
{
try
{
File.Delete(output);
break;
}
catch
{
fails++;
if (fails >= 10)
break;
await Task.Delay(1000);
}
}
}
if (exception != null)
{
throw new Exception("Failed to download file", exception);
}
if (cancelled)
{
throw new Exception($"Failed to download file (timeout reached: {timeout})");
}
}
}

Personally, if I were to make a robust download solution, I would add a Network connection monitor because that's what we are actually waiting for. For simplicity, something like this will be enough.
online = true;
NetworkChange.NetworkAvailabilityChanged += NetworkChange_NetworkAvailabilityChanged;
_isNetworkOnline = NetworkInterface.GetIsNetworkAvailable();
void NetworkChange_NetworkAvailabilityChanged(object sender, NetworkAvailabilityEventArgs e)
{
online = e.IsAvailable;
}
Then you can actually check for network availability and wait as appropriate before you attempt to download or progress... I will definitely accept that a simple ping solution seems to work better than this at times based on experience.
Depending on the size of what you're downloading, monitoring the network speed may also help so you can decide how to chunk in case of choppy connections. Take a look at this project for ideas.