We Keep Coding

C# - HTTP Get run in parallel tasks randomly fails with: Error while copying content to stream

In my C# application (.NET Core 3.1) there's an automatic task that every X hours starts another task, which gets run in parallel multiple times with different parameters.
At the end of this automatic task, there's a call to await Task.WhenAll(tasksList). to wait for parallel tasks completion.
Every task issues an HTTPClient (using IHttpClientFactory factory method) and issues a GET request, with the following syntax:
var res = await client.GetAsync(url);
if (res.IsSuccessStatusCode)
{
var exit = await res.Content.ReadAsStringAsync();
[...omitted]
}
The issue occurs randomly when two tasks, that share the same GET URL, run at a distance of max 60-70ms. Sometimes both tasks fail, one after another, each with this same exception:
System.Net.Http.HttpRequestException: Error while copying content to a stream.
---> System.IO.IOException: The response ended prematurely.
at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
From the logs, I can see how two different HTTP Requests are correctly started and received by the server.
If I remove the ReadAsStringAsync part, the issue never occurs so I presume it's related to the content reading (after the status code check), almost as if the two tasks end sharing the Get result (while having two different active connections issued). I tried using a ReadAsStreamAsync but the issue still occurs (this helps to reduce the occurrence, although).
Another thing that could be related is that the result retrieved is quite heavy (the last time I downloaded it, it ended being a .json file of 4.5MB, more or less).
Should I run each task sequentially? Or am I issuing the HTTP Requests wrong?
IF you want to test this issue, here you can find the source code of a console app I'm using to reproduce the issue (if it doesn't occurs by the first 20calls, restart the app until it occurs):
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;
using System.Collections.Generic;
using System.IO;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApp1
{
class Program
{
static async Task<int> Main(string[] args)
{
var builder = new HostBuilder()
.ConfigureServices((hostContext, services) =>
{
services.AddHttpClient();
services.AddTransient<TaskRunner>();
}).UseConsoleLifetime();
var host = builder.Build();
using (var serviceScope = host.Services.CreateScope())
{
var services = serviceScope.ServiceProvider;
try
{
var x = services.GetRequiredService<TaskRunner>();
var result = await x.Run();
Console.WriteLine(result);
}
catch (Exception ex)
{
Console.WriteLine("Error Occured");
}
}
return 0;
}
public class TaskRunner
{
private static IHttpClientFactory _httpFactory { get; set; }
public TaskRunner(IHttpClientFactory httpFactory)
{
_httpFactory = httpFactory;
}
public async Task<string> Run()
{
Console.WriteLine("Starting loop...");
do
{
await Task.Delay(2500); // wait app loading
await SendRequest();
} while (true);
}
private static async Task SendRequest()
{
await Task.WhenAll(new Task[] { ExecuteCall(), ExecuteCall()};
}
private async static Task<bool> ExecuteCall()
{
try
{
var client = _httpFactory.CreateClient();
// fake heavy API call (> 5MB data)
var api = "https://api.npoint.io/5896085b486eed6483ce";
Console.WriteLine("Starting call at " + DateTime.Now.ToUniversalTime().ToString("o"));
var res = await client.GetAsync(api);
if (res.IsSuccessStatusCode)
{
var exit = await res.Content.ReadAsStringAsync();
/* STREAM read alternative
var ed = await res.Content.ReadAsStreamAsync();
StringBuilder result = new StringBuilder();
using var sr = new StreamReader(ed);
while (!sr.EndOfStream)
{
result.Append(await sr.ReadLineAsync());
}
var exit = result.ToString();
*/
Console.WriteLine(exit.Substring(0, 10));
//Console.WriteLine(exit);
Console.WriteLine("Ending call at " + DateTime.Now.ToUniversalTime().ToString("o"));
return true;
}
Console.WriteLine(res.StatusCode);
Console.WriteLine("Ending call at " + DateTime.Now.ToUniversalTime().ToString("o"));
return false;
}
catch (Exception ex)
{
// put breakpoint here
// Exception => called on line:78 but if content isn't read it never occurs
Console.WriteLine(ex.ToString());
return false;
}
}
}
}
}
Thanks for any help/suggestion you can give me!

I'm answering my question to leave the solution I applied, for anyone who can encounter the same issue :)
I added the following line before the Api Call:
var client = _httpFactory.CreateClient();
var api = "https://api.npoint.io/5896085b486eed6483ce";
>>> client.DefaultRequestVersion = HttpVersion.Version10; // new line
var res = await client.GetAsync(api);
The issue seems to be related to the endpoint server, that drops concurrent connections when the HttpVersion is 11. It's possible it relates to the Keep-Alive Connection header, since on 10 v. the header is set to Close.

Asynchronous async await method to generate pdf

I decided to convert the method that generates a PDF to asynchronous call. The asynchronous call doesn't produce the requested PDF and I have no idea why this is happening
The client who calls the async operation is as follows:
public QuotationResponse CreateQuotation(IQuotation quotation)
{
...
// Create Quotation PDF
_pdfWriter.GeneratePdfAsync(response);
return response;
}
The class responsible for generating the PDF is the following:
public class Writer
{
//... other class methods
public async Task GeneratePdfAsync(QuotationResponse response)
{
await new Task(() =>
{
var currentPath = System.Web.Hosting.HostingEnvironment.ApplicationPhysicalPath;
var formFile = Path.Combine(currentPath, Settings.Default.QUOTATION_TEMPLATE_PATH);
var newFile = Path.Combine(currentPath, Settings.Default.QUOTATION_PDF_PATH);
var reader = new PdfReader(formFile);
using (PdfStamper stamper = new PdfStamper(reader, new FileStream(newFile, FileMode.Create)))
{
....
// flatten form fields and close document
stamper.FormFlattening = true;
stamper.Close();
}
});
}
}
I suspect that I'm not doing something right with the async - await operation but don't know what. Could you please assist?

You should never use the Task constructor. Ever. It has absolutely no valid use cases. At all. For any kind of approach to any kind of problem. Full details on my blog.
Since there is no asynchronous work to do, you should just use synchronous methods:
public class Writer
{
//... other class methods
public void GeneratePdfAsync(QuotationResponse response)
{
var currentPath = System.Web.Hosting.HostingEnvironment.ApplicationPhysicalPath;
...
}
}
If you want to call this from a GUI application and don't want to block the UI thread, then you can call it on a background thread by using await Task.Run, as such:
QuotationResponse response = await Task.Run(() => CreateQuotation(quotation));

Reading/Writing Async Files for Universal App

im trying to Reading/Writing Async Files for an Universal App in c#.
When i write and read a file for first time, it works... But when i retry it immeadiatly, there are two Errors: 1. UnauthorizedAccess 2. Handle with the OPLOCK has been closed
It seems that the methods arent finished yet and so the data is not free
(in my frame is a button which adds a new member to a List, then the list shall serialized in an XML data. When i reNavigate to that page, that XML sheet shall be deserialized back to that List, because the Content shall be displayed)
List<Immobilie> immoListe = new List<Immobilie>();
private const string FileName_ImmoObjects = "ImmoObjects.xml";
StorageFolder sFolder = Windows.Storage.ApplicationData.Current.LocalFolder;
IStorageFile latestImmoListFile;
public Startmenue()
{
this.InitializeComponent();
immoListe.Add(new Immobilie()); // for testing creating an XML first
immoListe[0].adresse = "Foo1";
immoListe.Add(new Immobilie());
immoListe[1].adresse = "Foo2";
WriteImmoListAsync();
ReadImmoListAsync(); // These two steps working
WriteImmoListAsync(); // everything more causes error
ReadImmoListAsync();
}
public async void WriteImmoListAsync()
{
try
{
IStorageFolder folder = await sFolder.CreateFolderAsync("Saves", CreationCollisionOption.OpenIfExists);
latestImmoListFile = await folder.CreateFileAsync(FileName_ImmoObjects, CreationCollisionOption.ReplaceExisting);
using (IRandomAccessStream stream = await latestImmoListFile.OpenAsync(FileAccessMode.ReadWrite))
using (Stream outputStream = stream.AsStreamForWrite())
{
DataContractSerializer serializer = new DataContractSerializer(typeof(List<Immobilie>));
serializer.WriteObject(outputStream, immoListe);
}
}
catch (Exception e)
{
var d = new MessageDialog(e.ToString());
await d.ShowAsync();
}
}
public async void ReadImmoListAsync()
{
int i = 0;
try
{
IStorageFolder folder = await sFolder.GetFolderAsync("Saves");
i = 1;
latestImmoListFile = await folder.GetFileAsync(FileName_ImmoObjects);
i = 2;
using (IRandomAccessStream stream = await latestImmoListFile.OpenAsync(FileAccessMode.Read))
{
i = 3;
using (Stream inputStream = stream.AsStreamForRead())
{
i = 4;
DataContractSerializer deserializer = new DataContractSerializer(typeof(List<Immobilie>));
i = 5;
immoListe = (List<Immobilie>)deserializer.ReadObject(inputStream);
}
}
}
catch (Exception e)
{
var d = new MessageDialog("Fehler I = " + i + "\n" + e.ToString());
await d.ShowAsync();
}
}
So what can i do and why is it so difficult??(normal I/O is easy-peasy).-.

As I describe in my MSDN article on async best practices, you should avoid async void:
public async Task WriteImmoListAsync();
public async Task ReadImmoListAsync();
Once your methods are properly async Task, then you can await them:
await WriteImmoListAsync();
await ReadImmoListAsync();
await WriteImmoListAsync();
await ReadImmoListAsync();

You can't start the methods again until you wait for them to complete. What that above code is trying to do is to write to a file, but while that's processing, it tries to open the file and write to it while the first method call hasn't completed. You need to wait for those method calls to finish before running them again - using the await keyword would be helpful here

It might be that the process writing/reading the file are still attached to the file. You might want to take a look at this pattern for async file read/write from Microsoft:
https://msdn.microsoft.com/en-ca/library/mt674879.aspx
Also, note that if the read and write are done from differents process, you're going to have to use a mutex. Here's a great explanation on how it works:
What is a good pattern for using a Global Mutex in C#?

Using HttpClient for Asynchronous File Downloads

I have a service which returns a csv file to a POST request. I would like to download said file using asynchronous techniques. While I can get the file, my code has a couple of outstanding problems and questions:
1) Is this really asynchronous?
2) Is there a way to know the length of the content even though it is being sent in chunked format? Think progress bars).
3) How can I best monitor progress in order to hold off the program exit until all work is complete.
using System;
using System.IO;
using System.Net.Http;
namespace TestHttpClient2
{
class Program
{
/*
* Use Yahoo portal to access quotes for stocks - perform asynchronous operations.
*/
static string baseUrl = "http://real-chart.finance.yahoo.com/";
static string requestUrlFormat = "/table.csv?s={0}&d=0&e=9&f=2015&g=d&a=4&b=5&c=2000&ignore=.csv";
static void Main(string[] args)
{
while (true)
{
Console.Write("Enter a symbol to research or [ENTER] to exit: ");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
break;
DownloadDataForStockAsync(symbol);
}
}
static async void DownloadDataForStockAsync(string symbol)
{
try
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.Timeout = TimeSpan.FromMinutes(5);
string requestUrl = string.Format(requestUrlFormat, symbol);
//var content = new KeyValuePair<string, string>[] {
// };
//var formUrlEncodedContent = new FormUrlEncodedContent(content);
var request = new HttpRequestMessage(HttpMethod.Post, requestUrl);
var sendTask = client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);
var response = sendTask.Result.EnsureSuccessStatusCode();
var httpStream = await response.Content.ReadAsStreamAsync();
string OutputDirectory = "StockQuotes";
if (!Directory.Exists(OutputDirectory))
{
Directory.CreateDirectory(OutputDirectory);
}
DateTime currentDateTime = DateTime.Now;
var filePath = Path.Combine(OutputDirectory, string.Format("{1:D4}_{2:D2}_{3:D2}_{4:D2}_{5:D2}_{6:D2}_{7:D3}_{0}.csv",
symbol,
currentDateTime.Year, currentDateTime.Month, currentDateTime.Day,
currentDateTime.Hour, currentDateTime.Minute, currentDateTime.Second, currentDateTime.Millisecond
));
using (var fileStream = File.Create(filePath))
using (var reader = new StreamReader(httpStream))
{
httpStream.CopyTo(fileStream);
fileStream.Flush();
}
}
}
catch (Exception ex)
{
Console.WriteLine("Error, try again!");
}
}
}
}

"Is this really asynchronous?"
Yes, mostly. The DownloadDataForStockAsync() method will return before the operation is complete, at the await response.Content.ReadAsStreamAsync() statement.
The main exception is near the end of the method, where you call Stream.CopyTo(). This isn't asynchronous, and because it's a potentially lengthy operation could result in noticeable delays. However, in a console program you won't notice, because the continuation of the method is executed in the thread pool rather than the original calling thread.
If you intend to move this code to a GUI framework, such as Winforms or WPF, you should change the statement to read await httpStream.CopyToAsync(fileStream);
Is there a way to know the length of the content even though it is being sent in chunked format? Think progress bars).
Assuming the server includes the Content-Length in the headers (and it should), yes. This should be possible.
Note that if you were using HttpWebRequest, the response object would have a ContentLength property giving you this value directly. You are using HttpRequestMessage here instead, which I'm less familiar with. But as near as I can tell, you should be able to access the Content-Length value like this:
long? contentLength = response.Content.Headers.ContentLength;
if (contentLength != null)
{
// use value to initialize "determinate" progress indication
}
else
{
// no content-length provided; will need to display progress as "indeterminate"
}
How can I best monitor progress in order to hold off the program exit until all work is complete.
There are lots of ways. I will point out that any reasonable way will require that you change the DownloadDataForStockAsync() method so that it returns Task and not void. Otherwise, you don't have access to the task that's created. You should do this anyway though, so that's not a big deal. :)
The simplest would be to just keep a list of all the tasks you start, and then wait on them before exiting:
static void Main(string[] args)
{
List<Task> tasks = new List<Task>();
while (true)
{
Console.Write("Enter a symbol to research or [ENTER] to exit: ");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
break;
tasks.Add(DownloadDataForStockAsync(symbol));
}
Task.WaitAll(tasks);
}
Of course, this requires that you explicitly maintain a list of each Task object, including those which have already completed. If you intend for this to run for a long time and process a very large number of symbols, that might be prohibitive. In that case, you might prefer to use the CountDownEvent object:
static void Main(string[] args)
{
CountDownEvent countDown = new CountDownEvent();
while (true)
{
Console.Write("Enter a symbol to research or [ENTER] to exit: ");
string symbol = Console.ReadLine();
if (string.IsNullOrEmpty(symbol))
break;
countDown.AddCount();
DownloadDataForStockAsync(symbol).ContinueWith(task => countdown.Signal()) ;
}
countDown.Wait();
}
This simply increments the CountDownEvent counter for each task you create, and attaches a continuation to each task to decrement the counter. When the counter reaches zero, the event is set, allowing a call to Wait() to return.

Read a file from background task

I'm trying to call a method from inside the Run method of a background task which among other it desirializes a xml file. The problem is that I end up in a deadlock. This is the methos that reads the file
protected async Task<Anniversaries> readFile(string fileName)
{
IStorageFile file;
Anniversaries tempAnniversaries;
file = await ApplicationData.Current.LocalFolder.GetFileAsync(fileName);
using (IRandomAccessStream stream =
await file.OpenAsync(FileAccessMode.Read))
using (Stream inputStream = stream.AsStreamForRead())
{
DataContractSerializer serializer = new DataContractSerializer(typeof(Anniversaries));
tempAnniversaries = serializer.ReadObject(inputStream) as Anniversaries;
}
return tempAnniversaries;
}
and here is the Run method
public sealed class TileUpdater : IBackgroundTask
{
GeneralAnniversariesManager generalManager = new GeneralAnniversariesManager();
Anniversaries tempAnn = new Anniversaries();
string test = "skata";
public async void Run(IBackgroundTaskInstance taskInstance)
{
DateTime curentTime = new DateTime();
var defferal = taskInstance.GetDeferral();
await generalManager.InitializeAnniversariesAsync().AsAsyncAction();
curentTime = DateTime.Now;
var updater = TileUpdateManager.CreateTileUpdaterForApplication();
updater.EnableNotificationQueue(true);
updater.Clear();
for (int i = 1; i < 6; i++)
{
var tile = TileUpdateManager.GetTemplateContent(TileTemplateType.TileWide310x150BlockAndText01);
tile.GetElementsByTagName("text")[0].InnerText = test + i;
tile.GetElementsByTagName("text")[1].InnerText = curentTime.ToString();
updater.Update(new TileNotification(tile));
}
defferal.Complete();
}

I'm assuming that by deadlock you mean that the deserialization method finishes too late and your original program tries to read the data before it's finished loading.
It depends on how complicated/reliable you want your solution to be and how you're intending to use the program. The simplest way relies on the fact that the directory creation function is always 100% atomic in Windows/Unix and OSX. For example at the top of your readFile function have something like this.
Directory.CreateDirectory("lock");
Before you start parsing the results of your async action in TileUpdater, have a loop that looks like this.
while (Directory.Exists("lock"))
{
Thread.Sleep(50);
}
This assumes that everything is happening in the same directory, generally you'll want to replace "lock" with a path that leads to the user's temp directory for their version of Windows/Linux/OSX.
If you want to implement something more complicated where you're reading from a series of files while at the same time reading the deserialized output into your class, you'll want to use something like a System.Collections.Concurrent.ConcurrentQueue that allows your threads to act completely independently without blocking each other.
Incidentally I'm assuming that you know that the class Process and the function .waitfor() exists. You can spin off a thread and then at a later point, halt the main thread until the spawned thread finishes.

Actually I think I've found where the problem is. At the namespaces, I've tried a try and catch and I got an exception about using different namespaces at the datacontract serealizer. I have updated the code like this
file = await ApplicationData.Current.LocalFolder.GetFileAsync("EortologioMovingEntries.xml");
try
{
using (IRandomAccessStream stream =
await file.OpenAsync(FileAccessMode.Read))
using (Stream inputStream = stream.AsStreamForRead())
{
DataContractSerializer serializer = new DataContractSerializer(typeof(Anniversaries), "Anniversaries", "http://schemas.datacontract.org/2004/07/Eortologio.Model");
tempAnniversaries = serializer.ReadObject(inputStream) as Anniversaries;
}
}
catch (Exception ex)
{
error = ex.ToString();
tempAnniversaries.Entries.Add(new AnniversaryEntry("Ena", DateTime.Now, "skata", PriorityEnum.High));
}
I don't get any exceptions now but the tempAnniversaries returns null. Any ideas?