We Keep Coding

Does these HttpClient call be called in parallel?

I've a .NET Core web app that, once a web method (i.e. Test()) is invoked, call another remote api.
Basically, I do it this way (here's a POST example, called within a web method Test()):
public T PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
T returnValue = default(T);
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
HttpResponseMessage res = null;
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var task = Task.Run(() => client.PostAsync($"{baseAddress}/{url}", body));
task.Wait();
res = task.Result;
succeded = res.IsSuccessStatusCode;
if (succeded)
{
returnValue = JsonConvert.DeserializeObject<T>(res.Content.ReadAsStringAsync().Result);
}
else
{
Log($"PostRead failed, error: {JsonConvert.SerializeObject(res)}");
}
}
}
catch (Exception ex)
{
Log($"PostRead error: {baseAddress}/{url} entity: {JsonConvert.SerializeObject(entity)}");
}
return returnValue;
}
Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
Because if they will be called in serial, the last one will take the time of the previous ones, and that's not what I want.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Question is: if Test() in my Web App is called 10 times, in parallel, do the POST requests to the remote server are be called in parallel or in serial?
They will be called in parallel. Nothing in the code would make it in a blocking way.
In fact, when I have huge list of requests, I often receive the message A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
There are at least 2 things that need to be fixed here:
Properly using async/await
Properly using HttpClient
The first one is easy:
public async Task<T> PostRead<T>(string baseAddress, string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
using (HttpClient client = new HttpClient())
{
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var responseMessage = await client.PostAsync($"{baseAddress}/{url}", body);
var responseContent = await responseMessage.Content.ReadAsStringAsync();
if (responseMessage.IsSuccessStatusCode)
{
succeeded = true;
return JsonConvert.DeserializeObject<T>();
}
else
{
// log...
}
}
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}
The second one is a bit more tricky and can quite possibly be the cause of the problems you're seeing. Generally, unless you have some very specific scenario, new HttpClient() is plain wrong. It wastes resources, hides dependencies and prevents mocking. The correct way to do this is by using IHttpClientFactory, which can be very easily abstracted by using the AddHttpClient extension methods when registering the service.
Basically, this would look like this:
services.AddHttpClient<YourClassName>(x =>
{
x.BaseAddress = new Uri(baseAddress);
x.DefaultRequestHeaders.Add("Authorization", MyApiKey);
}
Then it's a matter of getting rid of all using HttpClient in the class and just:
private readonly HttpClient _client;
public MyClassName(HttpClient client) { _client = client; }
public async Task<T> PostRead<T>(string url, out bool succeded, object entity = null)
{
succeded = false;
try
{
string json = JsonConvert.SerializeObject(entity);
var responseMessage = await _client.PostAsync($"{baseAddress}/{url}", body);
//...
}
catch (Exception ex)
{
// log...
}
return default; // or default(T) depending on the c# version
}

[HttpClient call using parallel and solve JSON Parse error]
You can use task. when for parallel call . but when you use N number of calls and try to parse as JSON this may throw an error because result concatenation change the Json format so we can use Jarray. merge for combining the result .
Code
public async Task < string > (string baseAddress, string url, out bool succeded, object entity = null) {
using(HttpClient client = new HttpClient()) {
string responseContent = string.Empty;
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Add("Authorization", MyApiKey);
string json = JsonConvert.SerializeObject(entity);
var body = new StringContent(json, UnicodeEncoding.UTF8, "application/json");
var tasks = new List < Task < string >> ();
var jArrayResponse = new JArray();
int count = 10; // number of call required
for (int i = 0; i <= count; i++) {
async Task < string > RemoteCall() {
var response = await client.PostAsync($"{baseAddress}/{url}", body);
return await response.Content.ReadAsStringAsync();
}
tasks.Add(RemoteCall());
}
await Task.WhenAll(tasks);
foreach(var task in tasks) {
string postResponse = await task;
if (postResponse != "[]") {
var userJArray = JArray.Parse(postResponse);
jArrayResponse.Merge(userJArray);
}
}
responseContent = jArrayResponse.ToString();
return responseContent;
}
}

Stream URL to Android.Graphics.Bitmap using BitmapFactory.DecodeStreamAsync on GZIP file

I need to decode a url to a Bitmap. I am using the following code to attempt to decode a stream.
async Task <Android.Graphics.Bitmap> GetImageFromUrl(string url) {
using(var client = new HttpClient()) {
var response = await client.GetAsync(url);
if (response != null && response.StatusCode == HttpStatusCode.OK) {
using(var stream = await response.Content.ReadAsStreamAsync()) {
return await BitmapFactory.DecodeStreamAsync(stream);
}
}
return null;
}
}
The url being supplied is a blank black image: https://assets.radiowave.io/prod/StationAssets/1/image-md.jpg
Why does the BitmapFactory.DecodeStreamAsync always return null?

You are trying to pass a GZIP'd stream to DecodeStream and thus it fails to determine the type image you are passing in. Add a HttpClientHandler to your HttpClient with automatic decompression enabled.
FYI: If this method is being called multiple times, you should treat HttpClient as a singleton and only createi t once.
async Task<Android.Graphics.Bitmap> GetBitmapFromUrlAsync(string url)
{
var handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip
};
using (var client = new HttpClient(handler))
{
var response = await client.GetAsync(url);
if (response != null && response.StatusCode == HttpStatusCode.OK)
{
using (var stream = await response.Content.ReadAsStreamAsync())
{
return await BitmapFactory.DecodeStreamAsync(stream);
}
}
return null;
}
}

I usually prefer to use the available Java API's since there are tons of examples that do work so the code should look something like this:
public async Task<Bitmap> GetBitmapFromUrlAsync(String src)
{
try
{
URL url = new URL(src);
HttpURLConnection connection = (HttpURLConnection)url.OpenConnection();
connection.DoInput = (true);
await connection.ConnectAsync();
Stream input = connection.InputStream;
Bitmap myBitmap = await BitmapFactory.DecodeStreamAsync(input);
return myBitmap;
}
catch (IOException e)
{
// Log exception
return null;
}
}
And you will need the following using statements:
using Java.Net;
using System;
using System.IO;
using System.Threading.Tasks;
using Android.Graphics;
Also since you are making an API call make sure you have the try-catch ready.

Null message on hTTPResponse.Content.ReadAsStringAsync().Result after 1st foreach loop

I have issue with null result messages when calling a HttpClient getAsync within a foreach loop.
I need to iterate with a list of objects for values to call an API via HttpClient. After the first loop, the HttpResponseMessage's result() comes up with null message.
I tried CacheControl.NoCache = true. It's not working.
public async Task<List<Common.ProgressResponse>> RunAsync()
{
List<Response> ListOfResponses = new List<Responses>();
try
{
_client.BaseAddress = new Uri([URL]);
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Clear();
_client.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue() { NoCache = true };
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
_client.DefaultRequestHeaders.Add([KEY], [VALUE]);
ListOfResponses = await SomeFunction();
}
catch (Exception ex)
{
//Exceptions here...
}
return ListOfResponses;
}
private async Task<List<ListOfResponses>> SomeFunction()
{
List<Response> responses = new List<Response>();
string aPISuffix = string.Format("{0}/{1}", [APISUFFIX1], [APISUFFIX2]);
foreach (Object obj in ListOfObject)
{
_client.DefaultRequestHeaders.Add("Param1", obj.Param1);
if (!string.IsNullOrEmpty(obj.Param2))
_client.DefaultRequestHeaders.Add("Param2", obj.Param2);
Response response = new Response();
/*This is where the issue is .begin.*/
HttpResponseMessage hTTPResponse = await _client.GetAsync(aPISuffix).ConfigureAwait(false);
string result = hTTPResponse.Content.ReadAsStringAsync().Result;
/*This is where the issue is .end.*/
if (hTTPResponse.IsSuccessStatusCode)
{
response = [Code here...]
//Codes here...
}
responses.Add(response);
}
return responses;
}
On: 'string result = hTTPResponse.Content.ReadAsStringAsync().Result;'
I would expect the result message would have values as it loops through ListOfObjects.

Update: Per comment
First off, I generally try to avoid using .Result with tasks. I also don't think adding the cache headers is the issue since you are making a server to server call.
One thing I tend to always do is evaluate the response before I try to read it. I also prefer to use the HttpRequestMessage so I can manage the disposal of it.
Here is a quick example demonstrating how call out:
using System;
using System.Net.Http;
using System.Threading.Tasks;
namespace Testing
{
class Program
{
static void Main(string[] args)
{
TestAsync().Wait();
}
static async Task TestAsync()
{
var urls = new string[]
{
"https://stackoverflow.com/questions/57084989/null-message-on-httpresponse-content-readasstringasync-result-after-1st-foreac",
"https://stackoverflow.com/users/2025711/rogala"
};
using (var httpClient = new HttpClient())
{
foreach (var url in urls)
{
using (var request = new HttpRequestMessage(HttpMethod.Get, url))
{
Console.WriteLine($"Request: {url}".PadLeft(5,'*').PadRight(5, '*'));
var response = await httpClient.SendAsync(request)
.ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
var body = await response.Content.ReadAsStringAsync()
.ConfigureAwait(false);
Console.WriteLine($"{body.Length}{Environment.NewLine}");
}
else
{
Console.WriteLine($"*Bad request: {response.StatusCode}");
}
}
}
}
Console.ReadKey();
}
}
}
One additional thing to keep in mind is socket exhaustion. If this code is running on a server, then you will run into socket exhaustion once your service gets some load. I would highly recommend using an HttpClientFactory to handle the HttpClients.
With respect to why the content string is empty, it could be because the server didn't return content which could be a server issue. The server may have also throttled you, which resulted in no content. I would recommend looking into Polly to handle certain response codes.

Solved it... I moved the .DefaultRequestHeaders values inside the foreach() loop.
I suspected that they were pilling up at every loop iteration. I had to clear them and set them up before the GetAsync()
Updated code here:
public async Task<List<Common.ProgressResponse>> RunAsync()
{
List<Response> ListOfResponses = new List<Responses>();
try
{
_client.BaseAddress = new Uri([URL]);
ListOfResponses = await SomeFunction();
}
catch (Exception ex)
{
//Exceptions here...
}
return ListOfResponses;
}
private async Task<List<ListOfResponses>> SomeFunction()
{
List<Response> responses = new List<Response>();
string aPISuffix = string.Format("{0}/{1}", [APISUFFIX1], [APISUFFIX2]);
foreach (Object obj in ListOfObject)
{
/*Moved Code .begin.*/
_client.DefaultRequestHeaders.Accept.Clear();
_client.DefaultRequestHeaders.Clear();
_client.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue() { NoCache = true };
_client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
_client.DefaultRequestHeaders.Add([KEY], [VALUE]);
/*Moved Code .end.*/
_client.DefaultRequestHeaders.Add("Param1", obj.Param1);
if (!string.IsNullOrEmpty(obj.Param2))
_client.DefaultRequestHeaders.Add("Param2", obj.Param2);
Response response = new Response();
HttpResponseMessage hTTPResponse = await _client.GetAsync(aPISuffix).ConfigureAwait(false);
if (hTTPResponse.IsSuccessStatusCode)
{
string result = hTTPResponse.Content.ReadAsStringAsync().Result;
response = [Code here...]
//Codes here...
}
responses.Add(response);
}
return responses;
}

Task Was Canceled

I am trying to download PDF from the database by API call..
All PDF are getting downloaded but only for one row I am getting task was canceled exception
public async System.Threading.Tasks.Task<ActionResult> Record(string empNo)
{
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/pdf"));
var response = client.GetAsync("URL?empNo=" + empNo).Result;
if (response.IsSuccessStatusCode)
{
var files = Directory.GetFiles(Server.MapPath(#"~/EmpPDF/"));
foreach (var file in files)
{
try
{
System.IO.File.Delete(file);
}
catch
{ }
}
var bytes = await response.Content.ReadAsByteArrayAsync();
using (Stream writer = System.IO.File.Create(System.Web.HttpContext.Current.Server.MapPath(#"~EmpPDF/" + empNo + ".pdf")))
{
writer.Write(bytes, 0, bytes.Length);
writer.Flush();
responsePDFPath = #"/EmpPDF/" + empNo + ".pdf"; //+ response.Content.Headers.ContentDisposition.FileName;
}
ViewBag.PathToPdf = responsePDFPath;
return View();
}
}
}
I am getting Task was cancelled exception for this code.

Deadlock because of mixing of blocking calls (.Result) and async-await code.
var response = client.GetAsync("URL?empNo=" + empNo).Result;
That line should be awaited as well
var response = await client.GetAsync("URL?empNo=" + empNo);

Try making sure to dispose the response object:
using(var response = await client.GetAsync("URL?empNo=" + empNo))
{
// Rest of function
}
Leaving too many HttpResponseMessage objects un-disposed can tie up network resources, meaning that later requests will block waiting for one of those resources (which will never get freed, because there is no memory pressure to trigger a GC) and eventually time out.
I've had tons of projects that experience this very same deadlock. It's very important to dispose things that are IDiposable as soon as possible, especially HttpResponseMessage objects.

asp.net Core 1.1 chunked responses

Hopefully, someone can enlighten me on the following.
I have a client that will do a request to a controller endpoint (there is no view, c# to c# or even C++ later). That controller will have to send responses as it fetches them asynchronously as json (sends json1 to client, then json2, then json3 until it closes the connection or send a null terminated text or similar). The purpose is to stream the results back to the client so it can start processing while the server still works.
My controller endpoint looks like this:
[HttpGet("testStream")]
public async Task testStream()
{
var response = HttpContext.Response;
response.Headers[HeaderNames.TransferEncoding] = "chunked";
for (var i = 0; i < 10; ++i)
{
await response.WriteAsync($"6\r\ntest {i}\r\n");
await response.Body.FlushAsync();
await Task.Delay(1 * 1000);
}
await response.WriteAsync("0\r\n\r\n");
await response.Body.FlushAsync();
}
My test looks like this:
static async void DownloadPageAsync()
{
// ... Target page.
string page = "http://localhost:8080/api/Stream/testStream";
Console.WriteLine("test");
while (!Debugger.IsAttached) Thread.Sleep(500);
// ... Use HttpClient.
using (HttpClient client = new HttpClient())
using (var response = await client.GetAsync(page, HttpCompletionOption.ResponseHeadersRead))
using (HttpContent content = response.Content)
{
string result = await content.ReadAsStringAsync();
do
{
Console.WriteLine(result);
result = await content.ReadAsStringAsync();
}
while (result != "null");
}
Console.WriteLine("END");
}
[Fact]
public void Test1()
{
TestSurvey.DownloadPageAsync();
}
I am getting exception when I call content.ReadAsStringAsync();
System.Net.Http.HttpRequestException : Error while copying content to a stream.
[xUnit.net 00:01:14.5836121] ---- System.IO.IOException : The read operation failed, see inner exception.
[xUnit.net 00:01:14.5836496] -------- System.Net.Http.CurlException : Failure when receiving data from the peer
[xUnit.net 00:01:14.5846837] Stack Trace:
[xUnit.net 00:01:14.5857807] at System.Net.Http.HttpContent.<LoadIntoBufferAsyncCore>d__48.MoveNext()
EDIT: Exception was due to not sending the size of the chunk
await response.WriteAsync($"6\r\ntest {i}\r\n");
but now on the test/client side, I get all the chunks at once...

To solve this, I made use of SSE or Server Side Events.
here is the server side in asp.net core:
[HttpGet("testStream")]
public async Task testStream()
{
var response = HttpContext.Response;
response.StatusCode = 200;
response.ContentType = "text/event-stream";
for (var i = 0; i < 10; ++i)
{
//the tags are either 'events:' or 'data:' and two \n indicates ends of the msg
//event: xyz \n\n
//data: xyz \n\n
await response.WriteAsync($"data: test {i}\n\n");
response.Body.Flush();
await Task.Delay(5 * 1000);
}
await response.WriteAsync("data:\n\n");
await response.Body.FlushAsync();
}
and here is the client side:
string page = "http://localhost:8080/api/Stream/testStream";
//while (!Debugger.IsAttached) Thread.Sleep(500);
using (HttpClient client = new HttpClient())
using (var s = await client.GetStreamAsync(page))
{
using (StreamReader r = new StreamReader(s))
{
string line = null;
while (null != (line = r.ReadLine()))
{
Console.WriteLine(line);
}
}
}
Usage of ReadAsStringAsync forced wait of all the message in order to proceed.