I am working on a system that uses HTTPClient to get a HTTP route like /api/GUID and holds the connection open until another system sends a message to SQL Server to notify that the work is completed. Once the message is received the connection ends and my client continues on with its data it was requesting.
When I test this with curl or postman I get the intended results.
I get intermittent issues with c# httpclient. Sometimes I get my data immediately. Other times httpclient will stay open and after its timeout expires the message is shown in my test app despite throwing a task cancellation exception. Other times nothing is returned and a timeout occurs OR I only receive an empty string from my data. This very well could be related to our service broker implementation but i want to rule out everything about HTTPclient. I have tried using threads/tasks/ lots of async await using.Result and I am getting the same results so far. It is highly likely related to our servicebroker waitfor but any insight would be great. The issue tends to be connections that come in rapidly from a single source tend to have this pattern.
public string GetSynchronous(string url)
{
var tokenSource = new CancellationTokenSource();
var cancellationToken = tokenSource.Token;
return _httpClient.GetAsync(url, cancellationToken).Result.Content.ReadAsStringAsync().Result;
}
public void PostAsync(string url, JObject jsonBody)
{
var content = new StringContent(jsonBody.ToString(), Encoding.UTF8, "application/json");
var tokenSource = new CancellationTokenSource();
var cancellationToken = tokenSource.Token;
var result = _httpClient.PostAsync(url, content, cancellationToken).Result;
}
private static readonly HttpClient _httpClient = new HttpClient {
Timeout = TimeSpan.FromSeconds(60),
DefaultRequestHeaders =
{
Accept = { new MediaTypeWithQualityHeaderValue("application/json")},
ConnectionClose = true
},
};
class Program
{
static void Main(string[] args)
{
Console.WriteLine("Hit enter to start");
Console.ReadLine();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.DefaultConnectionLimit = 100;
var tests = new Tests();
tests.CallSynchronizer();
tests.CallSynchronizerPostFirst();
Console.WriteLine("Hit enter to end");
Console.ReadLine();
}
}
Service Broker is designed for long exchanges (hours, days, weeks). Waiting for a response in the HTTP requests is definitely an anti-pattern. And more fundamentally, if concurrent requests are using this patter, there is nothing preventing them from each receiving the response intended for the other request.
You HTTP requests should post the work to-do (ie. issue a SEND) and return immediately. The response, when it comes, should activate a stored procedure and this should update some state, write the result. From HTTP you could then poll for the result and return current status immediately. Your app should work with service broker responses that return after a week same way as if the response returned in 10ms.
Ultimately this ended up being related to how we are load balancing our application and sqldependency.start queue name being set. This is a service that is setup in an anti-pattern way to migrate old code off of a really unreliable set of services.
Related
I have an external endpoint which I call to get some Json response.
This endpoint will initiate a session to a POS device, so the device will show the request details and ask the customer to enter his credit card to complete the payment, then when the customer finishes; the POS will call the endpoint and it will return the result back to my application.
The problem here is that I need the operation to complete as described in this scenario (synchronously).
When I do the call to this endpoint from postman; it waits a lot of time (until the POS receives the request and customer do his entries then returns the results back to endpoint and endpoint returns the results back to Postman) ... this is all works fine.
The problem is when I do this from an ASP.NET Core app, the request is not waited for endpoint and the response is returned with null directly.
I need something to wait for it.
using (var client = new HttpClient())
{
client.DefaultRequestHeaders.Add("x-API-Key", "ApiKey");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var postTask = client.PostAsJsonAsync(new Uri("terminalEndpoint here"), dto);//dto is the request payload
postTask.Wait();
var result = postTask.Result;
if (result.IsSuccessStatusCode)
{
//Should hang after this line to wait for POS
var terminalPaymentResponseDto = result.Content.ReadAsAsync<InitiateTerminalPaymentResponseDto>().Result;
//Should hit this line after customer finishes with POS device
return terminalPaymentResponseDto;
}
}
First of all, there's no need to block. In fact, in an ASP.NET Core application you should avoid blocking as much as possible. Use async and await instead. This allows ASP.NET Core to use the freed threadpool thread for other work.
Second, HttpClient is thread-safe and meant to be reused. Creating a new one every time in a using block leaks sockets. You could use a static instance but a better solution is to use IHttpClientFactory as Make HTTP requests using IHttpClientFactory in ASP.NET Core shows, to both reuse and recycle HttpClient instances automatically.
Finally, there's no reason to add these headers on every call. The Content-Type is set by PostAsJsonAsync anyway. I also suspect the API key doesn't change when calling the same server either.
In your Startup.cs or Program.cs you can use AddHttpClient to configure the API Key :
builder.Services.AddHttpClient(client=>{
client.DefaultRequestHeaders.Add("x-API-Key", "ApiKey");
});
After that you can inject IHttpClientFactory into your controllers or pages and call it asynchronously in asynchronous actions or handlers :
public class MyController:ControllerBase
{
private readonly IHttpClientFactory _httpClientFactory;
public MyController:ControllerBase(IHttpClientFactory httpClientFactory) =>
_httpClientFactory = httpClientFactory;
public async Task<InitiateTerminalPaymentResponseDto> PostAsync(MyDTO dto)
{
var client=_httpClientFactory.CreateClient();
var uri=new Uri("terminalEndpoint here");
var result = client.PostAsJsonAsync(uri, dto);payload
if (result.IsSuccessStatusCode)
{
//Should hang after this line to wait for POS
var paymentDto= await result.Content.ReadAsAsync<InitiateTerminalPaymentResponseDto>();
//Should hit this line after customer finishes with POS device
return paymentDto;
}
else {
//Do whatever is needed in case of error
}
}
}
Using HttpClientFactory allows adding retry strategies using Polly eg, to recover from a temporary network disconnection.
Why not use the await like below? And make sure to change the function to async
var postTask = await client.PostAsJsonAsync(new Uri("terminalEndpoint here"), dto);
Console App Net Core 3.0
I'm sending batch requests to Dynamic CRM Api, basically I read one folder looking for text files that are generated by another process (Nodejs) those files have the request content, then I iterate throught those files, take the text content as string and send it. The problem is when the content of any of those files is bigger (500kb+) The problem is when the requests takes longer (5 minutes or longer) then the HttpClient.SendAsync starts but NEVER completes, the system hangs there, I know the sending was successful cause I check in the api and the changes are applied but the automated process stops there and I have to kill the app. If the content is not that big then everything runs smoothly.
The most frustrating thing is there's no any error or feedback that leads me to any solution or workaround.
Important details:
I think this shouldn't be a deadlock since every single async call in the chain is awaited, the sendasync is also set to ConfigureAwait(false). No .Result, no Wait() or any other blocking operation is called.
I'm not exhausting netword sockets by creating one HttpClient instance per request, I'm using IHttpClientFactory to dispatch the httpClient instance as Singleton. And the issue happens at the first long content request it comes across.
Update
I realized the problem has nothing to do with the payload size but with the time the request takes to complete, I noticed files of 200kb or less hangs because the server takes more than 8 minutes to complete.
The code:
HttpRequestMessage GetRequest(string batchId, string requestBody)
{
var request = new HttpRequestMessage(HttpMethod.Post,"$batch");
request.Headers.Add("OData-MaxVersion", "4.0");
request.Headers.Add("OData-Version", "4.0");
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
request.Content = GetContent(batchId, requestBody);
return request;
}
StringContent GetContent(string batchId, string requestBody)
{
var content = new StringContent(requestBody);
content.Headers.Remove("Content-Type");
content.Headers.Add("Content-Type", $"multipart/mixed;boundary=batch_{batchId}");
return content;
}
string GetRequestBody(string file)
{
var requestBody = "";
using (var sr = new StreamReader(file))
{
requestBody = sr.ReadToEnd();
}
return requestBody;
}
private async Task<HttpResponseMessage> SendRequest(string batchId, string file)
{
// this call is awaited since the access token may expire then is necessary to request a new one
var httpClient = await _httpClientBuilder.BuildHttpClient().ConfigureAwait(false);
var requestBody = GetRequestBody(file); // Here I read the text from the file
using(var request = GetRequest(batchId, requestBody)){
using(var response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false)){ // after this call nothing is executed
if (response.IsSuccessStatusCode)
{
CustomConsole.Success("({0})Successfully Sent batch: {1}", response.StatusCode, batchId);
File.Delete(file);
}
else
{
CustomConsole.Error("({0}) Error sending batch: {1}", response.StatusCode, batchId);
var responseBody = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
CustomConsole.Info("Response Body:");
CustomConsole.Info("{0}",responseBody);
}
return response;
}
}
}
After testing with the .net HttpClient I'm having the following issue. It will do the first 5-6 requests just fine (within 200ms or so), but after that it'll be a full 60 seconds before the rest complete - and they all complete nearly at once
Here is how I've been testing
var tasks = new List<Task<HttpResponseMessage>>();
var client = new HttpClient();
client.DefaultRequestHeaders.Authorization = new System.Net.Http.Headers.AuthenticationHeaderValue("Bearer", "secret_jwt");
client.BaseAddress = new Uri("http://myapi/api/");
for (int i = 0; i < 50; i++)
{
tasks.Add(ProcessUrlAsync("organizations/id/185", client));
}
await Task.WhenAll(tasks);
-
static private async Task<HttpResponseMessage> ProcessUrlAsync(string url, HttpClient client)
{
var sw = System.Diagnostics.Stopwatch.StartNew();
var message = await client.GetAsync(url);
sw.Stop();
Console.WriteLine(sw.Elapsed.TotalMilliseconds.ToString());
return message;
}
and my output is typically
174.5346
127.0873
141.9458
141.7396
153.6638
153.3449
61241.5598
61241.8476
61283.9076
61287.406
61326.0361
61328.7341
61368.6317
etc.
This is not the API I'm using that's the issue - I can point the HttpClient to any address and the same issue occurs
If I write my own GetAsync method that sets the Http version to 1.0...
public new async Task<HttpResponseMessage> GetAsync(string url)
{
using (var request = new HttpRequestMessage(HttpMethod.Get, url))
{
request.Version = HttpVersion.Version10; //removing this will reproduce the issue!
return await SendAsync(request);
}
}
It works fine (all complete within a few hundred ms). Why is this, and what can I do to fix it whilst still using Http 1.1? I'm assuming it's something to do with 1.0 having connection : close and 1.1 having connection: keep-alive
I feel rather silly, the solution is as simple as
ServicePointManager.DefaultConnectionLimit = 100;
(Edited to complete)
This looks like Socket exhaustion with HttpClient. There's lots of info available online if you search for that.
One solution is to only create one HttpClient and re-use that.
Another option is to add the header Connection: close to your request
This would explain why switching to HTTP/1.0 seemed to solve the issue -- as leaving connections open was added in 1.1.
As a side note to anyone who comes across this, you can't send the same HttpRequestMessage twice. You will need to create a new one for the second request.
So I have 2 http Post requests consuming a web api as follow:-
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("https://your_url.com:8443/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// 1st request //
var datatobeSent = new ApiSendData()
{
UserID = "xxx",
UserPsw = "yyy",
ApiFunc = "zzz",
clearData = "x1y1z1"
};
HttpResponseMessage response = await client.PostAsJsonAsync("WebApi", datatobeSent);
var resultData = await response.Content.ReadAsStringAsync();
#region Extract Secured Data response from WebApi
JObject JsonObject = JObject.Parse(resultData);
datatobeSent.SecureData = (string)JsonObject["SecureResult"];
#endregion
// 2nd request //
var datatobeSent2 = new ApiSendData()
{
UserID = "xxx",
UserPsw = "yyy",
ApiFunc = "zzz",
SecureData = datatobeSent.SecureData
};
HttpResponseMessage response2 = await client.PostAsJsonAsync("WebApi", datatobeSent2);
var resultData2 = await response2.Content.ReadAsStringAsync();
}
So now I need some clarifications...
1) Are both my http POST requests sent over the same SSL session ?
2) If they are not, then how can I combine the two and send the 2 requests over a single connections/session ?
3) How may I improve performance of this code? currently 100 requests take 11secs to process and respond. ( i just used a for loop that counts 100 http post requests for the above 2 samples)
They are over the same SSL session and connection. The same HttpClient instance shares some configurations and underlying TCP connections. Therefore, you should reuse the same instance, which you are already doing with the using statement.
I would try to improve the performance of your code by asynchronously making the post requests and processing the results. Here's an option:
Create a new class to handle these async requests
public class WebHelper
{
public async Task<string> MakePostRequest(HttpClient client, string route, object dataToBeSent)
{
try{
HttpResponseMessage response = await client.PostAsJsonAsync(route, datatobeSent);
string resultData = await response.Content.ReadAsStringAsync();
return resultData;
}
catch (Exception ex){
return ex.Message;
}
}
}
Notice that the same httpClient instance is being used. In the main code, you could test your performance like this (to simplify our test, I'm just making the post request with the same parameters 101 times):
//Start time measurement
List<Task> TaskList = new List<Task>();
for (int i = 0; i < 100; i++)
{
Task postTask = Task.Run(async () =>
{
WebHelper webRequest = new WebHelper();
string response = await webRequest.MakePostRequest(client, "WebApi", dataToBeSent);
Console.WriteLine(response);
});
TaskList.Add(postTask);
}
Task.WaitAll(TaskList.ToArray());
//end time measurement
And just an improvement for your code: use try/catches to make the requests!
While HttpClient will aim to use the same Ssl session and connection, this cannot be guaranteed as it depends on the server also maintaining the session and connection. If the server drops them, HttpClient will renegotiate new connections and sessions transparently.
That said, while there is some overhead from connection and ssl session establishment, it's unlikely to be the root cause of any performance issues.
Given the simplicity of your code above, I strongly suspect that the performance issue is not in your client code, but in the service you are connecting to.
To determine that, you need to isolate the performance of that service. A few options for how to do that, depending on the level of control you have over that service:
If you are the owner of that service, carry out over the wire performance testing directly against the service, using a tool like https://www.blitz.io/. If 100 sequential requests from blitz.io takes 11 seconds, then the code you have posted is not the culprit, as the service itself just has an average response time of 110ms.
If you are not the owner of that service, use a tool like Mountebank to create an over the wire test double of that service, and use your existing test loop. If the test loop using Mountebank executes quickly, then again, the code you have posted is not the culprit. (Mountebank supports https, but you will need to either have the keypair, or use their self-signed cert and disable certificate verification in your client.)
It's worth noting that it's not unusual for complex web services to take 110ms to respond. Your test appears to be doing a sequential series of requests - one request at a time - most web services are optimised around handling many requests from independent users in parallel. The scalability challenge is in terms of how many concurrent users can I service while still ensuring that the average response time is 110ms. But if only a single user is using the service at a time, it will still take ~110ms.
So - an even earlier validation step you could take is to work out if your test of 100 sequential requests in a loop is a valid representation of your actual performance requirements, or if you should call your method 100 times in parallel to see if you can have 100 concurrent users access your service.
Our WinRT app is incredibly slow when opening connections to our servers. Requests take ~500ms to run. This is blocking some of our scenarios.
When debugging, we noticed that when Fiddler is active, the requests are much faster - ~100ms per request. Some searches later we understood that was because Fiddler was using Keep-Alive connections when proxying calls, which makes our proxied calls much faster.
We double-checked this in two ways.
We set UseProxy to false and observed that the request went back to being slow.
We turned off Fiddler's "reuse connections" option and observed that the requests went back to being slow.
We tried enabling keep-alive through the Connection header (.Connection.Add("Keep-Alive")) but this does not seem to have any effect - in fact, the header seems to be blatantly ignored by the .NET component and is not being sent on the request (again, by inspecting thru Fiddler).
Does anyone know how to set keep-alive on requests in Windows 8, WinRT, HttpClient class?
The following sets the correct headers to turn on keep-alive for me (client is an HttpClient)
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = false;
// The next line isn't needed in HTTP/1.1
client.DefaultRequestHeaders.Connection.Add("Keep-Alive");
If you want to turn keep-alive off, use
client.DefaultRequestHeaders.Connection.Clear();
client.DefaultRequestHeaders.ConnectionClose = true;
Try using the HttpContent class to add the headers - something like this based on (but untested) http://social.msdn.microsoft.com/Forums/en-CA/winappswithcsharp/thread/ce2563d1-cd96-4380-ad41-6b0257164130
Behind the scenes HttpClient uses HttpWebRequest which would give you direct access to KeepAlive but since you are going through HttpClient you can't directly access that property on the HttpWebRequest class.
public static async Task KeepAliveRequest()
{
var handler = new HttpClientHandler();
var client = new HttpClient(handler as HttpMessageHandler);
HttpContent content = new StringContent(post data here if doing a post);
content.Headers.Add("Keep-Alive", "true");
//choose your type depending what you are sending to the server
content.Headers.ContentType = new MediaTypeHeaderValue("application/x-www-form-urlencoded");
HttpResponseMessage response = await client.PostAsync(url, content);
Stream stream = await response.Content.ReadAsStreamAsync();
return new StreamReader(stream).ReadToEnd();
}
EDIT
Since you only want GET, you can do that with:
public static async Task KeepAliveRequest(string url)
{
var client = new HttpClient();
var request = new HttpRequestMessage()
{
RequestUri = new Uri("http://www.bing.com"),
Method = HttpMethod.Get,
};
request.Headers.Add("Connection", new string[] { "Keep-Alive" });
var responseMessage = await client.SendAsync(request);
return await responseMessage.Content.ReadAsStringAsync();
}