We Keep Coding

ASP.NET Core 5.0 Web API deployed in App service call third party API fails intermittently and raise 500 (Internal Server Error)

ASP.Net Core Web API Call Thirds party API fails intermittently.
The following exception raises intermittently when load test with postman.
"Call failed with status code 500 (Internal Server Error): POST https://sample.com/apiendpoint."
I tried the Named/Typed with HttpClient/IHttpClientFactory approach and the problem continues.
How to make sure it uses connection pooling and not create new on one.
what is the right value for SetHandlerLifetime to keep the connection in the pool for future call to use.
services.AddHttpClient<IRestService, RestServiceOne>()
.SetHandlerLifetime(TimeSpan.FromMinutes(5)) //Set lifetime to five minutes
.AddPolicyHandler(GetRetryPolicy());
The following code is in RestServiceOne.cs
public class RestServiceOne : IRestService
{
private readonly IHttpClientFactory _httpClientFactory;
public RestServiceOne(IHttpClientFactory httpClientFactory)
{
_httpClientFactory = httpClientFactory;
}
public async Task<HttpResponseMessage> GetDataAsync(string destinationUrl, string user,
string password, string requestXml, string orderNumber, CancellationToken cancellationToken)
{
var endpoint = $"{destinationUrl}";
var authToken = Encoding.ASCII.GetBytes($"{user}:{password}");
var data = new StringContent(requestXml, Encoding.UTF8, "text/xml");
var httpRequestMessage = new HttpRequestMessage(
HttpMethod.Post,
endpoint)
{
Headers =
{
{ "Accept", "application/vnd.github.v3+json" },
{ "User-Agent", "HttpRequestsConsoleSample" }
}
};
httpRequestMessage.Content = data;
var httpClient = _httpClientFactory.CreateClient();
httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(authToken));
var httpResponseMessage = await httpClient.SendAsync(httpRequestMessage);
return httpResponseMessage;
}
}
I also tried HttpClient injection given in Microsoft type example.
public class RestService
{
private readonly HttpClient _httpClient;
public RestService(HttpClient httpClient)
{
_httpClient = httpClient;
try
{
_httpClient.BaseAddress = new Uri("https://testxx.com/test");
// GitHub API versioning
_httpClient.DefaultRequestHeaders.Add("Accept",
"application/vnd.github.v3+json");
// GitHub requires a user-agent
_httpClient.DefaultRequestHeaders.Add("User-Agent",
"HttpClientFactory-Sample");
}
catch(Exception ex)
{
}
}
public async Task<HttpResponseMessage> GetDataAsync(string destinationUrl, string user,
string password, string requestXml, string orderNumber, CancellationToken cancellationToken)
{
var endpoint = $"{destinationUrl}";
var authToken = Encoding.ASCII.GetBytes($"{user}:{password}");
_httpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Basic",
Convert.ToBase64String(authToken));
var data = new StringContent(requestXml, Encoding.UTF8, "text/xml");
try
{
var response = await _httpClient.PostAsync(endpoint, data);
return response;
}
catch(Exception ex)
{
throw;
}
}
}
I tried with Flurl directly in service layer
var endpoint = $"{destinationUrl}";
var authToken = Encoding.ASCII.GetBytes($"{user}:{password}");
var data = new StringContent(requestXml, Encoding.UTF8, "text/xml");
try
{
var response = await endpoint
.WithHeader("Content-Type", "text/xml")
.WithHeader("app-bu-id", "SANIDERMMEDICAL")
.WithHeader("Authorization", new AuthenticationHeaderValue("Basic", Convert.ToBase64String(authToken)))
.PostAsync(data);
The above all 3 approach failed.
what is right value for .SetHandlerLifetime() to keep the connection in the pool and avoid creating new.
I saw some other example using the following approach but how to use this with IHttpClientFactory / Flurl.
var socketsHandler = new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromMinutes(10),
PooledConnectionIdleTimeout = TimeSpan.FromMinutes(5),
MaxConnectionsPerServer = 10
};
var client = new HttpClient(socketsHandler);
How can I ensure it use connection pooling and avoid the 500 error when calling 3rd party API from Azure.

I found solution when I use httpclient as given below.
private static readonly HttpClient httpClient = new HttpClient(new SocketsHttpHandler
{
PooledConnectionLifetime = TimeSpan.FromSeconds(60),
PooledConnectionIdleTimeout = TimeSpan.FromSeconds(20),
MaxConnectionsPerServer = 10
});
This article helped me
Tried couple of value for SocketsHttpHandler but finally choose this since no error.

HttpClient usage in polly

Wanted to verify if HttpCLient instance should be created outside method passed to polly for ExecuteAsync, or in?
My current usage varies between the two options and I am not sure which is the correct one?
Also, if it incurs some drawbacks, or possible memory leaks, etc. ?
Get:
var client = new HttpClient(new NativeMessageHandler()) { Timeout = new TimeSpan(0, 0, TimeOutSec) };
var httpResponse = await AuthenticationOnUnauthorizePolicy.ExecuteAsync(async () =>
{
UpdateClientHeader(client, correlationId);
return await client.GetAsync(url, token);
});
Post:
var httpResponse = await AuthenticationOnUnauthorizePolicy.ExecuteAsync(async () =>
{
using (var client = new HttpClient(new NativeMessageHandler()) { Timeout = new TimeSpan(0, 0, TimeOutSec) })
{
UpdateClientHeader(client, correlationId);
WriteNetworkAccessStatusToLog();
return await client.PostAsync(url, content);
}
});
The policy used here:
AuthenticationOnUnauthorizePolicy = Policy
.HandleResult<HttpResponseMessage>(reposnse => reposnse.StatusCode == HttpStatusCode.Unauthorized)
.RetryAsync(1, onRetryAsync:
async (response, count, context) =>
{
_logger.Info("Unauthorized Response! Retrying Authentication...");
await Authenticate();
});
Appreciates any comments on the code above.
Is there a correct way?
Do I need to use the Context to get the client again, or is my usage okay?
Update:
Authenticate method:
public virtual async Task Authenticate()
{
// lock it - only one request can request token
if (Interlocked.CompareExchange(ref _isAuthenticated, 1, 0) == 0)
{
var result = new WebResult();
var loginModel = new LoginModel
{
email = _settingService.Email,
password = _settingService.Password
};
var url = ......
var correlationId = Guid.NewGuid().ToString();
try
{
var stringObj = JsonHelper.SerializeObject(loginModel);
HttpContent content = new StringContent(stringObj, Encoding.UTF8, HttpConsts.JsonMediaType);
using (var client = new HttpClient(new NativeMessageHandler()) { Timeout = new TimeSpan(0, 0, TimeOutSec) }
)
{
UpdateClientHeader(client, correlationId, useToken: false); // not token, we need new one
using (var httpResponse = await client.PostAsync(url, content))
{
var sReader = await httpResponse.Content.ReadAsStringAsync();
await HandleRequest(result, sReader, httpResponse, correlationId, url, "result");
}
}
if (result != null && !result.HasError)
{
_loginToken = result.Token;
}
}
catch (Exception ex)
{
// Log error
}
finally
{
_isAuthenticated = 0;
}
}
}
Update client headr method:
if (_loginToken != null &&
!client.DefaultRequestHeaders.Contains("Token"))
{
client.DefaultRequestHeaders.Add("Token", _loginToken );
}
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue(HttpConsts.JsonMediaType));

Very slow HttpClient SendAsync call

After reading other answers I can't realize why SendAsync is so slow.
Calling same endpoint from Postman, I got a response in 160ms.
Calling from the code below, takes 10 seconds. I'm using a c# desktop application to make the call.
public static async Task<string> GetToken()
{
var url = "....";
var dict = new Dictionary<string, string>();
dict.Add("username", "foo");
dict.Add("password", "bar");
using (var client = new HttpClient(
new HttpClientHandler
{
Proxy = null,
UseProxy = false
}))
{
//bypass SSL
ServicePointManager.ServerCertificateValidationCallback = new
RemoteCertificateValidationCallback
(
delegate { return true; }
);
var req = new HttpRequestMessage(HttpMethod.Post, url) { Content = new FormUrlEncodedContent(dict) };
var res = await client.SendAsync(req); //10 seconds here!
if (res.StatusCode != HttpStatusCode.OK)
return string.Empty;
var token = await JsonConvert.DeserializeObject<TokenResponse>(res.Content.ReadAsStringAsync());
return token.access_token;
}
}

Your code is tangled and ignores IDisposable and this: HttpClient is intended to be instantiated once per application, rather than per-use.
Make reusable method for other-type requests
private static readonly HttpClient client = new HttpClient();
private async Task<T> PostDataAsync<T>(string url, Dictionary<string, string> formData)
{
using (HttpContent content = new FormUrlEncodedContent(formData))
using (HttpResponseMessage response = await client.PostAsync(url, content).ConfigureAwait(false))
{
response.EnsureSuccessStatusCode(); // throws if 404, 500, etc.
string responseText = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
return JsonConvert.DeserializeObject<T>(responseText);
}
}
Usage
public static async Task<string> GetToken()
{
var url = "....";
var dict = new Dictionary<string, string>();
dict.Add("username", "foo");
dict.Add("password", "bar");
try
{
TokenResponse token = await PostDataAsync<TokenResponse>(url, dict);
return token.access_token;
}
catch (HttpRequestException ex)
{
// handle Exception here
return string.Empty;
}
}

Set AllowAutoRedirect false in existing HttpClient for just one request

This answer to the question on how to make HttpClient not follow redirects gives a solution to be set upon creating the actual client:
var handler = new HttpClientHandler { AllowAutoRedirect = false };
var client = new HttpClient(handler);
The comment below the answer is my actual question:
Is it possible to do this on a per-request basis without needing two separate HttpClient instances (i.e. one that allows redirects and one that does not)?
I have a specific reason too for now wanting separate clients: I want the client to retain its cookies from earlier requests. I'm trying to do a few requests first that include valid redirects, but only the last one in the chain I don't want to be a redirect.
I've searched, and looked through the overloads of .GetAsync(url, ...), and looked through the properties and methods of HttpClient, but found no solution yet.
Is this possible?

Yes, you can set the properties of the HttpClientHandler per each request, like so:
using (var handler = new HttpClientHandler())
using (var client = new HttpClient(handler))
{
handler.AllowAutoRedirect = false;
// do your job
handler.AllowAutoRedirect = true;
}
Just make sure that only one thread consumes the HttpClient at a time, if the client handler settings are different.
Example (note: only works in test environment)
Dummy remote server with Node.js runnin on localhost:
const express = require('express')
const app = express()
const cookieParser = require('cookie-parser')
const session = require('express-session')
const port = 3000
app.use(cookieParser());
app.use(session({secret: "super secret"}))
app.get('/set-cookie/:cookieName', (req, res) => {
const cookie = Math.random().toString()
req.session[req.params.cookieName] = cookie
res.send(cookie)
});
app.get('/ok', (req, res) => res.send('OK!'))
app.get('/redirect-301', (req, res) => {
res.writeHead(301, {'Location': '/ok'})
res.end();
})
app.get('/get-cookie/:cookieName', (req, res) => res.send(req.session[req.params.cookieName]))
app.listen(port, () => console.log(`App listening on port ${port}!`))
Tests
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using NUnit.Framework;
public class Tests
{
private HttpClientHandler handler;
private HttpClient client;
private CookieContainer cookieJar = new CookieContainer();
private string cookieName = "myCookie";
private string cookieValue;
[SetUp]
public void Setup()
{
handler = new HttpClientHandler()
{
AllowAutoRedirect = true,
CookieContainer = cookieJar
};
client = new HttpClient(handler);
}
[Test]
public async Task Test0()
{
using (var response = await client.GetAsync($"http://localhost:3000/set-cookie/{cookieName}"))
{
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
cookieValue = await response.Content.ReadAsStringAsync();
}
}
[Test]
public async Task Test1()
{
handler.AllowAutoRedirect = true;
using (var response = await client.GetAsync("http://localhost:3000/redirect-301"))
{
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
Assert.AreEqual(await response.Content.ReadAsStringAsync(), "OK!");
}
}
[Test]
public async Task Test2()
{
handler.AllowAutoRedirect = false;
using (var response = await client.GetAsync("http://localhost:3000/redirect-301"))
{
Assert.AreEqual(HttpStatusCode.MovedPermanently, response.StatusCode);
}
}
[Test]
public async Task Test3()
{
using (var response = await client.GetAsync($"http://localhost:3000/get-cookie/{cookieName}"))
{
Assert.AreEqual(await response.Content.ReadAsStringAsync(), cookieValue);
}
}
}
Output via dotnet test:
Test Run Successful.
Total tests: 4
Passed: 4
Total time: 0.9352 Seconds

The question asks whether following redirects can be done on a case-by-case basis. While certainly useful for many common cases, I found the existing answers lacking in that regard.
The following implementation allows the decision on whether to follow a redirect or not to be configured on a true case-by-case basis via a predicate.
The solution is to override the SendAsync() method of HttpClientHandler.
using System;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
namespace HttpClientCustomRedirectBehavior
{
static class Program
{
private const string REDIRECTING_URL = "http://stackoverflow.com/";
static async Task Main(string[] args)
{
HttpMessageHandler followRedirectAlwaysHandler = new RestrictedRedirectFollowingHttpClientHandler(
response => true);
HttpMessageHandler followRedirectOnlyToSpecificHostHandler = new RestrictedRedirectFollowingHttpClientHandler(
response => response.Headers.Location.Host == "example.com");
HttpResponseMessage response;
using (HttpClient followRedirectAlwaysHttpClient = new HttpClient(followRedirectAlwaysHandler))
{
response = await followRedirectAlwaysHttpClient.GetAsync(REDIRECTING_URL);
Console.WriteLine(response.StatusCode); // OK
}
using (HttpClient followRedirectOnlyToSpecificHostHttpClient = new HttpClient(followRedirectOnlyToSpecificHostHandler))
{
response = await followRedirectOnlyToSpecificHostHttpClient.GetAsync(REDIRECTING_URL);
Console.WriteLine(response.StatusCode); // Moved
}
followRedirectOnlyToSpecificHostHandler = new RestrictedRedirectFollowingHttpClientHandler(
response => response.Headers.Location.Host == "stackoverflow.com");
using (HttpClient followRedirectOnlyToSpecificHostHttpClient = new HttpClient(followRedirectOnlyToSpecificHostHandler))
{
response = await followRedirectOnlyToSpecificHostHttpClient.GetAsync(REDIRECTING_URL);
Console.WriteLine(response.StatusCode); // OK
}
}
}
public class RestrictedRedirectFollowingHttpClientHandler : HttpClientHandler
{
private static readonly HttpStatusCode[] redirectStatusCodes = new[] {
HttpStatusCode.Moved,
HttpStatusCode.Redirect,
HttpStatusCode.RedirectMethod,
HttpStatusCode.TemporaryRedirect,
HttpStatusCode.PermanentRedirect
};
private readonly Predicate<HttpResponseMessage> isRedirectAllowed;
public override bool SupportsRedirectConfiguration { get; }
public RestrictedRedirectFollowingHttpClientHandler(Predicate<HttpResponseMessage> isRedirectAllowed)
{
AllowAutoRedirect = false;
SupportsRedirectConfiguration = false;
this.isRedirectAllowed = response => {
return Array.BinarySearch(redirectStatusCodes, response.StatusCode) >= 0
&& isRedirectAllowed.Invoke(response);
};
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
int redirectCount = 0;
HttpResponseMessage response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
while (isRedirectAllowed.Invoke(response)
&& (response.Headers.Location != request.RequestUri || response.StatusCode == HttpStatusCode.RedirectMethod && request.Method != HttpMethod.Get)
&& redirectCount < this.MaxAutomaticRedirections)
{
if (response.StatusCode == HttpStatusCode.RedirectMethod)
{
request.Method = HttpMethod.Get;
}
request.RequestUri = response.Headers.Location;
response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
++redirectCount;
}
return response;
}
}
}
The Main method shows three example requests to http://stackoverflow.com (which is a URI that redirects to https://stackoverflow.com):
The first GET request will follow the redirect and therefore we see the status code OK of the response to the redirected request, because the handler is configured to follow all redirects.
The second GET request will not follow the redirect and therefore we see the status code Moved, because the handler is configured to follow redirects to the host example.com exclusively.
The third GET request will follow the redirect and therefore we see the status code OK of the response to the redirected request, because the handler is configured to follow redirects to the host stackoverflow.com exclusively.
Of course, you can substitute any custom logic for the predicate.

As you've probably discovered, you're not allowed to change the HttpClientHandler configuration after a request has been made.
Because your motivation for wanting to do this is to maintain the cookies between requests, then I propose something more like this (no exception/null reference handling included):
static CookieContainer cookieJar = new CookieContainer();
static async Task<HttpResponseMessage> GetAsync(string url, bool autoRedirect)
{
HttpResponseMessage result = null;
using (var handler = new HttpClientHandler())
using (var client = new HttpClient(handler))
{
handler.AllowAutoRedirect = autoRedirect;
handler.CookieContainer = cookieJar;
result = await client.GetAsync(url);
cookieJar = handler.CookieContainer;
}
return result;
}
Test:
static async Task Main(string[] args)
{
string url = #"http://stackoverflow.com";
using (var response = await GetAsync(url, autoRedirect: false))
{
Console.WriteLine($"HTTP {(int)response.StatusCode} {response.StatusCode}");
Console.WriteLine($"{response.Headers}");
Console.WriteLine("Cookies:");
Console.WriteLine($"{cookieJar.GetCookieHeader(new Uri(url))}\r\n");
}
Console.WriteLine(new string('-', 30));
using (var response = await GetAsync(url, autoRedirect: true))
{
Console.WriteLine($"HTTP {(int)response.StatusCode} {response.StatusCode}");
Console.WriteLine($"{response.Headers}");
Console.WriteLine("Cookies:");
Console.WriteLine($"{cookieJar.GetCookieHeader(new Uri(url))}\r\n");
}
Console.ReadLine();
}

How do I resolve 302 Error when using HttpClient?

I've created a custom DNN module that uses HTTPClient to send information to an external API. My HttpClient method is as follows:
public static async Task<string> CreatePayerResponse()
{
var credentials = GetCredentials();
var objEventLog = new EventLogController();
var gatewaySettings = new GatewaySetting_ProPay();
SignupResult_ProPay result = new SignupResult_ProPay();
using (HttpClient client = new HttpClient(new LoggingHandler(new HttpClientHandler())))
{
HttpRequestMessage request = new HttpRequestMessage();
HttpResponseMessage response = new HttpResponseMessage();
response.EnsureSuccessStatusCode();
client.BaseAddress = new Uri("https://xmltestapi.propay.com/ProPayAPI/signup");
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
string responseBody;
GatewaySetting_ProPay gatewaySetting = new GatewaySetting_ProPay();
SignupRequest payerRequest = new SignupRequest();
HttpContent content = new StringContent(payerRequest.ToString());
try
{
request.Headers.Add("Authorization", credentials);
request.Headers.Add("accept", "application/json");
response = await client.PutAsync("https://xmltestapi.propay.com/ProPayAPI/signup", content);
responseBody = await response.Content.ReadAsStringAsync();
objEventLog.AddLog("Merchant Onboarding Request Sent", portalSettings, userId,
response.ToString(), EventLogController.EventLogType.ADMIN_ALERT);
Console.WriteLine(responseBody);
}
catch (HttpRequestException ex)
{
result.Succeeded = false;
result.Status = ex.Message;
objEventLog.AddLog("Merchant Onboarding Error!", portalSettings, userId, response.ToString(),
EventLogController.EventLogType.ADMIN_ALERT);
Console.WriteLine("\nException Caught!");
Console.WriteLine("Message :{0} ", ex.Message);
}
return response.Content.ToString();
}
}
public static string GetCredentials()
{
GatewaySetting_ProPay gatewaySettings = new GatewaySetting_ProPay();
var billerAccountId = "mycreds";
var authToken = "mycreds";
var encodedCredentials =
Convert.ToBase64String(Encoding.Default.GetBytes(billerAccountId + ":" + authToken));
var credentials = string.Format("Basic {0}", encodedCredentials);
return credentials;
}
When I wire this method up to a click event, an HTTP 302 response is received and nothing is sent to the API. What modifications are needed to ensure proper transmission?
Update
I still receive the following response:
Error code 302 was received from server response.
This is despite implementing the AllowAutoRedirect property and setting it to true. Here's the LoggingHandler class I've written:
public LoggingHandler(HttpMessageHandler innerHandler) : base(innerHandler)
{
}
protected async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken, HttpClientHandler handler)
{
Console.WriteLine("Request:");
Console.WriteLine(request.ToString());
if (request.Content != null)
{
Console.WriteLine(await request.Content.ReadAsStringAsync());
}
Console.WriteLine();
handler.AllowAutoRedirect = true;
HttpResponseMessage response = await base.SendAsync(request, cancellationToken);
Console.WriteLine("Response:");
Console.WriteLine(response.ToString());
if (response.Content != null)
{
Console.WriteLine(await response.Content.ReadAsStringAsync());
}
Console.WriteLine();
return response;
}
Is this the proper way to implement HttpClientHandler and associated properties?

Make sure that instance of HttpClientHandler has AllowAutoRedirect property set true and is used by your HttpClient.

After implementing trace logs for System.Net.Http and related namespaces, the logs stated that the connection was forcibly closed. After further research, it turns out the .NET Framework 4.5 is not compatible with more modern Transport Layer Security (TLS) versions. As such, the approach of calling the API from our DNN application had to be jettisoned because the source code we're extending targets .NET Framework version 4.5.