We Keep Coding

C# Grpc System.InvalidOperationException: 'Cannot write message after request is complete.'

At runtime I get the following error:
System.InvalidOperationException: 'Cannot write message after request
is complete.'
I'm using Grpc and reactive X to get data.
public class SensorService: Protos.Vehicule.VehiculeBase
{
private readonly ILogger<SensorService> _logger;
private DataProvider _dataProvider;
private CarSim.Vehicule _vehicule;
public SensorService(ILogger<SensorService> logger)
{
_dataProvider = new DataProvider(new CarSim.Vehicule());
_vehicule = _dataProvider.getVehicule();
_logger = logger;
_dataProvider.Start();
}
public System.IObserver<CarSim.Vehicule> GetData { get; private set; }
public override async Task Status(All request, IServerStreamWriter<StatusVehicule> responseStream, ServerCallContext context)
{
while (!context.CancellationToken.IsCancellationRequested)
{
_vehicule.VehiculeChangedState.Subscribe(onNext: new Action<CarSim.Vehicule>(async (t) =>
{
await responseStream.WriteAsync(new StatusVehicule()
{
Camera = t.Camera,
FuelLevel = t.FuelLevel,
GunStatus = true,
Light = t.Light,
OilLevel = t.OilLevel,
Peed = t.Speed,
Tempature = t.Tempature
}
);
; }));
}
}
}

This is likely not a bug. The exception means that you're trying to send a response after the RPC has actually finished. Usually this happens when the RPC deadline is exceeded (at which point the RPC is automatically cancelled) or when it was cancelled by the client. Both of these situations can happen at any time (from the server side handler's perspective) and they are basically an inherent race condition (the RPC could have been cancelled just before you decide to send the response).
The exception is just gRPC's way of informing you that the response could not be sent (and there is really no way to send a response AFTER the RPC has finished).

You must register SensorService in your gRpc Server code:
like this:
app.UseEndpoints(endpoints => { endpoints.MapGrpcService<SensorService >(); .....

I have found a solution that works. But i still have the same error.
public override async Task Status(All request, IServerStreamWriter<StatusVehicule> responseStream, ServerCallContext context){
try
{
var eventLoop = new EventLoopScheduler();
await _vehicule.VehiculeChangedState.ObserveOn(eventLoop).ForEachAsync<CarSim.Vehicule>(t =>
{
try
{
responseStream.WriteAsync(new StatusVehicule()
{
Camera = t.Camera,
FuelLevel = t.FuelLevel,
GunStatus = true,
Light = t.Light,
OilLevel = t.OilLevel,
Peed = t.Speed,
Status = t.GetState().ToString(),
StatusDoors = t.StatusDoors.ToString(),
Tempature = t.Tempature
}
).Wait();
}
catch (Exception)
{
// ignored
}
},
context.CancellationToken);
}
catch (Exception e)
{
_logger.LogError(e.Message);
}
}

AsyncRetryPolicy (with wait) not retry the delegate when handling exception thrown from the action (works fine for the AsyncRetry without wait)

I am using Polly 7.1.0 in my Xamarin mobile solution for the REST API call. As it is a mobile app, the key problem for us is to retry a few times when the network connection exceptions (HttpRequestException, WebException, etc...) occurs.
Basically, for most of the service calls, I am applying a policy wrap with Fallback -> Timeout -> RetryAndWait policies. The most inner policy will do the retry with wait when HttpRequestException/WebException etc happens, then Timeout policy for the total time for the retries, and then the outer fallback policy to get a failure return with error message depends on the Exception type.
But when I do the test with the internet disconnected, only a few of the API calls work as expected, most of other will not execute the retry policy (just timeout and then fallback).
If I put try-catch (with throw) statement in the HandleRequestAsync method of the base class, HttpRequestException was thrown, technically the retry policy should handle the exception and execute the action again, but for some reason, it doesn't work as I expected.
If I use retry (instead of retry and wait) for the inner policy, it does retry as expected.
A few major classes:
PolicyHelper (to get the policies)
public static class PolicyHelper
{
public static AsyncFallbackPolicy<TResponse> GetFallbackPolicy<TResponse>(TResponse fallbackValue,
Action<Exception, IDictionary<string,string>> fallbackHandler = null,
bool showAlert = true)
{
return Policy<TResponse>.Handle<WebException>()
.Or<HttpRequestException>()
.Or<TaskCanceledException>()
.Or<TimeoutRejectedException>()
.Or<Exception>(IsAndroidHttpClientException)
.FallbackAsync
(
fallbackAction : (context, ct) =>
{
return Task.FromResult(fallbackValue);
},
onFallbackAsync : async (result, context) =>
{
Exception exception = result?.Exception ?? new APIServiceException("API call failed.");
if (fallbackHandler != null)
{
IDictionary<string, string> properties = null;
if (context != null)
{
properties = new Dictionary<string, string>();
if (context.TryGetValue("CallerMethod", out object callMethod) && callMethod != null)
properties.Add("CallerMethod", callMethod.ToString());
if (context.TryGetValue("EndPoint", out object endpoint) && endpoint != null)
properties.Add("EndPoint", endpoint.ToString());
}
fallbackHandler.Invoke(exception, properties);
}
//Always to dismiss the possible loading
UserDialogs.Instance.HideLoading();
//Manipulate the err message according to the exception type.
if (fallbackValue != null && typeof(UbiApiResonse).IsAssignableFrom(typeof(TResponse)))
{
if (exception is TimeoutRejectedException)
(fallbackValue as UbiApiResonse).ErrMsg = GlobalConfig.TimeoutErrorMessage;
else if (exception is WebException ||
(exception.InnerException != null && exception.InnerException is WebException) ||
IsAndroidHttpClientException(exception))
(fallbackValue as UbiApiResonse).ErrMsg = GlobalConfig.NetworkErrorMessage;
if (showAlert)
UserDialogs.Instance.Alert((fallbackValue as UbiApiResonse).ErrMsg);
}
else
{
if (showAlert)
UserDialogs.Instance.Alert(GlobalConfig.GeneralErrorMessage);
}
await Task.FromResult(true);
}
);
}
public static AsyncTimeoutPolicy<TResponse> GetTimeoutPolicy<TResponse>(int timeoutInMilliSecs)
{
return Policy.TimeoutAsync<TResponse>(TimeSpan.FromMilliseconds(timeoutInMilliSecs));
}
public static AsyncRetryPolicy<TResponse> GetWaitAndRetryPolicy<TResponse>(int maxRetryCount, TimeSpan retryInterval)
{
return Policy<TResponse>.Handle<WebException>()
.Or<HttpRequestException>()
.Or<TaskCanceledException>()
.Or<TimeoutRejectedException>()
.Or<Exception>(IsAndroidHttpClientException)
.WaitAndRetryAsync
(
maxRetryCount,
retryAttempt => retryInterval
);
}
public static AsyncRetryPolicy<TResponse> GetRetryPolicy<TResponse>(int maxRetryCount)
{
return Policy<TResponse>.Handle<WebException>()
.Or<HttpRequestException>()
.Or<TaskCanceledException>()
.Or<TimeoutRejectedException>()
.Or<Exception>(IsAndroidHttpClientException)
.RetryAsync
(
maxRetryCount
);
}
//As Xamarin.Android http request handler only throw java excepotion
//we need catch all Excetpion by check the type name with string comparison
private static bool IsAndroidHttpClientException(Exception exception)
{
string exceptionTypeName = exception.GetType().FullName.ToLower();
return exceptionTypeName.Contains("javax.net.ssl.sslexception") ||
exceptionTypeName.Contains("java.net.socketexception") ||
exceptionTypeName.Contains("java.net.unknownhostexception") ||
exceptionTypeName.Contains("java.io.ioexception") ||
exceptionTypeName.Contains("java.lang.exception");
}
}
APIBaseService (the base class of all the service class. this class has the actual policy executions for the service methods)
public abstract class APIBaseService
{
private static HttpClient _httpClient = new HttpClient();
private object _httpClientLocker = new object();
//private int _tryCounter;
protected APIBaseService()
{
}
protected async Task<TResponse> HandleRequestWithPolicyAndFallbackAsync<TResponse>(string endPoint,
object payload = null,
AsyncPolicy<TResponse> asyncPolicy = null,
bool showAlert = false,
[CallerMemberName] string callerMethod = null) where TResponse : new()
{
var fallbackValue = GetFallbackValue<TResponse>();
var fallbackPolicy = PolicyHelper.GetFallbackPolicy<TResponse>(fallbackValue, HandleExceptionAfterPolicy, showAlert);
//default inner policy
if (asyncPolicy == null)
{
asyncPolicy = Policy.WrapAsync<TResponse>(PolicyHelper.GetTimeoutPolicy<TResponse>(5000),
PolicyHelper.GetWaitAndRetryPolicy<TResponse>(3, TimeSpan.FromSeconds(100)));
}
//Wrap fallback and inner policy
var policyWrap = Policy.WrapAsync<TResponse>(fallbackPolicy, asyncPolicy);
try
{
return await DoRequestWithPolicy(endPoint, payload, policyWrap, callerMethod);
}
//ideally other exception type should be caught by
catch (Exception ex)
{
IDictionary<string, string> properties = GetCallerMetaData(endPoint, callerMethod);
HandleExceptionAfterPolicy(ex, properties);
return GetFallbackValue<TResponse>(ex.Message);
}
}
private async Task<TResponse> DoRequestWithPolicy<TResponse>(string endPoint,
object payload,
AsyncPolicy<TResponse> asyncPolicy,
string callerMethod)
{
//_tryCounter = 0;
Context context = new Context("HandleRequest", new Dictionary<string, object>());
if (!string.IsNullOrEmpty(callerMethod))
context.Add("CallerMethod", callerMethod);
context.Add("EndPoint", endPoint);
using (CancellationTokenSource cts = new CancellationTokenSource())
{
TResponse response = await asyncPolicy.ExecuteAsync(async (ctx, ct) =>
{
return await HandleRequestAsync<TResponse>(endPoint, payload, ct);
}, context, cts.Token);
return response;
}
}
private async Task<TResponse> HandleRequestAsync<TResponse>(string endPoint, object payload, CancellationToken cancellationToken)
{
HttpClientSetup();
using (var httpContent = CreateHttpContent(payload))
{
using (var response = await _httpClient.PostAsync(endPoint, httpContent, cancellationToken))
using (var stream = await response.Content.ReadAsStreamAsync())
{
response.EnsureSuccessStatusCode();
if (response.IsSuccessStatusCode)
return DeserializeJsonFromStream<TResponse>(stream);
//Ideally, here should throw an exception.
//To keep backwards compatibility, we just return a failure response or a value for model
var errorDetails = await StreamToStringAsync(stream);
return GetFallbackValue<TResponse>(errorDetails);
}
}
}
The service classes
The below service method works fine as expected (ie when network is disconnected, it will retry 3 times for the HttpRequestException and then return a fallback value with the proper error message)
public class SearchAPIService : APIBaseService, ISearchAPIService
{
public async Task<SearchResponse> SearchAsync(double latitude, double longitude, DateTime timeIn, DateTime timeOut, BoundsModel bounds, FilterModel filter)
{
var endPoint = AppConstants.BaseURL + GlobalConfig.APIVersion + "/search";
var request = new SearchRequest
{
Location = new LatLngModel(latitude, longitude),
Filter = filter,
Bounds = bounds,
ArrivalTime = timeIn,
DepartureTime = timeOut,
};
var asyncPolicy = Policy.WrapAsync(PolicyHelper.GetTimeoutPolicy<SearchResponse>(10000),
PolicyHelper.GetWaitAndRetryPolicy<SearchResponse>(3, TimeSpan.FromMilliseconds(500)));
return await HandleRequestWithPolicyAndFallbackAsync(endPoint, request, asyncPolicy, true);
}
}
However, another service which has exactly policy applied will never do the retry and JUST get timeout by the outer Timeout policy
If I put try-catch (with throw) statement in the HandleRequestAsync method of the base class, HttpRequestException was thrown, technically the retry policy should handle the exception and execute the action again, but for some reason, it doesn't work as I expected.
If I use retry (instead of retry and wait) for the inner policy, it does retry as expected.
Not sure if I did anything wrong.
public class PaymentAPIService : APIBaseService, IPaymentAPIService
{
public async Task<PayStatusResponse> PayStatusAsync()
{
var endPoint = AppConstants.BaseURL + GlobalConfig.APIVersion + "/pay/status";
var asyncPolicy = Policy.WrapAsync(PolicyHelper.GetTimeoutPolicy<PayStatusResponse>(3500),
PolicyHelper.GetWaitAndRetryPolicy<PayStatusResponse>(3, TimeSpan.FromSeconds(100)));
return await HandleRequestWithPolicyAndFallbackAsync<PayStatusResponse>(endPoint, null, asyncPolicy, true);
}
}

How can I safely throw exceptions from within a DelegatingHandler

I have a delegating handler that throws specific exceptions based on fault messages received from the server.
This allows the client to retry (using polly, but as you'll see the mechanism doesn't matter)
After two failed attempts the third attempt simply hangs. No data hits the server, and nothing further happens. This seems to be because the HttpClient has a max of two connections, which suggests these connections are not being closed properly.
Is there something I can do to close the connection or should I change the design so that the delegating handlers do not throw exceptions?
See simple example below to reproduce
private class Test : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
throw new Exception("afesaf");
}
}
[HttpGet]
[Route("B")]
public async Task B()
{
var handler = new HttpClientHandler();
var pipeline = HttpClientFactory.CreatePipeline(handler, new[] { new Test() });
var http = new HttpClient(pipeline);
http.BaseAddress = new Uri("http://google.com");
var count = 0;
while (count++ < 5)
try
{
Log.Info("A");
var request = new HttpRequestMessage(HttpMethod.Get, "");
await http.SendAsync(request, HttpCompletionOption.ResponseContentRead, CancellationToken.None).ConfigureAwait(false);
}
catch
{
}
}

Dispose of the response object.
I've gone with the following in the delegating handler as I have multiple points where the exception could be thrown in the real code
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
HttpResponseMessage response = null;
try
{
response = await base.SendAsync(request, cancellationToken);
throw new Exception("afesaf");
}
catch
{
response?.Dispose();
throw
}
}

Retrying HttpClient Unsuccessful Requests

I am building a function that given an HttpContent Object, will issues request and retry on failure. However I get exceptions saying that HttpContent Object is disposed after issuing the request. Is there anyway to copy or duplicate the HttpContent Object so that I can issue multiple requests.
public HttpResponseMessage ExecuteWithRetry(string url, HttpContent content)
{
HttpResponseMessage result = null;
bool success = false;
do
{
using (var client = new HttpClient())
{
result = client.PostAsync(url, content).Result;
success = result.IsSuccessStatusCode;
}
}
while (!success);
return result;
}
// Works with no exception if first request is successful
ExecuteWithRetry("http://www.requestb.in/xfxcva" /*valid url*/, new StringContent("Hello World"));
// Throws if request has to be retried ...
ExecuteWithRetry("http://www.requestb.in/badurl" /*invalid url*/, new StringContent("Hello World"));
(Obviously I don't try indefinitely but the code above is essentially what i want).
It yields this exception
System.AggregateException: One or more errors occurred. ---> System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Http.StringContent'.
at System.Net.Http.HttpContent.CheckDisposed()
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at Submission#8.ExecuteWithRetry(String url, HttpContent content)
Is there anyway to duplicate an HttpContent Object or reuse it?

Instead of implementing retry functionality that wraps the HttpClient, consider constructing the HttpClient with a HttpMessageHandler that performs the retry logic internally. For example:
public class RetryHandler : DelegatingHandler
{
// Strongly consider limiting the number of retries - "retry forever" is
// probably not the most user friendly way you could respond to "the
// network cable got pulled out."
private const int MaxRetries = 3;
public RetryHandler(HttpMessageHandler innerHandler)
: base(innerHandler)
{ }
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
HttpResponseMessage response = null;
for (int i = 0; i < MaxRetries; i++)
{
response = await base.SendAsync(request, cancellationToken);
if (response.IsSuccessStatusCode) {
return response;
}
}
return response;
}
}
public class BusinessLogic
{
public void FetchSomeThingsSynchronously()
{
// ...
// Consider abstracting this construction to a factory or IoC container
using (var client = new HttpClient(new RetryHandler(new HttpClientHandler())))
{
myResult = client.PostAsync(yourUri, yourHttpContent).Result;
}
// ...
}
}

ASP.NET Core 2.1 Answer
ASP.NET Core 2.1 added support for Polly directly. Here UnreliableEndpointCallerService is a class which accepts a HttpClient in its constructor. Failed requests will retry with an exponential back-off so that the next retry takes place in an exponentially longer time after the previous one:
services
.AddHttpClient<UnreliableEndpointCallerService>()
.AddTransientHttpErrorPolicy(
x => x.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)));
Also, consider reading my blog post "Optimally Configuring HttpClientFactory".
Other Platforms Answer
This implementation uses Polly to retry with an exponential back-off so that the next retry takes place in an exponentially longer time after the previous one. It also retries if a HttpRequestException or TaskCanceledException is thrown due to a timeout. Polly is much easier to use than Topaz.
public class HttpRetryMessageHandler : DelegatingHandler
{
public HttpRetryMessageHandler(HttpClientHandler handler) : base(handler) {}
protected override Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken) =>
Policy
.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.OrResult<HttpResponseMessage>(x => !x.IsSuccessStatusCode)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
.ExecuteAsync(() => base.SendAsync(request, cancellationToken));
}
using (var client = new HttpClient(new HttpRetryMessageHandler(new HttpClientHandler())))
{
var result = await client.GetAsync("http://example.com");
}

The current answers won't work as expected in all cases, specifically in the very common case of request timeout (see my comments there).
In addition, they implement a very naive retry strategy - many times you'd want something a bit more sophosticated, such as exponential backoff (which is the default in the Azure Storage Client API).
I stumbled upon TOPAZ while reading a related blog post (also offering the misguided internal retry approach). Here's what I came up with:
// sample usage: var response = await RequestAsync(() => httpClient.GetAsync(url));
Task<HttpResponseMessage> RequestAsync(Func<Task<HttpResponseMessage>> requester)
{
var retryPolicy = new RetryPolicy(transientErrorDetectionStrategy, retryStrategy);
//you can subscribe to the RetryPolicy.Retrying event here to be notified
//of retry attempts (e.g. for logging purposes)
return retryPolicy.ExecuteAsync(async () =>
{
HttpResponseMessage response;
try
{
response = await requester().ConfigureAwait(false);
}
catch (TaskCanceledException e) //HttpClient throws this on timeout
{
//we need to convert it to a different exception
//otherwise ExecuteAsync will think we requested cancellation
throw new HttpRequestException("Request timed out", e);
}
//assuming you treat an unsuccessful status code as an error
//otherwise just return the respone here
return response.EnsureSuccessStatusCode();
});
}
Note the requester delegate parameter. It should not be an HttpRequestMessage since you can't send the same request multiple times. As for the strategies, that depends on your use case. For example, a transient error detection strategy could be as simple as:
private sealed class TransientErrorCatchAllStrategy : ITransientErrorDetectionStrategy
{
public bool IsTransient(Exception ex)
{
return true;
}
}
As for the retry strategy, TOPAZ offers three options:
FixedInterval
Incremental
ExponentialBackoff
For example, here's the TOPAZ equivalent of what the Azure Client Storage Library uses for default:
int retries = 3;
var minBackoff = TimeSpan.FromSeconds(3.0);
var maxBackoff = TimeSpan.FromSeconds(120.0);
var deltaBackoff= TimeSpan.FromSeconds(4.0);
var strategy = new ExponentialBackoff(retries, minBackoff, maxBackoff, deltaBackoff);
For more information see http://msdn.microsoft.com/en-us/library/hh680901(v=pandp.50).aspx
EDIT Note that if your request contains an HttpContent object, you'll have to regenerate it every time as that will be disposed by HttpClient as well (thanks for catching that Alexandre Pepin). For example () => httpClient.PostAsync(url, new StringContent("foo"))).

Duplicating the StringContent isn't probably the best idea. But simple modification could fix the problem. Just modify the function and create the StringContent object inside of the loop, something like:
public HttpResponseMessage ExecuteWithRetry(string url, string contentString)
{
HttpResponseMessage result = null;
bool success = false;
using (var client = new HttpClient())
{
do
{
result = client.PostAsync(url, new StringContent(contentString)).Result;
success = result.IsSuccessStatusCode;
}
while (!success);
}
return result;
}
and then call it
ExecuteWithRetry("http://www.requestb.in/xfxcva" /*valid url*/, "Hello World");

This is what I achieved using polly.
nuget
https://www.nuget.org/packages/Microsoft.Extensions.Http.Polly
https://www.nuget.org/packages/Polly
using Polly;
using Polly.Extensions.Http;
//// inside configure service
services.AddHttpClient("RetryHttpClient", c =>
{
c.BaseAddress = new Uri($"{configuration["ExternalApis:MyApi"]}/");
c.DefaultRequestHeaders.Add("Accept", "application/json");
c.Timeout = TimeSpan.FromMinutes(5);
c.DefaultRequestHeaders.ConnectionClose = true;
}).AddPolicyHandler(GetRetryPolicy());
//// add this method to give retry policy
private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
return HttpPolicyExtensions
//// 408,5xx
.HandleTransientHttpError()
//// 404
.OrResult(msg => msg.StatusCode == HttpStatusCode.NotFound)
//// 401
.OrResult(msg => msg.StatusCode == HttpStatusCode.Unauthorized)
//// Retry 3 times, with wait 1,2 and 4 seconds.
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)));
}

This builds off the accepted answer but adds the ability to pass in the amount of retries, plus the ability to add non-blocking delays / wait time to each request. It also uses a try catch to ensure the retry continues to happen after an exception has occurred. And last, I added code to break out of the loop in the case of BadRequests, you don't want to resend the same bad request multiple times.
public class HttpRetryHandler : DelegatingHandler
{
private int MaxRetries;
private int WaitTime;
public HttpRetryHandler(HttpMessageHandler innerHandler, int maxRetries = 3, int waitSeconds = 0)
: base(innerHandler)
{
MaxRetries = maxRetries;
WaitTime = waitSeconds * 1000;
}
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
HttpResponseMessage response = null;
for (int i = 0; i < MaxRetries; i++)
{
try
{
response = await base.SendAsync(request, cancellationToken);
if (response.IsSuccessStatusCode)
{
return response;
}
else if(response.StatusCode == HttpStatusCode.BadRequest)
{
// Don't reattempt a bad request
break;
}
}
catch
{
// Ignore Error As We Will Attempt Again
}
finally
{
response.Dispose();
}
if(WaitTime > 0)
{
await Task.Delay(WaitTime);
}
}
return response;
}
}

With RestEase And Task, on retry with httpClient reused in many call (singleton) it frezze and throw TaskCanceledException.
To fix this whe need to Dispose() the failed response before retry
public class RetryHandler : DelegatingHandler
{
// Strongly consider limiting the number of retries - "retry forever" is
// probably not the most user friendly way you could respond to "the
// network cable got pulled out."
private const int MaxRetries = 3;
public RetryHandler(HttpMessageHandler innerHandler)
: base(innerHandler)
{ }
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
HttpResponseMessage response = null;
for (int i = 0; i < MaxRetries; i++)
{
response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
if (response.IsSuccessStatusCode) {
return response;
}
response.Dispose();
}
return response;
}
}

I tried it and worked while using unit and integration tests. However, it stuck when I actually called from REST URL. I found this interesting post which explains why it gets stuck at this line.
response = await base.SendAsync(request, cancellationToken);
The fix to this is that you have .ConfigureAwait(false) added at the end.
response = await base.SendAsync(request, token).ConfigureAwait(false);
I also added create linked token part there like this.
var linkedToken = cancellationToken.CreateLinkedSource();
linkedToken.CancelAfter(new TimeSpan(0, 0, 5, 0));
var token = linkedToken.Token;
HttpResponseMessage response = null;
for (int i = 0; i < MaxRetries; i++)
{
response = await base.SendAsync(request, token).ConfigureAwait(false);
if (response.IsSuccessStatusCode)
{
return response;
}
}
return response;

You also refer to Building a Transient Retry Handler for the .NET HttpClient.
Visit refer to KARTHIKEYAN VIJAYAKUMAR post.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Data.SqlClient;
using System.Net.Http;
using System.Threading;
using System.Diagnostics;
using System.Net;
using Microsoft.Practices.EnterpriseLibrary.TransientFaultHandling;
namespace HttpClientRetyDemo
{
class Program
{
static void Main(string[] args)
{
var url = "http://RestfulUrl";
var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, url);
var handler = new RetryDelegatingHandler
{
UseDefaultCredentials = true,
PreAuthenticate = true,
Proxy = null
};
HttpClient client = new HttpClient(handler);
var result = client.SendAsync(httpRequestMessage).Result.Content
.ReadAsStringAsync().Result;
Console.WriteLine(result.ToString());
Console.ReadKey();
}
}
/// <summary>
/// Retry Policy = Error Detection Strategy + Retry Strategy
/// </summary>
public static class CustomRetryPolicy
{
public static RetryPolicy MakeHttpRetryPolicy()
{
// The transient fault application block provides three retry policies
// that you can use. These are:
return new RetryPolicy(strategy, exponentialBackoff);
}
}
/// <summary>
/// This class is responsible for deciding whether the response was an intermittent
/// transient error or not.
/// </summary>
public class HttpTransientErrorDetectionStrategy : ITransientErrorDetectionStrategy
{
public bool IsTransient(Exception ex)
{
if (ex != null)
{
HttpRequestExceptionWithStatus httpException;
if ((httpException = ex as HttpRequestExceptionWithStatus) != null)
{
if (httpException.StatusCode == HttpStatusCode.ServiceUnavailable)
{
return true;
}
else if (httpException.StatusCode == HttpStatusCode.MethodNotAllowed)
{
return true;
}
return false;
}
}
return false;
}
}
/// <summary>
/// The retry handler logic is implementing within a Delegating Handler. This has a
/// number of advantages.
/// An instance of the HttpClient can be initialized with a delegating handler making
/// it super easy to add into the request pipeline.
/// It also allows you to apply your own custom logic before the HttpClient sends the
/// request, and after it receives the response.
/// Therefore it provides a perfect mechanism to wrap requests made by the HttpClient
/// with our own custom retry logic.
/// </summary>
class RetryDelegatingHandler : HttpClientHandler
{
public RetryPolicy retryPolicy { get; set; }
public RetryDelegatingHandler()
: base()
{
retryPolicy = CustomRetryPolicy.MakeHttpRetryPolicy();
}
protected async override Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
HttpResponseMessage responseMessage = null;
var currentRetryCount = 0;
//On Retry => increments the retry count
retryPolicy.Retrying += (sender, args) =>
{
currentRetryCount = args.CurrentRetryCount;
};
try
{
await retryPolicy.ExecuteAsync(async () =>
{
responseMessage = await base.SendAsync(request, cancellationToken)
.ConfigureAwait(false);
if ((int)responseMessage.StatusCode > 500)
{
// When it fails after the retries, it would throw the exception
throw new HttpRequestExceptionWithStatus(
string.Format("Response status code {0} indicates server error",
(int)responseMessage.StatusCode))
{
StatusCode = responseMessage.StatusCode,
CurrentRetryCount = currentRetryCount
};
}// returns the response to the main method(from the anonymous method)
return responseMessage;
}, cancellationToken).ConfigureAwait(false);
return responseMessage;// returns from the main method => SendAsync
}
catch (HttpRequestExceptionWithStatus exception)
{
if (exception.CurrentRetryCount >= 3)
{
//write to log
}
if (responseMessage != null)
{
return responseMessage;
}
throw;
}
catch (Exception)
{
if (responseMessage != null)
{
return responseMessage;
}
throw;
}
}
}
/// <summary>
/// Custom HttpRequestException to allow include additional properties on my exception,
/// which can be used to help determine whether the exception is a transient
/// error or not.
/// </summary>
public class HttpRequestExceptionWithStatus : HttpRequestException
{
public HttpStatusCode StatusCode { get; set; }
public int CurrentRetryCount { get; set; }
public HttpRequestExceptionWithStatus()
: base() { }
public HttpRequestExceptionWithStatus(string message)
: base(message) { }
public HttpRequestExceptionWithStatus(string message, Exception inner)
: base(message, inner) { }
}
}

i have almost the same issue.
HttpWebRequest queueing library, which guarantees request delivery
I just updated (see EDIT3) my approach to avoid crashes, but i still need general mechanism to guarantee message delivery (or re-delivery in case message was not delivered).

I have the same problem and solved . it's about the "StringContent" / "HttpContent"
Please check Amogh Natu's Blog which help me to solve this problem
The problem with this code is, when the first call to PostAsync is
made and it fails, the httpContent object is disposed. This is as
designed in the HttpClient class. Refer the comment in this method.
Although this seems odd, they intent to do this so that the user
doesn’t have to do this explicitly and also to avoid the same request
being Posted more than once.
So what happens is, when the first call fails, httpContent is
disposed, then as we have retry mechanism, it tries to make the post
call again, now with a disposed object and hence this time, the call
fails with an ObjectDisposedException.
An easy way to resolve this is to NOT use a variable to store
httpContent and instead, create http content directly while making the
call. Something like this.
http://amoghnatu.net/2017/01/12/cannot-access-a-disposed-object-system-net-http-stringcontent-while-having-retry-logic/

Adding an answer that uses both Polly + Retry policy + per-retry Timeout policy, as top answer does not address that:
Policy
.Handle<HttpRequestException>()
.Or<TaskCanceledException>()
.Or<TimeoutRejectedException>()
.OrResult<HttpResponseMessage>(x => !x.IsSuccessStatusCode)
.WaitAndRetryAsync(3, retryAttempt => TimeSpan.FromSeconds(Math.Pow(3, retryAttempt)))
.WrapAsync(
Policy.TimeoutAsync(TimeSpan.FromSeconds(1), delegate (Context ctx, TimeSpan timeSpan, Task task)
{
// Do some on-timeout action
return Task.CompletedTask;
})
)
.ExecuteAsync(() =>
{
return httpclient.PostAsync(url, httpRequest);
});

//Could retry say 5 times
HttpResponseMessage response;
int numberOfRetry = 0;
using (var httpClient = new HttpClient())
{
do
{
response = await httpClient.PostAsync(uri, content);
numberOfRetry++;
} while (response.IsSuccessStatusCode == false | numberOfRetry < 5);
}
return response;
.........

Web API audit logging

I need to audit log calls to my Web API, ideally I'd like to use an Attribute, something like:
[HttpPost, Auditing]
public dynamic MyAPICall()
The Attribute should be able to intercept the API call before and after execution in order to log the parameters and also, how long the API call took to run.
With MVC I could create an ActionFilterAttribute derivative and override OnActionExecuted and OnActionExecuting.
Is the equivalent possible in the Web API world?

Http message handler should be a good extensible point for such purposes. Be careful though, there can be some issues with concurrent request content reading. For instance, Model Binder may try to read request content while LoggingHandler is reading it and fail to deserialize a model. To prevent such issues just add Wait call to the LogRequestLoggingInfo method.
public class LoggingHandler : DelegatingHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
// Log the request information
LogRequestLoggingInfo(request);
// Execute the request
return base.SendAsync(request, cancellationToken).ContinueWith(task =>
{
var response = task.Result;
// Extract the response logging info then persist the information
LogResponseLoggingInfo(response);
return response;
});
}
private void LogRequestLoggingInfo(HttpRequestMessage request)
{
if (request.Content != null)
{
request.Content.ReadAsByteArrayAsync()
.ContinueWith(task =>
{
var result = Encoding.UTF8.GetString(task.Result);
// Log it somewhere
}).Wait(); // !!! Here is the fix !!!
}
}
private void LogResponseLoggingInfo(HttpResponseMessage response)
{
if (response.Content != null)
{
response.Content.ReadAsByteArrayAsync()
.ContinueWith(task =>
{
var responseMsg = Encoding.UTF8.GetString(task.Result);
// Log it somewhere
});
}
}
}
You can read more about it here.

I would use a message handler rather than attributes.
public class LoggingHandler : DelegatingHandler
{
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
LogRequest(request);
return base.SendAsync(request, cancellationToken).ContinueWith(task =>
{
var response = task.Result;
LogResponse(response);
return response;
});
}
private void LogRequest(HttpRequestMessage request)
{
(request.Content ?? new StringContent("")).ReadAsStringAsync().ContinueWith(x =>
{
Logger.Info("{4:yyyy-MM-dd HH:mm:ss} {5} {0} request [{1}]{2} - {3}", request.GetCorrelationId(), request.Method, request.RequestUri, x.Result, DateTime.Now, Username(request));
});
}
private void LogResponse(HttpResponseMessage response)
{
var request = response.RequestMessage;
(response.Content ?? new StringContent("")).ReadAsStringAsync().ContinueWith(x =>
{
Logger.Info("{3:yyyy-MM-dd HH:mm:ss} {4} {0} response [{1}] - {2}", request.GetCorrelationId(), response.StatusCode, x.Result, DateTime.Now, Username(request));
});
}
private string Username(HttpRequestMessage request)
{
var values = new List<string>().AsEnumerable();
if (request.Headers.TryGetValues("my-custom-header-for-current-user", out values) == false) return "<anonymous>";
return values.First();
}
}

I think you will be interested to take a look at Web API tracing http://www.asp.net/web-api/overview/testing-and-debugging/tracing-in-aspnet-web-api. It allows you to look into the internal mechanism of Web API.
In your case, I assume you're particularly interested in what's the input and output of actions. So you can right your TraceWriter like following sample to filter out the redundant information:
public class ActionAuditor : ITraceWriter
{
private const string TargetOperation = "ExecuteAsync";
private const string TargetOpeartor = "ReflectedHttpActionDescriptor";
public void Trace(HttpRequestMessage request, string category, TraceLevel level, Action<TraceRecord> traceAction)
{
var rec = new TraceRecord(request, category, level);
traceAction(rec);
if (rec.Operation == TargetOperation && rec.Operator == TargetOpeartor)
{
if (rec.Kind == TraceKind.Begin)
{
// log the input of the action
}
else
{
// log the output of the action
}
}
}
}

I've worked on a library that allows you to log interactions with ASP.NET Web API Controllers by using Action Filters.
It can record action method calls with caller info, arguments, output, duration, exceptions and more.
Take a look at Audit.WebApi.
You can quickly create a sample project that uses this library with the following commands:
> dotnet new -i Audit.WebApi.Template
> dotnet new webapiaudit