I'm running into an issue with the .NET HttpClient class (.NET 4.5.1, System.Net.Http v4.0.0.0). I'm calling HttpClient.GetAsync, passing in a CancellationToken (as part of a Nuget package that abstracts calls between webservices). If the token has been cancelled before the call is made, the request goes through without throwing an exception. This behavior doesn't seem correct.
My test (incomplete, not fully written - no exception check):
[TestMethod]
public async Task Should_Cancel_If_Cancellation_Token_Called()
{
var endpoint = "nonexistent";
var cancellationTokenSource = new CancellationTokenSource();
var _mockHttpMessageHandler = new MockHttpMessageHandler();
_mockHttpMessageHandler
.When("*")
.Respond(HttpStatusCode.OK);
var _apiClient = new ApiClientService(new HttpClient(_mockHttpMessageHandler));
cancellationTokenSource.Cancel();
var result = await _apiClient.Get<string>(endpoint, null, cancellationTokenSource.Token);
}
The method I'm testing:
public async Task<T> Get<T>(string endpoint, IEnumerable<KeyValuePair<string, string>> parameters = null, CancellationToken cancellationToken = default(CancellationToken))
{
var builder = new UriBuilder(Properties.Settings.Default.MyEndpointHost + endpoint);
builder.Query = buildQueryStringFromParameters(parameters);
_httpClient.DefaultRequestHeaders.Accept.Clear();
_httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
try
{
// After this, we really shouldn't continue.
var request = await _httpClient.GetAsync(builder.Uri, cancellationToken);
if (!request.IsSuccessStatusCode)
{
if (request.StatusCode >= HttpStatusCode.BadRequest && request.StatusCode < HttpStatusCode.InternalServerError)
{
throw new EndpointClientException("Service responded with an error message.", request.StatusCode, request.ReasonPhrase);
}
if (request.StatusCode >= HttpStatusCode.InternalServerError && (int)request.StatusCode < 600)
{
throw new EndpointServerException("An error occurred in the Service endpoint.", request.StatusCode, request.ReasonPhrase);
}
}
var json = await request.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<T>(json);
}
catch (Exception ex)
{
throw;
}
}
I know that I can check the status of the cancellation token before calling HttpClient.GetAsync and throw if cancellation has been requested. I know that I can also register a delegate to cancel the HttpClient request. However, it seems as though passing the token to the HttpClient method should take care of this for me (or, else, what's the point?) so I'm wondering if I'm missing something. I don't have access to the HttpClient source code.
Why is HttpClient.GetAsync not checking my cancellation token and aborting its process when I pass it in?
HttpClient doesn't check the cancellation token itself, it passes it on to the message handler when it calls its SendAsync method. It then registers to the continuation on the task returned from SendAsync and will set its own task as cancelled if the task returned from the message handler was cancelled.
So the problem in your scenario is in your implementation of MockHttpMessageHandler which seems doesn't check the cancellation token.
Note, that if HttpClient is called via its empty constructor, it internally uses HttpClientHandler which registers a delegate on the cancellation token that aborts the request and cancels the task.
Related
I have on ASP.Net C# web API with an endpoint for the import. Javascript client sends a list of items to this API and API process this list in another thread (long task) and immediately returns unique id (GUID) of process. Now I need the cancel the background task from the CLIENT. Is possible to somehow send the cancelation token from the client? I have tried to add CancellationToken as a parameter to my controller async action but I don't know how to pass it from the client. For simplification, we can use as the client the Postman app.
Sample server-side
[HttpPost]
[UserContextActionFilter]
[RequestBodyType(typeof(List<List<Item>>))]
[Route("api/bulk/ImportAsync")]
public async Task<IHttpActionResult> ImportAsync()
{
var body = await RequestHelper.GetRequestBody(this);
var queue = JsonConvert.DeserializeObject<List<List<Item>>>(body);
var resultWrapper = new AsynckResultWrapper(queue.Count);
HostingEnvironment.QueueBackgroundWorkItem(async ct =>
{
foreach (var item in queue)
{
var result = await ProcessItemList(item, false);
resultWrapper.AddResultItem(result);
}
});
return Ok(new
{
ProcessId = resultWrapper.ProcessId.ToString()
});
}
private async Task<ItemResult> ProcessItemList(<List<Item>>itemList, bool runInOneTransaction = false)
{
try
{
var result = await PerformBulkOperation(true, itemList);
return new ResultWrapper(result);
}
catch (Exception ex)
{
// process exception
return new ResultWrapper(ex);
}
}
On a high level what you could do is store the process id along with a cancellation token source when you queue the work. Then you can expose a new endpoint that accepts a process id, gets the cancellation token source from the store and cancels the associated token:
[HttpPost]
[UserContextActionFilter]
[RequestBodyType(typeof(List<List<Item>>))]
[Route("api/bulk/ImportAsync")]
public async Task<IHttpActionResult> ImportAsync()
{
var body = await RequestHelper.GetRequestBody(this);
var queue = JsonConvert.DeserializeObject<List<List<Item>>>(body);
var resultWrapper = new AsynckResultWrapper(queue.Count);
HostingEnvironment.QueueBackgroundWorkItem(async ct =>
{
var lts = CancellationTokenSource.CreateLinkedTokenSource(ct);
var ct = lts.Token;
TokenStore.Store(resultWrapper.ProcessId, lts);
foreach (var item in queue)
{
var result = await ProcessItemList(item, ct, false);
resultWrapper.AddResultItem(result);
}
TokenStore.Remove(processId) // remove the cancellation token source from storage when doen, because there is nothing to cancel
});
return Ok(new
{
ProcessId = resultWrapper.ProcessId.ToString()
});
}
private async Task<ItemResult> ProcessItemList(<List<Item>>itemList, CancellationToken token, bool runInOneTransaction = false)
{
try
{
var result = await PerformBulkOperation(true, itemList, token);
return new ResultWrapper(result);
}
catch (Exception ex)
{
// process exception
return new ResultWrapper(ex);
}
}
[Route("api/bulk/CancelImportAsync")]
public async Task<IHttpActionResult> CancelImportAsync(Guid processId)
{
var tokenSource = TokenStore.Get(processId);
tokenSource.Cancel();
TokenStore.Remove(processId) // remove the cancellation token source from storage when cancelled
}
In the above example I modified the ProcessItemList to accept a cancellation token and pass it to PerformBulkOperation, assuming that method has support for cancellation tokens. If not, you can manually call ThrowIfCancellationRequested(); on the cancellation token at certain points in the code to stop when cancellation is requested.
I've added a new endpoint that allows you to cancel a pending operation.
Disclaimer
There are for sure some things you need to think about, especially when it is a public api. You can extend the store to accepts some kind of security token and when cancellation is requested you check whether it matches with the security token that queued the work. My answer is focused on the basics of the question
Also, I left the implementation of the store to your own imagination ;-)
I'm working with a very flaky API. Sometimes I get 500 Server Error with Timeout, some other time I also get 500 Server Error because I gave it input that it can't handle
SqlDateTime overflow. Must be between 1/1/1753 12:00:00 AM and 12/31/9999 11:59:59 PM.
Both of these cases give me HttpRequestException but I can look into the reply message from the server and determine the cause of the exception. If it is a timeout error, I should try again. If it is a bad input I should re-throw the exception, because no amount of retries will fix the problem of bad data.
What I'd like to do with Polly is to check on response message before attempting to retry. But all the samples I've seen so far only included type of exception.
I've come up with this so far:
HttpResponseMessage response = null;
String stringContent = null;
Policy.Handle<FlakyApiException>()
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
response = await client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"));
stringContent = await response.Content.ReadAsStringAsync();
if (response.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("Timeout"))
{
throw new FlakyApiException(stringContent);
}
});
Is there a better way to do this kind of checking?
In general, you can configure Polly policies to respond to the results of an execution (not just an exception), for example check an HttpResponseMessage.StatusCode with a predicate. Examples here in the Polly readme.
There is not however an in-built way to configure a single Polly policy to respond additionally to the content of the response message. This is because (as your example shows) obtaining that content requires a second async call, which may itself raise network errors.
This tl;dr engenders complications about how to express (in a simple syntax) a single policy which manages two different async steps with potentially different error handling for each step. Prior related discussion on Polly github: comment welcome.
As such, where a sequence requires two separate async calls, the Polly team currently recommends expressing this as two separate policies, similar to the example in the end of this answer.
The particular example in your question may not work because the onRetryAsync delegate (throwing FlakyApiException) is not itself guarded by the policy. A policy only guards the execution of delegates executed through .Execute/ExecuteAsync(...).
One approach could be to use two policies, a retry policy which retries all typical http exceptions and status codes including 500s; then inside that a Polly FallbackPolicy which traps the status code 500 representing SqlDateTime overflow, and excludes that from being retried by rethrowing as some distinguishing exception (CustomSqlDateOverflowException).
IAsyncPolicy<HttpResponseMessage> rejectSqlError = Policy<HttpResponseMessage>
.HandleResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.FallbackAsync(async (delegateOutcome, context, token) =>
{
String stringContent = await delegateOutcome.Result.Content.ReadAsStringAsync(); // Could wrap this line in an additional policy as desired.
if (delegateOutcome.Result.StatusCode == HttpStatusCode.InternalServerError && stringContent.Contains("SqlDateTime overflow"))
{
throw new CustomSqlDateOverflowException(); // Replace 500 SqlDateTime overflow with something else.
}
else
{
return delegateOutcome.Result; // render all other 500s as they were
}
}, async (delegateOutcome, context) => { /* log (if desired) that InternalServerError was checked for what kind */ });
IAsyncPolicy<HttpResponseMessage> retryPolicy = Policy<HttpResponseMessage>
.Handle<HttpRequestException>()
.OrResult(r => r.StatusCode == HttpStatusCode.InternalServerError)
.OrResult(r => /* condition for any other errors you want to handle */)
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)),
async (exception, timeSpan, context) =>
{
/* log (if desired) retry being invoked */
});
HttpResponseMessage response = await retryPolicy.WrapAsync(rejectSqlError)
.ExecuteAsync(() => client.PostAsync(requestUri, new StringContent(serialisedParameters, Encoding.UTF8, "application/json"), cancellationToken));
For Http, I chose to solve this problem using DelegatingHandler (DH) pattern, and polly. There is no HandleResultAsync(), so the issue still exists for a generalized question.
With polly, I avoid a solution that has "coupling".
I've had great success with using a retry policy in a DelegatingHandler as it follows SRP, and provides a nice SoC (see this SO post). Here is the retry DH I use typically for reference.
For your question at hand, there are 2 things: retry, and conditions to retry on. Building on my retry DH, I exploded it into two DelegatingHandlers: a retry DH that retries on a "signal", and a latter retry signaling DH that signals a retry. HttpRequestMessage's .Properties (or .Options) bag is used to signal.
I find it easily maintainable, and is not complex by avoiding nested polly policies or blocking call. I have few APIs using the async request/reply pattern, so the retry DH (used for polling) is reusable (nugetized), and the retry signaling DH is different as per the API. You can obviously combine them into one by inlining the signaling code into the action arg.
HttpClient CoR (chain of responsibility):
... -> retry on signal DH -> retry signaling DH -> ...
Here is the retry signaling DH for your conditions to retry.
public class RetrySignalingOnConditionHandler : DelegatingHandler
{
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var response = await base.SendAsync(request, cancellationToken);
// tweak conditions accordingly
if (response.StatusCode == (HttpStatusCode)500)
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
var content = await response.Content.ReadAsStringAsync(cancellationToken);
if (content.Contains("Timeout"))
{
request.Properties[RequestProperties.RetrySignal] = true;
return response;
}
return response;
}
}
internal static class RequestProperties
{
internal static string RetrySignal = nameof(RetrySignal);
}
Here is the retry DH that retries on the signal. It resets the signal before the attempt.
public class ExponentialBackoffRetryOnSignalHandler : DelegatingHandler
{
private readonly IAsyncPolicy<(HttpRequestMessage request, HttpResponseMessage response)> retryPolicy;
public ExponentialBackoffRetryOnSignalHandler(
IRetrySettings retrySettings)
{
_ = retrySettings
?? throw new ArgumentNullException(nameof(retrySettings));
var sleepDurations = Backoff.ExponentialBackoff(
initialDelay: TimeSpan.FromMilliseconds(retrySettings.RetryDelayInMilliseconds),
retryCount: retrySettings.RetryCount);
retryPolicy = Policy
.HandleResult<(HttpRequestMessage request, HttpResponseMessage response)>(tuple =>
tuple.request.Properties.TryGetValue(RequestProperties.RetrySignal, out var retrySignaledObj) && (bool)retrySignaledObj)
.WaitAndRetryAsync(
sleepDurations: sleepDurations,
onRetry: (responseResult, delay, retryAttempt, context) =>
{
// note: response can be null in case of handled exception
responseResult.Result.response?.Dispose();
});
}
protected override async Task<HttpResponseMessage> SendAsync(
HttpRequestMessage request,
CancellationToken cancellationToken)
{
var tuple = await retryPolicy.ExecuteAsync(
action: async (ct) =>
{
request.Properties.Remove(RequestProperties.RetrySignal);
var response = await base.SendAsync(request, ct)
.ConfigureAwait(false);
return (request, response);
},
cancellationToken: cancellationToken)
.ConfigureAwait(false);
return tuple.response;
}
}
public interface IRetrySettings
{
int RetryCount { get; }
int RetryDelayInMilliseconds { get; }
}
Here is the full code that I use along with tests.
If I understand your question correctly then you want to retry only if the status code is 500 and the body contains Timeout. If that's the case then you can define your policy just like this
Policy<HttpResponseMessage>
.HandleResult(response =>
response.StatusCode == System.Net.HttpStatusCode.InternalServerError
&& response.Content.ReadAsStringAsync().GetAwaiter().GetResult().Contains("Timeout"))
.WaitAndRetry(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt);
UPDATE #1
Just to clarify. Even tough .GetAwaiter().GetResult() should be avoided whenever possible, here I consider it as a valid use case to utilize it:
There is no HandleResultAsync builder method, so we have to use HandleResult sync method here
First we filter for 500 status code and then we lazily evaluate the response body
I assumed the response body is fairly small due to the fact we should not expose too much information in case of Internal Server Error
I am using a static HttpClient (for scalability reasons - see What is the overhead of creating a new HttpClient per call in a WebAPI client?) and would like to be able to cancel individual requests that take too long. There is an overload on SendAsync that takes a CancellationToken - but I don't know if it's thread-safe since my HttpClient instance is static. For example, if I have several requests being sent thru the HttpClient simultaneously and I try to cancel one, does it cancel the right one?
I looked thru the HttpClient code and at first glance it doesn’t look like it is thread-safe to do so since the cancellation is sent to the HttpClientHandler (which is the same for all requests). But I could be missing something. So my questions are:
Can I cancel individual requests on a static HttpClient?
If not, how can I accomplish this?
NOTE:
Since testing this, requires a way to reliably create a race condition, in code that I do not control, I don't see a way to test this.
Each SendAsync call is totally independent from each other, canceling the token for one request does not cancel other outstanding requests.
Your assumption that because HttpClientHandler is shared for all requests that means all requests get canceled is incorrect. If you look in to the decompiled source of HttpClientHandler you will see
[__DynamicallyInvokable]
protected internal override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
if (request == null)
throw new ArgumentNullException(nameof (request), SR.net_http_handler_norequest);
this.CheckDisposed();
if (Logging.On)
Logging.Enter(Logging.Http, (object) this, nameof (SendAsync), (object) request);
this.SetOperationStarted();
TaskCompletionSource<HttpResponseMessage> completionSource = new TaskCompletionSource<HttpResponseMessage>();
HttpClientHandler.RequestState state = new HttpClientHandler.RequestState();
state.tcs = completionSource;
state.cancellationToken = cancellationToken;
state.requestMessage = request;
try
{
HttpWebRequest prepareWebRequest = this.CreateAndPrepareWebRequest(request);
state.webRequest = prepareWebRequest;
cancellationToken.Register(HttpClientHandler.onCancel, (object) prepareWebRequest);
if (ExecutionContext.IsFlowSuppressed())
{
IWebProxy webProxy = (IWebProxy) null;
if (this.useProxy)
webProxy = this.proxy ?? WebRequest.DefaultWebProxy;
if (this.UseDefaultCredentials || this.Credentials != null || webProxy != null && webProxy.Credentials != null)
this.SafeCaptureIdenity(state);
}
Task.Factory.StartNew(this.startRequest, (object) state);
}
catch (Exception ex)
{
this.HandleAsyncException(state, ex);
}
if (Logging.On)
Logging.Exit(Logging.Http, (object) this, nameof (SendAsync), (object) completionSource.Task);
return completionSource.Task;
}
The cancellation token is getting wrapped up in a new HttpClientHandler.RequestState state object every call of SendAsnyc, when the token is canceled only the state.webRequest associated with that state object is the one that will be canceled.
Just got confirmation from the Product Team at Microsoft:
Yes, it is completely safe to cancel an individual request using the
cancellation token passed into the various HttpClient.SendAsync,
.GetAsync, etc. methods. It does not matter that the HttpClient is
"static". The cancellation token passed into the method is used for
that particular request only.
I have an OWIN-based ASP.NET Web API hosted in a Windows Service. Most of my ApiController actions are async, and accept CancellationToken parameters:
[Route("data/{id}")]
public async Task<IHttpActionResult> GetSomeDataAsync(int id, CancellationToken token)
{
try
{
using (var _dataSource = ...)
{
return Ok(await _dataSource.GetDataAsync(id, token));
}
}
catch (OperationCanceledException ex)
{
return StatusCode(HttpStatusCode.NoContent);
}
}
Using the built-in request-cancellation features of Web API, if the client cancels the request, token is signaled and _dataSource handles it appropriately and throws the OperationCanceledException.
So far, so great.
But when my host process terminates (that is, the Windows Service stops), token isn't signaled and the cancellation-and-bail-out process isn't graceful.
I'm aware of the OWIN environment dictionary's host.onAppDisposing property, and I've dug into the source for the Microsoft.Owin[.*] and Microsoft.AspNet.WebApi.* packages to try and figure out where GetSomeDataAsync's token argument is coming from, but I'm not sure how to connect the pieces together.
I'd like to do something like
class WebServiceInAWindowsService : ServiceBase
{
private readonly CancellationTokenSource _cts = new CancellationTokenSource();
...
protected override void OnStop()
{
_cts.Cancel();
}
}
But I'm not sure how to get _cts to be the source of the CancellationTokens that get fed to my actions, while not breaking the request-cancellation feature that's working well.
I'm thinking that CancellationTokenSource.CreateLinkedTokenSource() might be useful, but I'm not seeing how to put the pieces together.
Can you help? Thanks!
host.onAppDisposing is triggered when you call Dispose on the value returned from WebApp.Start.
https://github.com/aspnet/AspNetKatana/blob/9f6e09af6bf203744feb5347121fe25f6eec06d8/src/Microsoft.Owin.Hosting/Engine/HostingEngine.cs#L302-L308
https://github.com/aspnet/AspNetKatana/blob/9f6e09af6bf203744feb5347121fe25f6eec06d8/src/Microsoft.Owin.Hosting/Engine/HostingEngine.cs#L112
GetSomeDataAsync's is only associated with the request disconnect token by default (e.g. owin.CallCancelled). Via middleware or otherwise you can replace it with a linked TCS that's also connected to host.onAppDisposing.
Something like:
app.Use(async (env, next) =>
{
var reqAbt = env.Get<CancellationToken>("owin.CallCancelled");
var appAbt = env.Get<CancellationToken>("host.onAppDisposing");
using (linked = CancellationTokenSource.CreateLinkedTokenSource(reqAbt, appAbt))
{
env["owin.CallCancelled"] = linked.Token;
await next();
env["owin.CallCancelled"] = reqAbt;
}
});
Could someone point me to a resource that would help explain how web api (specifically using Owin Self Host) handles request cancellation?
Here's the sequence of events i'm observing:
someone makes a GET from chrome
Api controller (through some layers) fires off an async SQL query
someone hits the X button in chrome (i don't know exactly what happens on socket for this)
What happens next in Web Api??
There was some code running in a controller, does the thread running it get aborted? If it was an async controller awaiting another Task, does that task still have an awaiter in case it returns with an exception?
For context: I do have an async controller awaiting a Task (this is the only call site) which looks to be throwing an unobserved exception in some edge cases. I haven't been able to isolate or re-produce yet :)
I did find something called HttpResponse.ClientDisconnectedToken, but don't know well that is supported in Owin Selfhost + is it even the good thing to use for all user cancels.
I've dealt with this by handing the System.OperationCanceledException in a custom middleware I've registered before WebApi.
public class ExceptionHanldingMiddleware : OwinMiddleware
{
public override async Task Invoke(IOwinContext context)
{
try
{
await Next.Invoke(context);
}
catch (OperationCanceledException) when (context.Request.CallCancelled.IsCancellationRequested)
{
//swallow user-agent cancelling request.
_log.Trace($"client disconnected on request for: {context.Request.Path}.");
}
catch (Exception ex)
{
_log.Error(ex);
context.Response.StatusCode = (int) HttpStatusCode.InternalServerError;
context.Response.ReasonPhrase = "Internal Server Error";
}
}
}
As you stated that your async controller is awaiting for a Task, which sometimes got some exception, I suggest you ContinueWith extension method for a task, which can be run only then your task is faulted, like this:
task.ContinueWith(
t =>
logger.Error(t.Exception.Message, t.Exception);
, TaskContinuationOptions.OnlyOnFaulted);
This is a default mechanism to handle the exceptions, and this will work in OWIN application.
Second, as for the cancellation: task can be started with a CancellationToken structure, which can be used for a cancelling the task during the execution. You can read more in the MSDN article.
HttpResponse.ClientDisconnectedToken is used for a situation when the client has been disconnected and the request should not be proceed in execution.
You can use this token, or create your own with CancellationTokenSource, like this:
var source = new CancellationTokenSource();
var token = source.Token;
var task = Task.Factory.StartNew(() =>
{
// Were we already canceled?
ct.ThrowIfCancellationRequested();
var moreToDo = true;
while (moreToDo)
{
// Poll on this property if you have to do
// other cleanup before throwing.
if (ct.IsCancellationRequested)
{
// Clean up here, then...
ct.ThrowIfCancellationRequested();
}
}
}, token);