I am developing an integration solution that accesses a rate limited API. I am performing a variety of CRUD operations on the API using multiple HTTP verbs on different endpoints (on the same server though). I have been pointed towards Polly multiple times, but I haven't managed to come up with a solution that actually works.
This is what I have in my startup:
builder.Services
.AddHttpClient("APIClient", client =>
{
client.BaseAddress = new Uri(C.Configuration.GetValue<string>("APIBaseAddress"));
})
.AddTransientHttpErrorPolicy(builder =>
builder.WaitAndRetryAsync(new []
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(5),
TimeSpan.FromSeconds(15),
}));
This is just resilience to retry in case of failure. I have a RateLimit policy in a singleton ApiWrapper class:
public sealed class ApiWrapper
{
private static readonly Lazy<ApiWrapper> lazy = new Lazy<ApiWrapper>(() => new ApiWrapper());
public static ApiWrapper Instance { get { return lazy.Value; } }
private IHttpClientFactory _httpClientFactory;
public readonly AsyncRateLimitPolicy RateLimit = Policy.RateLimitAsync(150, TimeSpan.FromSeconds(10), 50); // 150 actions within 10 sec, 50 burst
private ApiWrapper()
{
}
public void SetFactory(IHttpClientFactory httpClientFactory)
{
_httpClientFactory = httpClientFactory;
}
public HttpClient GetApiClient()
{
return _httpClientFactory.CreateClient("APIClient");
}
}
That policy is used in multiple other classes like this:
public class ApiConsumer
{
private HttpClient _httpClient = ApiWrapper.Instance.GetApiClient();
public async Task<bool> DoSomethingWithA(List<int> customerIDs)
{
foreach (int id in customerIDs)
{
HttpResponseMessage httpResponse = await ApiWrapper.Instance.RateLimit.ExecuteAsync(() => _httpClient.GetAsync($"http://some.endpoint"));
}
}
}
My expectation was that the rate limiter would not fire more requests than configured, but that does not seem to be true. From my understanding the way it works is that the rate limiter just throws an exception if there are more calls than the limit that has been configured. That's where I thought the Retry policy would come into play, so just try again after 5 or 15 seconds if it did not go through the limiter.
Then I played around a bit with Polly's Bulkhead policy, but as far as I can see that is meant to limit the amount of parallel executions.
I have multiple threads that may use different HttpClients (all created by the Factory like in the example above) with different methods and endpoints, but all use the same policies. Some threads run in parallel, some sequentially as I have to wait for their response before sending the next requests.
Any suggestions on how this can or should be achieved with Polly? (Or any other extension if there is good reason to)
Thanks again to #Neil and #Panagiotis for pointing me in the right direction. I wrongly assumed that the Polly rate limiter would actually delay API calls. I found a workaround that probably is not particularly nice, but for my purpose it does the trick.
I installed David Desmaisons RateLimiter package which is super simple to use. In my singleton I have now this:
public TimeLimiter RateLimiter = TimeLimiter.GetFromMaxCountByInterval(150, TimeSpan.FromSeconds(10));
I use this RateLimiter everywhere I make calls to an API endpoint like this:
HttpResponseMessage httpResponse = await ApiWrapper.Instance.RateLimiter.Enqueue(() => _httpClient.GetAsync($"http://some.endpoint"), _cancellationToken);
Does exactly what I originally expected from Polly.
In this post I would like to clarify things around rate limiter and rate gate
Similarity
Both concepts can be used to throttle requests.
They sit between the clients and the server and they know about the server's capacity.
Difference
The limiter as its name implies limits the transient traffic. It short-cuts the requests if there are too many.
The gate on the other hand holds/delays the requests until there is enough capacity.
Algorithms
The rate limiter usually implements the leaky-bucket or token-bucket algorithm
Polly's ratelimiter implements token bucket
The rate gate usually utilizies some queuing mechanism and timers
Sample implementation
Related
I have the following endpoint:
[HttpPost("Submit")]
public String post()
{
_ = _service.SubmitMetric("test", MetricType.Count, 60, 1);
return "done";
}
And the service implementation:
public Task<HttpResponseMessage> SubmitMetric(<params>)
{
// build payload
using (var httpClient = new HttpClient())
{
return httpClient.PostAsync(<params>);
}
}
When I run the code and call the endpoint, the HTTP POST is not triggered. However, if I change my code to:
public async Task<HttpResponseMessage> SubmitMetric(<params>)
{
// build payload
using (var httpClient = new HttpClient())
{
return await httpClient.PostAsync(<params>);
}
}
the POST is submitted as expected. Why is that happening, and what can I do if I don't really care about the HTTP response? I just want to submit it and continue my flow. Shouldn't I be able to use it without awaiting the result? For example:
public void SubmitMetric(<params>)
{
// build payload
using (var httpClient = new HttpClient())
{
httpClient.PostAsync(<params>);
}
}
There are two problems with this code. If either was fixed, there would be no problem:
The HttpClient is used incorrectly. An HttpClient object is thread-safe and meant to be reused, not disposed. Disposing it like this leaks sockets and can result in application crashes or worse, instability. An HttpClient resolves the URL's Host to a socket and caches that socket. The OS also caches opened sockets because opening them is expensive. They're kept alive for a while even if an application closes them because some packets may still be in transit
By not awaiting PostAsync execution exits the using block and the HttpClient instance is disposed before the request had a chance to even start.
In any case, making a POST doesn't take long so there's no need to make the method fire-and-forget. Besides, few applications are OK with losing metrics, especially when things go wrong. That's when metrics are most useful.
Which is why ASP.NET Core 6 adds built-in support for OpenTelemetry tracing and metrics. More on that at the end, but the supporting packages can be used in ASP.NET Framework as well. You may be able to replace your current service with a built-in one.
Use await - not enough
One way to fix this is to use await but that doesn't solve the HttpClient usage problem.
public async Task<HttpResponseMessage> SubmitMetric(<params>)
{
// build payload
using (var httpClient = new HttpClient())
{
return await httpClient.PostAsync(<params>);
}
}
At the very least the HttpClient should be stored in a field. Once that's done though, there's no longer any reason to await, provided the service itself is still around :
HttpClient httpClient = new HttpClient();
public Task<HttpResponseMessage> SubmitMetric(<params>)
{
return httpClient.PostAsync(<params>);
}
Long lived services
Which brings us to keeping the service around. In ASP.NET and ASP.NET Core each request is served by a separate thread, in a new instance of the Controller class. The request itself is used as a GC scope so anything created during a request is disposed once this concludes, including the HttpClient instance.
To keep the Metrics service around we need to either register it as Singleton in ASP.NET Core's DI, make it a BackgroundService or ensure it's a singleton in ASP.NET Framework. We could make the field static, but that leads to the next issue.
Proper HttpClient usage
HttpClient can still cause problems if used as a singleton. The HttpClient caches sockets to specific machines. If that machine goes away, the HttpClient will still try to communicate with it causing errors. This can happen easily when the remote services uses a load balancer or fails over to a new server. To fix this, the HttpClient instance or rather the sockets, need to be recycled periodically.
That's the job of the HttpClientFactory. This class caches and recycles SocketClientHandler instances, the classes that do the actual work in an HttpClient. These are recycled periodically, eg every 10 minutes. When asked for a new HttpClient instance, it creates a new instance wrapping one of the already available handlers.
When you use services.AddHttpClient in ASP.NET Core you're actually configuring an HttpClientFactory. When you add an HttpClient dependency in a controller, the instance will be created by the configured HttpClientFactory.
This means that the following action would work properly :
HttpClient _client;
public MyController(HttpClient client)
{
_client=client;
}
[HttpPost("Submit")]
public String post()
{
await _client.PostAsync(<params>);
return "done";
}
A scoped service with an HttpClient dependency would also work:
MyService _service;
public MyController(MyService service)
{
_service=service;
}
HttpPost("Submit")]
public String post()
{
await _service.SubmitMetric("test", MetricType.Count, 60, 1);
return "done";
}
where MyService is :
class MyService
{
HttpClient _client;
public MyService(HttpClient client)
{
_client=client;
}
public Task<HttpResponseMessage> SubmitMetric(<params>)
{
// build payload
return httpClient.PostAsync(<params>);
}
}
In this case there's no real need to await inside SubmitMetric, that's taken care of by the action.
Using the built-in OpenTelemetry tracing and metrics
ASP.NET Core 6, the upcoming Long-Term-Support version, adds native support for the OpenTelemetry standard for logging, tracing and metrics. This allows using a standard API to push metrics to a lot of different observability applications like Prometheus, Jaeger, Zipking, Elastic and Splunk.
Instead of rolling one's own metrics infrastructure it's better to use the standard API. OpenTelemetry for .NET supports this in ASP.NET Framework 4.6 and later. ASP.NET Core 5 and later are instrumented to publish metrics and tracing to OpenTelemetry providers through the built-in System.Diagnostics namespace and the Activity class.
In fact, Controller is already instrumented so you could get rid of the metrics service, adding any Tags and Baggage to the request's current activity:
[HttpPost("Submit")]
public String post()
{
Activity.Current?.AddTag("test");
...
return "done";
}
Metrics were added in ASP.NET Core 6 Preview 5:
Meter meter = new Meter("my.library.meter.name", "v1.0");
Counter<int> _counter;
public MyController(...)
{
_counter = meter.CreateCounter<int>("Requests");
}
[HttpPost("Submit")]
public String post()
{
counter.Add(60, KeyValuePair.Create<string, object>("request", "test"));
return "done";
}
Don't do it. Await for it even though you discard the result.
Fire and forget is an anti pattern and the context that you are performing the request can be invalidated/killed before the request could be completed, terminating the connection. Just await it, and don't do anything with the result.
httpClient will be disposed while the POST operation is running, probably resulting in killing the socket. If you use await, the object will remain inside the using clause while the operation is running, and it won't be terminated before it finishes.
Note that in your current implementation, you're creating a new connection on each API request, which might eventually lead to socket exhaustion. A better approach would be injecting IHttpClientFactory, which manage the lifetime of network connections for you, and reuses connections from the pool:
public class MyService
{
private readonly IHttpClientFactory _httpClient;
public MyService(IHttpClientFactory httpClient)
{
_httpClient = httpClient;
}
public async Task<HttpResponseMessage> SubmitMetric(/*<params>*/)
{
var httpClient = _httpClient.CreateClient();
return await httpClient.PostAsync(/*<params>*/);
}
}
Note: You need to add services.AddHttpClient() in ConfigureServices in your Startup.cs to enable injection.
I need to create an REST API that connect to a third party SOAP API. The third party API events are sent by callback to an URL I provide.
The typical steps my API go through is it starts a session with the third party by providing an ID and an callback URL. The third party can now send new events to my API through this URL when, for example, a new participant connects. Now sometimes i need to request specific info, like the list of participants for a given session(ID), and wait for the event containing the info.
Note that there may be multiple open sessions at the same time.
An example of what I need:
private string url = "http://myapi/callback";
[HttpGet]
[Route("createSession")]
public async Task<string> CreateSession()
{
var id = Guid.NewGuid().ToString();
var result = await ExternAPI.CreateSession(id, this.url);
return result; //contains the id
}
[HttpGet]
[Route("endSession")]
public async Task<string> EndSession([FromUri] string id)
{
var result = await ExternAPI.EndSession(id);
return result;
}
[HttpGet]
[Route("partipants")]
public async Task<string> Partipants([FromUri] string id)
{
ExternAPI.participants(id); // The results of this method will be sent to the callback function
results = // Wait for the results for this id
return results;
}
[HttpPost]
[Route("callback")]
public void Callback(body)
{
// notify waiting function and pass body
}
I came up with a solution using ReactiveX but I'm not really sure about its reliability in production. What I have in mind is to create a subject that never terminate and handle all the events but it is not a usual lifetime for a subject, what happens on error ? And I don't think I did it the "RX-way" (state concerns).
Here it is (you will need System.Reactive to run this code):
class Data
{
public int id;
public string value;
}
class Program
{
private static Subject<Data> sub;
static void Main(string[] args)
{
sub = new Subject<Data>();
Task.Run(async () => {
int id = 1;
ExternAPI(CallBackHook, id);
Data result = await sub.Where(data => data.id == id).FirstAsync();
Console.WriteLine("{0}", result.value);
});
Console.ReadLine();
}
static void CallBackHook(Data data)
{
sub.OnNext(data);
}
static String ExternAPI(Action<Data> callback, int id)
{
// Third-party API, access via SOAP. callback is normally an url (string)
Task.Run(() =>
{
Thread.Sleep(1000);
callback(new Data { id = id, value = "test" });
});
return "success";
}
}
An other way will be a dictionary of subjects, one for each session, so I could manage their lifetimes.
it is not a usual lifetime for a subject
what happens on error?
And I don't think I did it the "RX-way"
Yes, these are all perfectly valid concerns with this kind of approach. Personally, I don't much mind the last one, because even though Subjects are frowned-upon, many times they're just plain easier to use than the proper Rx way. With the learning curve of Rx what it is, I tend to optimize for developer maintainability, so I do "cheat" and use Subjects unless the alternative is equally understandable.
Regarding lifetime and errors, the solutions there depend on how you want your application to behave.
For lifetime, it looks like currently you have a WebAPI resource (the SOAP connection) requiring an explicit disconnect call from your client; this raises some red flags. At the very least, you'd want some kind of timeout there where that resource is disposed even if endSession is never called. Otherwise, it'll be all too easy to end up with dangling resources.
Also for errors, you'll need to decide the appropriate approach. You could "cache" the error and report it to each call that tries to use that resource, and "clear" the error when endSession is called. Or, if it's more appropriate, you could let an error take down your ASP.NET process. (ASP.NET will restart a new one for you).
To delay an API until you get some other event, use TaskCompletionSource<T>. When starting the SOAP call (e.g., ExternAPI.participants), you should create a new TCS<T>. The API call should then await the TaskCompletionSource<T>.Task. When the SOAP service responds with an event, it should take that TaskCompletionSource<T> and complete it. Points of note:
If you have multiple SOAP calls that are expecting responses over the same event, you'll need a collection of TaskCompletionSource<T> instances, along with some kind of message-identifier to match up which events are for which calls.
Be sure to watch your thread safety. Incoming SOAP events are most likely arriving on the thread pool, with (possibly multiple) API requests on other thread pool threads. TaskCompletionSource<T> itself is threadsafe, but you'd need to make your collection threadsafe as well.
You may want to write a Task-based wrapper for your SOAP service first (handling all the TaskCompletionSource<T> stuff), and then consume that from your WebAPI.
As a very broad alternative, instead of bridging SOAP with WebAPI, I would consider bridging SOAP with SignalR. You may find that this is a more natural translation. Among other things, SignalR will give you client-connect and client-disconnect events (complete with built-in timeouts for clients). So that may solve your lifetime issues more naturally. You can use the same Task-based wrapper for your SOAP service as well, or just expose the SOAP events directly as SignalR messages.
I'm pretty new to working with HTTP stuff so I'm rather confused as to what would be the best approach to request data from a HTTP address every few seconds or so.
The API I'm using has - at least to my knowledge no webhook support. So I imagine the way to update my data would be a rather crude way of doing so.
I want this to happen in the background so the GUI does not freeze and become unresponsive. So I know I (probably) need to fiddle with threads.
Best results I've had has been with a async/await Timer. I'm not entirely sure how to work with this and the only way for me to get it to work is to throw an exception after it has elapsed. If I don't - it says that not all nodes return a value and I can't even use return which really, really confuses me.
How should I be doing this?
If it's of any use, I'm working on creating my own RCON tool for a game which has all kinds of server data available via a HTTP API - but documentation for this API is very lackluster.
if you go to .net core you can see my previous answer on: Start multiple background threads inside Self Hosted ASP.NET Core Microservice
for .net framework you have to do a little more yourself. But still very do-able!
in your global.asax you have (or should I say: should) have your dependency injection. Something like:
protected void Application_Start()
{
Bootstrap();
//and do something more
}
private static void Bootstrap()
{
var container = new Container();
container.Register(() => new HttpClient());
container.RegisterSingleton<IApiCaller, ApiCaller>();
container.RegisterInitializer<IApiCaller>(ApiCaller=> apicaller.StartCallingAsync());
// Suppress warnings for HttpClient
var registration = container.GetRegistration(typeof(HttpClient)).Registration;
registration.SuppressDiagnosticWarning(DiagnosticType.DisposableTransientComponent, "Dispose is being called by code.");
registration.SuppressDiagnosticWarning(DiagnosticType.LifestyleMismatch, "Every HttpCient is unique for each dependency.");
container.Verify();
GlobalConfiguration.Configuration.DependencyResolver = new SimpleInjectorWebApiDependencyResolver(container);
}
In this case, I let SimpleInjector start my background thread to do a lot of work.
In the apicaller you can do your httpcalls.
something like:
public async Task StartCallingAsync(CancellationToken cancellationToken = (default)CancellationToken)
{
while(true)
{
var response = await _httpClient.GetAsync(url);
if (response.IsSuccessStatusCode)
{
//do work
}
await Task.Delay(10000, cancellationToken);
}
}
for the GetAsync there are extension methods that can cast it directly to your object.
can you work with this?
After reading the posts below about recommended usage of HttpClient, I changed my code from instantiating HttpClient per request within a using block to a long-lived object.
Do HttpClient and HttpClientHandler have to be disposed?
What is the overhead of creating a new HttpClient per call in a WebAPI client?
My implementation is part of a low-level api, and would be to make requests from from different parts of the app running on different threads, so thread-safety and concurrency when making requests needs to be guaranteed as well.
I even went on to make it a singleton as below, so there is just one instance of HttpClient used throughout the app. (fourth version form John SKeet's article)
http://csharpindepth.com/Articles/General/Singleton.aspx
public sealed class MyHttpClient
{
private static readonly volatile HttpClient _myHttpClient = new HttpClient();
static MyHttpClient() {}
private MyHttpClient(){ }
public static HttpClient MyHttpClientObj
{
get
{
return _myHttpClient;
}
}
}
And below is an example of how this gets used
public IEnumerable<string> GetSomeData(string url, FormUrlEncodedContent bodyParameters)
{
try
{
//is it possible to configure timeout here instead, such that every request will have its one timeout duration?
var response = MyHttpClient.MyHttpClientObj.PostAsync(url, bodyParameters);
var result = response.Result;
if (!result.IsSuccessStatusCode)
{
//log and return null
}
var data = JsonConvert.DeserializeObject<List<string>>(result.Content.ReadAsStringAsync().Result);
return data;
}
catch (Exception ex)
{
//logging exceptions
}
}
When making requests via HttpClient, I've made sure to use only the therad-safe methods listed below, but when deserializing the response, result.Content.ReadAsStringAsync().Result is used. This is because the higher level calls don't support async responses yet.
https://msdn.microsoft.com/en-us/library/system.net.http.httpclient(v=vs.110).aspx#Anchor_5
However, I still have a few questions.
Is this approach Thread-safe and stable enough to not cause any memory leaks?
How can I configure the Timeout for every request?
Is it necessary to specify 'Connection: keep-alive' in DefaultHeaders?
How can I add a custom header/modify a default header for every request?
And finally, Are there any known performance issues/drawbacks in using HttpClient this way?
It's thread safe, and recommended.
Depending on your usage, the biggest thing may be to raise the connection limit to your desired level of concurrency:
ServicePointManager.DefaultConnectionLimit = 16;
Without this set, concurrent requests to a single host will sit in a queue until they can be issued. And they'll time out if they're not got to in time.
I'd also recommend using pipelining to improve performance:
new HttpClient(new WebRequestHandler() { AllowPipelining = true });
Yes, this approach is thread-safe, as you call a thread-safe methods and do not you any synchronization logic, so your client threads simply independent from each other.
You can use an overload with CancellationTokenSource, with calling it method CancelAfter, this approach is recommended by MSDN.
No, but if your connection do require some interaction between client and server, it is highly recommended approach for HTTP/1.1, it reduces the overhead to recreating the socket connection and some handshakes between participating sides.
You can use the Headers property of the FormUrlEncodedContent class, simply add a header you need to it.
The huge drawback for your solution is .Result call, as it blocks the current thread. You can try to refactor your approach with TaskCompletionSource usage so you could possibly use async methods internally. This will provide you a possibility for the threads to do something else rather than wait for result.
The biggest performance issue you will encounter using HttpClient in a highly concurrent environment is the number of concurrent connections to any given url is limited to 2 by default. You can increase this for all endpoints using ServicePointManager.DefaultConnectionLimit or get a specific ServicePoint using ServicePointManager.FindServicePoint and set ServicePoint.ConnectionLimit
I have been tasked with maintaining an C# web service that pretty much boils down to the following code that is used in conjunction with IIS 8:
public class Handler1 : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
// here we process the request and return it after fetching some data
}
}
Now I need to create another web service, which in turn will request another web service for fetching data. What I've gathered is that the HttpClient() in C# is the new shiny, however finding out how to implement it in my case is not so easy (or if that would be a good solution).
My solution would something like
public class Handler1 : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
// make sure that the incoming request is valid
HttpClient client = new HttpClient();
// do stuff with ^client and request the other service
[..]
// return the data from the response from ^client to the first request
}
}
Am I on the right track or would this be a catastrophe? I would gladly take tips or pointers to relevant documentation.
There are two issues with your approach.
First, as others have said, HttpClient must not be created for each request, as this would result in your server creating one new TCP connection for each request and not dispose it from your connection pool until they reach their MaxIdleTimeout.
This is because each instance of HttpClient (or rather, each instance of HttpClientHandler) creates a unique ConnectionGroup within the ConnectionPool, so two HttpClient (created in such a way) will not share the same connections.
You can quite easily exhaust your ports (because connections are kept alive for quite some time), and even without that, you'll have all the performance issues you might find when deactivating keepalive (typically when connecting on a HTTPs server).
As for a solution, you might add a private readonly HttpClient field in your handler that you would initialize as you've done in your code (I believe HttpHandlers are instanciated once by IIS, not per-request, as long as it says it's reusable). (Edit: note that HttpClient instances are thread-safe).
The other problem you'll find is that IHttpHandler are synchronous by design. This means you'll have to return task.Result in your code. This will block the current thread (let's call it A), and all subsequent processing of the outbound request (sending the body of the request, reading the header of the response and reading the body of the response) will spawn use a thread from the thread pool to process this (let's call them B, C, and D, though they can all be the same thread since no step overlap in time).
But in Asp.net, you cannot safely do that. When your main thread A is processing an Asp.net inbound request, it will, simply put, lock the current context. And when threads B, C and D will be spawn to process the outbound request, they too, will try to acquire and lock the current context. But because your main thread A is blocked, the context has not been released. This will effectively result in deadlocks (A locks the context and wait for B/C/D, B/C/D wait for the context).
To work around this, I would advise you to use instead a IHttpAsyncHandler. Async process are not made to be used synchronous, and HttpClient can only be used in an async way.
However, this might prove much more challenging for a first usage scenario. It would be much more easier to use a HttpClient from the context of a WebApi server. However, without any knowledge as to why you have to use a HttpHandler, I do not know if this would be in the realm of acceptable answers.
There are other way to run asynchronous tasks and wait synchronously for them (such as changing temporary the CurrentContext), but they're all very dangerous.
As a side note, HttpClient is the new shiny thing, and works quite well with other new shiny things such as "all-the-way async infrastructure". But trying to use it HttpClient in an older infrastructure is not as easy.
Better solution example:
public class YourApp
{
private static HttpClient Client = new HttpClient();
public static void Main(string[] args)
{
Console.WriteLine("Requests about to start!");
for(int i = 0; i < 5; i++)
{
var result = Client.GetAsync("http://www.stackoverflow.com").Result;
Console.WriteLine(result);
}
Console.WriteLine("Requests are finished!");
Console.ReadLine();
}
}