I have a requirement from a client, to call their API, however, due to the throttling limit, we can only make 100 API calls in a minute. I am using SemaphoreSlim to handle that, Here is my code.
async Task<List<IRestResponse>> GetAllResponses(List<string> locationApiCalls)
{
var semaphoreSlim = new SemaphoreSlim(initialCount: 100, maxCount: 100);
var failedResponses = new ConcurrentBag<IReadOnlyCollection<IRestResponse>>();
var passedResponses = new ConcurrentBag<IReadOnlyCollection<IRestResponse>>();
var tasks = locationApiCalls.Select(async locationApiCall =>
{
await semaphoreSlim.WaitAsync();
try
{
var response = await RestApi.GetResponseAsync(locationApi);
if (response.IsSuccessful)
{
passedResponses.Add((IReadOnlyCollection<IRestResponse>)response);
}
else
{
failedResponses.Add((IReadOnlyCollection<IRestResponse>)response);
}
}
finally
{
semaphoreSlim.Release();
}
});
await Task.WhenAll(tasks);
var passedResponsesList = passedResponses.SelectMany(x => x).ToList();
}
However this line
var passedResponsesList = passedResponses.SelectMany(x => x).ToList();
never gets executed and I see Lots of failedResponses as well, I guess I have to add Task.Delay (for 1 minute) somewhere in the code as well.
You need to keep track of the time when each of the previous 100 requests was executed. In the sample implementation below, the ConcurrentQueue<TimeSpan> records the relative completion time of each of these previous 100 requests. By dequeuing the first (and hence earliest) time from this queue, you can check how much time has passed since 100 requests ago. If it's been less than a minute, then the next request needs to wait for the remainder of the minute before it can be executed.
async Task<List<IRestResponse>> GetAllResponses(List<string> locationApiCalls)
{
var semaphoreSlim = new SemaphoreSlim(initialCount: 100, maxCount: 100);
var total = 0;
var stopwatch = Stopwatch.StartNew();
var completionTimes = new ConcurrentQueue<TimeSpan>();
var failedResponses = new ConcurrentBag<IReadOnlyCollection<IRestResponse>>();
var passedResponses = new ConcurrentBag<IReadOnlyCollection<IRestResponse>>();
var tasks = locationApiCalls.Select(async locationApiCall =>
{
await semaphoreSlim.WaitAsync();
if (Interlocked.Increment(ref total) > 100)
{
completionTimes.TryDequeue(out var earliest);
var elapsed = stopwatch.Elapsed - earliest;
var delay = TimeSpan.FromSeconds(60) - elapsed;
if (delay > TimeSpan.Zero)
await Task.Delay(delay);
}
try
{
var response = await RestApi.GetResponseAsync(locationApi);
if (response.IsSuccessful)
{
passedResponses.Add((IReadOnlyCollection<IRestResponse>)response);
}
else
{
failedResponses.Add((IReadOnlyCollection<IRestResponse>)response);
}
}
finally
{
completionTimes.Enqueue(stopwatch.Elapsed);
semaphoreSlim.Release();
}
});
await Task.WhenAll(tasks);
var passedResponsesList = passedResponses.SelectMany(x => x).ToList();
}
If you're calling this method from the UI thread of a WinForms or WPF application, remember to add ConfigureAwait(false) to its await statements.
Related
I'm trying to execute multiple requests at the same time to a Pi Number API. The main problem is that despite the 'Task.WhenAll(ExecuteRequests()).Wait();' line, it isn't completing all tasks. It should execute 50 requests and add it results to pi Dictionary, but after code execution the dictionary has about 44~46 items.
I tried to add an 'availables threads at ThreadPool verification', so i could guarantee i have enough Threads, but nothing changed.
The other problem is that sometimes when I run the code, i have an error saying I'm trying to add an already added key to the dicitionary, but the lock statement wasn't supposed to guarantee this error doesn't occur?
const int TotalRequests = 50;
static int requestsCount = 0;
static Dictionary<int, string> pi = new();
static readonly object lockState = new();
static void Main(string[] args)
{
var timer = new Stopwatch();
timer.Start();
Task.WhenAll(ExecuteRequests()).Wait();
timer.Stop();
foreach (var item in pi.OrderBy(x => x.Key))
Console.Write(item.Value);
Console.WriteLine($"\n\n{timer.ElapsedMilliseconds}ms");
Console.WriteLine($"\n{pi.Count} items");
}
static List<Task> ExecuteRequests()
{
var tasks = new List<Task>();
for (int i = 0; i < TotalRequests; i++)
{
ThreadPool.GetAvailableThreads(out int workerThreads, out int completionPortThreads);
while (workerThreads < 1)
{
ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);
Thread.Sleep(100);
}
tasks.Add(Task.Run(async () =>
{
var currentRequestId = 0;
lock (lockState)
currentRequestId = requestsCount++;
var httpClient = new HttpClient();
var result = await httpClient.GetAsync($"https://api.pi.delivery/v1/pi?start={currentRequestId * 1000}&numberOfDigits=1000");
if (result.StatusCode == System.Net.HttpStatusCode.OK)
{
var json = await result.Content.ReadAsStringAsync();
var content = JsonSerializer.Deserialize<JsonObject>(json)!["content"]!.ToString();
//var content = (await JsonSerializer.DeserializeAsync<JsonObject>(new MemoryStream(System.Text.Encoding.UTF8.GetBytes(json)!)!)!)!["content"]!.ToString();
pi.Add(currentRequestId, content);
}
}));
}
return tasks;
}
There`s only one problem - you turned only one part of code, which have problem with threads:
lock (lockState)
currentRequestId = requestsCount++;
But, there`s another one:
pi.Add(currentRequestId, content);
The problem related to dictionary idea - a lot of readers and only one writer. So, you saw case with exception and if you write try catch, you will see AggregateException, which almost in every case mean thread issues, so, you need to do this:
lock (lockState)
pi.Add(currentRequestId, content);
I put a lock statement around the dicitionary manipulation as #AlexeiLevenkov mentioned and it worked fine.
tasks.Add(Task.Run(async () =>
{
var currentRequestId = 0;
lock (lockState)
currentRequestId = requestsCount++;
var httpClient = new HttpClient();
var result = await httpClient.GetAsync($"https://api.pi.delivery/v1/pi?start={currentRequestId * 1000}&numberOfDigits=1000");
if (result.StatusCode == System.Net.HttpStatusCode.OK)
{
var json = await result.Content.ReadAsStringAsync();
var content = JsonSerializer.Deserialize<JsonObject>(json)!["content"]!.ToString();
//var content = (await JsonSerializer.DeserializeAsync<JsonObject>(new MemoryStream(System.Text.Encoding.UTF8.GetBytes(json)!)!)!)!["content"]!.ToString();
lock (lockState)
pi.Add(currentRequestId, content);
}
}));
I'm not directly answering the question, just suggesting that you can use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:
static void Main(string[] args)
{
var timer = new Stopwatch();
timer.Start();
(int currentRequestId, string content)[] results = ExecuteRequests(50).ToArray().Wait()
timer.Stop();
foreach (var item in results.OrderBy(x => x.currentRequestId))
Console.Write(item.content);
Console.WriteLine($"\n\n{timer.ElapsedMilliseconds}ms");
Console.WriteLine($"\n{results.Count()} items");
}
static IObservable<(int currentRequestId, string content)> ExecuteRequests(int totalRequests) =>
Observable
.Defer(() =>
from currentRequestId in Observable.Range(0, totalRequests)
from content in Observable.Using(() => new HttpClient(), hc =>
from result in Observable.FromAsync(() => hc.GetAsync($"https://api.pi.delivery/v1/pi?start={currentRequestId * 1000}&numberOfDigits=1000"))
where result.StatusCode == System.Net.HttpStatusCode.OK
from json in Observable.FromAsync(() => result.Content.ReadAsStringAsync())
select JsonSerializer.Deserialize<JsonObject>(json)!["content"]!.ToString())
select new
{
currentRequestId,
content,
});
So I have 2 queries:
var a = await _aQuery.Execute(aId);
var b = await _bQuery.Execute(bId);
How can I retry execution of these queries with Polly Retry so that if any of them succeeds to get a non-null value, the code can proceed?
For example: 5 retries every 200ms and if query succeeds on a or on b, break the retries and proceed.
Let me alter your two operations a bit for the sake of simplicity.
static Random rnd = new();
static async Task<int?> OperationA()
{
await Task.Delay(100); //Simulate I/O call's latency
var number = rnd.Next();
return number % 5 == 0 ? number : null;
}
static async Task<int?> OperationB()
{
await Task.Delay(100); //Simulate I/O call's latency
var number = rnd.Next();
return number % 4 == 0 ? number + 1 : null;
}
We have two operations (OperationA and OperationB) and they will either return null or an int
Their implementation are not important, they just serve demonstration purposes
Now If you need to define a retry policy which
Triggers if both operations failed
Triggers at most 5 times
Delays 200 milliseconds between each attempt
then you can do that like this:
var retryUntilGetResult = Policy<(int?, int?)>
.HandleResult(((int? OpA, int? OpB) results) => !results.OpA.HasValue && !results.OpB.HasValue)
.WaitAndRetryAsync(retryCount: 5, _ => TimeSpan.FromMilliseconds(200));
The Policy<(int?, int?)> part says that your return type is a tuple with two nullable integers
The HandleResult part says that if the returned tuple (results) does not contain any integer (!...HasValue) then the policy should trigger
The WaitAndRetryAsync part says you want to decorate an async method with this policy
The policy should fire at most 5 times, but wait 200 milliseconds between each attempt
The usage of this policy looks like this:
var result = await retryUntilGetResult.ExecuteAsync(async () =>
{
Task<int?> opA = OperationA();
Task<int?> opB = OperationB();
await Task.WhenAll(opA, opB);
return (await opA, await opB);
});
We fire of two Tasks and we are waiting for both to complete
Then we return a new tuple where we pass the results of the async operations
For the sake of completeness here the full source code:
static async Task Main()
{
var retryUntilGetResult = Policy<(int?, int?)>
.HandleResult(((int? OpA, int? OpB) results) => !results.OpA.HasValue && !results.OpB.HasValue)
.WaitAndRetryAsync(retryCount: 5, _ => TimeSpan.FromMilliseconds(200));
var result = await retryUntilGetResult.ExecuteAsync(async () =>
{
Task<int?> opA = OperationA();
Task<int?> opB = OperationB();
await Task.WhenAll(opA, opB);
return (await opA, await opB);
});
Console.WriteLine(result);
}
static Random rnd = new();
static async Task<int?> OperationA()
{
await Task.Delay(100); //Simulate I/O call's latency
var number = rnd.Next();
return number % 5 == 0 ? number : null;
}
static async Task<int?> OperationB()
{
await Task.Delay(100); //Simulate I/O call's latency
var number = rnd.Next();
return number % 4 == 0 ? number + 1 : null;
}
I would like to call my API in parallel x number of times so processing can be done quickly.
I have three methods below that I have to call APIs in parallel. I am trying to understand which is the best way to perform this action.
Base Code
var client = new System.Net.Http.HttpClient();
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.BaseAddress = new Uri("https://jsonplaceholder.typicode.com");
var list = new List<int>();
var listResults = new List<string>();
for (int i = 1; i < 5; i++)
{
list.Add(i);
}
1st Method using Parallel.ForEach
Parallel.ForEach(list,new ParallelOptions() { MaxDegreeOfParallelism = 3 }, index =>
{
var response = client.GetAsync("posts/" + index).Result;
var contents = response.Content.ReadAsStringAsync().Result;
listResults.Add(contents);
Console.WriteLine(contents);
});
Console.WriteLine("After all parallel tasks are done with Parallel for each");
2nd Method with Tasks. I am not sure if this runs parallel. Let me know if it does
var loadPosts = new List<Task<string>>();
foreach(var post in list)
{
var response = await client.GetAsync("posts/" + post);
var contents = response.Content.ReadAsStringAsync();
loadPosts.Add(contents);
Console.WriteLine(contents.Result);
}
await Task.WhenAll(loadPosts);
Console.WriteLine("After all parallel tasks are done with Task When All");
3rd Method using Action Block - This is what I believe I should always do but I want to hear from community
var responses = new List<string>();
var block = new ActionBlock<int>(
async x => {
var response = await client.GetAsync("posts/" + x);
var contents = await response.Content.ReadAsStringAsync();
Console.WriteLine(contents);
responses.Add(contents);
},
new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 6, // Parallelize on all cores
});
for (int i = 1; i < 5; i++)
{
block.Post(i);
}
block.Complete();
await block.Completion;
Console.WriteLine("After all parallel tasks are done with Action block");
Approach number 2 is close. Here's a rule of thumb: I/O bound operations=> use Tasks/WhenAll (asynchrony), compute bound operations => use Parallelism. Http Requests are network I/O.
foreach (var post in list)
{
async Task<string> func()
{
var response = await client.GetAsync("posts/" + post);
return await response.Content.ReadAsStringAsync();
}
tasks.Add(func());
}
await Task.WhenAll(tasks);
var postResponses = new List<string>();
foreach (var t in tasks) {
var postResponse = await t; //t.Result would be okay too.
postResponses.Add(postResponse);
Console.WriteLine(postResponse);
}
I made a little console app to test all the Methods at pinging API "https://jsonplaceholder.typicode.com/todos/{i}" 200 times.
#MikeLimaSierra Method 1 or 3 were the fastest!
Method
DegreeOfParallelism
Time
Not Parallel
n/a
8.4 sec
#LearnAspNet (OP) Method 1
2
5.494 sec
#LearnAspNet (OP) Method 1
30
1.235 sec
#LearnAspNet (OP) Method 3
2
4.750 sec
#LearnAspNet (OP) Method 3
30
1.795 sec
#jamespconnor Method
n/a
21.5 sec
#YuliBonner Method
n/a
21.4 sec
I would use the following, it has no control of concurrency (it will dispatch all HTTP requests in parallel, unlike your 3rd Method) but it is a lot simpler - it only has a single await.
var client = new HttpClient();
var list = new[] { 1, 2, 3, 4, 5 };
var postTasks = list.Select(p => client.GetStringAsync("posts/" + p));
var posts = await Task.WhenAll(postTasks);
foreach (var postContent in posts)
{
Console.WriteLine(postContent);
}
I want to limit the total number of queries that I submit to my database server across all Dataflow blocks to 30. In the following scenario, the throttling of 30 concurrent tasks is per block so it always hits 60 concurrent tasks during execution. Obviously I could limit my parallelism to 15 per block to achieve a system wide total of 30 but this wouldn't be optimal.
How do I make this work? Do I limit (and block) my awaits using SemaphoreSlim, etc, or is there an intrinsic Dataflow approach that works better?
public class TPLTest
{
private long AsyncCount = 0;
private long MaxAsyncCount = 0;
private long TaskId = 0;
private object MetricsLock = new object();
public async Task Start()
{
ExecutionDataflowBlockOptions execOption
= new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 30 };
DataflowLinkOptions linkOption = new DataflowLinkOptions()
{ PropagateCompletion = true };
var doFirstIOWorkAsync = new TransformBlock<Data, Data>(
async data => await DoIOBoundWorkAsync(data), execOption);
var doCPUWork = new TransformBlock<Data, Data>(
data => DoCPUBoundWork(data));
var doSecondIOWorkAsync = new TransformBlock<Data, Data>(
async data => await DoIOBoundWorkAsync(data), execOption);
var doProcess = new TransformBlock<Data, string>(
i => $"Task finished, ID = : {i.TaskId}");
var doPrint = new ActionBlock<string>(
s => Debug.WriteLine(s));
doFirstIOWorkAsync.LinkTo(doCPUWork, linkOption);
doCPUWork.LinkTo(doSecondIOWorkAsync, linkOption);
doSecondIOWorkAsync.LinkTo(doProcess, linkOption);
doProcess.LinkTo(doPrint, linkOption);
int taskCount = 150;
for (int i = 0; i < taskCount; i++)
{
await doFirstIOWorkAsync.SendAsync(new Data() { Delay = 2500 });
}
doFirstIOWorkAsync.Complete();
await doPrint.Completion;
Debug.WriteLine("Max concurrent tasks: " + MaxAsyncCount.ToString());
}
private async Task<Data> DoIOBoundWorkAsync(Data data)
{
lock(MetricsLock)
{
AsyncCount++;
if (AsyncCount > MaxAsyncCount)
MaxAsyncCount = AsyncCount;
}
if (data.TaskId <= 0)
data.TaskId = Interlocked.Increment(ref TaskId);
await Task.Delay(data.Delay);
lock (MetricsLock)
AsyncCount--;
return data;
}
private Data DoCPUBoundWork(Data data)
{
data.Step = 1;
return data;
}
}
Data Class:
public class Data
{
public int Delay { get; set; }
public long TaskId { get; set; }
public int Step { get; set; }
}
Starting point:
TPLTest tpl = new TPLTest();
await tpl.Start();
Why don't you marshal everything to an action block that has the actual limitation?
var count = 0;
var ab1 = new TransformBlock<int, string>(l => $"1:{l}");
var ab2 = new TransformBlock<int, string>(l => $"2:{l}");
var doPrint = new ActionBlock<string>(
async s =>
{
var c = Interlocked.Increment(ref count);
Console.WriteLine($"{c}:{s}");
await Task.Delay(5);
Interlocked.Decrement(ref count);
},
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 15 });
ab1.LinkTo(doPrint);
ab2.LinkTo(doPrint);
for (var i = 100; i > 0; i--)
{
if (i % 3 == 0) await ab1.SendAsync(i);
if (i % 5 == 0) await ab2.SendAsync(i);
}
ab1.Complete();
ab2.Complete();
await ab1.Completion;
await ab2.Completion;
This is the solution I ended up going with (unless I can figure out how to use a single generic DataFlow block for marshalling every type of database access):
I defined a SemaphoreSlim at the class level:
private SemaphoreSlim ThrottleDatabaseQuerySemaphore = new SemaphoreSlim(30, 30);
I modified the I/O class to call a throttling class:
private async Task<Data> DoIOBoundWorkAsync(Data data)
{
if (data.TaskId <= 0)
data.TaskId = Interlocked.Increment(ref TaskId);
Task t = Task.Delay(data.Delay); ;
await ThrottleDatabaseQueryAsync(t);
return data;
}
The throttling class: (I also have a generic version of the throttling routine because I couldn't figure out how to write one routine to handle both Task and Task<TResult>)
private async Task ThrottleDatabaseQueryAsync(Task task)
{
await ThrottleDatabaseQuerySemaphore.WaitAsync();
try
{
lock (MetricsLock)
{
AsyncCount++;
if (AsyncCount > MaxAsyncCount)
MaxAsyncCount = AsyncCount;
}
await task;
}
finally
{
ThrottleDatabaseQuerySemaphore.Release();
lock (MetricsLock)
AsyncCount--;
}
}
}
The simplest solution to this problem is to configure all your blocks with a limited-concurrency TaskScheduler:
TaskScheduler scheduler = new ConcurrentExclusiveSchedulerPair(
TaskScheduler.Default, maxConcurrencyLevel: 30).ConcurrentScheduler;
ExecutionDataflowBlockOptions execOption = new()
{
TaskScheduler = scheduler,
MaxDegreeOfParallelism = scheduler.MaximumConcurrencyLevel,
};
TaskSchedulers can only limit the concurrency of work done on threads. They can't throttle asynchronous operations that are not running on threads. So in order to enforce the MaximumConcurrencyLevel policy, unfortunately you must pass synchronous delegates to all the Dataflow blocks. For example:
TransformBlock<Data, Data> doFirstIOWorkAsync = new(data =>
{
return DoIOBoundWorkAsync(data).GetAwaiter().GetResult();
}, execOption);
This change will increase the demand for ThreadPool threads, so you'd better increase the number of threads that the ThreadPool creates instantly on demand to a higher value than the default Environment.ProcessorCount:
ThreadPool.SetMinThreads(100, 100); // At the start of the program
I am proposing this solution not because it is optimal, but because it is easy to implement. My understanding is that wasting some RAM on ~30 threads that are going to be blocked most of the time, won't have any measurable negative effect on the type of application that you are working with.
I have the following code, what it does I don't believe is important, but I'm getting strange behavior.
When I run just the months on separate threads, it runs fine(how it is below), but when I multi-thread the years(uncomment the tasks), it will timeout every time. The timeout is set for 5 minutes for months/20 minutes for years and it will timeout within a minute.
Is there a known reason for this behavior? Am I missing something simple?
public List<PotentialBillingYearItem> GeneratePotentialBillingByYear()
{
var years = new List<PotentialBillingYearItem>();
//var tasks = new List<Task>();
var startYear = new DateTime(DateTime.Today.Year - 10, 1, 1);
var range = new DateRange(startYear, DateTime.Today.LastDayOfMonth());
for (var i = range.Start; i <= range.End; i = i.AddYears(1))
{
var yearDate = i;
//tasks.Add(Task.Run(() =>
//{
years.Add(new PotentialBillingYearItem
{
Total = GeneratePotentialBillingMonths(new PotentialBillingParameters { Year = yearDate.Year }).Average(s => s.Total),
Date = yearDate
});
//}));
}
//Task.WaitAll(tasks.ToArray(), TimeSpan.FromMinutes(20));
return years;
}
public List<PotentialBillingItem> GeneratePotentialBillingMonths(PotentialBillingParameters Parameters)
{
var items = new List<PotentialBillingItem>();
var tasks = new List<Task>();
var year = new DateTime(Parameters.Year, 1, 1);
var range = new DateRange(year, year.LastDayOfYear());
range.Start = range.Start == range.End ? DateTime.Now.FirstDayOfYear() : range.Start.FirstDayOfMonth();
if (range.End > DateTime.Today) range.End = DateTime.Today.LastDayOfMonth();
for (var i = range.Start; i <= range.End; i = i.AddMonths(1))
{
var firstDayOfMonth = i;
var lastDayOfMonth = i.LastDayOfMonth();
var monthRange = new DateRange(firstDayOfMonth, lastDayOfMonth);
tasks.Add(Task.Run(() =>
{
using (var db = new AlbionConnection())
{
var invoices = GetInvoices(lastDayOfMonth);
var timeslipSets = GetTimeslipSets();
var item = new PotentialBillingItem
{
Date = firstDayOfMonth,
PostedInvoices = CalculateInvoiceTotals(invoices.Where(w => w.post_date <= lastDayOfMonth), monthRange),
UnpostedInvoices = CalculateInvoiceTotals(invoices.Where(w => w.post_date == null || w.post_date > lastDayOfMonth), monthRange),
OutstandingDrafts = CalculateOutstandingDraftTotals(timeslipSets)
};
items.Add(item);
}
}));
}
Task.WaitAll(tasks.ToArray(), TimeSpan.FromMinutes(5));
return items;
}
You might consider pre-allocating a bigger number of threadpool threads. The threadpool is very slow to allocate new threads. The code below task only 10 seconds (the theoretical minimum) to run setting the minimum number of threadpool threads to 2.5k, but commenting out the SetMinThreads makes it take over 1:30 seconds.
static void Main(string[] args)
{
ThreadPool.SetMinThreads(2500, 10);
Stopwatch sw = Stopwatch.StartNew();
RunTasksOutter(10);
sw.Stop();
Console.WriteLine($"Finished in {sw.Elapsed}");
}
public static void RunTasksOutter(int num) => Task.WaitAll(Enumerable.Range(0, num).Select(x => Task.Run(() => RunTasksInner(10))).ToArray());
public static void RunTasksInner(int num) => Task.WaitAll(Enumerable.Range(0, num).Select(x => Task.Run(() => Thread.Sleep(10000))).ToArray());
You could also be running out of threadpool threads. Per: https://msdn.microsoft.com/en-us/library/0ka9477y(v=vs.110).aspx one of the times to not use the threadpool (which is used by tasks) is:
You have tasks that cause the thread to block for long periods of time. The thread pool has a maximum number of threads, so a large number of blocked thread pool threads might prevent tasks from starting.
Since IO is being done on these threads maybe consider replacing them with async code or starting them with the LongRunning option? https://msdn.microsoft.com/en-us/library/system.threading.tasks.taskcreationoptions(v=vs.110).aspx