We Keep Coding

Issue With HttpClient Bulk Parallel Request in .Net Core C#

So I have been struggling with this issue for like 3 weeks. Here's what I want to do.
So I have like 2000 stock options. I want to fetch 5 of them at a time and process but it all has to be parallel. I'll write them in steps to make it more clear.
Get 5 stock symbols from an array
Send it to fetch its data and process. Don't wait for a response keep on processing.
wait 2.6 seconds (as we are limited to 120 API requests per minute so this delay helps in keeping it throttled to 115 per minute)
Goto 1
All the steps above have to be parallel. I have written the code for it and it all seems to be working fine but randomly it crashes saying
"A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection
failed because connected host has failed to respond".
And sometimes it'll never happen and everything works like a charm.
This error is very random. It could show up on maybe 57th stock or maybe at 1829th stock. I have used HttpClient for it. I have tested this same scenario using Angular and creating custom requests and it never crashes there so it's not third-party server's fault.
What I have already done:
Changed HttpClient class usage from new instances every time to a single instance for the whole project.
Increases Service point manager Connection limit to a different number. (Default for .net core is 2)
Instead of HttpClient Queuing I have used SemaphoreSlim for queue and short-circuiting.
Forced ConnectionLeaseTimeout to 40 seconds to detect DNS changes if any.
Changed Async Tasks to threading.
Tried almost everything from the internet.
My doubts:
I doubt that it has something to do with the HttpClient class. I have read a lot of bad things about its misleading documentation etc.
My friend's doubt:
He said it could be because of concurrent tasks and I should change it to threads.
Here's the code:
// Inside Class Constructor
private readonly HttpClient HttpClient = new HttpClient();
SetMaxConcurrency(ApiBaseUrl, maxConcurrentRequests);
// SetMaxConcurrency function
private void SetMaxConcurrency(string url, int maxConcurrentRequests)
{
ServicePointManager.FindServicePoint(new Uri(url)).ConnectionLimit = maxConcurrentRequests;
ServicePointManager.FindServicePoint(new Uri(url)).ConnectionLeaseTimeout = 40*1000;
}
// code for looping through chunks of symbol each chunk has 5 symbols/stocks in it
foreach(var chunkedSymbol in chunkedSymbols)
{
//getting o auth token
string AuthToken = await OAuth();
if(String.IsNullOrEmpty(AuthToken))
{
throw new ArgumentNullException("Access Token is null!");
}
processingSymbols += chunkSize;
OptionChainReq.symbol = chunkedSymbol.ToArray();
async Task func()
{
//function that makes request
var response = await GetOptionChain(AuthToken, ClientId, OptionChainReq);
// concat the result in main list
appResponses = appResponses.Concat(response).ToList();
}
// if request reaches 115 process the remaning requests first
if(processingSymbols >= 115)
{
await Task.WhenAll(tasks);
processingSymbols = 0;
}
tasks.Add(func());
// 2600 millisecond delay to wait for all the data to process
await Task.Delay(delay);
}
//once the loop is completed process the remaining requests
await Task.WhenAll(tasks);
// This code processes every symbol. this code is inside GetOptionChain()
try{
var tasks = new List<Task>();
foreach (string symbol in OptionChainReq.symbol)
{
List<KeyValuePair<string, string>> Params = new List<KeyValuePair<string, string>>();
string requestParams = string.Empty;
// Converting Request Params to Key Value Pair.
Params.Add(new KeyValuePair<string, string>("apikey" , ClientId));
// URL Request Query parameters.
requestParams = new FormUrlEncodedContent(Params).ReadAsStringAsync().Result;
string endpoint = ApiBaseUrl + "/marketdata/chains?";
HttpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", OAuthToken);
Uri tosUri = new Uri(endpoint + requestParams, UriKind.Absolute);
async Task func()
{
try{
string responseString = await GetTosData(tosUri);
OptionChainResponse OptionChainRes = JsonConvert.DeserializeObject<OptionChainResponse>(responseString);
var mappedOptionAppRes = MapOptionsAppRes( OptionChainRes );
if(mappedOptionAppRes != null)
{
OptionsData.Add( mappedOptionAppRes );
}
}
catch(Exception ex)
{
throw new Exception("Crashed");
}
}
// asyncronusly processing each request
tasks.Add(func());
}
//making sure all 5 requests are processed
await Task.WhenAll(tasks);
}
catch (Exception ex)
{
failedSymbols += " "+ string.Join(",", OptionChainReq.symbol);
}
// The code below is for individual request
public async Task<string> GetTosData(Uri url)
{
try
{
await semaphore.WaitAsync();
if (IsTripped())
{
return UNAVAILABLE;
}
var response = await HttpClient.GetAsync(url);
if(response.StatusCode == System.Net.HttpStatusCode.Unauthorized)
{
string OAuthToken = await OAuth();
HttpClient.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", OAuthToken);
return await GetTosData(url);
}
else if(response.StatusCode != HttpStatusCode.OK)
{
TripCircuit(reason: $"Status not OK. Status={response.StatusCode}");
return UNAVAILABLE;
}
return await response.Content.ReadAsStringAsync();
}
catch(Exception ex) when (ex is OperationCanceledException || ex is TaskCanceledException)
{
Console.WriteLine("Timed out");
TripCircuit(reason: $"Timed out");
return UNAVAILABLE;
}
finally
{
semaphore.Release();
}
}

C# Asychronous REST query not working: WaitAll or WhenAll combined with .Result hangs up

I am building a C# Winforms application and I have many REST calls to process. Each call takes about 10 sec till I receive an answer, so in the end, my application is running quite a while. Mostly spending time waiting for the REST service to answer.
I am not coming forward because no matter what I try (configureAwait, waitAll or whenAll), the application hangs or when I want to access each tasks result, it is going back to the Main methods or hangs. Here is what I currently have:
I am building up a list of tasks to fill my objects :
List<Task> days = new List<Task>();
for (DateTime d = dtStart; d <= dtEnd; d = d.AddDays(1))
{
if (UseProduct)
{
Task _t = AsyncBuildDay(d, Project, Product, fixVersion);
var t = _t as Task<Day>;
days.Add(t);
}
else
{
Task _t = AsyncBuildDay(d, Project, fixVersion);
var t = _t as Task<Day>;
days.Add(t);
}
}
Then I am starting and waiting until every task is finished and the objects are built:
Task.WaitAll(days.ToArray());
When I try this, then the tasks are waiting for activation:
var tks = Task.WhenAll(days.ToArray());
What is running asynchronously inside the tasks (AsyncBuildDay
) is a query to JIRA:
private async Task<string> GetResponse(string url)
{
WebRequest request = WebRequest.Create(url);
request.Method = "GET";
request.Headers["Authorization"] = "Basic " + Convert.ToBase64String(Encoding.Default.GetBytes(JIRAUser + ":" + JIRAPassword));
request.Credentials = new NetworkCredential(JIRAUser, JIRAPassword);
WebResponse response = await request.GetResponseAsync().ConfigureAwait(false);
// Get the stream containing all content returned by the requested server.
Stream dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content fully up to the end.
string json = reader.ReadToEnd();
return json;
}
And now I would like to access all my objects with .Result, but then the whole code freezes again.
foreach (Task<Day> t in days)
{
dc.colDays.Add(t.Result);
}
I don't find a wait to get to my objects and I'm really going nuts with this stuff. Any ideas are much appreciated!

You're overcomplicating this.
Task.WhenAll is the way to go; it returns a new Task that completes when the provided tasks have all completed.
It's also non-blocking.
By awaiting the Task returned by Task.WhenAll, you unwrap it's results into an array:
List<Task<Day>> dayTasks = new();
// ...
Day[] days = await Task.WhenAll(dayTasks);
You can then add this to dc.colDays:
dc.colDays.AddRange(days);
Or if dc.colDays doesnt have an AddRange method:
foreach (var day in days) dc.colDays.Add(day);

It might be better to await any completion, and remove the completed task from the list.
while (days.Count > 0)
{
Task completedTask = await Task.WhenAny(days);
// Do something with result.
days.Remove(completedTask);
}

async method not run asynchronously

I am writing below code in WPF. However, it does not run asynchronously. I have commented the line where the code gets blocked. Please let me know where am I doing the mistake.
private async void txtSampleRequest_Click(object sender, RoutedEventArgs e)
{
await DownloadData();
}
async Task<string> DownloadData()
{
string authorization = txtAuthentication.Text;
string cookie = txtCookie.Text;
try
{
var vvv = Enumerable.Range(0, 50);
List<Task> TaskList = new List<Task>();
foreach (int s in vvv)
{
Task LastTask = ZerodhaOperations.MyHttpRequest("", authorization, cookie, "5minute");//Even this method is async method, and I have added dummy Task.Delay(1) to run method asynchronously. Below is the full definition of this class.
TaskList.Add(LastTask);
}
await Task.WhenAll(TaskList); //<------------ Here it stops working asynchronously and blocks UI of application.
MessageBox.Show("DONE");
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
return "";
}
class ZerodhaOperations
{
public static async Task<string> MyHttpRequest(string WEBSERVICE_URL,string authorization, string cookie,string timeFrame)
{
await Task.Delay(1);
if (authorization == "" || cookie == "" || authorization == null || cookie == null)
{
throw new ArgumentException("Aditya Says, No Authorization and cookie has been set");
}
string jsonResponse = "";
WEBSERVICE_URL = "https://kite.zerodha.com/oms/instruments/historical/519937/timeFrame?user_id=YD0744&oi=1&from=2020-04-24&to=2020-04-24";
WEBSERVICE_URL = $"https://kite.zerodha.com/oms/instruments/historical/806401/{timeFrame}?user_id=YD0744&oi=1&from=2020-12-03&to=2020-12-03";
try
{
var webRequest = System.Net.WebRequest.Create(WEBSERVICE_URL);
//ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3;
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls
| SecurityProtocolType.Tls11
| SecurityProtocolType.Tls12
| SecurityProtocolType.Ssl3;
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
if (webRequest != null)
{
webRequest.Method = "GET";
webRequest.Timeout = 5000;
if (authorization != "")
{
webRequest.Headers.Add("authority", "kite.zerodha.com");
webRequest.Headers.Add("authorization", authorization);
}
webRequest.Headers.Add("cookie", cookie);
webRequest.Headers.Add("method", "GET");
using (System.IO.Stream s = webRequest.GetResponse().GetResponseStream())
{
using (System.IO.StreamReader sr = new System.IO.StreamReader(s))
{
jsonResponse = sr.ReadToEnd();
if (jsonResponse == "")
throw new Exception("Empty Response");
//MessageBox.Show(jsonResponse);
}
}
}
}
catch (Exception ex)
{
if (ex.Message.Contains("Empty Response"))
throw ex;
else
MessageBox.Show(ex.ToString());
}
return jsonResponse;
}
}

The way await works is that it captures a "context" when it asynchronously yields. The details are in the link, but in this case the asynchronous method is running on the WPF UI thread, so it captures a context that will resume running on that UI thread.
So the await Task.Delay(1) is in fact forcing MyHttpRequest to act asynchronously, but it's what happens next that is tripping you up. MyHttpRequest does the await, which captures the UI context and returns an incomplete task. This happens multiple times in DownloadData, which collects the incomplete tasks and then awaits the Task.WhenAll of them. So then DownloadData returns an incomplete task which is awaited by the event handler, which returns control back to the WPF message loop.
Now, what happens next is that those Task.Delay timers fire off almost immediately. So, MyHttpRequest resumes its async method, and since its await captured that UI context, it resumes running on the UI thread. So the await Task.Delay(1) did cause a yield to the WPF message loop, but then practically the very next thing the UI thread has to do is resume those async methods.
And the remainder of those async methods are synchronous, blocking the UI thread.
So, to solve this, you can make the methods truly asynchronous. Delete the await Task.Delay(1) completely and instead replace the synchronous APIs with asynchronous ones (GetResponseAsync, ReadToEndAsync). Or if you want to modernize the code further, replace WebRequest with HttpClient.
Your other option is to keep the code synchronous and just use Task.Run to run the synchronous code on background threads. Again, you would delete the await Task.Delay(1), and this time you would change the method signature to be synchronous. Then you can wrap the call to the (synchronous) MyHttpRequest in a Task.Run, e.g., Task LastTask = Task.Run(() => ZerodhaOperations.MyHttpRequest("", authorization, cookie, "5minute")); The Task.Run solution is less ideal than the asynchronous solution, but it is acceptable if you have a lot of other code that still needs MyHttpRequest to be synchronous for now.
Note that either way, the usage of Task.Delay(1) is wrong. It does not "run [the] method asynchronously". The proper way to run a method asynchronously is to have it call asynchronous APIs and not call blocking APIs.

C# Threading with functions that return variables

Okay so basically I have a function that returns a string, but to get that string it uses webrequest which means while it's doing that webrequest the form is locking up unless I put it in a different thread.
But I can't figure out a way to capture the returned data in a thread since it's started using thread.start and that's a void.
Any help please?
Current code if it matters to anyone:
string CreateReqThread(string UrlReq)
{
System.Threading.Thread NewThread = new System.Threading.Thread(() => CreateReq(UrlReq));
string ReturnedData = "";
return ReturnedData;
}
string CreateReq(string url)
{
try
{
WebRequest SendReq = WebRequest.Create(url);
SendReq.Credentials = CredentialCache.DefaultCredentials;
SendReq.Proxy = WebRequest.DefaultWebProxy; //For closed port networks like colleges
SendReq.Proxy.Credentials = CredentialCache.DefaultCredentials;
SendReq.Timeout = 15000;
System.IO.StreamReader Reader = new System.IO.StreamReader(SendReq.GetResponse().GetResponseStream());
string Response = Reader.ReadToEnd();
Reader.Close();
return Response;
}
catch (WebException e)
{
EBox(e.Message, "Unknown Error While Connecting");
return null;
}
}

A common means of doing this is to use a Task<T> instead of a thread:
Task<string> CreateReqThread(string UrlReq)
{
return Task.Factory.StartNew() => CreateReq(UrlReq));
// In .NET 4.5, you can use (or better yet, reimplement using await/async directly)
// return Task.Run(() => CreateReq(UrlReq));
}
You can then call Task<T>.Result to get the returned value (later), when it's needed, or schedule a continuation on the task which will run when it completes.
This could look something like:
var request = CreateReqThread(theUri);
request.ContinueWith(t =>
{
// Shove results in a text box
this.textBox.Text = t.Result;
}, TaskScheduler.FromCurrentSynchronizationContext());
This also works perfectly with the new await/async support in C# 5.

httpRequest temporarily freezes my application

Okay, before I go on, let me state that my background is in web scripting; so applications are very foreign to me. I know very little about .NET and I've been skating by on my limited knowledge.
Anyways, in my application, I have an OAuth httpRequest. The request itself works fine, it gets the data I need from the web API. However, the problem is that whenever I click the button that activates the request, my program freezes for a few seconds until the request is finished. I also have another request which is done automatically every 60 seconds. Which of course means every 60 seconds, my program freezes for a few seconds. How to fix this?
private string twitchCallAPI(string accessKey, string accessSecret, string endpointURI, string httpMethod)
{
OAuthHttpWebRequest httpRequest = new OAuthHttpWebRequest();
httpRequest.ConsumerToken = new OAuthToken { Token = this.twitchConKey, TokenSecret = this.twitchConSecret };
httpRequest.Token = new OAuthToken() { Token = accessKey, TokenSecret = accessSecret };
httpRequest.SetUri(endpointURI);
httpRequest.Method = httpMethod;
try
{
using (var response = httpRequest.GetResponse())
{
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
}
catch (WebException ex)
{
using (var reader = new StreamReader(ex.Response.GetResponseStream()))
{
System.Windows.MessageBox.Show(reader.ReadToEnd());
}
}
catch (Exception ex)
{
System.Windows.MessageBox.Show(ex.ToString());
}
return string.Empty;
}

You could use a background worker

Shortly said, do request in task and update UI thread with UI synchronization context
TaskFactory.StartNew(()=>
{
//do web request
})
.ContinueWith(() =>
{
this.TextBlock1.Text = "Complete";
}, TaskScheduler.FromCurrentSynchronizationContext());

You can try using Async methods, that is, using a different thread to wait for the response of the request. Its a solution that you can explore.
http://msdn.microsoft.com/en-us/library/86wf6409%28v=vs.100%29.aspx

You can use await keyword:
private async void OnButtonClick()
{
TextBox.Text = await twitchCallAPIAsync(accessKey, accessSecret, endpointURI, httpMethod);
}

The main reason of this is because your application is waiting the methods you launch to finish. You have to take a look at the 'async' concept.
A program executing an 'async' method continue its workflow, and doesn't wait the method to produce a result.