We Keep Coding

RestSharp v107 not returning data

I’ve used RestSharp 106 in the past but now I’m trying to get a new project up and running with v107. I did a simple test below but I can’t get it to return any data so I must be missing something that I’m not understanding. I’ve put a breakpoint at the top of the method and can follow it all the way down until it makes the async call and then the app just quits. I see the initial logging but again nothing after the async call. I would have thought I would see some logging about the list count or even the “Application END” piece. Anyone have any ideas on what I’m doing wrong? I’ve kept but commented out some of the different things I’ve tried.
Link to the documentation I’m looking at.
https://restsharp.dev/intro.html#introduction
public async void Run()
{
_log.LogInformation("Application START");
try
{
var listPPC = new List<PriorPeriodCorrection>();
listPPC = await CallApi();
}
catch (Exception ex)
{
_log.LogError("Error: {0} | {1} | {2}", ex.Message, ex.StackTrace, ex.InnerException);
}
_log.LogInformation("Application END");
}
public async Task<List<PriorPeriodCorrection>> CallApi()
{
var listPPC = new List<PriorPeriodCorrection>();
var apiURL = "https://example.com";
var apiEndpoint = "api/payroll/getpriorpaycorrectiondata/from/2021-12-01/to/2021-12-07";
var proxyAddress = "http://example.com:9400";
var apiUsername = "someusername";
var apiPassword = "4PsaI69#tuv";
var options = new RestClientOptions(apiURL)
{
ThrowOnAnyError = true,
Timeout = 1000,
Proxy = new WebProxy(proxyAddress)
};
var client = new RestClient(options);
client.Authenticator = new HttpBasicAuthenticator(apiUsername, apiPassword);
var request = new RestRequest(apiEndpoint);
try
{
listPPC = await client.GetAsync<List<PriorPeriodCorrection>>(request);
//var response = await client.GetAsync<PriorPeriodCorrection>(request);
//var response = await client.GetAsync<List<PriorPeriodCorrection>>(request);
//var response = await client.GetAsync(request);
//if (!string.IsNullOrWhiteSpace(response.Content))
//{
// listPPC = JsonConvert.DeserializeObject<List<PriorPeriodCorrection>>(response.Content);
//}
//else
// _log.LogInformation("Response Content is Blank or NULL.");
}
catch (Exception ex)
{
_log.LogError("Error: {0} | {1} | {2}", ex.Message, ex.StackTrace, ex.InnerException);
}
_log.LogInformation("Response count: {0}", listPPC.Count);
return listPPC;
}

Is my approach correct for concurrent network requests?

I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).

windows service in c# microsoft graph api getAsync() call for users goes in hung state

I am reading different mailboxes in windows service using MS graph api using threads. Service runs well for few days, but after it goes in hung state.
By observing logs found that stuck point is while calling GetAsync() method for users, after that it is not updating log file also but service is showing in running state.
After restarting the service it runs normally for few days.
public static async Task MainAsync()
{
Task t1 = ParallelThreadOne.MainAsync();
Task t2 = ParallelThreadSecond.MainAsync();
Task t3 = ParallelThreadThird.MainAsync();
await Task.WhenAll(t1, t2, t3);
}
public static async Task MainAsync()
{
try
{
EmailMaster objEmailMaster = new EmailMaster();
List<MailBoxConfiguration> lstMailBoxConfiguration = objEmailMaster.GetMailBoxInformation(1,logger);
if (lstMailBoxConfiguration != null)
{
if (lstMailBoxConfiguration.Count != 0)
{
GraphServiceClient client = GetAuthenticatedClient();
if (client != null)
{
for (int j = 0; j < lstMailBoxConfiguration.Count; j++)
{
var users = await graphClient
.Users
.Request()
.Filter("startswith(Mail,'" + lstMailBoxConfiguration[j].EmailId + "')")
.GetAsync();
if (users.Count > 0)
{
var msgs = await graphClient
.Users[users[0].Id]
.MailFolders["Inbox"].Messages
.Request().Top(500)
.GetAsync();
if (msgs.Count > 0)
{
foreach (var item in msgs)
{
//business logic goes here
}
}
else
{
logger.Info("msg.Count is zero");
}
}
else
{
logger.Info("users.Count is zero");
}
}
}
else
{
logger.Info("client is null");
}
}
else
{
logger.Info("lstMailBoxConfiguration.Count is zero from the database");
}
}
else
{
logger.Info("lstMailBoxConfiguration is null from the database");
}
logger.Info("MainAsync(1) : End of MainAsync(1)");
}
catch (Exception ex)
{
logger.Error("MainAsync(1) : Exception : " + ex.Message);
}
}
public static GraphServiceClient GetAuthenticatedClient()
{
string clientId = ConfigurationManager.AppSettings["AzureClientId"];
string password = ConfigurationManager.AppSettings["password"];
string tenantId = ConfigurationManager.AppSettings["tenantId"];
string getTokenUrl = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";
const string grantType = "client_credentials";
const string myScopes = "https://graph.microsoft.com/.default";
string postBody = $"client_id={clientId}&scope={myScopes}&client_secret={password}&grant_type={grantType}";
try
{
if (graphClient == null)
{
graphClient = new GraphServiceClient(
"https://graph.microsoft.com/v1.0",
new DelegateAuthenticationProvider(
async (requestMessage) =>
{
HttpRequestMessage httpRequestMessage = new HttpRequestMessage(HttpMethod.Post, getTokenUrl);
httpRequestMessage.Content = new StringContent(postBody, Encoding.UTF8, "application/x-www-form-urlencoded");
HttpResponseMessage httpResponseMessage = await client.SendAsync(httpRequestMessage);
string responseBody = await httpResponseMessage.Content.ReadAsStringAsync();
userToken = JObject.Parse(responseBody).GetValue("access_token").ToString();
requestMessage.Headers.Authorization = new AuthenticationHeaderValue("bearer", userToken);
}));
}
return graphClient;
}
catch (Exception ex)
{
logger.Error("Could not create a graph client: " + ex.Message);
}
finally
{
logger.Info("GetAuthenticatedClient() :inside finally!");
}
return graphClient;
}

From looking through the source code of the GraphServiceClient, it is using HTTPClient as its underlying communication provider. The HTTPClient has an issue because windows keeps TCP/IP connections open for a certain amount of time after the HTTPClient is disposed. If you call new and then dispose on the HTTPClient class fast enough for long enough it can lead to socket starvation. (note, using(var client = new HTTPClient()) calls dispose under the covers when the instance goes out of scope)
Take a look at this blog post on the issue.
https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
You should be able to use a single instance of the GraphServiceClient as long as you are talking to the same GraphQL endpoint and fix your issues with the service hanging. If you add logging you will probably notice that the service hangs after a flurry of activity causing lots of new instances of the GraphServiceClient to be created then disposed in a short time frame and your open network connections on the server exploding causing an error that crashes one of your threads.

Recursive method is returning first iteration object C#

static void Main(string[] args)
{
token objtoken = new token();
var location = AddLocations();
OutPutResults outPutResultsApi = new OutPutResults();
GCPcall gCPcall = new GCPcall();
OutPutResults finaloutPutResultsApi = new OutPutResults();
var addressdt = new AddressDataDetails();
finaloutPutResultsApi.addressDatas = new List<AddressDataDetails>();
Console.WriteLine("Hello World!");
List<string> placeId = new List<string>();
var baseUrl = "https://maps.googleapis.com/maps/api/place/textsearch/json?";
var apiKey = "&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx".Trim();
foreach (var itemlocations in location)
{
var searchtext = "query=" + itemlocations.Trim();
var finalUrl = baseUrl + searchtext + apiKey;
gCPcall.RecursiveApiCall(finalUrl, ref placeId, objtoken.NextToken);
}
var ids = gCPcall.myPalceid;
}
public List<string> RecursiveApiCall(string finalUrl, ref List<string> placeId, string nextToken = null)
{
try
{
var token = "&pagetoken=" + nextToken;
using (var client = new HttpClient())
{
var responseTask = client.GetAsync(finalUrl + token);
responseTask.Wait();
var result = responseTask.Result;
if (result.IsSuccessStatusCode)
{
var readTask = result.Content.ReadAsStringAsync();
readTask.Wait();
var students = readTask.Result;
Rootobject studentsmodel = JsonConvert.DeserializeObject<Rootobject>(students);
nextToken = studentsmodel.next_page_token;
foreach (var item in studentsmodel.results)
{
placeId.Add(item.place_id);
}
}
}
if (nextToken != null)
{
RecursiveApiCall(finalUrl, ref placeId, nextToken);
}
return placeId;
}
catch (Exception ex)
{
throw;
}
}
My recursive method has some issue. Here whenever I am debugging this code it work fine. It goes in recursive call twice.
As debugging result I am getting list place_id with 20 items in first call and next call 9 items total 29 items in place_id object which is correct in static main method.
But if I run without debugging mode I am getting only 20 place_id. next recursive iteration data is missing even if it has next valid token.
I don't have any clue why this is happening. Can someone tell me what is the issue with my code?

Here are my suggestions, which may or may not solve the problem:
// First of all, let's fix the signature : Go async _all the way_
//public List<string> RecursiveApiCall(string finalUrl, ref List<string> placeId, string nextToken = null)
// also reuse the HttpClient!
public async Task ApiCallAsync(HttpClient client, string finalUrl, List<string> placeId, string nextToken = null)
{
// Loop, don't recurse
while(!(nextToken is null)) // C# 9: while(nextToken is not null)
{
try
{
var token = "&pagetoken=" + nextToken;
// again async all the way
var result = await client.GetAsync(finalUrl+token);
if (result.IsSuccessStatusCode)
{
// async all the way!
var students = await result.Content.ReadAsStringAsync();
Rootobject studentsmodel = JsonConvert.DeserializeObject<Rootobject>(students);
nextToken = studentsmodel.next_page_token;
foreach (var item in studentsmodel.results)
{
// Will be reflected in main, so no need to return or `ref` keyword
placeId.Add(item.place_id);
}
}
// NO recursion needed!
// if (nextToken != null)
// {
// RecursiveApiCall(finalUrl, ref placeId, nextToken);
// }
}
catch (Exception ex)
{
// rethrow, only is somewhat useless
// I'd suggest using a logging framework and
// log.Error(ex, "Some useful message");
throw;
// OR remove try/catch here all together and wrap the call to this method
// in try / catch with logging.
}
}
Mind that you'll need to make your main :
async Task Main(string[] args)
and call this as
await ApiCallAsync(client, finalUrl, placeId, nextToken);
Also create an HttpClient in main and reuse that:
"HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors." - Remarks
... which shouldn't do much harm here, but it's a "best practice" anyhow.
Now, as to why you get only 20 instead of expected 29 items, I cannot say if this resolves that issue. I'd highly recommend to introduce a Logging Framework and make log entries accordingly, so you may find the culprid.

HttpClient in while loop only executed once

I am trying to call HttpClient request inside for loop as follows. It needs to do multiple consecutive calls to third party rest api.
But it only gives me fist service call result while loop exit before getting result from rest of the service call.
private void Search()
{
try
{
var i = 1;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(jsonResult.ToString());
i++;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
When I run with debug points the program gives me all the result. But when I run it without debug points it gives me only the first result.
I tried this with using async, await methods too. It also gives me same result.
As I feel Program needs to wait until the async call returns data.
Please help me to solve this.
EDIT - async way
private async Task<string> SearchNew()
{
try
{
var i = 1;
var res = string.Empty;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = await response.Content.ReadAsStringAsync();
res = res + jsonResult + " --- ";
i++;
}
}
return res;
}
catch (Exception ex)
{
return ex.Message;
}
}
Both are giving same result.

There's a few things here that you should be doing. First, move the HttpClient creation outside of your method and make it static. You only need one of them and having multiple can be really bad for stability (see here):
private static HttpClient _client = new HttpClient();
Next, extract the calls to the HttpClient into a single method, something simple like this:
//Please choose a better name than this
private async Task<string> GetData(string url)
{
var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
And finally, you create a list of tasks and wait for them all to complete asynchronously using Task.WhenAll:
private async Task<string[]> SearchAsync()
{
var i = 1;
var tasks = new List<Task<string>>();
//Create the tasks
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
tasks.Add(GetData(url));
i++;
}
//Wait for the tasks to complete and return
return await Task.WhenAll(tasks);
}
And to call this method:
var results = await SearchAsync();
foreach (var result in results)
{
Console.WriteLine(result);
}