We Keep Coding

Is my approach correct for concurrent network requests?

I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).

Nsubstitute or moq an Async method so it will return a faulted task instead of exception

In the below code for each request we are calling rest client in an Async manner. Rest client is basically a http client which is calling some webapis in a batch. If suppose AsyncTaskCount value is 5 then 5 request will be called asynchronously and then in the while block we are getting the result for each call. If any response out of those 5 request has an exception then the response will be faulted and IsFaulted becomes true for that particular request and in the response we can get the inner exception.
private async Task<List<RequestResponse>> ProcessInvestment(List<Request> Requests, List<Result> Results, ILogger log)
{
var requestResponses = new List<RequestResponse>();
var asyncTaskCount = Convert.ToInt32(Environment.GetEnvironmentVariable("AsyncTaskCount"));
log.LogInformation($"Start processing {Requests.Count} in batches of {asyncTaskCount}");
for (int index = 0; index < Requests.Count; index = index + asyncTaskCount)
{
var requestBatch = Requests.Skip(index).Take(asyncTaskCount).ToList();
var requests = requestBatch.Select(x => _restClient.RequestResponse(x)).ToList();
while (requests.Count > 0)
{
// Identify the first task that completes.
Task<RequestResponse> requestResponseTask = await Task.WhenAny(requests);
var requestResponse = new RequestResponse();
// ***Remove the selected task from the list so that you don't process it more than once
requests.Remove(requestResponseTask);
if (!requestResponseTask.IsFaulted)
{
// Await the completed task.
requestResponse = await requestResponseTask;
requestResponses.Add(requestResponse);
}
else
{
if (requestResponseTask.Exception.InnerException != null && requestResponseTask.Exception.InnerException is Exception)
{
var result = new Result();
result = ResponseTransformComponent.ResponseToResult(((Exception)requestResponseTask.Exception.InnerException).Request, null);
result.SetBadRequestErrorDetails(((Exception)RequestResponseTask.Exception.InnerException).BadRequestResponse);
results.Add(Result);
}
else
{
throw requestResponseTask.Exception;
}
}
}
log.LogInformation($"Number of records processed = {requestResponses.Count}");
}
log.LogInformation($"Total invalid and Bad requests count = {results.Count}");
return RequestResponses;
}
Below is the code for restclient which is called from the above method.
public async Task<Response> RequestResponse(Request request)
{
var response = await GetDataFromService("calculation", "CalculateCapital", request);
return JsonConvert.DeserializeObject<Response>(response);
}
public async Task<string> GetDataFromService(string controller, string method, object request)
{
var client = _httpClientFactory.CreateClient(ServiceEnum.DCS);
string baseAddress = client.BaseAddress.ToString();
var requestUrl = $"api/{controller}/{method}";
var content = new StringContent(JsonConvert.SerializeObject(request), Encoding.UTF8, "application/json");
var response = await client.PostAsync(requestUrl, content).ConfigureAwait(false);
if (!response.IsSuccessStatusCode)
{
var responseResult = response.Content.ReadAsStringAsync().Result;
if (response.StatusCode == HttpStatusCode.BadRequest)
{
throw new CalculatorServiceException("Bad Request", JsonConvert.DeserializeObject<BadRequestResponse>(responseResult), (Request)request);
}
throw new Exception($"Status code: {response.StatusCode}. {responseResult}");
}
return response.Content.ReadAsStringAsync().Result;
}
GetDataFromService method is called from Calculate method. And GetDataFromService method will return a custom exception if the request is a Bad Request.
I am trying to write the unit test case for the above method and try to mock a request so that it will return a faulted task and then IsFaulted should become true. Below is the part of my unit test case.
_restClient
.When(a => a.RequestResponse(Arg.Any<Request>()))
.Do(a => {Task.FromException(new CalculatorServiceException(string.Empty, new BadRequestResponse { Message = string.Empty, ModelState = new Dictionary<string, string[]> { { "CalculationDates.StartDate", new string[] { "0002: Duration too short to execute calculations (CalculationDates.StartDate)" } } } }, Arg.Any<Request>())); });
If i mock my restclient method like above then it is throwing the exception instead of giving the response with IsFaulted to true. So how should i mock the restclient method so that it will return a faulted task which has an exception instead of throwing it. Please suggest.
Thanks in advance.

When..Do is for wiring up callbacks for when a member is called.
Try using Returns instead:
_restClient.RequestResponse(Arg.Any<Request>())
.Returns(x => Task.FromException<RequestResponse>(...));
// (assuming RequestResponse(..) returns a Task<RequestResponse>. Tweak as required)

HttpClient in while loop only executed once

I am trying to call HttpClient request inside for loop as follows. It needs to do multiple consecutive calls to third party rest api.
But it only gives me fist service call result while loop exit before getting result from rest of the service call.
private void Search()
{
try
{
var i = 1;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(jsonResult.ToString());
i++;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
When I run with debug points the program gives me all the result. But when I run it without debug points it gives me only the first result.
I tried this with using async, await methods too. It also gives me same result.
As I feel Program needs to wait until the async call returns data.
Please help me to solve this.
EDIT - async way
private async Task<string> SearchNew()
{
try
{
var i = 1;
var res = string.Empty;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = await response.Content.ReadAsStringAsync();
res = res + jsonResult + " --- ";
i++;
}
}
return res;
}
catch (Exception ex)
{
return ex.Message;
}
}
Both are giving same result.

There's a few things here that you should be doing. First, move the HttpClient creation outside of your method and make it static. You only need one of them and having multiple can be really bad for stability (see here):
private static HttpClient _client = new HttpClient();
Next, extract the calls to the HttpClient into a single method, something simple like this:
//Please choose a better name than this
private async Task<string> GetData(string url)
{
var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
And finally, you create a list of tasks and wait for them all to complete asynchronously using Task.WhenAll:
private async Task<string[]> SearchAsync()
{
var i = 1;
var tasks = new List<Task<string>>();
//Create the tasks
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
tasks.Add(GetData(url));
i++;
}
//Wait for the tasks to complete and return
return await Task.WhenAll(tasks);
}
And to call this method:
var results = await SearchAsync();
foreach (var result in results)
{
Console.WriteLine(result);
}

How to wait for a callback before returning

I am trying to port a code from .Net to Unity C# and I am stuck on a syntax including a callback.
Basically, I had to replace the .Net 'HttpClient' library by this one, also called 'HttpClient'. But the 'Get' syntax is not the same and uses a Callback. I am quite new to C# and Http queries and don't know how to deal with this syntax to get the expected return.
The original function written in .Net:
internal static JObject GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
var response =
httpClient.GetAsync(url
).Result;
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
var idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Content.ReadAsStringAsync().Result);
return idProcess;
}
else
return null;
The C# code I am writing for Unity:
internal class Utils
{
internal static JObject GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
JObject idProcess = new JObject();
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Data);
}
else
{
idProcess = null;
}
});
// Here I would like to wait for the response, so that idProcess is filled with the received data before returning
return idProcess;
}
public class Action
{
public bool SendData(string id, string secretKey, FakeData data)
{
var idProcess = Utils.GetToBackOffice(String.Format("Events/{0}/infos",id));
//...I do then something with idProcess
//Currently, when I use idProcess here, it is still empty since the GetString response hasn't been received yet when this line is executed
return true;
}
}
public class EventHubSimulator : MonoBehaviour
{
void Start()
{
//Fill the parameters (I skip the details)
string oId = ...;
string secretKey = ...;
var vh = ...;
Action action = new Action();
action.SendData(oId, secretKey, vh);
}
}
My issue is that after the GetToBackOffice function, my code directly uses 'idProcess' for something else but this object is empty because the response was not received yet. I would like to wait for the response before my function returns.
I hope I was clear enough. I know that similar question have already been posted but couldn't find a solution to my specific issue.
Edit:
Finally I used a coroutine as Nain suggested but couldn't get what I expected the way he said. This way seems to work (event though it might not be a good way to do it).
public class EventHubSimulator : MonoBehaviour
{
void Start()
{
//Fill the parameters (I skip the details)
string oId = ...;
string secretKey = ...;
var vh = ...;
Utils utils = new Utils();
StartCoroutine(utils.SendData(oId, secretKey, vh));
}
}
public class Utils: MonoBehaviour
{
private const string backOfficeUrl = "http://myurl/api/";
public CI.HttpClient.HttpResponseMessage<string> response;
public IEnumerator SendData(string id, string secretKey, FakeData data)
{
response = null;
yield return GetToBackOffice(String.Format("Events/{0}/infos", id)); //Make a Http Get request
//The next lines are executed once the response has been received
//Do something with response
Foo(response);
}
IEnumerator GetToBackOffice(string action)
{
var url = backOfficeUrl;
url = url + action;
//Make a Http Get request
HttpClient httpClient = new HttpClient();
httpClient.GetString(new Uri(url), (r) =>
{
// Raised when the download completes
if (r.StatusCode == System.Net.HttpStatusCode.OK)
{
//Once the response has been received, write it in the global variable
response = r;
Debug.Log("Response received : " + response);
}
else
{
Debug.Log("ERROR =============================================");
Debug.Log(r.ReasonPhrase);
throw new Exception(r.ReasonPhrase);
}
});
//Wait for the response to be received
yield return WaitForResponse();
Debug.Log("GetToBackOffice coroutine end ");
}
IEnumerator WaitForResponse()
{
Debug.Log("WaitForResponse Coroutine started");
//Wait for response to become be assigned
while (response == null)
{
yield return new WaitForSeconds(0.02f);
}
Debug.Log("WaitForResponse Coroutine ended");
}
}

One solution is polling for completion as Nain submitted as an answer. If you don't want polling you can use a TaskCompletionSource. This Q&A dives a bit deeper into the why and how.
Your code can then be written like this:
async Task CallerMethod()
{
JObject result = await GetToBackOffice(...);
// Do something with result
}
internal static Task<JObject> GetToBackOffice(string action)
{
var tsc = new TaskCompletionSource<JObject>();
var url = backOfficeUrl;
url = url + action;
HttpClient httpClient = new HttpClient();
JObject idProcess = new JObject();
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
tsc.SetResult(Newtonsoft.Json.Linq.JObject.Parse(response.Data));
}
else
{
tsc.SetResult(null);
}
});
return tsc.Task;
}
See also the section Async Method Calls a Coroutine And Wait for Completion of this msdn blogpost.
NOTE Tasks, and async/await support is only available as beta functionality in Unity. See also this post.

Write a coroutine like
//Class scope variable is neede to hold Response other wise it will
//be destroied as soon as function is ended
ResponseType response;
IEnumerator WaitForResponce(ResponseType response)
{
this.response = response;
while(this.response.Data == null)
yield return new WaitForSeconds (0.02f);
//do what you want here
}
and call the coroutine
httpClient.GetString(new Uri(url),
(response) =>
{
// Raised when the download completes
if (response.StatusCode == System.Net.HttpStatusCode.OK)
{
//idProcess = Newtonsoft.Json.Linq.JObject.Parse(response.Data);
StartCoroutine(WaitForResponse(response));
}
else
{
idProcess = null;
}
});

If the GetString method is returning a Task object then you can use the Wait() method to let the Task finished processing before a result is returned.
var task = httpClient.GetString({
impl here..
});
task.Wait();
return idProcess;

One or more errors occurred: PostAsJsonAsync

This is the error I get:
One or more errors occurred.
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
at System.Threading.Tasks.Task`1.get_Result()
at myProject.mymethod(Int32[] myIds) in path\MyClass.cs:line 758
And here's that method:
private void mymethod(int[] myIds)
{
var uri = new Uri(string.Format(UriPath, string.Format(MyPath)));
var client = GetHttpClient(uri);
var postModel = new PostModel
{
Ids = myIds,
LastUpdate = NewItem ? null : _lastUpdated
};
if (client != null)
{
var response = client.PostAsJsonAsync(uri, postModel).Result;//this line crashes
if (response.IsSuccessStatusCode)
{
//doSomething
}
}
}
I call a lot of methods like this and all of them work except this one. When it's hit, it takes a lot of time and then this exception is throws. With all of the other methods the error doesn't happen.
This is the inner exception:
Inner.System.Threading.Tasks.TaskCanceledException: A task was canceled.
Here's my GetCLient() method:
private HttpClient GetHttpClient(Uri uri)
{
var handler = new HttpClientHandler
{
CookieContainer = CoockieContainer
};
return new HttpClient(handler)
{
BaseAddress = uri
};
}
Here's the API method:
[HttpPost]
public IList<MyModel> MyAPIMethod(PostModel model)
{
List<MyModel> myTranslations;
using (var db = new myEntities(GetDbConnStrByUser(new GetCookies().GetUserName())))
{
myTranslations = db.tblTranslations.Where(it => model.Ids.Contains(it.id)
&& (!model.Update.HasValue || it.update > model.LastUpdate.Value))
.Select(it => new MyModel
{
Id = it.id,
Name = it.name,
Description = it.desc,
LanguageId = it.language_id
}).ToList();
}
return myTranslations.GroupBy(x => new { x.Id, x.LanguageId }).Select(x => x.First()).ToList();
}
Maybe a timeout occurs.
Fiddler returns this error: The wait operation timed out.

.Result tries to turn Task into T. It synchronously blocks waiting for the task to complete and will only return T if no exceptions occur. The two types of exceptions you should anticipate are OperationCanceledExceptions/TaskCanceledExceptions (when the HTTP request times out or a CancellationToken was used and the token was cancelled--I don't think the latter is applicable here) and general/HTTP-related exceptions.
You're probably running into the HTTP timeout scenario. Is it taking a long time for the exception to get thrown? If not, what does your GetHttpClient method look like? Is it explicitly setting timeouts or cancellation tokens?
If you want to stick with the synchronous approach and not returns Tasks, you should do something like this instead:
try
{
var response = client.PostAsJsonAsync(uri, postModel).Result;//this line crashes
}
catch (OperationCanceledException oce)
{
// Tell the user that the request timed our or you cancelled a CancellationToken
}
catch (Exception exc)
{
// Look at the HttpClient docs and try to catch something more specific. You don't want
// to swallow StackOverflowExceptions or OutOfMemoryExceptions.
}
If you're willing to take the async plunge, this is more what you're looking for:
private async Task mymethodAsync(int[] myIds)
{
var uri = new Uri(string.Format(UriPath, string.Format(MyPath)));
var client = GetHttpClient(uri);
var postModel = new PostModel { ... };
if (client != null)
{
try
{
var response = await client.PostAsJsonAsync(uri, postModel);
if (response.IsSuccessStatusCode)
{
//doSomething
}
}
catch (OperationCanceledException oce)
{
// because timeouts are still possible
}
catch (Exception exc) {
// Because other things can still go wrong too (HTTP 500, parsing errors)
}
}
}
See here for more info on how .Result call works w/HttpClient: What happens while waiting on a Task's Result?