We Keep Coding

Is my approach correct for concurrent network requests?

I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).

Recursive method is returning first iteration object C#

static void Main(string[] args)
{
token objtoken = new token();
var location = AddLocations();
OutPutResults outPutResultsApi = new OutPutResults();
GCPcall gCPcall = new GCPcall();
OutPutResults finaloutPutResultsApi = new OutPutResults();
var addressdt = new AddressDataDetails();
finaloutPutResultsApi.addressDatas = new List<AddressDataDetails>();
Console.WriteLine("Hello World!");
List<string> placeId = new List<string>();
var baseUrl = "https://maps.googleapis.com/maps/api/place/textsearch/json?";
var apiKey = "&key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx".Trim();
foreach (var itemlocations in location)
{
var searchtext = "query=" + itemlocations.Trim();
var finalUrl = baseUrl + searchtext + apiKey;
gCPcall.RecursiveApiCall(finalUrl, ref placeId, objtoken.NextToken);
}
var ids = gCPcall.myPalceid;
}
public List<string> RecursiveApiCall(string finalUrl, ref List<string> placeId, string nextToken = null)
{
try
{
var token = "&pagetoken=" + nextToken;
using (var client = new HttpClient())
{
var responseTask = client.GetAsync(finalUrl + token);
responseTask.Wait();
var result = responseTask.Result;
if (result.IsSuccessStatusCode)
{
var readTask = result.Content.ReadAsStringAsync();
readTask.Wait();
var students = readTask.Result;
Rootobject studentsmodel = JsonConvert.DeserializeObject<Rootobject>(students);
nextToken = studentsmodel.next_page_token;
foreach (var item in studentsmodel.results)
{
placeId.Add(item.place_id);
}
}
}
if (nextToken != null)
{
RecursiveApiCall(finalUrl, ref placeId, nextToken);
}
return placeId;
}
catch (Exception ex)
{
throw;
}
}
My recursive method has some issue. Here whenever I am debugging this code it work fine. It goes in recursive call twice.
As debugging result I am getting list place_id with 20 items in first call and next call 9 items total 29 items in place_id object which is correct in static main method.
But if I run without debugging mode I am getting only 20 place_id. next recursive iteration data is missing even if it has next valid token.
I don't have any clue why this is happening. Can someone tell me what is the issue with my code?

Here are my suggestions, which may or may not solve the problem:
// First of all, let's fix the signature : Go async _all the way_
//public List<string> RecursiveApiCall(string finalUrl, ref List<string> placeId, string nextToken = null)
// also reuse the HttpClient!
public async Task ApiCallAsync(HttpClient client, string finalUrl, List<string> placeId, string nextToken = null)
{
// Loop, don't recurse
while(!(nextToken is null)) // C# 9: while(nextToken is not null)
{
try
{
var token = "&pagetoken=" + nextToken;
// again async all the way
var result = await client.GetAsync(finalUrl+token);
if (result.IsSuccessStatusCode)
{
// async all the way!
var students = await result.Content.ReadAsStringAsync();
Rootobject studentsmodel = JsonConvert.DeserializeObject<Rootobject>(students);
nextToken = studentsmodel.next_page_token;
foreach (var item in studentsmodel.results)
{
// Will be reflected in main, so no need to return or `ref` keyword
placeId.Add(item.place_id);
}
}
// NO recursion needed!
// if (nextToken != null)
// {
// RecursiveApiCall(finalUrl, ref placeId, nextToken);
// }
}
catch (Exception ex)
{
// rethrow, only is somewhat useless
// I'd suggest using a logging framework and
// log.Error(ex, "Some useful message");
throw;
// OR remove try/catch here all together and wrap the call to this method
// in try / catch with logging.
}
}
Mind that you'll need to make your main :
async Task Main(string[] args)
and call this as
await ApiCallAsync(client, finalUrl, placeId, nextToken);
Also create an HttpClient in main and reuse that:
"HttpClient is intended to be instantiated once and re-used throughout the life of an application. Instantiating an HttpClient class for every request will exhaust the number of sockets available under heavy loads. This will result in SocketException errors." - Remarks
... which shouldn't do much harm here, but it's a "best practice" anyhow.
Now, as to why you get only 20 instead of expected 29 items, I cannot say if this resolves that issue. I'd highly recommend to introduce a Logging Framework and make log entries accordingly, so you may find the culprid.

How to find all id and autopaste as parameter using linq?

I have 2 projects. One of them aspnet core webapi and second one is console application which is consuming api.
Api method looks like:
[HttpPost]
public async Task<IActionResult> CreateBillingInfo(BillingSummary
billingSummaryCreateDto)
{
var role = User.FindFirst(ClaimTypes.Role).Value;
if (role != "admin")
{
return BadRequest("Available only for admin");
}
... other properties
billingSummaryCreateDto.Price = icu * roc.Price;
billingSummaryCreateDto.Project =
await _context.Projects.FirstOrDefaultAsync(x => x.Id ==
billingSummaryCreateDto.ProjectId);
await _context.BillingSummaries.AddAsync(billingSummaryCreateDto);
await _context.SaveChangesAsync();
return StatusCode(201);
}
Console application which consuming api:
public static async Task CreateBillingSummary(int projectId)
{
var json = JsonConvert.SerializeObject(new {projectId});
var data = new StringContent(json, Encoding.UTF8, "application/json");
using var client = new HttpClient();
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("Bearer", await Token.GetToken());
var loginResponse = await client.PostAsync(LibvirtUrls.createBillingSummaryUrl,
data);
WriteLine("Response Status Code: " + (int) loginResponse.StatusCode);
string result = loginResponse.Content.ReadAsStringAsync().Result;
WriteLine(result);
}
Program.cs main method looks like:
static async Task Main(string[] args)
{
if (Environment.GetEnvironmentVariable("TAIKUN_USER") == null ||
Environment.GetEnvironmentVariable("TAIKUN_PASSWORD") == null ||
Environment.GetEnvironmentVariable("TAIKUN_URL") == null)
{
Console.WriteLine("Please specify all credentials");
Environment.Exit(0);
}
Timer timer = new Timer(1000); // show time every second
timer.Elapsed += Timer_Elapsed;
timer.Start();
while (true)
{
Thread.Sleep(1000); // after 1 second begin
await PollerRequests.CreateBillingSummary(60); // auto id
await PollerRequests.CreateBillingSummary(59); // auto id
Thread.Sleep(3600000); // 1hour wait again requests
}
}
Is it possible find all id and paste it automatically instead of 59 and 60? Ids from projects table. _context.Projects
Tried also approach using method which returns ids
public static async Task<IEnumerable<int>> GetProjectIds2()
{
var json = await
Helpers.Transformer(LibvirtUrls.projectsUrl);
List<ProjectListDto> vmList =
JsonConvert.DeserializeObject<List<ProjectListDto>>(json);
return vmList.Select(x => x.Id).AsEnumerable(); // tried
ToList() as well
}
and in main method used:
foreach (var i in await PollerRequests.GetProjectIds2())
new List<int> { i }
.ForEach(async c => await
PollerRequests.CreateBillingSummary(c));
for first 3 ids it worked but does not get other ones,
tested with console writeline method returns all ids

First get all Ids:
var ids = await PollerRequests.GetProjectIds2();
Then create list of task and run all tasks:
var taskList = new List<Task>();
foreach(var id in ids)
taskList.Add(PollerRequests.CreateBillingSummary(id));
await Task.WhenAll(taskList);

Function returns before await async finishes

I'm trying to send 2 emails through the SendGrid API. Sometimes 0 send, sometimes 1 sends, sometimes both send. It seems that the function does not await the promise. How can I fix it so it always sends both emails?
My function looks like this:
private async Task<bool> SendMails(string email, string name, string pdfPath, string imgPath)
{
var client = new SendGridClient(_config["SendGrid:Key"]);
bool messagesSent = false;
var messageClient = new SendGridMessage
{
From = new EmailAddress(_config["SendGrid:Recipient"]),
Subject = "Testmail",
HtmlContent = _textManager.Get("getMailHtml")
};
var messageSecondClient = new SendGridMessage
{
From = new EmailAddress(_config["SendGrid:Recipient"]),
Subject = "Second Testmail",
HtmlContent = _textManager.Get("getSecondMailHtml")
};
messageClient.AddTo(email, name);
messageSecondClient.AddTo(email, name);
string[] fileListClient = new string[] { pdfPath };
string[] fileListSecond = new string[] { pdfPath, imgPath };
foreach (var file in fileListClient)
{
var fileInfo = new FileInfo(file);
if (fileInfo.Exists)
await messageClient.AddAttachmentAsync(fileInfo.Name, fileInfo.OpenRead());
}
foreach (var file in fileListSecond)
{
var fileInfo = new FileInfo(file);
if (fileInfo.Exists)
await messageSecondClient.AddAttachmentAsync(fileInfo.Name, fileInfo.OpenRead());
}
var responseClient = await client.SendEmailAsync(messageClient);
var responseSecond = await client.SendEmailAsync(messageSecondClient);
if (responseClient.StatusCode.ToString() == "202" && responseSecond.StatusCode.ToString() == "202")
{
messagesSent = true;
}
return messagesSent;
}
And this is how I'm calling it:
Task<bool> sendMails = await Task.FromResult(SendMails(formCollection["email"], formCollection["name"], pdfPath, imgPath));
if (!sendMails.Result)
{
errorMessage = "Error sending mails.";
}

You're blocking on the async task:
if (!sendMails.Result)
and this can cause a deadlock. Instead of blocking, use await.
And you can also get rid of the await Task.FromResult, which isn't doing anything at all:
bool sentMails = await SendMails(formCollection["email"], formCollection["name"], pdfPath, imgPath);
if (!sentMails)
{
errorMessage = "Error sending mails.";
}

Task.FromResult returns a new Task that is already completed, not the Task returned from SendMails.
Nothing is awaiting the completion of SendMails.
Just await the Task returned from the method:
bool result = await SendMails(formCollection["email"], formCollection["name"], pdfPath, imgPath);
The await keyword unwraps the Task.Result for you.

How to unit test dialogs that contains attachments or more objects in Bot Framework C#

Bot Info
SDK Platform: .NET
SDK Version: 3.14.0.7
Active Channels: Web
Deployment Environment: Local development with Emulator
Issue Description
We've trying to unit test every case that we have stored in a certain Dictionary, it seems to be working fine when the user sends and string and the test has to answer with a string. But we can't find any documentation on how to test the other kind of dialogs, like with attachments, buttons, etc.
We wish to make a dictionary of string,objects where the string is what we ask the bot and de object is either a string, Attachment, Dialog.
Code Example
This is how we store the answers:
public static Dictionary<string, object> data = new Dictionary<string, object>{
{"Nuevo", "Que quieres crear?"},
{"Ayuda", "Ya te ayudas!"},
{"Adios", "Nos vemos!"},
{
"Coche",
new Attachment() {
ContentUrl = "https://media.ed.edmunds-media.com/subaru/impreza/2006/oem/2006_subaru_impreza_sedan_sti_fq_oem_1_500.jpg",
ContentType = "image/png",
Name = "Subaru_Impreza.png"
}
},
{
"Moto",
new Attachment() {
ContentUrl = "http://motos.honda.com.co/sites/default/files/motos/cb-1000-r-cc-menu-honda.png",
ContentType = "image/png",
Name = "moto.png"
}
},
{
"Perro",
new Attachment() {
ContentUrl = "https://upload.wikimedia.org/wikipedia/commons/thumb/6/6b/Taka_Shiba.jpg/1200px-Taka_Shiba.jpg",
ContentType = "image/png",
Name = "ShibaInu.png"
}
}
};
This is how the bot works and returns everything, this is working as intended for at least text and attachments but we haven't done it for more type of messages.
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
var activity = await result as Activity;
var r = context.MakeMessage();
foreach (var item in data)
{
if (item.Key == activity.Text)
{
if (item.Value is Attachment)
{
r.Attachments = new List<Attachment>() { item.Value as Attachment };
}
if (item.Value is string)
{
r.Text = item.Value.ToString();
}
break;
}
}
// return our reply to the user
await context.PostAsync(r);
context.Wait(MessageReceivedAsync);
}
But when we want to make the test for it, it only works when what we send is a string not a IMessageActivity, which works in the emulator.
The code for the test:
[TestMethod]
public async Task Pregunta_respuesta_prueba()
{
foreach (var item in RootDialog.data)
{
var preg = item.Key;
var resp = item.Value;
if (item.Value is Attachment)
{
Attachment auxText = resp as Attachment;
resp = auxText.ContentUrl;
}
using (ShimsContext.Create())
{
// Arrange
var waitCalled = false;
object message = null;
var target = new RootDialog();
var activity = new Activity(ActivityTypes.Message)
{
Text = preg
};
var awaiter = new Microsoft.Bot.Builder.Internals.Fibers.Fakes.StubIAwaiter<IMessageActivity>()
{
IsCompletedGet = () => true,
GetResult = () => activity
};
var awaitable = new Microsoft.Bot.Builder.Dialogs.Fakes.StubIAwaitable<IMessageActivity>()
{
GetAwaiter = () => awaiter
};
var context = new Microsoft.Bot.Builder.Dialogs.Fakes.StubIDialogContext();
Microsoft.Bot.Builder.Dialogs.Fakes.ShimExtensions.PostAsyncIBotToUserStringStringCancellationToken = (user, s1, s2, token) =>
{
message = s1;
Console.WriteLine(message);
return Task.CompletedTask;
};
Microsoft.Bot.Builder.Dialogs.Fakes.ShimExtensions.WaitIDialogStackResumeAfterOfIMessageActivity = (stack, callback) =>
{
if (waitCalled) return;
waitCalled = true;
// The callback is what is being tested.
callback(context, awaitable);
};
// Act
await target.StartAsync(context);
// Assert
Assert.AreEqual(resp, message);
}
}
}
If you check this part of the code
Microsoft.Bot.Builder.Dialogs.Fakes.ShimExtensions.PostAsyncIBotToUserStringStringCancellationToken = (user, s1, s2, token) =>
{
message = s1;
Console.WriteLine(message);
return Task.CompletedTask;
};
```
It does only works when the bot is returning an string, we can't even check if it is an activiy, this happens because the Fake Context that we create for the test is not working as expected.
That IDialogContext that we are faking doesnt seem to work at all when it is an object, but it does work when it is a string.
private async Task MessageReceivedAsync(IDialogContext context, IAwaitable<object> result)
{
var activity = await result as Activity;
/// Here when the test is running, this context.MakeMessage is null, but when the bot
/// is working, it wors perfectly.
var r = context.MakeMessage();
foreach (var item in data)
{
if (item.Key == activity.Text)
{
if (item.Value is Attachment)
{
r.Attachments = new List<Attachment>() { item.Value as Attachment };
}
if (item.Value is string)
{
r.Text = item.Value.ToString();
}
break;
}
}
// return our reply to the user
await context.PostAsync(r);
context.Wait(MessageReceivedAsync);
}
Reproduction Steps
To try this out you can try to test with an attachment, code is in this repository.

In stead of using PostAsyncIBotToUserStringStringCancellationToken, you can use context.PostAsyncIMessageActivityCancellationToken And, in the RootDialog's MessageReceivedWithTextAsync respond with an activity reply instead of just a string.
public async Task MessageReceivedWithTextAsync(IDialogContext context, IAwaitable<object> result)
{
var activity = await result as Activity;
string r = "";
foreach (var item in dataText)
{
if (item.Key == activity.Text)
{
r = item.Value;
break;
}
}
var reply = activity.CreateReply(r);
foreach (var item in dataAtt)
{
if (item.Key == activity.Text)
{
reply.Attachments.Add(item.Value);
reply.Text = "attachment";
break;
}
}
if ((string.IsNullOrWhiteSpace(r) || r == null) && reply.Attachments.Count == 0)
{
reply.Text = "No tengo respuesta para eso.";
}
// return our reply to the user
await context.PostAsync(reply);
}
Here are the changes to the test method:
[TestMethod]
public async Task Bot_Test_Attachments()
{
foreach (var item in RootDialog.dataAtt)
{
var preg = item.Key;
var att = item.Value;
using (ShimsContext.Create())
{
var waitCalled = false;
IMessageActivity message = null;
var target = new RootDialog();
var activity = new Activity(ActivityTypes.Message)
{
Text = preg,
From = new ChannelAccount("id","name"),
Recipient = new ChannelAccount("recipid","recipname"),
Conversation = new ConversationAccount(false,"id","name")
};
var awaiter = new Microsoft.Bot.Builder.Internals.Fibers.Fakes.StubIAwaiter<IMessageActivity>()
{
IsCompletedGet = () => true,
GetResult = () => activity
};
var awaitable = new Microsoft.Bot.Builder.Dialogs.Fakes.StubIAwaitable<IMessageActivity>()
{
GetAwaiter = () => awaiter
};
var context = new Microsoft.Bot.Builder.Dialogs.Fakes.StubIDialogContext();
context.PostAsyncIMessageActivityCancellationToken = (messageActivity, token) => {
message = messageActivity;
return Task.CompletedTask;
};
Microsoft.Bot.Builder.Dialogs.Fakes.ShimExtensions.WaitIDialogStackResumeAfterOfIMessageActivity = (stack, callback) =>
{
if (waitCalled) return;
waitCalled = true;
callback(context, awaitable);
};
await target.MessageReceivedWithTextAsync(context, awaitable);
Assert.AreEqual(att, message.Attachments[0]);
}
}
}