We Keep Coding

Is my approach correct for concurrent network requests?

I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).

Never coming back from Task.WaitAll [duplicate]

This question already has answers here:
An async/await example that causes a deadlock
(5 answers)
Closed 2 years ago.
Still coming up to speed on Xamarin and C#.
I have some code like:
List<Task<int>> taskList = new List<Task<int>>();
ConfigEntry siteId = new ConfigEntry
{
ConfigKey = KEY_SITE_ID,
ConfigValue = siteInfo.siteId
};
taskList.Add(ConfigDatabase.SaveConfigAsync(siteId));
ConfigEntry productId = new ConfigEntry
{
ConfigKey = KEY_PRODUCT_ID1,
ConfigValue = siteInfo.products[0].productId.ToString()
};
taskList.Add(ConfigDatabase.SaveConfigAsync(productId));
There's a total of nine of these getting added to taskList. Each of these inserts stuff into SQLITE. Here is the code being run:
public async Task<int> SaveConfigAsync(ConfigEntry entry)
{
if (entry.ConfigKey == null)
{
throw new Square1Exception("Config entry key not defined:" + entry);
}
else
{
try
{
ConfigEntry existing = await GetConfigAsync(entry.ConfigKey);
if (existing == null)
{
return await _database.InsertAsync(entry);
}
else
{
existing.UpdateFrom(entry);
return await _database.UpdateAsync(entry);
}
}
catch (Exception ex)
{
Console.WriteLine("Error while saving value:" + entry.ConfigKey);
throw ex;
}
}
}
So at the end of the building of this tasklist, I have the following line:
Task.WaitAll(taskList.ToArray());
Which I had hoped would wait until all of the adds completed before exiting. Instead it is never coming back from this. It just hangs my whole app. Not seeing anything in the log either. Does it (potentially) start the task when created or wait until something like WaitAll?
If I replace each of the adds with an await and single thread them it works fine. Maybe blocking on the database or disk?
Ideas?

You shouldn't block on asynchronous code.
The best fix is to change Task.WaitAll(taskList.ToArray()); to await Task.WhenAll(taskList);.
If you must block, then you can use Task.Run to push the work to background threads, as such:
taskList.Add(Task.Run(() => ConfigDatabase.SaveConfigAsync(siteId)));
...
taskList.Add(Task.Run(() => ConfigDatabase.SaveConfigAsync(productId)));
But then you would be blocking your UI thread at the Task.WaitAll, so I don't recommend that approach. Using await Task.WhenAll is better.

You are missing await when doing the Task.WhenAll(Tasks).
Try the following solution:
ConfigEntry siteId = new ConfigEntry
{
ConfigKey = KEY_SITE_ID,
ConfigValue = siteInfo.siteId
};
ConfigEntry productId = new ConfigEntry
{
ConfigKey = KEY_PRODUCT_ID1,
ConfigValue = siteInfo.products[0].productId.ToString()
};
var insertResultFirstTask = ConfigDatabase.SaveConfigAsync(siteId)
var insertResultSecondTask = ConfigDatabase.SaveConfigAsync(productId)
IEnumerable<Task> tasks = new List<Task>() {
insertResultFirstTask,
insertResultSecondTask
};
await Task.WhenAll(tasks);
var insertResultFirst = insertResultFirstTask.Result;
var insertResultSecond = insertResultSecondTask.Result;

How to find all id and autopaste as parameter using linq?

I have 2 projects. One of them aspnet core webapi and second one is console application which is consuming api.
Api method looks like:
[HttpPost]
public async Task<IActionResult> CreateBillingInfo(BillingSummary
billingSummaryCreateDto)
{
var role = User.FindFirst(ClaimTypes.Role).Value;
if (role != "admin")
{
return BadRequest("Available only for admin");
}
... other properties
billingSummaryCreateDto.Price = icu * roc.Price;
billingSummaryCreateDto.Project =
await _context.Projects.FirstOrDefaultAsync(x => x.Id ==
billingSummaryCreateDto.ProjectId);
await _context.BillingSummaries.AddAsync(billingSummaryCreateDto);
await _context.SaveChangesAsync();
return StatusCode(201);
}
Console application which consuming api:
public static async Task CreateBillingSummary(int projectId)
{
var json = JsonConvert.SerializeObject(new {projectId});
var data = new StringContent(json, Encoding.UTF8, "application/json");
using var client = new HttpClient();
client.DefaultRequestHeaders.Authorization =
new AuthenticationHeaderValue("Bearer", await Token.GetToken());
var loginResponse = await client.PostAsync(LibvirtUrls.createBillingSummaryUrl,
data);
WriteLine("Response Status Code: " + (int) loginResponse.StatusCode);
string result = loginResponse.Content.ReadAsStringAsync().Result;
WriteLine(result);
}
Program.cs main method looks like:
static async Task Main(string[] args)
{
if (Environment.GetEnvironmentVariable("TAIKUN_USER") == null ||
Environment.GetEnvironmentVariable("TAIKUN_PASSWORD") == null ||
Environment.GetEnvironmentVariable("TAIKUN_URL") == null)
{
Console.WriteLine("Please specify all credentials");
Environment.Exit(0);
}
Timer timer = new Timer(1000); // show time every second
timer.Elapsed += Timer_Elapsed;
timer.Start();
while (true)
{
Thread.Sleep(1000); // after 1 second begin
await PollerRequests.CreateBillingSummary(60); // auto id
await PollerRequests.CreateBillingSummary(59); // auto id
Thread.Sleep(3600000); // 1hour wait again requests
}
}
Is it possible find all id and paste it automatically instead of 59 and 60? Ids from projects table. _context.Projects
Tried also approach using method which returns ids
public static async Task<IEnumerable<int>> GetProjectIds2()
{
var json = await
Helpers.Transformer(LibvirtUrls.projectsUrl);
List<ProjectListDto> vmList =
JsonConvert.DeserializeObject<List<ProjectListDto>>(json);
return vmList.Select(x => x.Id).AsEnumerable(); // tried
ToList() as well
}
and in main method used:
foreach (var i in await PollerRequests.GetProjectIds2())
new List<int> { i }
.ForEach(async c => await
PollerRequests.CreateBillingSummary(c));
for first 3 ids it worked but does not get other ones,
tested with console writeline method returns all ids

First get all Ids:
var ids = await PollerRequests.GetProjectIds2();
Then create list of task and run all tasks:
var taskList = new List<Task>();
foreach(var id in ids)
taskList.Add(PollerRequests.CreateBillingSummary(id));
await Task.WhenAll(taskList);

Why do I have to use (async method).result instead of await (async method)?

I am starting 2 channels in the mediaservices azure portal.
Starting a channel takes a long time to complete, about 25-30 seconds per channel. Hence, multithreading :)
However, the following is not clear to me:
I have 2 methods:
public async Task<bool> StartAsync(string programName, CancellationToken token = default(CancellationToken))
{
var workerThreads = new List<Thread>();
var results = new List<bool>();
foreach (var azureProgram in _accounts.GetPrograms(programName))
{
var thread = new Thread(() =>
{
var result = StartChannelAsync(azureProgram).Result;
lock (results)
{
results.Add(result);
}
});
workerThreads.Add(thread);
thread.Start();
}
foreach (var thread in workerThreads)
{
thread.Join();
}
return results.All(r => r);
}
and
private async Task<bool> StartChannelAsync(IProgram azureProgram)
{
var state = _channelFactory.ConvertToState(azureProgram.Channel.State);
if (state == State.Running)
{
return true;
}
if (state.IsTransitioning())
{
return false;
}
await azureProgram.Channel.StartAsync();
return true;
}
in the first method I use
var result = StartChannelAsync(azureProgram).Result;
In this case everything works fine. But if I use
var result = await StartChannelAsync(azureProgram);
Executing is not awaited and my results has zero entries.
What am I missing here?
And is this a correct way?
Any comments on the code is appreciated. I am not a multithreading king ;)
Cheers!

Don't span new Thread instances to execute tasks in parallel, instead use Task.WhenAll:
public async Task<bool> StartAsync(string programName, CancellationToken token = default(CancellationToken))
{
// Create a task for each program and fire them "at the same time"
Task<bool>[] startingChannels = _accounts.GetPrograms(programName))
.Select(n => StartChannelAsync(n))
.ToArray();
// Create a task that will be completed when all the supplied tasks are done
bool[] results = await Task.WhenAll(startingChannels);
return results.All(r => r);
}
Note: I see that you're passing a CancellationToken to your StartAsync method, but you're not actually using it. Consider passing it as an argument to StartChannelAsync, and then use it when calling azureProgram.Channel.StartAsync
If you love one-liners:
public async Task<bool> StartAsync(string programName, CancellationToken token = default(CancellationToken))
{
return (await Task.WhenAll(_accounts.GetPrograms(programName)
.Select(p => StartChannelAsync(p))
.ToArray())).All(r => r);
}

C# await lambda function

I'll start off by publishing the code that is troubled:
public async Task main()
{
Task t = func();
await t;
list.ItemsSource = jlist; //jlist previously defined
}
public async Task func()
{
TwitterService service = new TwitterService(_consumerKey, _consumerSecret);
service.AuthenticateWith(_accessToken, _accessTokenSecret);
TwitterGeoLocationSearch g = new TwitterGeoLocationSearch(40.758367, -73.982706, 25, 0);
SearchOptions s = new SearchOptions();
s.Geocode = g;
s.Q = "";
s.Count = 1;
service.Search(s, (statuses, response) => get_tweets(statuses, response));
void get_tweets(TwitterSearchResult statuses, TwitterResponse response)
{
//unimportant code
jlist.Add(info);
System.Diagnostics.Debug.WriteLine("done with get_tweets, jlist created");
}
I am having issues with the get_tweets(..) function running (on what I believe a different thread) and the Task t is not awaited like I have in the main function. Basically, my issue is that the list.Itemsource = jlist is ran before the get_tweets function is finished. Does anyone have a solution or the right direction to point me in?

First, create a TAP wrapper for TwitterService.Search, using TaskCompletionSource. So something like:
public static Task<Tuple<TwitterSearchResult, TwitterResponse>> SearchAsync(this TwitterService service, SearchOptions options)
{
var tcs = new TaskCompletionSource<Tuple<TwitterSearchResult, TwitterResponse>>();
service.Search(options, (status, response) => tcs.SetResult(Tuple.Create(status, response)));
return tcs.Task;
}
Then you can consume it using await:
SearchOptions s = new SearchOptions();
s.Geocode = g;
s.Q = "";
s.Count = 1;
var result = await service.SearchAsync(s);
get_tweets(result.Item1, result.Item2);