We Keep Coding

Is my approach correct for concurrent network requests?

I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).

My list of tasks only runs twice while it should run many times more

I want to run a function over all rows, but instead of it running for each row at a time, I want it to run in parallel. The issue is that it seems to only run twice (while there are 217 products on the list) and it seems rows are picked randomly. I'm not sure why it's happening - the synchronous version works as intended.
The function makes an API call to a SOAP service that uses row data from WPF SfDataGrid. Maybe the API closes connection after so many requests? It's really hard to tell and I don't know how I could check it besides writing a ticket and waiting a week.
A part of event from which the function is called:
var tasks = new List<Task>();
using (var db = new SqliteConnection("Data Source=file:products.sqlite"))
{
using (var client = new ApiOrdersPortTypeClient(binding, address))
{
db.Open();
// run ScanProduct() on each row in a WPF SfDataGrid
foreach (var row in ProductList.View.Records)
{
tasks.Add(Task.Factory.StartNew(
() => ScanProduct(row, desiredDays, deliveryTime, client, db)));
}
Task.WaitAll(tasks.ToArray());
}
}
ScanProduct():
private async Task ScanProduct(RecordEntry row, int desiredDays, int deliveryTime, ApiOrdersPortTypeClient client, SqliteConnection db)
{
await db.OpenAsync();
var selectedItem = (Product)row.Data;
if (selectedItem.IsIgnored == true | selectedItem.IsInDelivery == true)
{
return;
}
else
{
var pagesCount = await ApiRequests
.GetOrdersPages(int.Parse(selectedItem.Id), desiredDays, client);
var listSold = await ApiRequests
.GetOrdersFromApi(int.Parse(selectedItem.Id), desiredDays, pagesCount);
foat x = 100.0; // cut irrelevant stuff
if (x > selectedItem.Stock)
{
using (var command =
new SqliteCommand($"UPDATE products_settings SET requires_purchase = 1,"
+ $" was_checked_with = {desiredDays},"
+ $"sold_amount = {listSold.Count},"
+ $"average_daily = '{averageSales.ToString()}',"
+ $"average_delivery = '{x}'"
+ $"WHERE id = {selectedItem.Id};", db))
{
await db.OpenAsync();
command.ExecuteReader();
}
}
else
{
using (var command2 =
new SqliteCommand($"UPDATE products_settings SET requires_purchase = 0,"
+ $"was_checked_with = {desiredDays},"
+ $"sold_amount = {listSold.Count},"
+ $"average_daily = '{averageSales.ToString()}',"
+ $"average_delivery = '{x}"
+ $"WHERE id = {selectedItem.Id};", db))
{
await db.OpenAsync();
command2.ExecuteReader();
}
}
}
}

Your issue is here:
tasks.Add(Task.Factory.StartNew(() => ScanProduct(row, desiredDays, deliveryTime, client, db)));
ScanProduct is an asynchronous function, and thus needs to be awaited.
Each call to Task.Factory.StartNew returns a Task, which will be awaited using Task.WaitAll, however, within each task, a call to ScanProduct will return another Task, which is not being awaited by Task.WaitAll.
The solution is to use an async lambda in combination with Task.Run, then in each case, the outer Task will not complete until ScanProduct has also completed.
tasks.Add(Task.Run(async () => await ScanProduct(row, desiredDays, deliveryTime, client, db)));

Is Object created before or after method calls?

I have three different databases that I need to check that I am connected to. This is what I originally have, which works perfectly fine.
public async Task<ServiceAvailabilityDTO> ServiceAvailabilityStatus()
{
return new ServiceAvailabilityDTO
{
IsDb1Online = await IsDb1Available(),
IsDb2Online = IsDb2Available(),
IsDb3Online = await IsDb3Available()
};
}
private async Task<bool> IsDb1Available()
{
var count = await _db1Service.GetDbCount();
if (count > 0)
return true;
return false;
}
private bool IsDb2Available()
{
if (_db2Service.GetDbCount() > 0)
return true;
return false;
}
private async Task<bool> IsDb3Available()
{
var pong = await _db3Provider.PingDb();
if(pong.Success == true && pong.Version != null)
return true;
return false;
}
Now however, I need to log exception messages in my DTO for each DB check.
public async Task<ServiceAvailabilityDTO> ServiceAvailabilityStatus()
{
return new ServiceAvailabilityDTO
{
IsDb1Online = await IsDb1Available(),
IsDb2Online = IsDb2Available(),
IsDb3Online = await IsDb3Available(this) // This is an example. I want to pass the reference of **ServiceAvailabilityDTO** to **IsDb3Available**
};
}
private async Task<bool> IsDb3Available(ServiceAvailabilityDTO availability)
{
try
{
var pong = await _db3Provider.PingDb();
if(pong.Success == true && pong.Version != null)
return true;
return false;
}
catch (Exception e)
{
var exceptionMessage = e.Message;
if (e.InnerException != null)
{
// This is what I hope to put into the object reference
exceptionMessage = String.Join("\n", exceptionMessage, e.InnerException.Message);
availability.db3Exception = exceptionMessage ;
}
return false;
}
}
My question is;
Can I keep my return method the same as in the first example, and pass the object reference to each method to store the exception and still return my bool value.
Or does the object not get created until all of the method calls have happened, and then create the object with the returned values?
I know I could just create the object normally and pass it in each
method call, but it is specifically this way of doing it that has
inspired me to ask this question, purely to be informed and learn
from.
Thanks.

No, you cannot do it like this because in the context of what you're doing this does not refer to the object you're populating, it refers to the object containing the method you're calling.
public async Task<ServiceAvailabilityDTO> ServiceAvailabilityStatus()
{
return new ServiceAvailabilityDTO
{
IsDb1Online = await IsDb1Available(),
IsDb2Online = IsDb2Available(),
IsDb3Online = await IsDb3Available(this) // here 'this' does NOT ref to ServiceAvailabilityDTO
};
}
There is no keyword which does refer to ServiceAvailabilityDTO either, so you're left with creating the object, and passing it to each method. At this point, I dont think there is much point you returning the boolean either - you may as well set the boolean property in line
public async Task<ServiceAvailabilityDTO> ServiceAvailabilityStatus()
{
var sa = new ServiceAvailabilityDTO();
await CheckDb1Available(sa);
CheckDb2Available(sa);
await CheckDb3Available(sa);
return sa;
}
(Note I've renamed the methods from Is* to Check* as the former implies a return boolean, the latter implies something going on inline.)

Async await how to use return values

I have a windows service that I have inherited from another developer, it runs very slow and has numerous slow call to the eBay API. I wish to speed it up without too much refactoring.
I've just started to look at using c# async/await to try to get some of these slow call to run async.
Here's what i'm trying to achieve:
I have a 1 very busy method that makes lots of calls as below:
getProducts
getCategories
getVehicles
getImages
My thoughts were that I could simply change the methods to async and add Task<T> to the return type as below:
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem)
{
String additionalProductDetails = string.Empty;
if (oItem.ItemSpecifics.Count > 0)
{
foreach (NameValueListType nvl in oItem.ItemSpecifics)
{
if (nvl.Value.Count > 0)
{
foreach (string s in nvl.Value)
{
additionalProductDetails += "<li><strong>" + nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return additionalProductDetails;
}
Then call them with await:
Task<String> additionalProductDetials = ebayPartNumbers.ProcessAdditionalProductDetialsAsync(item);
Task<PartNumberCollection> partNumberCollection = ebayPartNumbers.ProcessPartNumbersAsync(item);
await Task.WhenAll(partNumberCollection, additionalProductDetials);
How do I get hold of the returned types so I can use them? I have tried just using partNumberCollection but it only has the await properties available.

Use Result property on Task class:
await Task.WhenAll(partNumberCollection, additionalProductDetials);
var partNumberCollectionResult = partNumberCollection.Result;
var additionalProductDetialsResult = additionalProductDetials.Result;

If the task returned by Task.WhenAll has completed, that means all of the tasks that you passed to it have completed too. That in turn means that you can use the Result property of each task, with no risk of it blocking.
string details = additionalProductDetials.Result;
Alternatively, you could await the tasks, for consistency with other async code:
string details = await additionalProductDetials;
Again, this is guaranteed not to block - and if you later remove the Task.WhenAll for some reason (e.g. you're happy to use the details to kick off another task before you've got the part number collection) then you don't need to change the code.

Your async method lacks of await operators and will run synchronously. while you are calling non blocking API you could use Task.Run() to do cpu-bound work on background thread.
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem)
{
return await Task.Run(() =>
{
String additionalProductDetails = string.Empty;
if (oItem.ItemSpecifics.Count > 0)
{
foreach (NameValueListType nvl in oItem.ItemSpecifics)
{
if (nvl.Value.Count > 0)
{
foreach (string s in nvl.Value)
{
additionalProductDetails += "<li><strong>" + nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return additionalProductDetails;
});
}
and get result
var detail = await ProcessAdditionalProductDetialsAsync(itemType);
var result = ProcessAdditionalProductDetialsAsync(itemType).Result;

Try this code:
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem) {
String additionalProductDetails = await Task.Run(() => {
if (oItem.ItemSpecifics.Count > 0) {
foreach (NameValueListType nvl in oItem.ItemSpecifics) {
if (nvl.Value.Count > 0) {
string retval = String.Empty;
foreach (string s in nvl.Value) {
retval += "<li><strong>"
+ nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return retval;
}
return additionalProductDetails;
}
Usage:
private async void GetAdditionalProductDetailsAsync(Action<string> callback) {
string apd = await ProcessAdditionalProductDetialsAsync();
callback(apd);
}
private void AdditionalProductDetailsRetrieved(string apd) {
// do anything with apd
}

Threading issues with SQLite3 and C# async

I am trying to save some data to a SQLite3 database. If I do not use async, I can save the data without any problems. As soon as I try to use the following code however, I receive the following error:
{Unable to evaluate expression because the code is optimized or a native frame is on top of the call stack.}
From my UI, I invoke the following SyncDomainTablesAsync method:
private readonly IDataCoordinator _coordinator;
public Configuration(IDataCoordinator coordinator)
{
_coordinator = coordinator;
}
public async Task<int> SyncDomainTablesAsync(IProgress<string> progress, CancellationToken ct, DateTime? lastDateSynced=null, string tableName = null)
{
//Determine the different type of sync calls
// 1) Force Resync (Drop/Create Tables and Insert)
// 2) Auto Update
var domainTable = await GetDomainTablesAsync(progress,ct,lastDateSynced, tableName);
var items = domainTable.Items;
int processCount = await Task.Run<int>( async () =>
{
int p = 0;
progress.Report(String.Format("Syncing Configurations..."));
foreach (var item in items)
{
progress.Report(String.Format("Syncing {0} Information",item.Key));
var task = await SyncTableAsync(item.Value); // INVOKED BELOW
if (task) progress.Report(String.Format("Sync'd {0} {1} records", item.Value.Count,item.Key));
if (ct.IsCancellationRequested) goto Cancelled;
p += item.Value.Count;
}
Cancelled:
if (ct.IsCancellationRequested)
{
//Update Last Sync'd Records
progress.Report(String.Format("Canceling Configuration Sync..."));
ct.ThrowIfCancellationRequested();
}
else
progress.Report(String.Format("Syncing Configurations Compleleted"));
return p;
},ct);
return processCount;
}
private async Task<bool> SyncTableAsync(IEnumerable<object> items, bool includeRelationships = false)
{
try
{
//TODO: Replace with SaveObjects method
var i = await Task.Run(() => _coordinator.SaveObjects(items, includeRelationships));
if (i == 0)
return false;
}
catch(Exception ex)
{
return false;
}
return true;
}
The UI invokes the SyncDomainTablesAsync method. I then create a new Task and loop through the items that were returned from the GetDomainTablesAsync method. During each iteration I await until the SyncTableAsync method completes. Within the SyncTableAsync I am calling a SaveObject method inside of a class that implements my IDataCoordinator interface.
public override int SaveObjects(IEnumerable<object> items, Type underlyingType, bool saveRelationships = true)
{
int result = 0;
if (items == null)
throw new ArgumentNullException("Can not save collection of objects. The collection is null.");
else if (items.Count() == 0)
return 0;
// Check if table exists.
foreach (var item in items)
this.CreateTable(item.GetType(), saveRelationships);
using (SQLiteConnection connection = new SQLiteConnection(this.StorageContainerPath))
{
connection.BeginTransaction();
foreach (var item in items)
{
result += ProcessSave(item, saveRelationships, connection);
}
try
{
connection.Commit();
}
catch (SQLiteException ex)
{
connection.Rollback();
throw ex;
}
}
return result;
}
public override int CreateTable(Type type, bool createRelationalTables = false)
{
if (this.TableExists(type) == 1)
return 1;
using (SQLiteConnection cn = new SQLiteConnection(this.StorageContainerPath))
{
try
{
// Check if the Table attribute is used to specify a table name not matching that of the Type.Name property.
// If so, we generate a Sql Statement and create the table based on the attribute name.
//if (Attribute.IsDefined(type, typeof(TableAttribute)))
//{
// TableAttribute attribute = type.GetAttribute<TableAttribute>();
// Strongly typed to SQLiteCoordinator just to get a SqlQuery instance. The CreateCommand method will create a table based on 'type'
var query = new SqlQuery<SQLiteCoordinator>().CreateCommand(DataProviderTypes.Sqlite3, type);
query = query.TrimEnd(';') + ";";
cn.Execute(query);
//}
// Otherwise create the table using the Type.
//else
//{
// cn.CreateTable(type);
//}
// If we are to create relationship tables, we cascade through all relationship properties
// and create tables for them as well.
if (createRelationalTables)
{
this.CreateCascadingTables(type, cn);
}
}
catch (Exception ex)
{
return 0;
}
}
return 1;
}
The flow of the code goes
UI->SyncDomainTablesAsync->SyncTableAsync->SaveObjects->SaveTable(type)
The issue that I have is within Save Table. If I just use SaveTable synchronously I have no issues. Using it in my async method above, causes a thread abort exception. The exception is thrown within the SQLite.cs file included with SQLite.net (within the . The weird thing is that the table is created in the database, even though the exception is thrown. The error is thrown some times when the Prepare() function is called and the rest of the time when the SQLite3.Step() function is called.
public int ExecuteNonQuery ()
{
if (_conn.Trace) {
Debug.WriteLine ("Executing: " + this);
}
var r = SQLite3.Result.OK;
var stmt = Prepare (); // THROWS THE ERRROR
r = SQLite3.Step(stmt); // THROWS THE ERRROR
Finalize(stmt);
if (r == SQLite3.Result.Done) {
int rowsAffected = SQLite3.Changes (_conn.Handle);
return rowsAffected;
} else if (r == SQLite3.Result.Error) {
string msg = SQLite3.GetErrmsg (_conn.Handle);
throw SQLiteException.New (r, msg);
} else {
throw SQLiteException.New (r, r.ToString ());
}
}
I assume that because my foreach statement awaits the return of SyncTableAsync that none of the threads are closed. I am also getting a system transaction critical exception that says "attempting to access a unloaded app domain".
Am I using await/async incorrectly with Sqlite3 or is this an issue with Sqlite3 that I am not aware of.
Attached is a photo of the Parallel's stack and the exception.
EDIT
When I try to run the code above as well in unit tests, the unit tests process never dies. I have to exit Visual Studio in order to get the process to die. I am assuming something in SQLite.dll is grabbing a hold of the process when the exception is thrown and not letting go, but I am not sure.
EDIT 2
I can modify the initial method SyncDomainTablesAsync to the following and the code runs without error. The issue is my use of async and await I believe.
public async Task<int> SyncDomainTablesAsync(IProgress<string> progress, CancellationToken ct, DateTime? lastDateSynced=null, string tableName = null)
{
var domainTable = await GetDomainTablesAsync(progress,ct,lastDateSynced, tableName);
var items = domainTable.Items;
foreach (var item in items)
{
_coordinator.SaveObjects(item.Value, typeof(object), true);
}
return 1;
}