I have a windows service that I have inherited from another developer, it runs very slow and has numerous slow call to the eBay API. I wish to speed it up without too much refactoring.
I've just started to look at using c# async/await to try to get some of these slow call to run async.
Here's what i'm trying to achieve:
I have a 1 very busy method that makes lots of calls as below:
getProducts
getCategories
getVehicles
getImages
My thoughts were that I could simply change the methods to async and add Task<T> to the return type as below:
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem)
{
String additionalProductDetails = string.Empty;
if (oItem.ItemSpecifics.Count > 0)
{
foreach (NameValueListType nvl in oItem.ItemSpecifics)
{
if (nvl.Value.Count > 0)
{
foreach (string s in nvl.Value)
{
additionalProductDetails += "<li><strong>" + nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return additionalProductDetails;
}
Then call them with await:
Task<String> additionalProductDetials = ebayPartNumbers.ProcessAdditionalProductDetialsAsync(item);
Task<PartNumberCollection> partNumberCollection = ebayPartNumbers.ProcessPartNumbersAsync(item);
await Task.WhenAll(partNumberCollection, additionalProductDetials);
How do I get hold of the returned types so I can use them? I have tried just using partNumberCollection but it only has the await properties available.
Use Result property on Task class:
await Task.WhenAll(partNumberCollection, additionalProductDetials);
var partNumberCollectionResult = partNumberCollection.Result;
var additionalProductDetialsResult = additionalProductDetials.Result;
If the task returned by Task.WhenAll has completed, that means all of the tasks that you passed to it have completed too. That in turn means that you can use the Result property of each task, with no risk of it blocking.
string details = additionalProductDetials.Result;
Alternatively, you could await the tasks, for consistency with other async code:
string details = await additionalProductDetials;
Again, this is guaranteed not to block - and if you later remove the Task.WhenAll for some reason (e.g. you're happy to use the details to kick off another task before you've got the part number collection) then you don't need to change the code.
Your async method lacks of await operators and will run synchronously. while you are calling non blocking API you could use Task.Run() to do cpu-bound work on background thread.
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem)
{
return await Task.Run(() =>
{
String additionalProductDetails = string.Empty;
if (oItem.ItemSpecifics.Count > 0)
{
foreach (NameValueListType nvl in oItem.ItemSpecifics)
{
if (nvl.Value.Count > 0)
{
foreach (string s in nvl.Value)
{
additionalProductDetails += "<li><strong>" + nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return additionalProductDetails;
});
}
and get result
var detail = await ProcessAdditionalProductDetialsAsync(itemType);
var result = ProcessAdditionalProductDetialsAsync(itemType).Result;
Try this code:
public async Task<String> ProcessAdditionalProductDetialsAsync(ItemType oItem) {
String additionalProductDetails = await Task.Run(() => {
if (oItem.ItemSpecifics.Count > 0) {
foreach (NameValueListType nvl in oItem.ItemSpecifics) {
if (nvl.Value.Count > 0) {
string retval = String.Empty;
foreach (string s in nvl.Value) {
retval += "<li><strong>"
+ nvl.Name + ":</strong> " + s + "</li>";
}
}
}
}
return retval;
}
return additionalProductDetails;
}
Usage:
private async void GetAdditionalProductDetailsAsync(Action<string> callback) {
string apd = await ProcessAdditionalProductDetialsAsync();
callback(apd);
}
private void AdditionalProductDetailsRetrieved(string apd) {
// do anything with apd
}
Related
I wrote a web crawler and I want to know if my approach is correct. The only issue I'm facing is that it stops after some hours of crawling. No exception, it just stops.
1 - the private members and the constructor:
private const int CONCURRENT_CONNECTIONS = 5;
private readonly HttpClient _client;
private readonly string[] _services = new string[2] {
"https://example.com/items?id=ID_HERE",
"https://another_example.com/items?id=ID_HERE"
}
private readonly List<SemaphoreSlim> _semaphores;
public Crawler() {
ServicePointManager.DefaultConnectionLimit = CONCURRENT_CONNECTIONS;
_client = new HttpClient();
_semaphores = new List<SemaphoreSlim>();
foreach (var _ in _services) {
_semaphores.Add(new SemaphoreSlim(CONCURRENT_CONNECTIONS));
}
}
Single HttpClient instance.
The _services is just a string array that contains the URL, they are not the same domain.
I'm using semaphores (one per domain) since I read that it's not a good idea to use the network queue (I don't remember how it calls).
2 - The Run method, which is the one I will call to start crawling.
public async Run(List<int> ids) {
const int BATCH_COUNT = 1000;
var svcIndex = 0;
var tasks = new List<Task<string>>(BATCH_COUNT);
foreach (var itemId in ids) {
tasks.Add(DownloadItem(svcIndex, _services[svcIndex].Replace("ID_HERE", $"{itemId}")));
if (++svcIndex >= _services.Length) {
svcIndex = 0;
}
if (tasks.Count >= BATCH_COUNT) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
if (tasks.Count > 0) {
var results = await Task.WhenAll(tasks);
await SaveDownloadedData(results);
tasks.Clear();
}
}
DownloadItem is an async function that actually makes the GET request, note that I'm not awaiting it here.
If the number of tasks reaches the BATCH_COUNT, I will await all to complete and save the results to file.
3 - The DownloadItem function.
private async Task<string> DownloadItem(int serviceIndex, string link) {
var needReleaseSemaphore = true;
var result = string.Empty;
try {
await _semaphores[serviceIndex].WaitAsync();
var r = await _client.GetStringAsync(link);
_semaphores[serviceIndex].Release();
needReleaseSemaphore = false;
// DUE TO JSON SIZE, I NEED TO REMOVE A VALUE (IT'S USELESS FOR ME)
var obj = JObject.Parse(r);
if (obj.ContainsKey("blah")) {
obj.Remove("blah");
}
result = obj.ToString(Formatting.None);
} catch {
result = string.Empty;
// SINCE I GOT AN EXCEPTION, I WILL 'LOCK' THIS SERVICE FOR 1 MINUTE.
// IF I RELEASED THIS SEMAPHORE, I WILL LOCK IT AGAIN FIRST.
if (!needReleaseSemaphore) {
await _semaphores[serviceIndex].WaitAsync();
needReleaseSemaphore = true;
}
await Task.Delay(60_000);
} finally {
// RELEASE THE SEMAPHORE, IF NEEDED.
if (needReleaseSemaphore) {
_semaphores[serviceIndex].Release();
}
}
return result;
}
4- The function that saves the result.
private async Task SaveDownloadedData(List<string> myData) {
using var fs = new FileStream("./output.dat", FileMode.Append);
foreach (var res in myData) {
var blob = Encoding.UTF8.GetBytes(res);
await fs.WriteAsync(BitConverter.GetBytes((uint)blob.Length));
await fs.WriteAsync(blob);
}
await fs.DisposeAsync();
}
5- Finally, the Main function.
static async Task Main(string[] args) {
var crawler = new Crawler();
var items = LoadItemIds();
await crawler.Run(items);
}
After all this, is my approach correct? I need to make millions of requests, will take some weeks/months to gather all data I need (due to the connection limit).
After 12 - 14 hours, it just stops and I need to manually restart the app (memory usage is ok, my VPS has 1 GB and it never used more than 60%).
I am trying to call HttpClient request inside for loop as follows. It needs to do multiple consecutive calls to third party rest api.
But it only gives me fist service call result while loop exit before getting result from rest of the service call.
private void Search()
{
try
{
var i = 1;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = response.Content.ReadAsStringAsync().Result;
Console.WriteLine(jsonResult.ToString());
i++;
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
When I run with debug points the program gives me all the result. But when I run it without debug points it gives me only the first result.
I tried this with using async, await methods too. It also gives me same result.
As I feel Program needs to wait until the async call returns data.
Please help me to solve this.
EDIT - async way
private async Task<string> SearchNew()
{
try
{
var i = 1;
var res = string.Empty;
using (var httpClient = new HttpClient())
{
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
var response = httpClient.GetAsync(url).Result;
string jsonResult = await response.Content.ReadAsStringAsync();
res = res + jsonResult + " --- ";
i++;
}
}
return res;
}
catch (Exception ex)
{
return ex.Message;
}
}
Both are giving same result.
There's a few things here that you should be doing. First, move the HttpClient creation outside of your method and make it static. You only need one of them and having multiple can be really bad for stability (see here):
private static HttpClient _client = new HttpClient();
Next, extract the calls to the HttpClient into a single method, something simple like this:
//Please choose a better name than this
private async Task<string> GetData(string url)
{
var response = await _client.GetAsync(url);
return await response.Content.ReadAsStringAsync();
}
And finally, you create a list of tasks and wait for them all to complete asynchronously using Task.WhenAll:
private async Task<string[]> SearchAsync()
{
var i = 1;
var tasks = new List<Task<string>>();
//Create the tasks
while (i < 5)
{
string url = "https://jsonplaceholder.typicode.com/posts/" + i;
tasks.Add(GetData(url));
i++;
}
//Wait for the tasks to complete and return
return await Task.WhenAll(tasks);
}
And to call this method:
var results = await SearchAsync();
foreach (var result in results)
{
Console.WriteLine(result);
}
I am trying to refactor my code below to increase its speed performance. I have noticed that the code in the foreach seems to be executed one after another, instead of in parallel to speed up the execution of the UpdateSites function. I just need UpdateSites to run in the background. What can I do to increase its performance?
await UpdateSites(currentUserSites);
private async Task<List<Site>> UpdateSites(List<Site> sites)
{
foreach (var site in sites)
{
var newSite = await FetchSite(site.SiteId);
site.address = newSite.address;
site.phone = newSite.phone;
}
return sites;
}
public async Task<SiteSimple> FetchSite(int siteId)
{
var url = $"/site/{siteId}/simple";
return await ExecuteRestRequest<SiteSimple>(url, Method.GET);
}
I refactored your code according to your need.
private Task<List<Site>> UpdateSites(List<Site> sites)
{
return sites.Select(x => FetchSite(site.SiteId))
}
public Task<SiteSimple> FetchSite(int siteId)
{
var url = $"/site/{siteId}/simple";
return ExecuteRestRequest<SiteSimple>(url, Method.GET);
}
then
var resultList = await Task.WhenAll(UpdateSites(currentUserSites));
foreach (var item in result)
{
//make your operation
}
You may want to use Task.WaitAll to fetch all tasks in parallel
private async Task<List<Site>> UpdateSites(List<Site> sites)
{
var newSites = (await Task.WhenAll(sites.Select(site => FetchSite(site.SiteId))
.ToList();
for(int i = 0; i < sites.Count; ++i)
{
sites[i].address = newSites[i].address;
sites[i].phone= newSites[i].phone;
}
return sites;
}
I have this method in my ViewModel:
private async void InicializarModulo(IModuloNeotek modulo)
{
var t = _service.InitializeModuloAsync(DataProvider.NombreInstanciaSqlServer,
modulo, Empresa.NombreEmpresa);
ServiceResult<IModuloNeotek> sResult = await t;
if (sResult.HasErrors)// this never runs
MessageBox.Show("Error");
}
This is my service call:
public async Task<ServiceResult<IModuloNeotek>> InitializeModuloAsync(string dataSource,
IModuloNeotek modulo, string nombreEmpresa)
{
return await Task.Run(() => InitializeModulo(dataSource, modulo, nombreEmpresa));
}
EDIT:
public ServiceResult<IModuloNeotek> InitializeModulo(string dataSource, IModuloNeotek modulo, string nombreEmpresa)
{
ServiceResult<IModuloNeotek> sResult;
if (DatabaseExists(dataSource, Strings.GetDbName(nombreEmpresa, modulo.NombreModulo)))
{
sResult = new ServiceResult<IModuloNeotek>(null);
sResult.Error =
string.Format(
"El modulo {0} ya esta inicializado para esta empresa.",
modulo.NombreModulo);
return sResult;
}
SqlQueryResult qResult = new SqlQueryResult();
string connString = GenerateConnectionString(dataSource);
switch (modulo.NombreModulo)
{
case "Contabilidad":
qResult = ExecuteScript(connString, GetScriptStream(TipoModulo.Contabilidad, nombreEmpresa));
if (qResult.Result)
{
string query =
string.Format("INSERT INTO Modulos (NombreModulo, Inicializado, EmpresaId)" +
"VALUES ('{0}', 'true', (SELECT IdEmpresa FROM Empresas WHERE NombreEmpresa=" +
"'{1}')) ", modulo.NombreModulo, nombreEmpresa);
qResult = ExeCuteQuery(query, dataSource, Resources.MaterDbName);
if (qResult.Result)
{
sResult = new ServiceResult<IModuloNeotek>(modulo);
return sResult;
}
}
break;
case "Proveedores":
break;
case "Produccion":
break;
}
sResult = new ServiceResult<IModuloNeotek>(null);
sResult.Error = qResult.Error;
sResult.InnerError = qResult.InnerError;
return sResult;
}
So, why it is not continuing execution? I tried everything, returning a Task from service and awaiting in ViewModel, not returning Task.Run, simply the task, I dont know that to do, any clues? thanks
Try changing:
private async void InicializarModulo(IModuloNeotek modulo)
to:
private async Task InicializarModulo(IModuloNeotek modulo)
Your service call is incorrect. It is already awaiting itself.
Update to not await.
private async void InicializarModulo(IModuloNeotek modulo)
{
var t = _service.InitializeModuloAsync(DataProvider.NombreInstanciaSqlServer, modulo, Empresa.NombreEmpresa);
ServiceResult<IModuloNeotek> sResult = await t;
if (sResult.HasErrors)// this never runs
MessageBox.Show("Error");
}
public Task<ServiceResult<IModuloNeotek>> InitializeModuloAsync(string dataSource, IModuloNeotek modulo, string nombreEmpresa)
{
return Task.Run(() => InitializeModulo(dataSource, modulo, nombreEmpresa));
}
For my app I make calls to a web service to get customer data. The problem I am encountering is when I make this particular call it gets the the asynchronous await call and loops back without finishing the call and then storing the results.
private void DatabaseTest()
{
cNum = Convert.ToString(db.selectCustomerNumber());
callC = "SELECT * FROM dashboardCustomer WHERE period = 'C' AND customerNumber = " + cNum;
callB = "SELECT * FROM dashboardCustomer WHERE period = 'B' AND customerNumber = " + cNum;
callF = "SELECT * FROM dashboardCustomer WHERE period = 'F' AND customerNumber = " + cNum;
if (db.selectDashboard(callC).Count == 0)
{
GetDataSummary(passC);
}
if (db.selectDashboard(callB).Count == 0)
{
GetDataSummary(passB);
}
if (db.selectDashboard(callF).Count == 0)
{
GetDataSummary(passF);
}
}
private async void GetDataSummary(string r)
{
long customerNum = db.selectCustomerNumber();
pin = db.selectPinByCustomerNumber(customerNum);
string cType = r;
try
{
Windows.Security.Credentials.PasswordVault vault = new Windows.Security.Credentials.PasswordVault();
IReadOnlyList<PasswordCredential> userCredential = vault.FindAllByResource(pin);
userCredential[0].RetrievePassword();
try
{
getCustomerBillSummaryResponse billSum = await
UBPclient.getCustomerBillSummaryAsync(userCredential[0].UserName, userCredential[0].Password, customerNum, cType);
invoiceSummaryBean[] summaryList = billSum.#return;
rh.DashboardHandler(summaryList, customerNum);
}
catch
{
}
}
catch
{
}
}
it runs to the following part
getCustomerBillSummaryResponse billSum = await
UBPclient.getCustomerBillSummaryAsync(userCredential[0].UserName, userCredential[0].Password, customerNum, cType);
and then loops back to the try and runs again until it has ran three times.
How do I make it return the data it is suppose to for each call and store it in my database?
Also I have tested the web service in SoapUI and the call is returning results, so the problem is not with the web service.
You need to do this:
private async Task GetDataSummary(string r)
You need to return Task instead of void because your caller needs to have something to wait for. When you return void, the caller must treat method as "fire-and-forget". When you return Task, the caller can create the necessary code to await for async method to finish.
And don't forget to add the await keyword when you call it: await GetDataSummaryAsync(...);
You should avoid async void. Change GetDataSummary to return Task and then await it from DatabaseTest:
private async Task DatabaseTestAsync()
{
...
if (db.selectDashboard(callC).Count == 0)
{
await GetDataSummaryAsync(passC);
}
if (db.selectDashboard(callB).Count == 0)
{
await GetDataSummaryAsync(passB);
}
if (db.selectDashboard(callF).Count == 0)
{
await GetDataSummaryAsync(passF);
}
}
private async Task GetDataSummaryAsync(string r)