We Keep Coding

How to design parallel web api in c#?

I am trying to design a web api that can get data from an external server but with limitations. I'm trying to figure out how best to design it to be efficient.
My api has an endpoint that takes an input. It is is a domain name like tom#domain.com. My endpoint then makes an http call to the domain to get an auth token, then makes another call to that domain with the username to get some data which is returned to the client. However my api can accept multiple usernames (comma delimited like ?users=tom#domain.a.com, bill#domain.b.com). My web server knows for each domain what is the max parallel connections I can make to get the data.
So the problem is how to organize the data so I can maximize parallel computing but stay within the limits.
Here's my thoughts:
First parse the user list and group them up. Then have a static dictionary. Key is domain, value is a custom object which has 2 queues. Both queues holds a list of Tasks (from async/await). However the first queue max length will be the value of the limit for that domain.
?users=bill#D.com, max#D.com, sarah#A.com, tom#D.com
dictionary = {
"D.com" : [
[],
["bill#D.com", "max#D.com", "tom#D.com"]
],
"A.com" : [
[],
["sarah#A.com"]
]
}
Then I can run a code every second, which loops through all dictionary values, and fills the first queue with as many Task objects from the second queue (.e. removing from 2nd queue and putting in first) so its within the limit.
As soon as its in the first queue, the task executes using Parallel.Invoke() then when the task is completed it gets removed from first queue (unless some request is waiting for it, explained in next paragraph).
I do this because if another api request is made to my endpoint with some names thats already from the first request, I want to reuse it. So If it's in the first queue, I call await on that Task.
Somehow when a task finishes, I need to know that no other people are waiting for that user in the task, and in that case, remove it from the first queue. Also if a client disconnects it should remove the watching of the users part for that client.
Does anyone know if this is a good approach?

Since it's parallel, you know right away you're probably going to need to use System.Collections.Concurrent, and since you need key/value lookup (user identifier/HTTP response) you need a ConcurrentDictionary. And since there is a common cache for all users, you will want to store it in a static variable, which is available to all threads and all HTTP requests.
Here is a simple example:
public class MyCacheClass
{
//Store the list of users/requests
static private ConcurrentDictionary<string, Task<HttpResponseMessage>> _cache = new ConcurrentDictionary<string, Task<HttpResponseMessage>>();
//Get from the ConcurrentDictionary or add if it's not there
public async Task<HttpResponseMessage> GetUser(string key)
{
return await _cache.GetOrAdd(key, GetResponse(key));
}
//You just to implement this method, potentially in a subclass, to get the data
protected virtual async Task<HttpResponseMessage> GetResponse(string key)
{
var httpClient = new HttpClient();
var url = string.Format(#"http://www.google.com?q={0}", key);
return await httpClient.GetAsync(url);
}
}
Then to get a user's information, just call:
var o = new MyCacheClass();
var userInfo = await o.GetUser(userID);
Note: If you're going to use code like this on a production system, you might consider adding some means of purging or trimming the cache after a period of time or when it reaches a certain size. Otherwise your solution may not scale the way you need it to.

Better way to post to an external api at a specific time everyday in Asp.Net Core C#

I have a requirement where the Asp.Net Core Application(Deployed to IIS) needs to send data external domain("http://example.com/api/statistics") at a given time everyday(only once a day; say 6PM localTime where application is running). I am hesitant to place code any place(like in Startup.cs or Program.cs) that might create problems later. Something like the following : Your insights highly appreciated. Thank you.
Task.Run(() =>
{
while (true)
{
using (var client = new HttpClient())
{
var response = client.PostAsync("http://example.com/api/statistics",
new StringContent(JsonConvert.SerializeObject("data"),
Encoding.UTF8, "application/json"));
}
}
});

There's a number of ways to approach this, however the way I think works well is make a controller with an action which will do the post. That way anything can trigger the posting of statistics ( you will want an authorization token of some sort so only things that are meant to trigger the action can do it )
so something like mysite/statistics/post?url=<destination url>&...any other options
then you can use a scheduled task, or manually trigger it, or trigger it via a webhook, or any other mechanisim. You can even still use a Task that waits for a particular time then calls your hook.

Send and return variable with c# API call?

I have a c# script task in an ssis package designed to geocode data through my company's proprietary system. It currently works like this:
1) Pull query of addresses and put in data table
2) Loop through that table and Foreach row, build request, send request, wait for response, then insert back into the database.
The issue is that each call takes forever to return, because before going out and getting a new address on the api side, it checks a current database(string match) to ensure the address does not already exist. If not exists, then go out and get me new data from a service like google.
Because I'm doing one at a time, it makes it easy to keep the ID field with the record when I go back to insert it into the database.
Now comes the issue at hand... I was told to configure this as multi-thread or asynchronous. Here is the page I was reading on here about this topic:
ASP.NET Multithreading Web Requests
var urls = new List<string>();
var results = new ConcurrentBag<OccupationSearch>();
Parallel.ForEach(urls, url =>
{
WebRequest request = WebRequest.Create(requestUrl);
string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));
results.Add(result);
});
Perhaps I'm thinking about this wrong, but if I send 2 requests(A & B) and lets say B actually returns first, how can I ensure that when I go back to update my database I'm updating the correct record? Can I send the ID with the API call and return it?
My thoughts are to create an array of requests, burn through them without waiting for a response and return those value in another array, that I will then loop through on my insert statement.
Is this a good way of going about this? I've never used Parrallel.ForEach, and all the info I find on it is too technical for me to visualize and apply to my situation.

Perhaps I'm thinking about this wrong, but if I send 2 requests(A & B) and lets say B actually returns first, how can I ensure that when I go back to update my database I'm updating the correct record? Can I send the ID with the API call and return it?
None of your code contains anything that looks like an "ID," but I assume everything you need is in the URL. If that is the case, one simple answer is to use a Dictionary instead of a Bag.
List<string> urls = GetListOfUrlsFromSomewhere();
var results = new ConcurrentDictionary<string, OccupationSearch>();
Parallel.ForEach(urls.Distinct(), url =>
{
WebRequest request = WebRequest.Create(url);
string response = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();
var result = JsonSerializer().Deserialize<OccupationSearch>(new JsonTextReader(new StringReader(response)));
results.TryAdd(url, result);
});
After this code is done, the results dictionary will contain entries that correlate each response back to the original URL.
Note: you might want to use HttpClient instead of WebClient, and you should take care to dispose of your disposable objects, e.g. StreamReader and StringReader.

Any downsides to replacing REST endpoints with SignalR?

I'm building a fairly simple single page app. It's basically a list of items, where each item has some details, an activity log, and a current status along with some buttons to trigger actions on the server to advance the status along a workflow.
It was originally written using MVC and REST/Web API but I got stuck on the problem of keeping concurrent users up to date. For example, if User A adds an item, we want the list on User B's screen to now update to include it.
To solve this I looked into SignalR which works great. But I had a problem.
When adding an item (using POST) the callback adds the item on the requesting client. This is fine.
I then triggered a SignalR broadcast on the server to tell all clients about the new item. This worked fine except the local client, who now has 2 items.
I was looking into filtering the duplicate id client-side, or sending the connection id with the POST, then broadcast to all clients except the requester but it seems a bit needlessly complicated.
Instead I'm just doing this.
public class UpdateHub : Hub
{
public void AddNewItem(NewItem item)
{
// and some server-side stuff, persist in the data store, etc
item.trackingID = new Guid();
item.addLogEntry("new item");
// ...
dataStore.addItem(item);
// send message type and data payload
Clients.All.broadcastMessage("add", item);
}
}
It seems a lot simpler to just get rid of all the REST stuff altogether, so am I missing anything important?
It'll run on an intranet for a handful of users using IE11+ and I guess we do lose some commonly-understood semantics around HTTP response codes for error handling, but I don't think that's a huge deal in this situation.

In order to solve duplicate you can try to use Clients.Others inside Hub class, or AllExcept(id) if you not in the Hub class.
Clients.Others.broadcastMessage("add", item);
In your case using SignalR shouldn`t have any downsides.

Calling method async from WCF service and return immediately

A third party is calling our WCF service. The caller wants confirmation, that the sent records have been received and stored, within a small timeframe.
The records that are stored need some lenghty processing. Can the processing be executed async, right after storing the records, so the confirmation can be send immediately?
Ofcourse there can be a separate process that does the processing, but the question is whether I can combine storage and processing without timing out.
Update:
It looks like this works:
var aTask = new Task(myService.TheMethod);
aTask.Start();
return aVariableAsync;
Or is this a very bad idea to do from within my WCF host, because.. ?

You can set "AsyncPattern" to true on the OperationContract attribute as described on MSDN.
You can then control the concurrency using the following attribute on the service method:
[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]

Yes it can be done. I dont have a ton of experience with it, but this is a snippet of some code showing that. The service calls this method on the controller that saves an xml message to the hard drive and then kicks off a separate task to process it into MongoDB and returns a message back to the service that it was successfully saved.
public string SaveTransaction(XElement pTransactionXml, string pSavePath)
{
//save the transaction to the drive locally
pTransactionXml.Save(pSavePath);
...
var mongoTask = Task.Run(async () =>
{
await SendXMLFilesToMongo(pSavePath);
});
return response.WithResult("Successfully saved to disk.");
}
public virtual async Task<int> SendXMLFilesToMongo(string pSavePath)
{
//call the code to save to mongo and do additional processing
}