We Keep Coding

Best approach to a class that would run in background polling other API and saving data

So I have a website written in .NET Core C# and I would like to run a process in the background that would make API calls to other website and save the data in database.
I have created ApiAccessor class and would like to invoke the method from the controller (which uses dependency injections for it's database connection), but if I pass them to the ApiAccessor (it would be async) the connection is already disposed of. I've tried injecting it from the get go, but it will still say that the interfaces are disposed, by the time it finishes. I can only do await on it, but this would cause user to wait for too long. What approach should I take with this one? I am a newbie at DI. Maybe some Singleton class? I would still don't know how to pass dependency injections to singleton
ApiAccessor:
IUserAccount _userAccounts;
public ApiAccessor(IConfiguration configuration, IUserAccount userAccounts)
{
_configuration = configuration;
_userAccounts = userAccounts;
}
//...
MethodToPollApi(){
var newUserIdToAdd = // just some kind of new data from api
_userAccounts.Add(newUserIdToAdd) // accessing DB, that causes errors
}
Controller:
void Index(){
MethodToPollApi();
return View();
}

I would consider an idea of using as called background jobs. There are a few popular frameworks for this type of solutions. Within them: custom implementation based on IHostedService, Quartz.NET, Hangfire, and many more available.
I used to play with many of them, personally prefer Hangfire as it self bootstrapped, provide nice UI for jobs dashboard, and really easy to use - for instance, that is how triggering jobs light look like with it:
Run once immediately:
var jobId = BackgroundJob.Enqueue(() => Console.WriteLine("Fire-and-forget!"));
Run delayed:
BackgroundJob.Schedule(() => Console.WriteLine("Delayed!"), TimeSpan.FromDays(7));
Run repeating:
var jobId = RecurringJob.AddOrUpdate(() => Console.WriteLine("Recurring!"), Cron.Daily);
Pick up completed job and continue:
BackgroundJob.ContinueWith(jobId, () => Console.WriteLine("Continuation!"));

Continuing the answer from #Dmitry. With Hangfire you can do something like this.
services.AddHangfire(x => x.UseSqlServerStorage("<Your connection string>"));
Hope this helps.

Wait for a third-party API callback

I need to create an REST API that connect to a third party SOAP API. The third party API events are sent by callback to an URL I provide.
The typical steps my API go through is it starts a session with the third party by providing an ID and an callback URL. The third party can now send new events to my API through this URL when, for example, a new participant connects. Now sometimes i need to request specific info, like the list of participants for a given session(ID), and wait for the event containing the info.
Note that there may be multiple open sessions at the same time.
An example of what I need:
private string url = "http://myapi/callback";
[HttpGet]
[Route("createSession")]
public async Task<string> CreateSession()
{
var id = Guid.NewGuid().ToString();
var result = await ExternAPI.CreateSession(id, this.url);
return result; //contains the id
}
[HttpGet]
[Route("endSession")]
public async Task<string> EndSession([FromUri] string id)
{
var result = await ExternAPI.EndSession(id);
return result;
}
[HttpGet]
[Route("partipants")]
public async Task<string> Partipants([FromUri] string id)
{
ExternAPI.participants(id); // The results of this method will be sent to the callback function
results = // Wait for the results for this id
return results;
}
[HttpPost]
[Route("callback")]
public void Callback(body)
{
// notify waiting function and pass body
}
I came up with a solution using ReactiveX but I'm not really sure about its reliability in production. What I have in mind is to create a subject that never terminate and handle all the events but it is not a usual lifetime for a subject, what happens on error ? And I don't think I did it the "RX-way" (state concerns).
Here it is (you will need System.Reactive to run this code):
class Data
{
public int id;
public string value;
}
class Program
{
private static Subject<Data> sub;
static void Main(string[] args)
{
sub = new Subject<Data>();
Task.Run(async () => {
int id = 1;
ExternAPI(CallBackHook, id);
Data result = await sub.Where(data => data.id == id).FirstAsync();
Console.WriteLine("{0}", result.value);
});
Console.ReadLine();
}
static void CallBackHook(Data data)
{
sub.OnNext(data);
}
static String ExternAPI(Action<Data> callback, int id)
{
// Third-party API, access via SOAP. callback is normally an url (string)
Task.Run(() =>
{
Thread.Sleep(1000);
callback(new Data { id = id, value = "test" });
});
return "success";
}
}
An other way will be a dictionary of subjects, one for each session, so I could manage their lifetimes.

it is not a usual lifetime for a subject
what happens on error?
And I don't think I did it the "RX-way"
Yes, these are all perfectly valid concerns with this kind of approach. Personally, I don't much mind the last one, because even though Subjects are frowned-upon, many times they're just plain easier to use than the proper Rx way. With the learning curve of Rx what it is, I tend to optimize for developer maintainability, so I do "cheat" and use Subjects unless the alternative is equally understandable.
Regarding lifetime and errors, the solutions there depend on how you want your application to behave.
For lifetime, it looks like currently you have a WebAPI resource (the SOAP connection) requiring an explicit disconnect call from your client; this raises some red flags. At the very least, you'd want some kind of timeout there where that resource is disposed even if endSession is never called. Otherwise, it'll be all too easy to end up with dangling resources.
Also for errors, you'll need to decide the appropriate approach. You could "cache" the error and report it to each call that tries to use that resource, and "clear" the error when endSession is called. Or, if it's more appropriate, you could let an error take down your ASP.NET process. (ASP.NET will restart a new one for you).
To delay an API until you get some other event, use TaskCompletionSource<T>. When starting the SOAP call (e.g., ExternAPI.participants), you should create a new TCS<T>. The API call should then await the TaskCompletionSource<T>.Task. When the SOAP service responds with an event, it should take that TaskCompletionSource<T> and complete it. Points of note:
If you have multiple SOAP calls that are expecting responses over the same event, you'll need a collection of TaskCompletionSource<T> instances, along with some kind of message-identifier to match up which events are for which calls.
Be sure to watch your thread safety. Incoming SOAP events are most likely arriving on the thread pool, with (possibly multiple) API requests on other thread pool threads. TaskCompletionSource<T> itself is threadsafe, but you'd need to make your collection threadsafe as well.
You may want to write a Task-based wrapper for your SOAP service first (handling all the TaskCompletionSource<T> stuff), and then consume that from your WebAPI.
As a very broad alternative, instead of bridging SOAP with WebAPI, I would consider bridging SOAP with SignalR. You may find that this is a more natural translation. Among other things, SignalR will give you client-connect and client-disconnect events (complete with built-in timeouts for clients). So that may solve your lifetime issues more naturally. You can use the same Task-based wrapper for your SOAP service as well, or just expose the SOAP events directly as SignalR messages.

How to build architecture better

I have an ASP.NET Core application which calls a service from another library. The
service works with an external API, which requires a sessionId. We have to call a Login API method to get the sessionId. How long this sessionId lives and when it can be changed - we don't know. Rule is: sessionId can be valid for 1 request, for 10 requests, for 100 requests, can be valid 1 minute, 10 minutes, 1 day... Nobody knows it.
The service has many methods to call similar APIs:
public class BillRequest
{
private readonly HttpClient client;
public BillRequest()
{
client = new HttpClient
{
BaseAddress = new Uri("https://myapi.com/api/v2/")
};
}
public async Task<List<Dto1>> CustomerBankAccountListAsync(int start, int count)
{
List<KeyValuePair<string, string>> nvc = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("sessionId", CURRENT_SESSION_ID)
};
var customerStream = await client.PostAsync("List/CustomerBankAccount.json", new FormUrlEncodedContent(nvc));
var customerString = await customerStream.Content.ReadAsStringAsync();
//....
}
public async Task<List<Dto2>> Method2(int start, int count)
{
List<KeyValuePair<string, string>> nvc = new List<KeyValuePair<string, string>>
{
new KeyValuePair<string, string>("sessionId", CURRENT_SESSION_ID)
};
var customerStream = await client.PostAsync("List/Method2.json", new FormUrlEncodedContent(nvc));
var customerString = await customerStream.Content.ReadAsStringAsync();
//....
}
// logic to get SessionId here
public async Task LoginAsync()
{
}
How to implement to save this sessionId inside service?
There are many options to implement:
Call Login method every time before calling a method. Easy to implement, but bad approach, because we have many unnecessary requests then and use the sessionId only once
Save the sessionId on web application level and try to catch exception, when any method gets an 'invalid sessionId' back, and then call Login method, which will return a new sessionId. In this case we have to pass sessionId to constructor of BillRequest class. It works, but I don't like to move responsibility of service to other, because it's internal responsibility of service how to work with API.
Save sessionId inside the service itself and recall Login method inside service, when old sessionId is considered invalid, rewrite it by new etc. But how to save it as "static" in memory? I don't want to save it to any external places (file system, cloud etc), but I can't save to variable of class too, because object of class can be recreated...

I'd suggest certain mental shift here towards functional programming.
Think of sessionID as of a stream of independet values rather than a single object. Then your problem can be redefined in a following (semantically equivalent) way: given a typed stream (string in your case), how to observe its flow and react on incomming changes, which your source code does not control?
Well, there is an answer, proven by an Enterprise™: reactive extensions.
Techinically such a shift impliest that you're dealing with an IObservable<string> inside of your controller, which either can be injected via the standard .NET Core DI approach, or simply defined by the constructor. That's quite flexible, since rX gives your fully testable, unbelivable powerful toolset to deal with taks of this kind; rX is also compatible with native Task and hence, async/await feature. Nice fact is that it is really easy to inject required behavior from an outerworld and decorate exising observable with a more appropriate one: so, you're safe: once 3rd party's service logic changes, you can adopt your codebase almost instantly and painlessly.
What is gonna be inside that IObservable<string>? Well, I can't say, since you did not give enough information. It might be an interval asking remote server whether current sessionID is still valid and in case not - runs relogin procedure and notifies it's subscrivers about new value; it might be a timer responsible for compile-time known rule of expiration, it might be as sophisticated logic as you need: rX is flexible enough not to limit you on what can be achieved with it as long as you deal with (possible infinite) streams.
As a consequence, it means that you don't need any global value. Just subscribe to a stream of session ids and take latest - the one which is currently valid, do the job and dispose your subscription. It is not expensive and won't hit performance; neither would mess up concurency. Wrap rX into Task and await it, if you'd like to stick to a common .NET fashion.
P.S. 99% of what you would need to deliver an implementation is already there; you just need to combine it.

C# IHttpHandler and the use of httpclient for a web service

I have been tasked with maintaining an C# web service that pretty much boils down to the following code that is used in conjunction with IIS 8:
public class Handler1 : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
// here we process the request and return it after fetching some data
}
}
Now I need to create another web service, which in turn will request another web service for fetching data. What I've gathered is that the HttpClient() in C# is the new shiny, however finding out how to implement it in my case is not so easy (or if that would be a good solution).
My solution would something like
public class Handler1 : IHttpHandler
{
public void ProcessRequest(HttpContext context)
{
// make sure that the incoming request is valid
HttpClient client = new HttpClient();
// do stuff with ^client and request the other service
[..]
// return the data from the response from ^client to the first request
}
}
Am I on the right track or would this be a catastrophe? I would gladly take tips or pointers to relevant documentation.

There are two issues with your approach.
First, as others have said, HttpClient must not be created for each request, as this would result in your server creating one new TCP connection for each request and not dispose it from your connection pool until they reach their MaxIdleTimeout.
This is because each instance of HttpClient (or rather, each instance of HttpClientHandler) creates a unique ConnectionGroup within the ConnectionPool, so two HttpClient (created in such a way) will not share the same connections.
You can quite easily exhaust your ports (because connections are kept alive for quite some time), and even without that, you'll have all the performance issues you might find when deactivating keepalive (typically when connecting on a HTTPs server).
As for a solution, you might add a private readonly HttpClient field in your handler that you would initialize as you've done in your code (I believe HttpHandlers are instanciated once by IIS, not per-request, as long as it says it's reusable). (Edit: note that HttpClient instances are thread-safe).
The other problem you'll find is that IHttpHandler are synchronous by design. This means you'll have to return task.Result in your code. This will block the current thread (let's call it A), and all subsequent processing of the outbound request (sending the body of the request, reading the header of the response and reading the body of the response) will spawn use a thread from the thread pool to process this (let's call them B, C, and D, though they can all be the same thread since no step overlap in time).
But in Asp.net, you cannot safely do that. When your main thread A is processing an Asp.net inbound request, it will, simply put, lock the current context. And when threads B, C and D will be spawn to process the outbound request, they too, will try to acquire and lock the current context. But because your main thread A is blocked, the context has not been released. This will effectively result in deadlocks (A locks the context and wait for B/C/D, B/C/D wait for the context).
To work around this, I would advise you to use instead a IHttpAsyncHandler. Async process are not made to be used synchronous, and HttpClient can only be used in an async way.
However, this might prove much more challenging for a first usage scenario. It would be much more easier to use a HttpClient from the context of a WebApi server. However, without any knowledge as to why you have to use a HttpHandler, I do not know if this would be in the realm of acceptable answers.
There are other way to run asynchronous tasks and wait synchronously for them (such as changing temporary the CurrentContext), but they're all very dangerous.
As a side note, HttpClient is the new shiny thing, and works quite well with other new shiny things such as "all-the-way async infrastructure". But trying to use it HttpClient in an older infrastructure is not as easy.

Better solution example:
public class YourApp
{
private static HttpClient Client = new HttpClient();
public static void Main(string[] args)
{
Console.WriteLine("Requests about to start!");
for(int i = 0; i < 5; i++)
{
var result = Client.GetAsync("http://www.stackoverflow.com").Result;
Console.WriteLine(result);
}
Console.WriteLine("Requests are finished!");
Console.ReadLine();
}
}

What do I need to do to prevent a client waiting on a long WCF service method?

I am writing a WCF webservice that includes a method that accepts an array of objects and inserts them into the database. This could take a long time, so I can't just expect the client to wait.
My colleague thinks that I don't need to do anything, that it's the client's job to call my service asynchronously. I just write a normal method. This doesn't sound right to me, although I hope it's true because looking at WCF async tutorials and SO questions has just confused me so far.
Is he correct? If not, how do I actually write the method in a way that would allow the client to call the method asynchronously or otherwise avoid hanging?
If he is correct (as appears to be the case), then what is the point of defining an asynchronous method ([OperationContract (AsyncPattern=true)], Begin, End, etc.). Is it a way explicitly handling asynchronous calls, or allowing interactivity, or what?

It should fall on the client's side. They are the ones that have to prevent their app/UI from hanging.
Have your client call your method asynchronously. If they are using a service reference, all methods/events are generated automatically.
myWcfClient.myMethodCompleted
+= new EventHandler<myMethodCompletedEventArgs>(myCallBack);
myWcfClient.myMethodAsync(args);
public void myCallback(object sender, myMethodCompletedEventArgs e)
{
var myResult = e.Result;
}

If your client doesn't care what happens with the service call, you want a simple fire and forget operation and you can do this.
The AsyncPattern property tells the runtime that your operations implement the .NET Framework asynchronous method design pattern. See here. If you want your client application to know what has happened with your service call then you can use this pattern. There are other ways to get the results though.

This is only on the client side, I've skipped the old event driven async bleh pattern and replaced it with the async-await pattern. Not waiting for webmethod calls async, and blocking the UI... doesn't even belong in this century ;)
If you are using .net 4.5+ you are getting the async-await pattern for free (Unless wp8, where you still have to wrap it). The async methods should already be avaliable through the service. I recommend the AsyncBridge if you are using old frameworks, which allows you to use the async-await pattern for cases like this. The alternative is to stick to the old event driven async nightmare. The examples below is only possible if you are using C#5.0 or never.
Ensure to start in a new thread from a non async method.
Task.Factory.StartNew(client.DoSomethingAsync("blabla") ).ContinueWith(...);
The last part is run after your method has completed, check for exceptions to completion code etc.
Or in some async method
public async Task<string> DoSomethingAsync(String text) {
// Exception handling etc
return await client.DoSomethingAsync(text);
}
wrapping APM to async-await pattern:
public class ServiceWrapper : IServiceWrapper
{
readonly YourServiceClient client;
public ServiceWrapper(YourServiceClient client)
{
this.client = client;
}
public async Task<string> DoSomethingAsync(string someParameter)
{
return await Task<string>.Factory.FromAsync(client.BeginDoSomeStuff, client.EndDoSomeStuff, someParameter, new object());
}
}
EDIT
Opening and closing connections in a wrapped service. (I don't have my devbox avaliable right now but this should work).
public class ServiceWrapper : IServiceWrapper
{
EndpointAddress address;
public ServiceWrapper(EndpointAddress clientAddress)
{
address = clientAddress;
}
public async Task<string> DoSomethingAsync(string someParameter)
{
// handle exceptions etc here, can be done some cleaner..
var client = new YourServiceClient();
client.Endpoint.Address = address.Address; // can skip this..
await client.OpenAsync()
var res = await Task<string>.Factory.FromAsync(client.BeginDoSomeStuff, client.EndDoSomeStuff, someParameter, new object());
await client.CloseAsync();
return res;
}
}
One last thing I'm not sure how you generate your proxy, if you are using vs make sure to hook of the checkbox allowing async methods when configuring the service. If you are using svcutil add the appropriate flags.
Async-await pattern
Old event driven async pattern
Hope it helps,
Cheers,
Stian