We Keep Coding

Running a long running background task from OData/Web API request running on Azure App Service (.Net Framework)

We have a pretty memory heavy OData/Web API written in C# and running on an Azure App Service instance. Some specific endpoints which get used rarely can take a long time to execute (20s) or they can take an extremely long time (10min) depending on the query. We would like in that case to have some of sort of background task mechanism for executing these kind of tasks, ideally a header we can attach to any call to tell the API to run it in the background.
We have tried:
Returning the request immediately with a tracking id and then running the SendAsync task in a HostingEnvironment.QueueBackgroundWorkItem, however when the work is completed it throws an exception as its trying to do something with the initial web request which is already disposed
Returning the request immediately with a tracking id and then passing the SendAsync to HostingEnvironment.QueueBackgroundWorkItem as a function, however that also throws an error upon activating
Returning the request immediately with a tracking id and then enqueuing a job in the Hangfire which will make a direct call to the app service and can wait as long as it takes. However even using the internal dns, the azure app service has a timeout of 230 seconds, and we can not change that in any way
We've also tried running the business code itself in a different thread, but we use autofac DI and if we try and get a new scope it still closes when the original request returns.
Similar SO questions have gotten answers that I don't think apply to us:
Running the work itself in Hangfire, that would mean spinning up the whole API again, just for these rare calls, also we can't do that because our Hangfire is on AKS on Linux, and the API is .net framework.
Using a webjob - similar issue as the api is very complex and deploying it to a webjob seems very dangerous, also we will then have to pay for 24/7 webjob.
Using a queue, the problem is that I don't really know how we can take a message from a queue and then execute it against the odata controllers, which we will need to do to populate various contexts and then get data from the DB that we need for these request.
Using IHostedService - Our API is .net framework and not .net core
Basically any thoughts on how we can achieve this, running an odata web request for as long as needed after we return a tracking id to the client (we understand the risks), without having to do another deployment of our API?

c# Web Api service call with no return value - async processing

I am looking to create a Web Api c# fire and forget service that I can post a payload to and immediately return a success or failure based on some initial checks, but then do the rest of the heavy processing asynchronously after the return 200 has been made.
What is the best approach for situations like this?
I am struggling to find a concrete example on the web to be honest.
Thanks
Neil

I wouldn't call the best approach but one possible approach is using message queues.
Your flow would be:
Call the RESTful API.
The RESTful API resource enqueues a message and returns 200/OK if it could be done successfully.
Some asynchronous worker (a different process than the Web API one) dequeues messages overtime and processes them. The whole process can be a Windows service (do you know Topshelf?), a Windows Task Scheduler task...
If you're already in Azure, take a look at Service Bus, otherwise, you might want to learn more about a message queue server like RabbitMQ.

How do I keep a task alive after the method that called it has exited?

I'm writing a webservice that drops off a long-running bulk insert command to a sql db through a stored proc. I don't want the webservice hung up while waiting for a response from the db, so I'd like to just return an http response that lets the client know the request has been sent to the db after I start the task. But as soon as I return the response, the task will lose context and get trashed, right? How should I keep this alive?

In general, it's not a good idea to spin off something to do work from IIS. What happens if the AppPool restarts? What happens if there is an exception?
Instead, I would recommend writing a Windows Service and have it responsible for the work.
Based on your comments, I would see if you can ask for the following requirements (theoretically):
All external calls are done through the web service. The web service uses a separate assembly for the actual data access.
A separate windows service is used for long running processes, which would also use the same data access assembly the web service uses.
That is really the best way to go (but not necessarily doable based on requirements).

I think it's more of a architecture question than just about maintaining the 'context'. And talking about architecture, I think WCF webservices would help in your scenario.
What you would need is a service with callback contract. Where the service takes a request, returns an ack, stores the client context (for callback), and triggers off a long running database task in background. When the task completes, it reads client context and calls the callback handler with the result.
This article at MSDN suggests how to do a callback contract in webservice.
Hope this helps!

Using Task with ASP.NET

I am trying to get some asynchronous work done with the System.Threading.Tasks.Task class. The scenario is simple. I have a web app and in one button click event I start a Task which must run to check some outside service for a couple of minutes. It is not a heavy task. All it's going to do is send a request every 5 seconds and get a response. But it must do it for at least a couple of minutes. So, I don't want user to wait until this task gets job done. After I have started the task, I immediately return to the user saying that the task started and he/she will be informed when it is done. I wonder if this task I created will cause any problems, since I returned and ended the HTTP response.

This type of "asynchronous work" isn't possible by using the Task type. As I mention on my blog, async does not change the HTTP protocol; you still get one response per request, that's it!
The ideal ASP.NET app does not do any work outside of a request/response pair. There are ways to make it work (also described on my blog), but it's almost never recommended.
The proper solution is to split up the processing. A web site (or service) should start the processing by placing a request into persistent storage (e.g., Azure queue), a separate worker service (e.g., Azure worker role / Win32 service) would do the polling and put the results into persistent storage (e.g., Azure table), and the web site/service could poll that.

You can consider using message based service bus, and a good tutorial on MSDN Building
Distributed Apps with NHibernate and Rhino Service Bus will be
very useful.

If you just return from a standard asp.net request then wouldn't you expect the HttpResponse to end? Starting up a task in itself won't hold the HttpResponse open, to that you'd need to stream your response and block on the server until your task is finished which is presumably not what you want to do?
Maybe you should look at some ajax on the client that periodically pings the server to see if the task has finished, or at HTML 5 push notifications if you know your browser is going to support it.

You can use this http://www.asp.net/web-forms/tutorials/aspnet-45/using-asynchronous-methods-in-aspnet-45 but imho ajax with web service much better

Website Calling Windows Service With Callback

I have a WCF service hosted in a Windows Service. I want a website to be able to call it asynchronously and then when the work is finished the WCF service will let the website know the result. I've looked at various ways of achieving this but I would like to get some more advice on which way would be best. I've looked into using callbacks but also read they can be unreliable. I've read about not doing it this way at all and just having another interface in my service for the website to query the status. I've looked at using MSMQ which at the moment looks like my preferred way forward but would like some more info on how to set this up or whether I shouldn't do it this way.
Does anyone have any advice please?

The nature of any communication on a network is unreliable. The statement:
I've looked into using callbacks but also read they can be unreliable
Assuming you mean WCF callbacks, they are as unreliable as the clients/servers themselves, they all use the same mechanism.
That said, you can store the client of your WCF service in the HttpApplicationState (if the call is application-wide) or HttpSessionState (if the call is local to a session).
When generating the proxy, make sure that you check the option (or specify on the contract) that you are using asynchronous calls.
Then, you would make the call, using a callback (delegate) to indicate when the async call completed.
When the call completes, you would then store the result in the session state.
If this is something that a client on the front end needs to be aware of, then the browser will have to poll your site, checking for the return result, redirecting to a page that can display the results when the result is populated.

Selecting a binding for your application depends on
Architecture of your application
Requirements
interoperability required or not.
response time of the application
availability of time to implement
Infrastructure you are using or want to use.
As your application is a web application and is built on a request/response model, you will not be able to use asyncronous or msmq style for this architecture(or is not adviceable), because there will not be any thread listining for a delayed async response or msmq call.
you can make use of one way Methods and direct calls to methods. in this case to reduce response time you have to device ways to optimize your service methods and the processing it is doing.