I have a application that will allow a user to upload a file. After the upload is complete there are a number of processing steps that must be done on the server (decompression, storage, validation, etc ...) thus the user will be informed sometime later by email when everything is complete.
I have seen a number of examples where the BackGroundWorker from System.ComponentModel is used to allow asynchronous work to be done in another thread. However, it seems like they lead to the user eventually getting a response. In our case no web response is necessary - the code can take up to 30 minutes to complete.
Is there another way to start a completely separate thread/process that will keep running even after the user completely closes their session?
If there is no need to respond immediately, you want to offload to some other process to do the heavy lifting. I would dump it in a DB, folder or post to a Message Queue. The worker processes (Windows Services?) would process the files, reading from the db, file system or queue. When the work is done, your worker process can call out to your ASP app (webhook style) if it needs to know when it's done. Just a suggestion.
Write a Windows Service that will run on the ASP.NET server. Architect it in such a way that it can accept and queue job requests. The queue will allow you to create the optimal number of threads in a ThreadPool for executing a subset of the queued jobs concurrently. Submit jobs to the Windows Service using either .NET Remoting, or WCF.
If processing can take up to 30 minutes, I'd recommend skipping using a background thread from the the web worker process and using something like a Windows service instead, or running a console application on a schedule using the Windows scheduler.
Once the file is uploaded, you would add it to a queue of some sort (either in a database, or using a message queuing system like RabbitMQ if you're feeling adventurous). Your web request could then return immediately and let the user know that the file is being processed, and the background service would pick the item up off the queue and continue the processing, emailing the user when it is complete.
Related
Our webserver generates a file on the fly for download. This takes about two minutes of heavy processing because it's taking 1000 word templates, mail merging them, converting them to pdf, then making a zip out of those 1000 files. The processing is freezing the web server from being able to do anything else in the mean time which is a problem since it's hosting 23 subdomains for clients and they noticed it freezing.
How can i force the UI thread to get some work done? I've been looking at Thread.Sleep and Thread.Yield but perhaps i'm not implementing them correctly. I'm very new to the concept of threading.
When starting the processing on the web server, generate a "job ID" and store it somewhere (such as a database). Add an endpoint so the client can query the status of the job. When the processing is complete, the user can use the job ID to get the resulting file(s). It works like this:
User wants to process files. They call the start endpoint, which returns a JobId.
The server begins processing that job in a non-request thread, or the job is picked up and processed by another server dedicated to that task. When the thread completes, it updates the job's status accordingly.
Later...
User wants to know the status of their process. They call the status endpoint using their JobId periodically. The server replies with some status information to show the user.
Later...
Once the job's status has changed to 'complete', the user can call the result endpoint with their JobId to get the final file(s).
The heavy processing should be done in a non-UI, non-request thread so other users are unaffected.
Using this approach, you can even do the processing on another server entirely. All the web server is doing is allowing the user to create and query processing jobs.
So here is the problem statement. I have a service which services mobile devices. Each user on the trigger of an event sends message to the service to register itself. After which the service performs a particular set of tasks at regular intervals(say 5 min) from the time of registration. So the execution time will be different for each user based on registration time.
I Implemented this using threads and Timers, it worked to an extent but as the users increased, the threads are killed and the tasks are not completed. Also this service is hosted on azure. I have created a WCF service with WebHtpp binding which accepts and returns JSON data.
Web jobs are a suggestion given to me. But since the execution times vary I am unable to use that as well. Is it even possible to perform such a task using C# and asp.net or am i going i the wrong direction entirely.
You need to identify where's the bottleneck that stops your threads before completion.
I would solve this using another approach: every new user, put a message in a queue, and create one Azure Function to dequeue the message and perform the logic of your service. This way your application will scale better, and you'll save money with the serverless approach.
After so much research, I thought I should ask the experts.
I am working on a project for my corporate employer, we have android and iPhone mobile apps that make request to a web service, the request is logged in pending state for processing.
A windows service retrieves the pending requests and spins a new thread for every request. This is because the request could be directed to different providers who process request in different manners. One could immediately process the request and return feedback, others could receive the request and take up to 30 seconds to return a feedback which you have to poll for the status.
The mobile app will also be polling for the status of the request.
Now my challenge is:
I am thinking of creating a list of threads say 100 and assign each thread to execute a request, once finished the thread will be recreated and assigned a new request. It is a high response based platform so I am thinking of not using ThreadPool.
Is it advisable to spin new threads in a fire and forget sequence or manage these list of threads and if it is to managed, then what is the best approach to manage these list of threads and ensure high performance in C# as the mobile apps will be polling for a response.
Regards
I have a windows service written in C# that reads from MSMQ and based on the type of the message it assigns them to Agents that process that message in a worker thread. The application starts with no agents and are created dynamically at runtime as messages arrive in the MSMQ
Here is a basic figure of how it works:
If the agent worker thread is busy doing work the message is queued to its local queue. So far so good. But if for some reason if the service is stopped, the local queue content is lost.
I am trying to figure out what could be the best way to handle this scenario. Right now the local queues are a System.Concurrent.ConcurrentQueue. I could probably use a Sql Ce db or some other persistent storage, but i am worried about performance. The other thing in my mind is to read from MSMQ only when agents are ready to process message, but the problem is that I don't know what message the MSMQ will contain.
What possible approaches can I take on this issue?
Your design is basically implements the following pattern: http://www.eaipatterns.com/MessageDispatcher.html
However, rather than using actual messaging you are choosing to implement the dispatcher in multithreaded code.
Rather, each processing agent should be an autonomous process with it's own physical message queue. This is what will provide message durability in case of failure. It also allows you to scale simply by hosting more instances of the processing agent.
I have built a similar system dependent on Redis. The idea is that it provides memory - fast data access isolated from the rest of the application, and will not shut down when my service does. Furthermore, it will eventually persist my data to the disk, so I get a good compromise between reliability and speed.
If you designed it so that each client read from its own message queue that would be hosted in Redis, you could keep the queue independent from the service's downtime, and each worker's load apportioned when you next start the service.
Why don't you simply create two new msms queues to receive the messages for Agenta and agentb, and create a new agent that ( transactionally ) fetch the command from the main queue and dispatch the message to the proper agent queue ?
I have a webpage with a button that generates some files to a server path. (It takes somewhere from 5 to 20 minutes). I want to create an async task that will continue executing even after the user closes the browser. Is it possible to do this with asp.net 4 and C#?
You do not control the thread pool in an asp.net application. You cannot even guarantee that a request will be completed on the same thread that it started with. Creating threads uses the same application pool that the web server uses, and you can use up all the request threads leaving your web server unavailable to process requests.
You should implement a windows service that hosts a WCF service that you can call from within your web application. In the service you can then fire off a thread to process the long running process. At the end of that process you can then update a status flag (e.g from Processing to Complete) that the user can view to determine if the files are done processing.
I would recommend using Topshelf to implement your windows service, it will save you much headache.
Actually, it is recommended that you not do this. Instead, the recommended way is to create a service (e.g. a windows service) that performs your processing asynchronously. In your web application, you create methods that starts the process, and another method that polls the service to determine if processing has completed.
There are a number of reasons for this, but one of the biggest is that the default and recommended configuration for webservers allows the server to kill long-running requests.
Or that I didn't understand what you want to do, or that you don't need to do a thing.
After the request was sent, the request process continues no matter if the user browser was closed or not. You don't need to do a thing
Fabulous nature of stateless WEB applications...
Creating new thread / using thread pool is the easiest approach to create run away tasks.
Note that there is no guarantees that process will stay alive for duration of a long task - so be prepared to handle partial completion and manual restarts. I.e. AppPoll recycle due to config change.
Easiest way is to put your task on the ThreadPool. The thread pool threads will stay alive even after the web page has completed rendering. The code would look like the following:
/* Beginning Method */
object someData = new object();
ThreadPool.QueueUserWorkItem(new WaitCallback(ProcessAsync), someData);
/* Ending Method */
static void ProcessAsync(Object stateInfo)
{
string dataToString = stateInfo.ToString();
}
you have to create a thread that does the long running task
have a look at the below:
http://kiranpatils.wordpress.com/2010/01/01/performing-a-long-running-task-with-asp-net/
Anyway what ever you start on the server it will continue running even if the user close the browser(until you recycle the app-pool or restart the web server).