I am developing an Application where I am submitting POST Requests to a .NET Web Service.
Current implementation is to process the request instantly and give response. In actual deployment, there will be huge amount of data that needs to be processed and thus the request must be processed offline.
What are the strategies that can have the task accomplished
Should I implement a Windows Service, or a scheduled task that invokes an application to perform the desired task.
This might be a good case for MSMQ. Your webservice can fill the queue with incoming data, and another process can read those messages and perform the necessary processing.
Here's a good overview of MSMQ:
http://www.primaryobjects.com/CMS/Article77.aspx
If you have so much data it cannot be processed in real-time, I would probably setup the service to do the following:
ProcessRecordViaPost
Create new record in "Queue" database with UniqueID, and all other info to be processed
Return UniqueID to client immediatly
ReadRecordViaGet
Check queue, if processed return data if not return status code (number of items in queue before it?)
I would also have a windows service that continually grabs the oldest item from the Queue, and processes it and moves on to the next oldest.
Related
A Web API receives a customer's credit card request data at an endpoint.
The endpoint sends a message with this data to the Kafka.
Several pods/containers will be connected to Kafka in this topic for processing each request in parallel. The application requires high asynchronous processing power.
After sending the request, the frontend will display a progress bar and will wait. It needs a response as soon as the process is finished.
The question
How to return in the same call to this endpoint the result of a
processing that will be done in another web API project?
(asynchronous)
What I thought
Creating a topic in Kafka to be notified of the completion of processing and subscribe to it in the endpoint after sending the CreditCardRequest message to process on Kafka.
Using a query on the mongo on every 3~4 seconds (pooling) and check if the record has been included by the Worker / Pod / Processing Container. (URRGGH)
Creating another endpoint for the frontend to query the operation status, this endpoint will also do a query in the mongo to check the current status of the process.
I wonder deeply if there is no framework/standard already used for these situations.
yes, there are frameworks that handle this.
From a .NET perspective, I have used nServiceBus todo something similar in the past (coordinate long-running processes).
They have a concept called Sagas https://docs.particular.net/nservicebus/sagas/
A saga will wait for all messages that are required to finish processing before notifying the next step of the process to continue.
If the framework is not useful, hopefully, the processes are and you can discover how to implement in a Kafka/Mongo environment.
Our webserver generates a file on the fly for download. This takes about two minutes of heavy processing because it's taking 1000 word templates, mail merging them, converting them to pdf, then making a zip out of those 1000 files. The processing is freezing the web server from being able to do anything else in the mean time which is a problem since it's hosting 23 subdomains for clients and they noticed it freezing.
How can i force the UI thread to get some work done? I've been looking at Thread.Sleep and Thread.Yield but perhaps i'm not implementing them correctly. I'm very new to the concept of threading.
When starting the processing on the web server, generate a "job ID" and store it somewhere (such as a database). Add an endpoint so the client can query the status of the job. When the processing is complete, the user can use the job ID to get the resulting file(s). It works like this:
User wants to process files. They call the start endpoint, which returns a JobId.
The server begins processing that job in a non-request thread, or the job is picked up and processed by another server dedicated to that task. When the thread completes, it updates the job's status accordingly.
Later...
User wants to know the status of their process. They call the status endpoint using their JobId periodically. The server replies with some status information to show the user.
Later...
Once the job's status has changed to 'complete', the user can call the result endpoint with their JobId to get the final file(s).
The heavy processing should be done in a non-UI, non-request thread so other users are unaffected.
Using this approach, you can even do the processing on another server entirely. All the web server is doing is allowing the user to create and query processing jobs.
I have a WCF service in which I have long running methods like get and process some kind of data and then return it to client. I have tried to use contracts similar to following
Task<string> ServiceMethod(string message);
The problem is if I want to return the same data to multiple clients from the service then how can I do it (How do I get and store information about clients who are requesting data).
Also if I need to call a background worker from the above method then how do I process in the runworker_completed and return the result to the above.
Additional info
The returning of same data to multiple clients is only in case the clients request the same data but since it takes time to get and process it so whenever it is available I want to return to all the clients who have requested it.
if I understand your question correctly, you want the service to callback clients when it finishes a long running process that generates data. As you have to take care of multiple clients, I would recommend that you use Duplex WCF. A duplex service contract provides for calling back a method on invoking client. The following code project link is good example for Duplex and has a lot more details
http://www.codeproject.com/Articles/491844/A-Beginners-Guide-to-Duplex-WCF
Note that you should have your own logic for maintaining the list of callback channels
I have stored a cache of Task in the service with the keys requested. Whenever the task for that key is completed I send back tasks to all the clients which had requested the same key.
Also for already existing functions with event based completion I have used TaskCompletionSource and stored it in the cache and again using it to send async response.
After so much research, I thought I should ask the experts.
I am working on a project for my corporate employer, we have android and iPhone mobile apps that make request to a web service, the request is logged in pending state for processing.
A windows service retrieves the pending requests and spins a new thread for every request. This is because the request could be directed to different providers who process request in different manners. One could immediately process the request and return feedback, others could receive the request and take up to 30 seconds to return a feedback which you have to poll for the status.
The mobile app will also be polling for the status of the request.
Now my challenge is:
I am thinking of creating a list of threads say 100 and assign each thread to execute a request, once finished the thread will be recreated and assigned a new request. It is a high response based platform so I am thinking of not using ThreadPool.
Is it advisable to spin new threads in a fire and forget sequence or manage these list of threads and if it is to managed, then what is the best approach to manage these list of threads and ensure high performance in C# as the mobile apps will be polling for a response.
Regards
I have a application that will allow a user to upload a file. After the upload is complete there are a number of processing steps that must be done on the server (decompression, storage, validation, etc ...) thus the user will be informed sometime later by email when everything is complete.
I have seen a number of examples where the BackGroundWorker from System.ComponentModel is used to allow asynchronous work to be done in another thread. However, it seems like they lead to the user eventually getting a response. In our case no web response is necessary - the code can take up to 30 minutes to complete.
Is there another way to start a completely separate thread/process that will keep running even after the user completely closes their session?
If there is no need to respond immediately, you want to offload to some other process to do the heavy lifting. I would dump it in a DB, folder or post to a Message Queue. The worker processes (Windows Services?) would process the files, reading from the db, file system or queue. When the work is done, your worker process can call out to your ASP app (webhook style) if it needs to know when it's done. Just a suggestion.
Write a Windows Service that will run on the ASP.NET server. Architect it in such a way that it can accept and queue job requests. The queue will allow you to create the optimal number of threads in a ThreadPool for executing a subset of the queued jobs concurrently. Submit jobs to the Windows Service using either .NET Remoting, or WCF.
If processing can take up to 30 minutes, I'd recommend skipping using a background thread from the the web worker process and using something like a Windows service instead, or running a console application on a schedule using the Windows scheduler.
Once the file is uploaded, you would add it to a queue of some sort (either in a database, or using a message queuing system like RabbitMQ if you're feeling adventurous). Your web request could then return immediately and let the user know that the file is being processed, and the background service would pick the item up off the queue and continue the processing, emailing the user when it is complete.