A Web API receives a customer's credit card request data at an endpoint.
The endpoint sends a message with this data to the Kafka.
Several pods/containers will be connected to Kafka in this topic for processing each request in parallel. The application requires high asynchronous processing power.
After sending the request, the frontend will display a progress bar and will wait. It needs a response as soon as the process is finished.
The question
How to return in the same call to this endpoint the result of a
processing that will be done in another web API project?
(asynchronous)
What I thought
Creating a topic in Kafka to be notified of the completion of processing and subscribe to it in the endpoint after sending the CreditCardRequest message to process on Kafka.
Using a query on the mongo on every 3~4 seconds (pooling) and check if the record has been included by the Worker / Pod / Processing Container. (URRGGH)
Creating another endpoint for the frontend to query the operation status, this endpoint will also do a query in the mongo to check the current status of the process.
I wonder deeply if there is no framework/standard already used for these situations.
yes, there are frameworks that handle this.
From a .NET perspective, I have used nServiceBus todo something similar in the past (coordinate long-running processes).
They have a concept called Sagas https://docs.particular.net/nservicebus/sagas/
A saga will wait for all messages that are required to finish processing before notifying the next step of the process to continue.
If the framework is not useful, hopefully, the processes are and you can discover how to implement in a Kafka/Mongo environment.
Related
I need to consume an external API to update or create data, I need to handle service down cases and don't lose the data if the service went down for more than expected, what is the best approach to do this... thanks in advance.
If downtimes are happening a lot and you have also high traffic taking an asynchronous microservice approach could be best.
Using a message queuing system like RabbitMQ you can store all requests with their data as a message on a Queue.
Then another microservice which consumes those messages from the Queue can reach the external service and do the operation. If the external service is up, the operation is completed and the messages is removed from the Queue.
If not RabbitMQ has dozens of configs for Retrying (reque after x secs etc)
I think you can have some kind of internal database, maybe NoSQL database that stores all of the failed requests you tried to send.
So when you send a request and it fails for any reason, then you will insert the failed request to that database.
And you will make a recurring task that pulls all the data from that internal db, and retries to send the requests again. If the request is succeeded, then remove it from the internal db.
I am looking to create a Web Api c# fire and forget service that I can post a payload to and immediately return a success or failure based on some initial checks, but then do the rest of the heavy processing asynchronously after the return 200 has been made.
What is the best approach for situations like this?
I am struggling to find a concrete example on the web to be honest.
Thanks
Neil
I wouldn't call the best approach but one possible approach is using message queues.
Your flow would be:
Call the RESTful API.
The RESTful API resource enqueues a message and returns 200/OK if it could be done successfully.
Some asynchronous worker (a different process than the Web API one) dequeues messages overtime and processes them. The whole process can be a Windows service (do you know Topshelf?), a Windows Task Scheduler task...
If you're already in Azure, take a look at Service Bus, otherwise, you might want to learn more about a message queue server like RabbitMQ.
I am trying to get some asynchronous work done with the System.Threading.Tasks.Task class. The scenario is simple. I have a web app and in one button click event I start a Task which must run to check some outside service for a couple of minutes. It is not a heavy task. All it's going to do is send a request every 5 seconds and get a response. But it must do it for at least a couple of minutes. So, I don't want user to wait until this task gets job done. After I have started the task, I immediately return to the user saying that the task started and he/she will be informed when it is done. I wonder if this task I created will cause any problems, since I returned and ended the HTTP response.
This type of "asynchronous work" isn't possible by using the Task type. As I mention on my blog, async does not change the HTTP protocol; you still get one response per request, that's it!
The ideal ASP.NET app does not do any work outside of a request/response pair. There are ways to make it work (also described on my blog), but it's almost never recommended.
The proper solution is to split up the processing. A web site (or service) should start the processing by placing a request into persistent storage (e.g., Azure queue), a separate worker service (e.g., Azure worker role / Win32 service) would do the polling and put the results into persistent storage (e.g., Azure table), and the web site/service could poll that.
You can consider using message based service bus, and a good tutorial on MSDN Building
Distributed Apps with NHibernate and Rhino Service Bus will be
very useful.
If you just return from a standard asp.net request then wouldn't you expect the HttpResponse to end? Starting up a task in itself won't hold the HttpResponse open, to that you'd need to stream your response and block on the server until your task is finished which is presumably not what you want to do?
Maybe you should look at some ajax on the client that periodically pings the server to see if the task has finished, or at HTML 5 push notifications if you know your browser is going to support it.
You can use this http://www.asp.net/web-forms/tutorials/aspnet-45/using-asynchronous-methods-in-aspnet-45 but imho ajax with web service much better
What design has someone successfully used to implement job processing on Windows Azure?
Requirements:
Ability to push a Job into a queue.
N workers can consume Jobs from the queue and process them.
Invoker of the job should be able to be alerted (push, not polling) of the job being completed.
Research thus far:
Create a "Job" Queue using Azure Service Bus Queues (http://blogs.msdn.com/b/appfabric/archive/2011/05/17/an-introduction-to-service-bus-queues.aspx)
Web front-end pushes Jobs to the queue, workers block on Receive() indefinitely (see http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.aspx) until a Job is ready (to avoid "null" long polling, which costs money due to API call transaction costs)
With regards to being notified of Job completion:
There is no apparent ability to be alerted to when a Job has been completed.
I thought I could leverage Service Bus Topics/Subscriptions (https://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/) and have a caller "subscribe to" a "Job Finished Notifications" topic, however:
You apparently can't subscribe more than once to the same topic, unless you create multiple "Subscription" entries (which does not scale)
Unless we did create a "Subscription" for each Job Id, and had the caller block on a Receive() API call (using I/O completion ports) on that subscription, we can't get real time notifications of when a Job has been processed.
Has anyone had any experience implementing this sort of Job system (real time, low latency, with completion notifications for the caller) before?
Thanks
Actually, queue does not stand by push. The whole idea about queue is the receiver does not need to receive the message in real time, and wants to check the message periodically. If you need real time communication, you can create an HTTP/TCP listener on the receiver side, and let the sender make an HTTP/TCP request.
Thus, one approach is to create a web service on the web role, using internal endpoints. You send the service's address along with the message to worker role using queue. When the job is finished, the worker role invokes the service to notify the web role that job is done.
This approach is fine, but it does not provide much value. It cannot display something on the UI(unless you implement web socket), since a server cannot notify the browser. So if you want to display a notification in a browser client, I would like to suggest you to use a pull solution (unless you implement web socket). If you're using a rich client, you can host a web service on the client machine, and let the worker role notify the client by invoking the service.
Best Regards,
Ming Xu.
I am developing an Application where I am submitting POST Requests to a .NET Web Service.
Current implementation is to process the request instantly and give response. In actual deployment, there will be huge amount of data that needs to be processed and thus the request must be processed offline.
What are the strategies that can have the task accomplished
Should I implement a Windows Service, or a scheduled task that invokes an application to perform the desired task.
This might be a good case for MSMQ. Your webservice can fill the queue with incoming data, and another process can read those messages and perform the necessary processing.
Here's a good overview of MSMQ:
http://www.primaryobjects.com/CMS/Article77.aspx
If you have so much data it cannot be processed in real-time, I would probably setup the service to do the following:
ProcessRecordViaPost
Create new record in "Queue" database with UniqueID, and all other info to be processed
Return UniqueID to client immediatly
ReadRecordViaGet
Check queue, if processed return data if not return status code (number of items in queue before it?)
I would also have a windows service that continually grabs the oldest item from the Queue, and processes it and moves on to the next oldest.