I have a pricing application. It sends pricing requests to an Azure Service Bus Queue (could be any queue) "PricingRequestQueue". There are a number of workers that pick these up, process them and return the results to a PricingResponse Queue.
I would like to create an Observable over the PricingResponse queue. I do not require any filtering, but would like to read the messages off using the batch interface (QueueClient.BeginReceiveBatch). The queue has the number of messages expected, and has a session to read from (QueueClient.AcceptMessageSession(correlationIdentifier).
I'm still trying to get my head around RX, and this would really clear things up.
There is the CloudFx library that adds Rx extensions to Azure.
https://www.nuget.org/packages/Microsoft.Experience.CloudFx/ (Updated link)
However I must warn you that we have found some thread leaks in the current CloudFx libraries (in particular with the table storage one - however you have not needed the Rx extensions since table storage 2.0).
Related
I have been requested to use Amazon SQS in our new system. Our business depends on having some tasks/requests from the clients to our support agents, and once the client submit his task/request, it should be queued in my SQL Server database, and all queued tasks should be assigned to the non-busy agent because the flow says that the agent can process or handle one task/request at the meantime, so, If I have 10 tasks/requests came to my system, all should be queued, then, the system should forward the task to the agent who is free now and once the agent solves the task, he should get the next one if any, otherwise, the system should wait for any agent until finishing his current task to assign a new one, and for sure, there should not be any duplication in tasks/requests handling ... and so on.
What do I need, now?
Simple reference which can clarify what is Amazon SQS as this is my first time to use queuing service?
How can I use the same with C# and SQL Server? I have read this topic but I still feel that there is something messing as I am not able to start. I am just aiming at the way which I can process the task in run-time and assign it to an agent, then close it and getting a new one as I explained above.
Asking us to design a system based on a paragraph of prose is a pretty tall order.
SQS is simply a cloud queue system. Based on your description, I'm not sure it would make your system any better.
First off, you are already storing everything in your database, so why do you need to store things in the queue as well? If you want to have queue semantics while storing stuff in your database you could consider SQL Server Service Broker (https://technet.microsoft.com/en-us/library/ms345108(v=sql.90).aspx#sqlsvcbr_topic2) which supports queues within SQL. Alternatively unless your scale is pretty high (100+ tasks/second maybe) you could just query the table for tasks which need to be picked up.
Secondly, it sounds like you might have a workflow around tasks that could extend to more than just a single queue for agents to pick them up. For example, do you have any follow up on the tasks (emailing clients to ask them how their service was, putting a task on hold until a client gets back to you, etc)? If so, you might want to look at Simple Workflow Service (https://aws.amazon.com/swf/) or since you are already on Microsoft's stack you can look at Windows Workflow (https://msdn.microsoft.com/en-us/library/ee342461.aspx)
BTW, SQS does not guarantee "only one" delivery by default, so if duplication is a big problem for you then you will either have to do your own deduplication or use FIFO queues (http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html) which support deduplication, but are limited to 300 transactions/second (aka: roughly 100 messages/second accounting for the standard send -> receive -> delete APIs. Using batching obviously that number could be much higher, but considering your use case it doesn't sound like you would be able to use batching without a lot of work).
I have a subscription model, and want to perform renew-related logic like issue new invoice, send emails, etc. For example, user would purchase the subscription today, and the renewal is in a year's time. I've been using an Azure Queue recently, and think it would apply for such a renewal.
Is it possible to use the Azure Queue by pushing messages using BrokeredMessage.ScheduledEnqueueTimeUtc (http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.scheduledenqueuetimeutc.aspx) for such long term scheduled messages?
I've used it for shorter-term, like sending notifications in 1 minute time and it works great.
This way, I can have even multiple processes listening to the queue, and be sure that only one process would perform the renewal logic. This would solve a lot of locking-related problems, as that is kind of built-in the Azure Queue via leasing and related features.
Yes, you can use it for long-term scheduling, scheduled messages have the same guaranties as normal ones. But there are few things you need to be aware of:
ScheduledEnqueueTimeUtc is a time when message is going to be available (within hundreds of miliseconds) on the queue but not necessary delivered, this depends on load and state of the queue. So it's fine for business processes but not for time sensitive (milliseconds) usage. Not a problem in your case, unless your subscription cancellation is really time sensitive.
It affects your storage quota ( Not really a problem with current quotas, but if you think about years this might be a problem)
As far as I'm aware you can't access scheduled messages before ScheduledEnqueueTimeUtc, they are invisible.
Extremely awesome source of informations on azure messaging
From technological perspective it's fine but in your case I would also think about other potential problems if you think about years:
Message versioning
What happens when you would like to change Azure to something else (AWS?)
What if you decide to change in next year Azure Service Bus for NServiceBus
Just worked through a quick sandbox version of what I posted earlier.
The original requirements still stand - and this worked nicely in my console sandbox verison:
Read a file (of account IDs), CSV format
Download the account data
file from the web for each account (by Id) (REST API)
Pass the file to a converter that will produce a report (financial predictions etc)
[~20ms]
If the prediction threshold is within limits, run a parser to analyse the data [400ms]
Generate a report for the analysis above [80ms]
Upload all files generated to the web (REST API)
I'm trying to make use of NServiceBus this time as suggested but I'm finding it hard to workout how to make things fit.
I'm pushing a bulk of 20 account Ids into a message, that's read by NServiceBus Handler and they get Posted it to the BufferBlock one at a time. The reason for bulk loading is our systems use a very slow web-service to get account info - especially archived accounts (deceased etc) that take 3-4s sometimes! (don't worry about that for the moment)
How do I keep popping stuff into the BufferBlock and still have my TPL Dataflow active as I read messages etc? Do I configure the TPL Dataflow separately and wait for ever and just Post when a new batch of messages come through?
There are roughly 2 million accounts to process. Ideally I'd like the Account Report Generator (TPL-DF) to be alive all the time and I just push messages to its buffer which it will work through. It's the lifetime that confuses me.
I've been tasked with taking a RabbitMQ queue, processing the messages (key and values) to filter out unneeded items (based on the key), and delaying the results before making them available via a webservice.
Being new to RabbitMQ, it seems like my best approach was to write a windows client that retrieves messages from the queue, filters it accordingly and puts it into a custom class collection (System.Collection.Queue). Whenever an item in this collection has been stored for X seconds, the message data would be pushed into public collection to overwrite the existing value based on the key.
This publicly accessible collection would be exposed as a REST service returning json data.
This would loop indefinitely for as long as the client was running.
The end client is a javascript widget that will connect to this webservice, and probably poll it every second. It seems like my approach would work, but I am concerned the process would be too intensive? I get the feeling there might be a better solution.
I was originally thinking node.js might be a good fit for this project, but I'm predominantly an asp.net developer, so I'm happy to consider other solutions, perhaps like SignalR, Web API, WCF?
It seems what you're worried about is polling, no RabbitMQ. A push solution is a better fit, if you can dictate technology that supports it. Since SignalR and other websocket solutions allow push notifications, it's a great fit for this.
I don't think personally you need a windows application - you can do worker tasks in ASP easily. The .NET parallels library now has great support for producer / consumer patterns using blocking collections as well.
So you could just do
-message recieved - add to blocking queue
-blocking queue consumer gets message and
-starts new task for message
-sleep task for amount of time wanted
-add to web APIs output queue
-push new messages out through websocket
I'm doing a project with some timing constraints right now. Setup is: A web service accepts (tiny) xml files and I have to process these, fast.
First and most naive idea was to handle this processing in the request dispatcher itself, but that didn't scale and was doomed from the start.
So now I'm looking at a varying load of incoming requests that each produce ~ 50 jobs on my side. Technologies available for use are limited due to the customers' rules. If it's not Sql Server or MS MQ it probably won't fly.
I thought about going down the MS MQ route (Web service just submitting messages, multiple consumer processes lateron) and small proof of concept modules worked like a charm.
There's one problem though: The priority of these jobs might change a lot, in the queue. The system is fairly time critical, so if we - for whatever reasons - cannot process incoming jobs in a timely fashion, we need to prefer the latest ones.
Basically the usecase changes from reliable messaging in general to LIFO under (too) heavy load. Old entries still have to be processed, but just lost all of their priority.
Is there any manageable way to build something like this in MS MQ?
Expanding the business side, as requested:
The processing of the incoming job is bound to some tracks, where physical goods are moved around. If I cannot process the messages in time, the things are "gone".
I still want the results for statistical purpose, but really need to focus on the newer messages now.
Think of me being able to influence mechanical things and reroute things moving on a track - if they didn't move past point X yet..
So, if i understand this, you want to be able to switch between sorting the queue by priority OR by arrival time, depending on the situation. MSMQ can only sort the queue by priority AND by arrival time.
Although I understand what you are trying to do, I don't quite see the business justification for it. Can you expand on this?
I would propose using a service to move messages from the incoming queue to a number of work queues for processing. Under normal load, there would be a several queues, each with a monitoring thread.
Under heavy load, new traffic would all go to just one "panic" queue under the load dropped. The threads on the other work queues could be paused if necessary.
CheersJohn Breakwell