I have a javascript logging utility that sends requests in bulk to my server which then relays them to a Queue Client (Microsoft.ServiceBus.Messaging.QueueClient). I want to send them in batch asynchronously to the ServiceBus and still have them processed in the order they are placed into the batch I am sending. The documentation for SendBatchAsync shows that the method is for "batch" processing. This makes me think I can send it a batch of requests and have them processed as a single unit (i.e.: sequentially). Although, it appears that the messages are getting processed out of order. I'm using OnMessage to receive the messages; I'm not sure if this is a limitation or what am I missing?
I get that async doesn't guarantee order vs. other async requests, but this is a single request. I don't want to have to wait for a response before responding to the javascript client as I'm just trying to send them off, but I still need to ensure they stay in order since they are sequential events.
Here is how I send them to the queue:
MyQueueClient.SendBatchAsync(MyListOfBrokerMessages);
Then I process them:
ServiceBus.TrackerClient.OnMessage((m) =>
{
try
{
ProcessMessage(m);
}
I don't get the point of the batch processing if it doesn't process as a batch other than maybe making a single request. There must be some way to send a batch and have it process in order??
EDIT:
I've tried using Send instead of SendBatchAsync and I've set MaxConcurrentCalls to 1 and yet the messages are still not in order.
Taken from MSDN:
SessionId: If a message has the
Microsoft.ServiceBus.Messaging.BrokeredMessage.SessionId property set, then Service Bus
uses the SessionId property as the partition key. This way, all messages that belong to
the same session are handled by the same message broker. This enables Service Bus to
guarantee message ordering as well as the consistency of session states.
For a coding sample employing SessionId and AcceptSessionReceiver see.
What you can do is to use Sessions here,
Set the same sessions id to all the messages in the batch
Receiving side, AcceptMessageSession() will give you a session
Call receive on the session (ReceiveBatch). This session will give you all the messages in that batch alone.
Related
Is it possible to bulk read messages from Solace queue rather than receiving them one by one on callback?
Currently MessageEventHandler receives about 20 messages per minute, this is too slow for our application.
Does anyone have a better solution to speed things up in Solace?
This is a C# application.
We used
ISession.CreateFlow(FlowProperties, IEndpoint, ISubscription,
EventHandler<MessageEventArgs>, EventHandler<FlowEventArgs>)
Passing in a MessageEventHandler which gets the message via MessageEventArgs.Message
queue = CreateQueue();
Flow = Session.CreateFlow(flowProperties, queue, null, OnHandleMessageEvent, OnHandleFlowEvent);
..
void OnHandleMessageEvent(object sender, MessageEventArgs args)
{
var msgObj = args.Message.BinaryAttachment;
..
}
```
No, there is no API call for a user to read messages in bulk.
By default, the API already obtaining messages from the message broker in batches, with each message being individualy delivered to the application in the message receive callback.
FlowProperties.WindowSize and FlowProperties.MaxUnackedMessages can change this behavior.
20 messages per minute is extremely slow.
One common reason for slowness is that the application is taking a long time to process messages in the message receive callback ("OnHandleMessageEvent").
Blocking in the message receive callback will prevent the API from delivering another message to the application.
Refer to Do Not Block in Callbacks for details.
I'd like to write parallel execution module based on Solace. And I use request-reply schema for this.
I have:
Multiple message consumers, which publish messages into the same queue.
Multiple message producers, which read queue and create reply messages.
Message execution time is between 10 seconds to 10 minutes.
Queue access type is non-exclusive (e.g. it does round-robin between all consumers).
Each producer and consumer is asynchronous, e.g. Solace API blocks execution during the connection only.
What I'd like to have: if produces works on the message, it should not receive any other messages. This is extremely important, because some tasks blocks executor for several minutes, however other executors can be free after couple of seconds.
Scheme below can be workable (possible), however blocking code appears below. I'd like to avoid it.
while(true)
{
var inputMessage = flow.ReceiveMsg( /*timeout 1s*/1_000); // <--- blocking code, I'd like to avoid it
flow.Ack(inputMessage.ADMessageId);
var reply = await ProcessMessageAsync(inputMessage); // execute plus handle exceptions
session.SendReply(inputMessage, reply)
}
Messages are only pushed to the consuming applications.
That being said, your desired behavior can be obtained by setting the "max-delivered-unacked-msgs-per-flow" on your queue to 1.
This means that each consumer bound to the queue is only allowed to have 1 outstanding unacknowledged messages.
The next message will be only sent to the consumer after it has acknowledged the message.
Details about this feature can be found here.
Do note that your code snippet does not appear to be valid.
IFlow.ReceiveMsg is only used in transacted sessions, which makes use of ITransactedSession.Commit to acknowledge messages.
I have a problem with async TCP listener in C#. The main problem is I want to create async TCP listener in order to handle multiple connections. I have tons of requests from devices and webpages. Also I have to use database to write specific information from these connections (read/write to/from SQL Server).
The scenario of our task is this: One REST request will post from a webpage with a unique identifier to our Web API. Then our Web API makes a TCP connection to our listener, so we must halt this connection until we get another connection from a device with that unique identifier. Then we send data which we got it before (webpage connection) to this connected device and again we must halt this connection too. After processing this data in the device it will send us some other data again, and we must send this data to webpage which we halted it before.
How can I find halted connection in our listener?
Is there a better solution for us? (except using async TCP listener)
Because of some customer reasons we are unable to use signalR or self-hosted Web API in C#.
Regards,
Sara
'Halt' isn't the best word to describe what you need. If you need two-way communication with a web page over a REST request, you simply need to keep that request pending until the response is ready (not recommended, it could take really long and the connection could be dropped due to network conditions). Do reconsider your choice of avoiding SignalR. However, if need be, you can keep the request thread waiting. To do that, you'd need either a TaskCompletionSource (if you're processing the request within a Task) or a synchronization primitive such as a ManualResetEvent. I can't really give you more details without knowing the conditions your code will run under.
On the device side of things, again you need two way communication. You could implement this in one of two ways:
The device opens a TCP connection and keeps it open. The server receives the ID, and then sends the data back over the connection. The device then processes this data in some way and sends its response back to the server over the same connection and terminates the connection.
The device makes the equivalent of a REST GET request to the server to grab the data from the web page. It then processes the data and makes the equivalent of a POST request to send its own data back to the server.
After this is done, you still have the connection from the web page waiting for a response. Simply let it know the transaction has completed, using TaskCompletionSource.SetResult or ManualResetEvent.Signal. The server can then write whatever data it needs in the response to the web page's request and close that connection too.
Also note that there is no such thing as a halted connection. You just intentionally delay writing a response.
EDIT: You can't really hold the connection (not with the normal execution flow of most web servers at least), but you can stop the thread processing that connection. This is a heavily simplified (and completely inappropriate for any real system) example:
// ConnectionManager.cs
public static Dictionary<Guid, TaskCompletionSource<DataToSendToWebPage>> connectionTCSs;
// WebPageRequestHandler.cs
async Task HandleClientRequest() {
// do some stuff
var tcs = new TaskCompletionSource<DataToSendToWebPage>();
ConnectionManager.connectionTCSs[deviceID] = tcs;
var result = await tcs.Task; // This is where you wait for the other flow to complete
// Write response to connection
}
// DeviceRequestHandler.cs
void HandleRequest() {
// do stuff
ConnectionManager.connectionTCSs[clientID].SetResult(result);
}
The general idea is that you keep the thread (or task) processing the web page request waiting, and then signal it to continue from the other thread when the device's connection is handled and data is received.
I have a couple of queues where certain information is queued. Let us say I have "success" and "failed" queues in which Server side component has continuously written some data to these queues for clients.
Clients read this data and display it on a UI for end users. Now, I have a situation to purge any message in these queues older than 30 days. Clients would then only be able to see only 30 days of information at any point of time.
I have searched a lot and could see some command line options to purge whole queue but could not find a relevant suggestion.
Any help in the right direction is appreciated. Thanks
I don't think this is possible; looks like you're trying to use RabbitMq as data storage instead of message server.
The only way to understand if a message is "older" than 30, is to process the message, and by doing this you are removing the messagge from the queue.
Best thing to do here is to process the messages and store them in a long term storage; then you can implement a deletion policy to eliminate the older elements.
If you really want to go down this path, RabbitMQ implements TTL at queue level or message level; take a look at this: https://www.rabbitmq.com/ttl.html
[As discussed in comments]
To keep the message in the queue you can try to use a NACK instead of ACK as confirmation; this way RabbitMQ will consider the message undelivered and it will try to deliver it again and again. Remember to create a durable queue (https://www.rabbitmq.com/confirms.html).
You can also check this answer: Rabbitmq Ack or Nack, leaving messages on the queue
Feature Description
The NServiceBus gateway, http://docs.particular.net/nservicebus/gateway/, seems to be a way to achieve an internal webhook using the NServiceBus infrastructure.
We need to go further with this concept to open up a few event to any 3rd party subscriber that has access to register a webhook url in our system.
Review
We plan to create two initial window services
1) WebHookBatchService, that can be added as a subscriber to specific messages of interest.
<UnicastBusConfig>
<MessageEndpointMappings>
.......
<add Messages="MyMessages.MyImportantMessage, MyMessages" Endpoint="WebHookBatchService.Queue"/>
.......
</MessageEndpointMappings>
</UnicastBusConfig>
2) WebHookProcessService - actually processes 1 message sent by the WebHookBatchService.
Once messages are received on the WebHookBatchService.Queue our WebHookBatchService will look up all the subscribers for the specific tenant + message type and foreach send individual messages to WebHookProcessService.Queue for the WebHookProcessService (which we can make an instance of nservicebus loadbalancer to bridge the batch and actual processor) to actually process the real messages probably using http://restsharp.org/.
Questions
Are there any existing open source projects that do this today?
Now since we have no control of the durability of the subscribers how should we manage errors?
http://wiki.shopify.com/WebHook
A webhook will be deleted if there are 19 consecutive failures for the exact same webhook.
It doesn't mention any delays in the webhook.. What have people experienced with standard delay in retry logic?
Here are some other thoughts:
proposal 0: MaxRetries="1". Purge WebHookProcessService.ErrorQueue nightly. (no retry - guaranteed message loss if it fails the first time)
proposal 1:
MaxRetries="1" on exception catch send email containing xml version of the message that would have been delivered over http.
Purge WebHookProcessService.ErrorQueue nightly.
-- I see potential a spam issues.
proposal 2: The nservicebus MaxRetries retries right away without delay. So i would need to create (1hr - 24hr) bucket queues and use a RetrySchedulerService although I see this as difficult to maintain and confusing for subscribers when they all at once get 25 messages in a non DateCreated ordered fashion when there service endpoint begins to work.
Digging for ideas...
The Gateway is typically used for communication between physical sites over HTTP. Since you are exposing an endpoint to the world to accept callbacks, I'm thinking you could just use the built-in WCF hosting and expose your endpoint through the firewall to 3rd parties. The rest of your setup sounds appropriate to me.
As for errors, you are correct, NSB retries immediately, but if you using web call backs this may get you by in the cases there are small hiccups. You will need to determine how you want to process the error queues, we just build in a new endpoint to process the error queues with logic to determine the retries, delay etc. A nice way to accomplish this is to use a Saga, which includes a Timeout manager. This enables a workflow where you can retry a specified number of times, try another communication, log everything, and ultimately notify someone who can contact the 3rd party to let them know there stuff is busted.