I want to create two messages within a pipeline component and pass this onto the Assemble stage where it would go through the BTAHL7 Microsoft Accelerator.
The pipeline component would be placed in the Send Pipeline 'Pre-Assemble' stage.
This is the code which would return the messages.
private System.Collections.Queue qOutputMsgs = new System.Collections.Queue();
IBaseMessage[] allMessages = new IBaseMessage[] { pInMsg, pInMsg };
foreach (IBaseMessage msg in allMessages)
{
qOutputMsgs.Enqueue(msg);
}
return (IBaseMessage)qOutputMsgs.Dequeue();
For testing purpose, I have added the two pInMsg into the an array, and looped through this array to add it to the queue. Then, returning the messages separated.
However, although the queue has 2 values, in the other parts of the pipeline (Assemble and Encode), it returns only one message.
Why does it do this? How can I get it to return two messages?
This logic won’t work in Execute method. You either need to implement IDisassembler interface for such pipeline and need to use it on a receive location. Or you use an orchestration to create multiple messages before sending to the send port.
Related
I am using ZMQ NetMQ package in c# to receive the message from the subscriber. I am able to receive the msg but I am sticking in the while loop. I want to break the while loop if the publisher is stopped sending data.
Here is my subscriber code:
using (var subscriber = new SubscriberSocket())
{
subscriber.Connect("tcp://127.0.0.1:4000");
subscriber.Subscribe("A");
while (true)
{
var msg = subscriber.ReceiveFrameString();
Console.WriteLine(msg);
}
Q : "How to check ZMQ publisher is alive or not in c# ?"
A :There are at least two ways to do so :
a )modify the code on both the PUB-side and SUB-side, so that the Publisher sends both the PUB/SUB-channel messages, and independently of that also PUSH/PULL-keep-alive messages to prove to the SUB-side it is still alive, as being autonomously received as confirmations from the PULL-AccessPoint on the SUB-side loop. Not receiving such soft-keep-alive message for some time may trigger the SUB-side loop to become sure to break. The same principle may get served by a reversed PUSH/PULL-channel, where SUB-side, from time to time, asks the PUB-side, listening on the PULL-side, using asynchronously sent soft-request message to inject a soft-keep-alive message into the PUB-channel ( remember the TOPIC-filter is a plain ASCII-filtering from the left to the right of the message-payload, so PUSH-delivered message could as easily send the exact text to be looped-back via PUB/SUB back to the sender, matching the locally known TOPIC-filter maintained by the very same SUB-side entity )
b )in cases, where you cannot modify the PUB-side code, we still can setup a time-based counter, after expiring which, without receiving a single message ( be it using a loop of a known multiple of precisely timed-aSUB.poll( ... )-s, which allows for a few, priority-ordered interleaved control-loops to be operated without uncontrolled mutual blocking, or by using a straight, non-blocking form of aSUB.recv( zmq.NOBLOCK ) aligned within the loop with some busy-loop avoiding, CPU-relieving sleep()-s ). In case such timeout happens, having received no actual message so far, we can autonomously break the SUB-side loop, as requested above.
Q.E.D.
var msgs = new List<string> {“msg1”, “msg2”, “msg3”};
var tasks = new List<Task>();
Foreach(var msg in msgs) {
tasks.add(_producer.ProduceAsync(...)); }
var deliveryReports = Task.WhenAll(tasks).Result;
My Kafka producer config:
Batch size: 10
Linger:100 ms
My question is, do the tasks get completed in the order they were
created. Can I guarantee that the task representing msg1 completes
before the task representing msg2 or msg3.
Thanks.
Ok I think I now understand how the producer and the broker works to achieve ordering.
So, when ProduceAsync is called, it adds the message to the send buffer, creates promise that is used to complete future and returns future.So, it creates task completion source object and returns its task.
The client library(librdkafka) waits until it receives the configured number of messages or timeout period to batch the messages. A batch is created containing the messages in the same order as in the send buffer. The batch is partitioned (randomly if the default partitioner is used) based on their destination partitions/topics, i.e. split into smaller batches. Each post-split batch is sent to the respective leader broker/ISR (the individual send()’s happen sequentially), and each is acked by its respective leader broker according to request.required.acks. The client library invokes a callback on each ack it receives and the callback completes its respective future i.e taskCompletionSource.Set();
There's a couple of things here.
First, librdkafka has the capability to manage re-tries for you and by default it does ('retries' is set to 2) - so this can cause re-ordering of message delivery and delivery reports. To ensure this doesn't happen you can set 'max.in.flight' to 1 (or 'retries' to 0 and manage this yourself).
With librdkafka configured to supply delivery-reports back to .net in the order the messages are sent, the question becomes one of Task completion ordering guarantees. I need to think about this for more than 5 minutes to give a good answer, but for now assume ordering is not guaranteed (I will write more later). You can get guaranteed ordering by using the variants of ProduceAsync that accept an IDeliveryReport handler. Note that in version 1.0, these methods will be changed somewhat and will be called BeginProduce.
I have a console application to read all the brokered messages present in the subscription on the Azure Service Bus. I have around 3500 messages in there. This is my code to read the messages:
SubscriptionClient client = messagingFactory.CreateSubscriptionClient(topic, subscription);
long count = namespaceManager.GetSubscription(topic, subscription).MessageCountDetails.ActiveMessageCount;
Console.WriteLine("Total messages to process : {0}", count.ToString()); //Here the number is showing correctly
IEnumerable<BrokeredMessage> dlIE = null;
dlIE = client.ReceiveBatch(Convert.ToInt32(count));
When I execute the code, in the dlIE, I can see only 256 messages. I have also tried giving the prefetch count like this client.PrefetchCountbut then also it returns 256 messages only.
I think there is some limit to the number of messages that can be retrieved at a time.However there is no such thing mentioned on the msdn page for the RecieveBatch method. What can I do to retrieve all messages at a time?
Note:
I only want to read the message and then let it exist on the service bus. Therefore I do not use message.complete method.
I cannot remove and re-create the topic/subscription from the Service Bus.
Edit:
I used PeekBatch instead of ReceiveBatch like this:
IEnumerable<BrokeredMessage> dlIE = null;
List<BrokeredMessage> bmList = new List<BrokeredMessage>();
long i = 0;
dlIE = subsciptionClient.PeekBatch(Convert.ToInt32(count)); // count is the total number of messages in the subscription.
bmList.AddRange(dlIE);
i = dlIE.Count();
if(i < count)
{
while(i < count)
{
IEnumerable<BrokeredMessage> dlTemp = null;
dlTemp = subsciptionClient.PeekBatch(i, Convert.ToInt32(count));
bmList.AddRange(dlTemp);
i = i + dlTemp.Count();
}
}
I have 3255 messages in the subscription. When the first time peekBatch is called it gets 250 messages. so it goes into the while loop with PeekBatch(250,3225). Every time 250 messages are only received. The final total messages I am having in the output list is 3500 with duplicates. I am not able to understand how this is happening.
I have figured it out. The subscription client remembers the last batch it retrieved and when called again, retrieves the next batch.
So the code would be :
IEnumerable<BrokeredMessage> dlIE = null;
List<BrokeredMessage> bmList = new List<BrokeredMessage>();
long i = 0;
while (i < count)
{
dlIE = subsciptionClient.PeekBatch(Convert.ToInt32(count));
bmList.AddRange(dlIE);
i = i + dlIE.Count();
}
Thanks to MikeWo for guidance
Note: There seems to be some kind of a size limit on the number of messages you can peek at a time. I tried with different subscriptions and the number of messages fetched were different for each.
Is the topic you are writing to partitioned by chance? When you receive messages from a partitioned entity it will only fetch from one of the partitions at a time. From MSDN:
"When a client wants to receive a message from a partitioned queue, or from a subscription of a partitioned topic, Service Bus queries all fragments for messages, then returns the first message that is returned from any of the messaging stores to the receiver. Service Bus caches the other messages and returns them when it receives additional receive requests. A receiving client is not aware of the partitioning; the client-facing behavior of a partitioned queue or topic (for example, read, complete, defer, deadletter, prefetching) is identical to the behavior of a regular entity."
It's probably not a good idea to assume that even with a non partitioned entity that you'd get all messages in one go with really either the Receive or Peek methods. It would be much more efficient to loop through the messages in much smaller batches, especially if your message have any decent size to them or are indeterminate in size.
Since you don't actually want to remove the message from the queue I'd suggest using PeekBatch instead of ReceiveBatch. This lets you get a copy of the message and doesn't lock it. I'd highly suggest a loop using the same SubscriptionClient in conjunction with PeekBatch. By using the same SubscriptionClient with PeekBatch under the hood the last pulled sequence number is kept as as you loop through it should keep track and go through the whole queue. This would essentially let you read through the entire queue.
I came across a similar issue where client.ReceiveBatchAsync(....) would not retrieve any data from the subscription in the azure service bus.
After some digging around I found out that there is a bit for each subscriber to enable batch operations. This can only be enabled through powershell. Below is the command I used:
$subObject = Get-AzureRmServiceBusSubscription -ResourceGroup '#resourceName' -NamespaceName '#namespaceName' -Topic '#topicName' -SubscriptionName '#subscriptionName'
$subObject.EnableBatchedOperations = $True
Set-AzureRmServiceBusSubscription -ResourceGroup '#resourceName' -NamespaceName '#namespaceName' -Topic '#topicName'-SubscriptionObj $subObject
More details can be found here. While it still didn't load all the messages at least it started to clear the queue. As far as I'm aware, the batch size parameter is only there as a suggestion to the service bus but not a rule.
Hope it helps!
Right now, our publishing code for large amounts of messages looks like so:
foreach (var message in messages)
{
publisher.Publish(message);
}
Does there exist the ability to send more than one message over the channel at once?
publisher.Publish(messages);
or as so if we chunk
var chunks = messages.Chunk(100);
foreach (var chunk in chunks)
{
publisher.Publish(chunk);
}
With current version of RabbitMq(3.8.2) you can send batch messages as below for c# client sdk:
basicPublishBatch = channel.CreateBasicPublishBatch();
basicPublishBatch.Add("exchange", "routeKey", false, null, new byte[]{1});
basicPublishBatch.Add("exchange", "routeKey", false, null, new byte[]{1});
basicPublishBatch.Publish();
Check this PR:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/368
For RabbitMQ the AMQP protocol is asynchronous for produce and consume operations so it is not clear how to benefit from batch consumer endpoint given out of the box.
What you can do is to create endpoints for chunked messages and process them inside workflow if you can speed up operations. So one solution would be for example to have batching component included before publisher class and send custom messages.
I'm looking at using MassTransit to publish a event from one process to all other processes that care about that event. In my environment, several processes that are subscribed to that event could be running on the same box.
Based on the MassTransit's example code, here's a VERY simple proof-of-concept that does not attempt to deal with multi-cast (processes will also be distributed across many machines). Consider the "Message" below as an "Event", and that I want every instance of this process to get that message:
using System;
using MassTransit;
namespace BasicPubSub
{
public class Message
{
public String Text { get; set; }
}
public class Program
{
static void Main(string[] args)
{
//initialize the bus
var bus = ServiceBusFactory.New(sbc =>
{
sbc.UseMsmq();
//cause the DTC and MSMQ to get installed/fixed if needed.
sbc.VerifyMsDtcConfiguration();
sbc.VerifyMsmqConfiguration();
//sbc.UseMulticastSubscriptionClient();//Doesn't behave for a private queue.
sbc.ReceiveFrom("msmq://localhost/test_queue");
//Listen for interesting events.
sbc.Subscribe(subs =>
{
subs.Handler<Message>(msg => Console.WriteLine(msg.Text));
});
});
//stay open until the user types "exit"
var message = "";
do
{
message = Console.ReadLine();
//broadcast the message
bus.Publish(new Message { Text = message });
} while (message != "exit");
}
}
}
If I run multiple instances of this C# app, only one of the instances receives the message, but I need all of the instances to receive it. This behavior is consistent with the classical notion of a queue, so I'm fine with that.
What I am looking to know is whether I can use MSMQ/MassTransit to publish a message that all subscribers will receive, as opposed to just a single subscriber dequeuing it and the others not receiving it. Perhaps I'm trying to use a hammer and I need a paintbrush?
Just to be clear here, I am not looking at sharing a single queue for different kinds of messages, I'm looking to setup "peer-to-peer" pub-sub for a specific type of message, using MT and MSMQ.
I'm not sure about hammers or paintbrushes, but I think you're heading in the right direction.
To start with, each instance of a bus requires it's own queue to read from. If you are running multiple versions of this same program, make sure they are reading off different queues. Attempting to read from the same queue leads to Bad Stuff(TM).
Join the mailing list if you have more questions. https://groups.google.com/forum/#!forum/masstransit-discuss.
When publishing, all consumers should recieve the message once they have been subscribed. It might take a second or two for the subscription data to pass to all processes when starting up.
I would also suggest taking a look at RabbitMQ. Unless you absolutely need DTC, RabbitMQ is a much more robust messaging platform.