I have few topics and subscriptions on google pub/sub. I created .net core application which publish and receives messages using Google.Cloud.PubSub.V1 nuget library.
Code is pretty easy:
// send a message
var result = await publisher.PublishAsync(topicName, messages);
...
// receive pending messages in loop
var response = await subscriber.PullAsync(subscriptionName, true, count);
As I see from logs publisher successfully publish a message but subscriber never receive it. Same application does not work on cloud but works on local machine. Configuration is same on both environments (my machine get access to same Pub/Sub service as production). Is it possible somehow make a diagnostics and find out what the problem is?
Related
My Azure Service Bus has only one topic and there is only one publisher. The publisher sends messages to the topic with this code:
public void Publish<T>(T messageObject)
{
var jsonString = JsonSerializer.Serialize(messageObject);
var message = new ServiceBusMessage(jsonString);
message.ApplicationProperties["messageType"] = typeof(T).Name;
serviceBusSender.SendMessageAsync(message);
}
In my application code, I call this method consecutively to send message1, message2 and message3, respectivly. However, when I go to Azure, and receive messages on Service Bus Explorer, I see the messages' order is not necessarily the same.
Is this behavior expected? Or am I missing something here?
If you have created the non-partitioned entity then you can enable the Support ordering feature as documented here.
The Support ordering feature allows you to specify whether messages
that are sent to a topic will be forwarded to the subscription in the
same order in which they were sent. This feature doesn't support
partitioned topics. For more information, see
TopicProperties.SupportOrdering in .NET or
TopicProperties.setOrderingSupported in Java.
In case of partitioned entity your can leverage the session while sending the message. A session will give you related messages in the exact arrival order if you process them in sequence, meaning using one thread and without prefetching. When the consumer fails processing a message and abandons it, that message will again be the first to be delivered until it exceeds its delivery count.
how to detect if the message broker configuration is valid or if the connection to the message broker is lost using Mass Transit to RabbitMQ? When publishing messages when RabbitMQ is present does not seems to complain if there is no broker connection right away and seems to recover when the RabbitMQ server comes up. Is there a way to listen in on the connection events and warn if the configuration is not valid?
If you use .NET Core and configure MassTransit as per the docs, you can resolve the instance of IBusHealth and use it in your service.
The AddMassTransit method registers the default instance, which you can ask for the bus health status at any time. That's the method code:
public HealthResult CheckHealth()
{
var endpointHealthResult = _endpointHealth.CheckHealth();
var data = new Dictionary<string, object> {["Endpoints"] = endpointHealthResult.Data};
return _healthy && endpointHealthResult.Status == BusHealthStatus.Healthy
? HealthResult.Healthy("Ready", data)
: HealthResult.Unhealthy($"Not ready: {_failureMessage}", data: data);
}
As you can see, if you call busHealth.CheckHealth() it will return either Healthy or Unhealthy and in the latter case would also give you the list of failing endpoints.
Since BusHealth only monitors the bus itself and all its receive endpoints, you might not get notified when your service failed to publish messages.
You can use the diagnostics listener or create your own publish or send observer, which is called before and after publish/send and on any failure.
I am using NService to create an endpoint.
The endpoint is listening to an event and do some calculation, then publish result (success or fail) to other endpoints
I know that NServiceBus support ImmediateRetry and DelayRetry, and they are configurable.
Now, I want to publish a fail result event to other endpoints after all retries (before sending to error queue).
public async Task Handle(MyEvent message, IMessageHandlerContext context)
{
Console.WriteLine($"Received MyEvent, ID = {message.Id}");
//Connect to other services to get data and do some calculation
Thread.Sleep(1000);
Console.WriteLine($"Processed MyEvent, ID = { message.Id}");
await context.Publish(new MyEventResult { IsSucceed = true });
}
Above is my current code. It will publish a successful result if there is no exception throw. But If it has a fatal exception, I don't know how to publish a fail result event before the message is sent to the error queue.
Thanks in advance.
Notes: I am using NServiceBus 6.4.3
I'm not sure why you want this but have you looked at NServiceBus sagas? They are intended to be used when having to doing blocking IO via (external) services. You can take alternative actions based on the fact if a specific task hasn't been performed within an allocated period or because the returned result was incorrect.
https://docs.particular.net/nservicebus/sagas/
See the following sample of a saga:
https://docs.particular.net/samples/saga/simple/
The following is a sample showing the usage of saga timeouts. If specific task has not been performed within a specific duration an alternative action can be performed like publishing an event or performing a ReplyToOriginator
https://docs.particular.net/nservicebus/sagas/timeouts
https://docs.particular.net/nservicebus/sagas/reply-replytooriginator-differences
https://docs.particular.net/nservicebus/sagas/#notifying-callers-of-status
By using sagas you are making your process explicit. I would avoid hooking into the recovery mechanism for this.
The recovery mechanism is meant to deal with transient errors like network connectivity issues, database deadlocks, etc. but not with expected failure results. You should properly process these and continue your modeled process in its unhappy path.
I have 3 machines, let's call them A, B (both servers) and Dev (my local machine).
I want to send a message queue item from A to B.
The actual C# code I have is rather simple and honestly, I really do not think it is problem here. (It's a webapi that takes a POSTed object and just shoves it down the queue).
I can send these messages just fine from Dev to B (while logged into a domain admin account) without a problem and I can inspect the body of the messages. However I cannot send messages from A to B. The private queue on B is set to allow "Everyone" the "Full Control" permissions.
If I pause the outbound queue on A and send the messages, they sit in the outbound queue and the body is exactly as I would expect it to be, but when I resume that outbound queue again, they are never received on the other end at B.
I can't figure out what's going on for the life of me. I tried the 'TCP' method but I need to refer to my queues by machine name not IP.
For reference, the code used to send the message is:
using (var queue = new MessageQueue($"FormatName:Direct=OS:MachineB\\private$\\Queue"))
{
var queueItem = new QueueItem();
queueItem.Object = this.postedObject;
var message = new System.Messaging.Message(queueItem);
queue.Send(message);
}
For reference and anyone else who comes across this:
The project sending the messages to the queue was a webapi. It was running under the AppPoolIdentity account in IIS and despite the receiving queue allowing Everyone access, it was denying these messages, but gracefully. So the webapi would fire the message and wouldn't throw any exceptions, it'd hit the outbound, but then never arrive.
We switched the account to NetworkService and it worked just fine.
I have the following setup and problem with MSMQ. Based on previous experience with MSMQ I'm betting that it is something simple I'm missing but I just don't know what it is.
The Setup
I have 3 load-balanced web servers (lets call them Servers W1, W2 and W3) and 1 server which processes certain events/data away from web requests (which I'll call P). All 3 of the web servers, once a particular event occurs within the web application, will send a message to a remote private queue on Server P, which will then process each message from the queue and carry out some task.
The Problem
For the most part - at a guess 95% of the time - everything runs fine, but occasionally Server P does not receive messages from the web servers. This is either because W1, W2 or W3 are not sending them or they are not being received by P, I just can't tell. This means I'm missing vital events happening from the users on the web application but I cannot find any errors listed in my own logs.
The Details
Here are all the details I can think of which may help explain my setup and what I've figured out so far:
The private queue on Server P is non-transactional.
The private queue has permissions setup for Everyone to both Send and Receive Messages.
This is the code I use (C#) to send the message to the remote private queue:
var queue = new MessageQueue(#"FormatName:DIRECT=OS:ServerP\PRIVATE$\MyMessageQueue");
var defaultProperties = queue.DefaultPropertiesToSend;
defaultProperties.AcknowledgeType = AcknowledgeTypes.FullReachQueue | AcknowledgeTypes.FullReceive;
defaultProperties.Recoverable = true;
defaultProperties.UseDeadLetterQueue = true;
defaultProperties.UseJournalQueue = true;
queue.Send(requestData);
Sending the message using the code above does not appear to throw an exception - if it did my error handler in the web application would have caught and logged it, so I'm assuming it is sent.
There are outgoing queues on W1, W2 and W3 all pointing to the private queue on P - all these are empty.
On W1, W2 and W3 I cannot see any "dead-letter" messages.
On P the private queue is empty so messages are being processed (which I can verify from my database).
On P there are no "dead-letter" messages. There are journal messages but they don't seem to correspond to any recent date/times.
All servers are running Windows Server 2012.
Most of the time messages are sent, received and processed just fine but, without any pattern visible to me, sometimes they are not. Can anyone see what is going wrong? Or explain to me how I can try and figure out what is happening?
Are you sure that the receiver on P does not crash/lose the message somehow? Because your queue is not transactional, if somehow processing fails then that's one lost message.
Anyway, there are many possible causes why this could fail.
What kind of logging do you have (DEBUG/INFO levels)?
I think the following will help tracking down the issue:
When an event is generated in the web app.
Right before you send an event from the web app, via MSMQ.
In the receiver when you get a message from the queue.
This way you could at least match sent messages to received messages and to processed messages.
As a side note, when you check for dead-letter messages you do so on the source computer and on any intermediary hops, not on the destination one. If you don't have any hops, then they will be relayed to the non-transactional dead-letter queue on the web servers.