I have a method that reads a list of messages from a message queue. The signature is:
IList<TMsg> Read<TMsg>(MessageQueue queue, int timeout, MessageQueueTransaction tx);
The basic functionality is that it will reads as many messages as it can from the queue, within the timeout, using the given transaction. The problem I'm having is deciding on how best to enforce the timeout. I have two working versions at the moment. These are:
Using BeginPeek with a timeout. If it succeeds, the message is removed with a transactional Receive call. The timeout for BeginPeek is recalculated after each call, based on the time the Read began, and the current time.
Using Receive with the timeout value, and catching the exception when the timeout expires.
The problem with the first approach is that it requires the queue to be read in DenySharedReceive mode, otherwise you can't guarantee the message will still be there between the Peek and Receive. The problem with the second method is that an exception needs to be thrown and handled (albeit, internally and transparently) which is probably not a great design since each call will always end in an exception which goes against the idea of throwing exceptions only in exceptional circumstances.
Does anyone have any other suggestions how I might achieve this, or comments on these two techniques and my concerns?
You could use the reactive excensions to create a producer of messages, use Observable.Buffer to manage the timeout and later subscribe to this producer
public IEnumerable<Message> GetMessage()
{
//do the peek and receive a single message
yield return message;
}
//and then something like
var producer = GetMessage().ToObservable();
// this is where your timeout goes
var bufferedMessages = producer.Buffer(TimeSpan.FromSeconds(3));
var disp = bufferedMessages.Subscribe(messages =>
{
Console.WriteLine("You've got {0} new messages", messages.Count());
foreach (var message in messages)
Console.WriteLine("> {0}", message); // process messages here
});
disp.Dispose(); // when you no longer want to subscribe to the messages
For more reactive examples look here
After a bit of investigation, hatchet's comment is the closest to the 'answer', at least as far as .NET is concerned. The wrapped native methods provide a return value (rather than error value) for 'TIMEOUT', but this is considered an exception by .NET and re-wrapping the native code is just not worth it. I tried. :p
Related
I would like to track the number of message retries and redelivers that occur while using MassTransit 3. I have both retries and redeliveries configured:
config.UseDelayedRedelivery(r => r.Immediate(2));
config.UseRetry(r => r.Immediate(3));
I have set up a IConsumeObserver and a IReceiveObserver as described here. And I can inspect the ConsumeContext/ReceiveContext in PostConsume<T>(ConsumeContext<T> context)/PostReceive(ReceiveContext context).
But when inspecting the contexts I cannot see a difference between the context for a message which was consumed without exception and one that threw an exception during consumption and will be redelivered.
How can I, in the PostConsume, method of an IConsumeObserver or IReceiveObserver determine if context represents a message that will be redelivered or one that has completed sucesfully?
You can do it. MassTransit keeps the redelivery count in the message headers, otherwise, it won't know when to stop redelivering, according to your policy.
If this line returns a non-zero (or not null, I am not sure) - you are dealing with a redelivered message.
context.Headers.Get(MessageHeaders.RedeliveryCount, default(int?)));
If your message is being retried (not redelivered), check this answer from Chris: Get MassTransit message retries amount
The consumer can influence whether or not a message will be redelivered, but it doesn't have full control or knowledge of it.
For example, everything succeeds on the consuming side, but it just takes too long, the publisher will retry and the consumer has no simple way to know that this will happen.
It's often best to design your application so that consuming the same message multiple times has the same effect as consuming it one time.
Additionally, you check the MessageId on consuming the message if you want to see if you've consumed it before.
The ConsumeContext also has a RetryCount, but I don't believe it's incremented until the next time the consumer runs.
I am reading from a REST service and need to handle "Wait and retry" for a heavily used service that will give me an error:
Too many queries per second
or
Server Busy
Generally speaking, since I have many REST services to call, how can I generically handle backoff logic that would occur when an exception occurs?
Is there any framework that has this built in? I'm just looking to write clean code that doesn't worry too much about plumbing and infrastructure.
You can wrap the attempt up within a method that handles the retry logic for you. For example, if you're using WebClient's async methods:
public async Task<T> RetryQuery<T>(Func<Task<T>> operation, int numberOfAttempts, int msecsBetweenRetries = 500)
{
while (numberOfAttempts > 0)
{
try
{
T value = await operation();
return value;
}
catch
{
// Failed case - retry
--numberOfAttempts;
}
await Task.Delay(msecsBetweenRetries);
}
throw new ApplicationException("Operation failed repeatedly");
}
You could then use this via:
// Try 3 times with 500 ms wait times in between
string result = await RetryQuery(async () => webClient.DownloadStringTaskAsync(url), 3);
Try and determine how many active requests can be active at a time and use a Semaphore.
It is a way to handle resource locking where the are multiple identical resources, but only a limited number of them.
Here's the MSDN documentation on semaphores
I recommend you look into the Transient Fault Handling Application Block, part of the Enterprise Library.
In the past, the EL has IMO been over-engineered and not that useful, but they've taken steps to address that; the TFHAB is one of the newer blocks that follows better design guidelines (again, IMO).
I have written a socket for a server in C++ CLI that is using winsock. The sockets are using async methods for sending, receiving and accepting connections. After implementing my socket in the production environment, the send function stops working giving me the error WSAEWOULDBLOCK. Out from my research on the net, this means the network buffer for socket IO is full or the networking is too busy to do my operation at this moment. However, I have not seen any specific solution which can address this problem. My temporary solution was to create a do-while loop around the WSASend function, making the thread sleep for X amount of MS and then try again. This resulted in far higher latency than the previous socket (.NET socket class) and large lag spikes.
My code for sending data is as following:
void Connectivity::ConnectionInformation::SendData(unsigned char data[], const int length)
{
if (isClosed || sendError)
return;
Monitor::Enter(this->syncRoot);
try
{
sendInfo->buf = (char*)data;
sendInfo->len = length;
do
{
state = 0;
if (WSASend(connection, sendInfo, 1, bytesSent, 0, NULL, NULL) == SOCKET_ERROR)
{
state = WSAGetLastError();
if (state == WSAEWOULDBLOCK)
{
Thread::Sleep(SleepTime);
//Means the networking is busy and we need to wait a bit for data to be sent
//Might wanna decrease the value since this could potentially lead to lagg
}
else if (state != WSA_IO_PENDING)
{
this->sendError = true;
//The send error bool makes sure that the close function doesn't get called
//during packet processing which could cause a lot of null reffernce exceptions.
}
}
}
while (state == WSAEWOULDBLOCK);
}
finally
{
Monitor::Exit(this->syncRoot);
}
}
Is there a way to use for example the WSAEventSelect method in order to get a callback when I am able to send data? Out from the documentation on MSDN, the wait for data method could also get stuck in this error. Anyone got any solutions for getting around this?
The error code WSAEWOULDBLOCK means that you attempted to operate on a non-blocking socket but the operation could not be completed immediately. This is not a real error - it means that you can retry later or schedule an asynchronous IO (which wouldn't fail). But this is not what you want in the first place. Let me explain:
You are supposed to use sockets in one of two ways:
Synchronous, blocking.
Asynchronous, non-blocking, callback-based.
You are mixing the two which gets you the worst of both. You created a non-blocking socket and use it in a potentially blocking way.
Alas I'm not full qualified to give best-practices for native-code sockets. I suggest you read all of the docs for WSASend because they seem to explain all of this.
Now, why would this strange error code even exist? It is a performance optimization. You can speculatively try to send synchronously (which is very fast). And only if it fails you are supposed to schedule an asynchronous IO. If you don't need that optimization (which you don't) don't do it.
As #usr says, I need to have either LPWSAOVERLAPPED or LPWSAOVERLAPPED_COMPLETION_ROUTINE set to a value in order to make the operation non-blocking. However, after testing, I found out I need t have a LPWSAOVERLAPPED object in order to make the completion routine called. It is also mentioned on MSDN on the documentation of the WSASend function that if the overlapped object and the completion routine is NULL, the socket would behave as a blocking socket.
Thanks, and merry xmas everyone! :)
I want to take message from one queue and send it to database. I want to do it only if it's in specific format.
If i use Receive method directly and some exception occurs while accessing Body of the Message, I lose the message since Receive method of the MessageQueue removes the message from the queue.
To avoid loss of message, now i first Peek the message, and if its well formatted, I use Receive method to remove it from the queue to send it to database.
Code I have written is like this:
Message msg = _queue.Peek(new TimeSpan(0, 0, LoggingService.Configuration.ReceiveTimeout));
// LogMessage is my own class which is adding some more stuff to original message from MessageQueue
LogMessage message = null;
if (msg != null)
{
if (!(msg.Formatter is BinaryMessageFormatter))
msg.Formatter = new BinaryMessageFormatter();
message = LogMessage.GetLogMessageFromFormattedString((string) msg.Body);
// Use Receive method to remove the message from queue. This line will we executed only if the above line does not
// throw any exception i.e. if msg.Body does not have any problem
Message wellFormattedMsg =
_queue.ReceiveById(msg.Id);
SendMessageToDatabase(message);
}
Is this logic right to first using Peek and then Receive? Or is there any other better way f achieving the same thing? Please note that I dont want to get all messages at a time. MessageQueue is non transactional.
This is the same approach that I take when manually dequeuing message one at a time and I've not come across any issues with it.
The one thing that you do not appear to dealing with is how to handle a message on the queue that does not have the required format. Is your intention to leave it on the queue? If so, you might end up with a very large queue and have all sorts of issues with peeking at messages further up the queue that have not yet been expected. It would appear to make more sense to also de-queue those messages that do not have the required format and store them elsewhere if they cannot be deleted.
"If i use Receive method directly and some exception occurs while accessing Body of the Message, I lose the message since Receive method of the MessageQueue removes the message from the queue."
You should be using transactional receives so that the message returns to the queue when/if the transaction aborts.
Cheers
John Breakwell
My original question from a while ago is MSMQ Slow Queue Reading, however I have advanced from that and now think I know the problem a bit more clearer.
My code (well actually part of an open source library I am using) looks like this:
queue.Receive(TimeSpan.FromSeconds(10), MessageQueueTransactionType.Automatic);
Which is using the Messaging.MessageQueue.Receive function and queue is a MessageQueue. The problem is as follows.
The above line of code will be called with the specified timeout (10 seconds). The Receive(...) function is a blocking function, and is supposed to block until a message arrives in the queue at which time it will return. If no message is received before the timeout is hit, it will return at the timeout. If a message is in the queue when the function is called, it will return that message immediately.
However, what is happening is the Receive(...) function is being called, seeing that there is no message in the queue, and hence waiting for a new message to come in. When a new message comes in (before the timeout), it isn't detecting this new message and continues waiting. The timeout is eventually hit, at which point the code continues and calls Receive(...) again, where it picks up the message and processes it.
Now, this problem only occurs after a number of days/weeks. I can make it work normally again by deleting & recreating the queue. It happens on different computers, and different queues. So it seems like something is building up, until some point when it breaks the triggering/notification ability that the Receive(...) function uses.
I've checked a lot of different things, and everything seems normal & isn't different from a queue that is working normally. There is plenty of disk space (13gig free) and RAM (about 350MB free out of 1GB from what I can tell). I have checked registry entries which all appear the same as other queues, and the performance monitor doesn't show anything out of the normal. I have also run the TMQ tool and can't see anything noticably wrong from that.
I am using Windows XP on all the machines and they all have service pack 3 installed. I am not sending a large amount of messages to the queues, at most it would be 1 every 2 seconds but generally a lot less frequent than that. The messages are only small too and nowhere near the 4MB limit.
The only thing I have just noticed is the p0000001.mq and r0000067.mq files in C:\WINDOWS\system32\msmq\storage are both 4,096KB however they are that size on other computers also which are not currently experiencing the problem. The problem does not happen to every queue on the computer at once, as I can recreate 1 problem queue on the computer and the other queues still experience the problem.
I am not very experienced with MSMQ so if you post possible things to check can you please explain how to check them or where I can find more details on what you are talking about.
Currently the situation is:
ComputerA - 4 queues normal
ComputerB - 2 queues experiencing problem, 1 queue normal
ComputerC - 2 queues experiencing problem
ComputerD - 1 queue normal
ComputerE - 2 queues normal
So I have a large number of computers/queues to compare and test against.
Any particular reason you aren't using an event handler to listen to the queues? The System.Messaging library allows you to attach a handler to a queue instead of, if I understand what you are doing correctly, looping Receive every 10 seconds. Try something like this:
class MSMQListener
{
public void StartListening(string queuePath)
{
MessageQueue msQueue = new MessageQueue(queuePath);
msQueue.ReceiveCompleted += QueueMessageReceived;
msQueue.BeginReceive();
}
private void QueueMessageReceived(object source, ReceiveCompletedEventArgs args)
{
MessageQueue msQueue = (MessageQueue)source;
//once a message is received, stop receiving
Message msMessage = null;
msMessage = msQueue.EndReceive(args.AsyncResult);
//do something with the message
//begin receiving again
msQueue.BeginReceive();
}
}
We are also using NServiceBus and had a similar problem inside our network.
Basically, MSMQ is using UDP with two-phase commits. After a message is received, it has to be acknowledged. Until it is acknowledged, it cannot be received on the client side as the receive transaction hasn't been finalized.
This was caused by different things in different times for us:
once, this was due to the Distributed Transaction Coordinator unable to communicate between machines as firewall misconfiguration
another time, we were using cloned virtual machines without sysprep which made internal MSMQ ids non-unique and made it receive a message to one machine and ack to another. Eventually, MSMQ figures things out but it takes quite a while.
Try this
public Message Receive( TimeSpan timeout, Cursor cursor )
overloaded function.
To get a cursor for a MessageQueue, call the CreateCursor method for that queue.
A Cursor is used with such methods as Peek(TimeSpan, Cursor, PeekAction) and Receive(TimeSpan, Cursor) when you need to read messages that are not at the front of the queue. This includes reading messages synchronously or asynchronously. Cursors do not need to be used to read only the first message in a queue.
When reading messages within a transaction, Message Queuing does not roll back cursor movement if the transaction is aborted. For example, suppose there is a queue with two messages, A1 and A2. If you remove message A1 while in a transaction, Message Queuing moves the cursor to message A2. However, if the transaction is aborted for any reason, message A1 is inserted back into the queue but the cursor remains pointing at message A2.
To close the cursor, call Close.
If you want to use something completely synchronous and without event you can test this method
public object Receive(string path, int millisecondsTimeout)
{
var mq = new System.Messaging.MessageQueue(path);
var asyncResult = mq.BeginReceive();
var handles = new System.Threading.WaitHandle[] { asyncResult.AsyncWaitHandle };
var index = System.Threading.WaitHandle.WaitAny(handles, millisecondsTimeout);
if (index == 258) // Timeout
{
mq.Close();
return null;
}
var result = mq.EndReceive(asyncResult);
return result;
}