When hitting the endpoint the main method gets invoked but subsequent handlers do not pick messages from queue. The queue keeps building.
Has anyone encountered this issue before?
Thanks,
So far we do not know much about neither your architecture nor your code, we have to guess here. So please bear with me if I say something that may not apply to your case.
Several things come to my mind:
1.) Did you see this tread? NServiceBus Bus.Send().Register(callback) Not Working on IIS/Windows Server 2008
2.) There is some unhandled exception or there is one that gets silently caught and causes your handler method to halt for some reason.
3.) Are you using WCF with NServiceBus? Last week my colleague implemented a WCF service that was hosted with NServiceBus and had a problem that sounds similar to yours. It hit the message handler for the first message, but never handled any subsequent message.
The reason was the handler never returned an enum value as a return code (represented as YourEnum in the code snippet), like it was defined in the service inheritance definition (NServiceBus documentation):
public class YourWcfService : WcfService<YourMessage, YourEnum> { }
The calling service did not need any return values, so my collegue did not return it (I cannot exactly tell why there where no compilation errors, it was not my code...). Therefore the handler method never finished correctly and no other messages could be handled.
Related
I am working on SOAP-client in WCF to communicate with a self-hosted WCF service for remote controlling a piece of software that I am developing. This software has a very long running operation (lets call it Print) that will run for several minutes, perhaps up to an hour. I am trying to figure out how to implement the method Print given the following requirements:
The server should be able to raise FaultExceptions to the client, in case something goes wrong.
The client should be informed ASAP should the connection to the service be lost.
The server-side process of Print should continue to run if disconnected, so that the client can reconnect and continue to monitor the process and abort it if necessary.
Since I am new to WCF, I am unsure how to implement this. I currently see two alternatives:
Make Print an async method so that I can "fire and forget" it until it finishes or throws a FaultException. This seems straight-forward, but I see this "problem": There is a client-side request timeout in WCF with default value of 1 minute, which also applies to async methods (if I am not mistaken) and which I would therefore have to increase significantly. This seems a bit like a hack.
Implement the async behavior of Print myself by splitting its behavior into a non-async method StartPringing that starts a server-side task for printing and returns directly (or throws an exception in case something goes wrong) and a client-callback method PrintingFinished. I could then use the callback PrintingFinished to signal to the client, when the print-process has finished or a use an additional callback PrintingFailed to send an exceptions in case something goes wrong. This implementation would be "hidden" behind the async method Print, so that it behaves like any other async method that might throw an exception. Here I see the following challenge: I will have to implement the whole exception callback-stuff myself, to handle exceptions that occur after StartPringing has returned (from StartPringing itself I can throw FaultExceptions).
For both cases I will have to work out how to detect, when the connection is servered (which I am currently doing using a ping method on the service) and then somehow get that event to throw an exception from within the method Print. Implementation-wise this seems more aligned with alternative (2), since I need to already implement all the other event handlers for when the print-process finishes or an exception is thrown. However I am unsure how I would implement this for alternative (1).
So which one of the two alternatives is "better". By better I mean the following considerations:
1. Aligned with the "standard" way in WCF for implementing such a long running method.
2. Maintainability and extensibility.
If I should consider any other alternative, I would be grateful for any other suggestion.
For what I understand of your problem I think if you need a real async communication with reliability to use a message queue like MSMQ. You can use it with WCF : https://msdn.microsoft.com/en-us/library/ms789048(v=vs.110).aspx
Update
In your case, you can use a SOAP call to send print command to the server because it sync and you need to know if the server handle the request. After in the printing operation is long and async. When this operation finish (exception or not) it need to notify client(s). But client could be shutdown for example. For that communication a Message Queue is the solution, MQ ensure that the message will be transmit.
https://en.wikipedia.org/wiki/Message_queue
If you don't want use MSMQ, you can implement a web service on client side to be notified by the printing server, using for example a UUID to match call and notification in a map in memory or in a Data Base.
Consider the scenario, an Azure service bus with message deduplication enabled, with a single topic, with a single subscription and an application that is subscribed to that queue.
How can I ensure that the application receives messages from the queue once and only once ?
Here is the code I'm using in my application to receive messages :
public abstract class ServiceBusListener<T> : IServiceBusListener
{
private SubscriptionClient subscriptionClient;
// ..... snip
private void ReceiveMessages()
{
message = this.subscriptionClient.Receive(TimeSpan.FromSeconds(5));
if (message != null)
{
T payload = message.GetBody<T>(message);
try
{
DoWork(payload);
message.Complete();
}
catch (Exception exception)
{
// message.Complete failed
}
}
}
}
The problem I forsee is that if message.Complete() fails for whatever reason, then that message that has just been processed will remain on the subscription's queue in Azure. When ReceiveMessages() is called again it will pick up that same message from the queue and the application would do the same work again.
Whilst the best solution would be to have idempotent domain logic (DoWork(payload)), this would be very difficult to write in this instance.
The only method I can see to ensure once and only once delivery to an application is by building another queue to act as an intermediary between the Azure service bus and the application. I believe this is called a 'Durable client-side queue'.
However I can see that this would be a potential issue for a lot of applications that use Azure service bus, so is a durable client-side queue the only solution ?
The default behavior when you dequeue a message is called "Peek-Lock" it will lock the message so no one else can get it while your processing it and will remove it when you commit. It will unlock if you fail to commit, so it could be picked up again. This is probably what you are experiencing. You can change the behavior to use "Receive and Delete" which will delete it from the queue as soon as you receive it for processing.
https://msdn.microsoft.com/en-us/library/azure/hh780770.aspx
https://azure.microsoft.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/#how-to-receive-messages-from-a-subscription
I have similar challenges in a large scale Azure platform I am responsible for. I use a logical combination of the concepts embodied by the Compensating Transaction pattern (https://msdn.microsoft.com/en-us/library/dn589804.aspx), and Event sourcing Pattern (https://msdn.microsoft.com/en-us/library/dn589792.aspx). Exactly how you incorporate these concepts will vary, but ultimately, you may need to plan on your own "rollback" logic, or detecting that a previous process completed 100% successfully minus the removal of the message. If there is something you could check upfront, you will know that a message was simply not removed, then complete it and move on. How expensive that "check" is may make this a bad idea. You can even "create" an artificial final step, like adding a row to a DB, that runs only when the DoWork reaches the end. You can then check for that row before processing any other messages.
IMO, the best approach is to make sure that all of the steps in your DoWork() check for the existence of the work as having already been performed (if possible). For example, if it's creating a DB table, run a "IF NOT EXISTS(SELECT TABLE_NAME FROM INFORMATION_SCHEMA...". In that scenario, even in the unlikely event this happens, it's safe to process the message again.
Other approaches I use are to store the MessageID's (the sequential bigint on each message) of the previous X messages (i.e. 10,000), and then check for their existence (NOT IN) before I proceed with processing a message. Not as expensive as you might think and very safe. If found, simply Complete() the message and move on. In other situations, I update the message with a "starting" type status (inline in certain queue types, persisted elsewhere in others), then proceed. If you read a message and this is already set to "started", you know something either failed or did not clear appropriately.
Sorry this is not a clear cut answer, but there are a lot of considerations.
Kindest regards...
You can continue to use a single subscription if you include the logic to detect if the message has been successfully processed already or the stage it had reached into your message handling.
For example, I use service bus messages to insert payments from an external payment system into a CRM system. The message handling logic first checks to see if the payment already exists in CRM (using unique ids associated with the payment) before inserting. This was required because very occasionally the payment would be successfully added to CRM but not reported back as such (timeout or connectivity). Using Receive/Delete when picking up a message would mean that payments would potentially be lost, not checking that the payment already existed could result in duplicate payments.
If this is not possible then another solution I have applied is updating table storage to record the progress of handling a message. When picking up a message the table is checked to see if any stages have already been completed. This allows a message to continue from the stage it had reached previously.
The most likely cause of the scenario you outline is that the time taken to DoWork exceeds the lock on the message. The message lock timeout can be adjusted to a value that safely exceeds the expected DoWork period.
It also possible to call RenewLock on a message within the handler if you are able to track time taken to process against the message lock expiry.
Maybe I misunderstand the design principle of a second queue but it seems as if this would be just as vulnerable to the original scenario you outlined.
Hard to give a definitive answer without knowing what your DoWork() involves but I would consider one or combination of the above as a better solution.
So - I have this external assembly that I'm using. It fires an event DataReceived. Then I'm doing some database operations which may fail due to problems with the data or because of some errors in the code. It would be great if I could "bubble up" the exception into the GUI. In my case I would need a blocking call to the GUI because of the way the assembly works. I'm not sure if this is a good idea but right now that's the only thing that comes to mind based on how the external code works.
The assembly assumes that if the callback (the event) returned safely then the data was processed succesfully - which may not be the case. Of course I would have to deal with the error in some way but that would mean that the server on the other side would always assume that the data was handled correctly.
My questions are:
Can I throw the exception into the GUI? If so, how?
How can I handle the exception in my event so that the assembly doesn't think I processed the data? Do I need some kind of blocking call/exception into the GUI? (Is this even possible?)
On a side note: Isn't that assembly broken by design somehow? Why would it automatically assume that everything went fine just based on if the callback returned?
I don't think that this is broken by design. If you receive the event you'll get informed that something has changed in the source. Now you should only do what is needed to get the informations you need from the source and do any further processing decoupled from the source. For that purpose I would within the event handler simply grab the data (maybe from the source; maybe from the event args) and put them into a ConcurrentQueue. Within my class another Task is running that using the BlockingCollection the retrieve the elements out of this queue to process them. If anything fails, simply call Invoke() to the gui thread and inform the user about what happened.
Ah, and another approach instead of using ConcurrentQueue would be to use Rx. With that you can subscribe to an event and observe it on a different thread by using ObserveOn() which would lead to nearly the same (in this case) but using a more LINQish syntax.
I am used to register callbacks from within ASP/MVC applications in order to get notified of responses to events send/published. In order to do so, NServiceBus provides some methods (Register/RegisterWebCallback) which can be invoked on the IAsync context which is returned by bus.Send(..).
Is it there any equivalent on Rebus side? I could define an IHandleMessage and then manually do internal dispatching of received responses, but it seems a bit overkill.
To be honest, I never really got why NServiceBus would allow you to register an in-memory callback when calling bus.Send.
I've actually only seen it used in weird hacky scenarios where people use it to implement a blocking request/response API by waiting on the wait handle of the returned IAsyncResult.
Is it something that you're seriously missing?
How would you use it?
I'm trying to implement an IErrorHandler in my WCF service in order to log every exception that hits the service boundary before it's passed to the client. I already use IErrorHandlers for translating Exceptions to typed FaultExceptions, which has been very useful. According to the MSDN for IErrorHandler.HandleError(), it's also intended to be used for logging at the boundary.
The problem is, the HandleError function isn't guaranteed to be called on the operation thread, so I can't figure out how to get information about what operation triggered the exception. I can get the TargetSite out of the exception itself, but that gives me the interior method instead of the operation. I could also parse through the StackTrace string to figure out where it was thrown, but this seems a little fragile and hokey. Is there any consistent, supported way to get any state information (messages, operationdescription, anything) while in the HandleError function? Or any other ways to automatically log exceptions for service calls?
I'm looking for a solution to implement on production, using my existing logging framework, so SvcTraceViewer won't do it for me.
Thanks.
I ended up putting the logging in IErrorHandler.ProvideFault() instead of IErrorHandler.HandlerError(). The ProvideFault call is made in the operation thread, so I can use OperationContext.Current to get some information to log.
I use the IErrorHanlder in the same way that you describe, but not for logging. Instead on service classes (WCF or not) I use an interceptor as described here. I believe that this technique will capture the information you are interested in.
You could stash any context information you need to log in the Exception's Data dictionary in the ProvideFault method which is called on the operation thread...then reference it in the HandleError method for logging purposes.
Have you used the Service Trace Viewer?
The ProvideFault() operations is being called on the incoming call thread and the client is still blocked waiting for response. I don't think it is good idea to add a lengthy process(like logging) inside this method. That is why they exposed another operation HandleError whch gets called on a separate worker thread.
But I understand your situation. Please share if you found a solution other than logging inside ProvideFault.
What about creating an instance and saving the request message on that instance?