We Keep Coding

How I can poll messages from Solace queue (instead of default pushing behavior)?

I'd like to write parallel execution module based on Solace. And I use request-reply schema for this.
I have:
Multiple message consumers, which publish messages into the same queue.
Multiple message producers, which read queue and create reply messages.
Message execution time is between 10 seconds to 10 minutes.
Queue access type is non-exclusive (e.g. it does round-robin between all consumers).
Each producer and consumer is asynchronous, e.g. Solace API blocks execution during the connection only.
What I'd like to have: if produces works on the message, it should not receive any other messages. This is extremely important, because some tasks blocks executor for several minutes, however other executors can be free after couple of seconds.
Scheme below can be workable (possible), however blocking code appears below. I'd like to avoid it.
while(true)
{
var inputMessage = flow.ReceiveMsg( /*timeout 1s*/1_000); // <--- blocking code, I'd like to avoid it
flow.Ack(inputMessage.ADMessageId);
var reply = await ProcessMessageAsync(inputMessage); // execute plus handle exceptions
session.SendReply(inputMessage, reply)
}

Messages are only pushed to the consuming applications.
That being said, your desired behavior can be obtained by setting the "max-delivered-unacked-msgs-per-flow" on your queue to 1.
This means that each consumer bound to the queue is only allowed to have 1 outstanding unacknowledged messages.
The next message will be only sent to the consumer after it has acknowledged the message.
Details about this feature can be found here.
Do note that your code snippet does not appear to be valid.
IFlow.ReceiveMsg is only used in transacted sessions, which makes use of ITransactedSession.Commit to acknowledge messages.

How can I wait for an Azure CloudQueue to be deleted?

I am running some tests that use Azure CloudQueue, and as setup/teardown I am calling CreateIfNotExistsAsync() and DeleteIfExistsAsync(). However when I am running my tests back to back I got a Microsoft.WindowsAzure.Storage.StorageException,"The remote server returned an error: (409) Conflict."
await cloudQueue.CreateIfNotExistsAsync();
// do work 1
await cloudQueue.DeleteIfExistsAsync();
await cloudQueue.CreateIfNotExistsAsync(); // throws exception
// do work 2
After taking a closer look at the server's response, I found the StatusDescription says "The specified queue is being deleted."
Is there a method that I can call so that once it returns, I know for sure the queue is already deleted?
=========================================================================
UPDATE Now that I think of it. If Azure Queue server wants to reply with deletion result, it will have to keep track of unfinished incoming request, which is obviously bad desgin (vulnerable to DOS attack)...

Is there a method that I can call so that once it returns, I know for
sure the queue is already deleted?
Unfortunately no. Deleting a queue (or blob container/table/file share) is an asynchronous operation. When you send a request to delete a queue, Azure Storage marks that queue for deletion (so that no operations can be performed on it) and then actually deletes the queue through a background process. Based on the documentation, it can take up to 30 seconds to delete a queue. However it may be more depending on how much data is held in there.
From the documentation:
When a queue is successfully deleted, the queue is immediately marked
for deletion and is no longer accessible to clients. The queue is
later removed from the Queue service during garbage collection.
Possible Workaround:
Since there's no method that you can call which will tell you for sure that a queue is already deleted, what you would need to do is try to create the queue using CreateIfNotExistsAsync and catch any error. If the HTTP status code is Conflict (409) and error code is QueueBeingDeleted, you should wait for some time and retry the operation. If you want, you can put incremental delay between retries.

Microsoft ServiceBus Receiving BrokeredMessages Out of Order

I have a javascript logging utility that sends requests in bulk to my server which then relays them to a Queue Client (Microsoft.ServiceBus.Messaging.QueueClient). I want to send them in batch asynchronously to the ServiceBus and still have them processed in the order they are placed into the batch I am sending. The documentation for SendBatchAsync shows that the method is for "batch" processing. This makes me think I can send it a batch of requests and have them processed as a single unit (i.e.: sequentially). Although, it appears that the messages are getting processed out of order. I'm using OnMessage to receive the messages; I'm not sure if this is a limitation or what am I missing?
I get that async doesn't guarantee order vs. other async requests, but this is a single request. I don't want to have to wait for a response before responding to the javascript client as I'm just trying to send them off, but I still need to ensure they stay in order since they are sequential events.
Here is how I send them to the queue:
MyQueueClient.SendBatchAsync(MyListOfBrokerMessages);
Then I process them:
ServiceBus.TrackerClient.OnMessage((m) =>
{
try
{
ProcessMessage(m);
}
I don't get the point of the batch processing if it doesn't process as a batch other than maybe making a single request. There must be some way to send a batch and have it process in order??
EDIT:
I've tried using Send instead of SendBatchAsync and I've set MaxConcurrentCalls to 1 and yet the messages are still not in order.

Taken from MSDN:
SessionId: If a message has the
Microsoft.ServiceBus.Messaging.BrokeredMessage.SessionId property set, then Service Bus
uses the SessionId property as the partition key. This way, all messages that belong to
the same session are handled by the same message broker. This enables Service Bus to
guarantee message ordering as well as the consistency of session states.
For a coding sample employing SessionId and AcceptSessionReceiver see.

What you can do is to use Sessions here,
Set the same sessions id to all the messages in the batch
Receiving side, AcceptMessageSession() will give you a session
Call receive on the session (ReceiveBatch). This session will give you all the messages in that batch alone.

Is it possible to implement Async Webservice call in .Net not only by one connection?

I have a .net Webservice. It has a method that does a task that takes 60 seconds and after that, returns the string result to client.
In some networks especially low band widths I get timeout error in 40 seconds before webservice method do its task in 60 second.
Now I want to implement webservice Async call to support low band width networks.
In Async webservice call an approach is using a thread that runs webservice method and returns webservice result to main thread that is shown in the following picture.
But my problem will not be solved in this approach because that thread uses one connection.
I need another approach. A thread in my client call webservice method and method starts its operation and when the task is done, 1) webservice sends a message that your response is ready or, 2) client checks if the webservice response is ready (I think polling mechanism) like the following picture.
How can I implement the second approach in .net? Is it possible?
Thanks.

Create a table on your database to store the state of the process.
UniqClientId, ProcessId, StartTime, EndTime and any other state if required.
Client sends a request to the server by passing its unique id.
Server logs the process on the above table and initiates the process.
Clients contacts the server instantly(2-3 sec or 15-20 sec depending on your application) to check the process completion.
If the client get a response that process has been completed, then it requests the server to send the response.
In between, the server does the following job.
When the process completes, stores the EndTime on above table.
Provides a method to send the process state by checking the above table.
Provides a method to send the response.
I'm not sure what exactly your service is doing, but if the operation of your process is just to modify some table on the database, then is is not difficult to implement this.

WCF Windows Service - Long operations/Callback to calling module

I have a Windows Service that takes the name of a bunch of files and do operations on them (zip/unzip, updating db etc). The operations can take time depending on size and number of files sent to the service.
(1) The module that is sending a request to this service waits until the files are processed. I want to know if there is a way to provide a callback in the service that will notify the calling module when it is finished processing the files. Please note that multiple modules can call the service at a time to process files so the service will need to provide some kind of a TaskId I guess.
(2) If a service method is called and is running and another call is made to the same service, then how will that call be processed(I think there is only one thread asociated with the service). I have seen that when the service is taking time in processing a method, the threads associated with the service begin to increase.

WCF does indeed offer duplex bindings which allow you to specify a callback contract, so that the service can call back to the calling client to notify.
However, in my opinion, this mechanism is rather flaky and not really to be recommended.
In such a case, when the call causes a fairly long running operation to happen, I would do something like this:
If you want to stick to HTTP/NetTcp bindings, I would:
drop off the request with the service, and then "let go" - this would be a one-way call, you just drop off what you want to have done, and then your client is done
have a status call that the client could call after a given time to find out whether or not the results of the request are ready by now
if they are, there should be a third service call to retrieve the results
So in your case, you could drop off the request to zip some files. The service would go off and do its work and store the resulting ZIP in a temporary location. Then later on the client could check to see whether the ZIP is ready, and if so, retrieve it.
This works even better over a message queue (MSMQ) which is present in every Windows server machine (but not a lot of people seem to know about it or use it):
your client drops off the request on a request queue
the service listens on that request queue and fetches request after request and does it works
the service can then post the results to a result queue, on which your callers in turn are listening
Check out how to do all of this efficiently by reading the excellent MSDN article Foudnations: Build a queue WCF Response Service - highly recommended!
A message-queue based systems tends to be much more stable and less error-prone that a duplex-/callback-contract based system, in my opinion.

(1) The simplest way to achieve this is with a taskId as you note, and then have another method called IsTaskComplete with which client can check whether the task has been completed.
(2) Additional calls made to the service will start new threads.
edit: the default service behaviour is to start new threads per call. The configurable property is Instance Context Mode, and can be set to PerCall, PerSession, or Shareable.

The question has a solution, but I'm using a WCF duplex service to get the result of a long operation, and even though I found a problem that has cost me several hours to solve (and that's why I searched this question earlier), now it works perfectly, and I believe it is a simple solution within the WCF duplex service framework.
What is the problem with a long operation? The main problem is blocking the client interface while the server performs the operation, and with the WCF duplex service we can use a call back to the client to avoid the blockage (It is an old method to avoid blocking but it can easily be transformed into the async/await framework using a TaskCompletionSource).
In short, the solution uses a method to start the operation asynchronously on the server and returns immediately. When the results are ready, the server returns them by means of the client call back.
First, you have to follow any standard guide to create WCF duplex services and clients, and I found these two useful:
msdn duplex service
Codeproject Article WCF Duplex Service
Then follow these steps adding your own code:
Define the call back interface with an event manager method to send results from the server and receive them in the client.
public interface ILongOperationCallBack
{
[OperationContract(IsOneWay = true)]
void OnResultsSend(....);
}
Define the Service Interface with a method to pass the parameters needed by the long operation (refer the previous ILongOperationCallBack interface in the CallBackContractAttribute)
[ServiceContract(CallbackContract=typeof(ILongOperationCallBack))]
public interface ILongOperationService
{
[OperationContract]
bool StartLongOperation(...);
}
In the Service class that implements the Service Interface, first get the proxy of the client call back and save it in a class field, then start the long operation work asynchronously and return the bool value immediately. When the long operation work is finished send the results to the client using the client call back proxy field.
public class LongOperationService:ILongOperationService
{
ILongOperationCallBack clientCallBackProxy;
public ILongOperationCallBack ClientCallBackProxy
{
get
{
return OperationContext.Current.GetCallbackChannel<ITrialServiceCallBack>());
}
}
public bool StartLongOperation(....)
{
if(!server.IsBusy)
{
//set server busy state
//**Important get the client call back proxy here and save it in a class field.**
this.clientCallBackProxy=ClientCallBackProxy;
//start long operation in any asynchronous way
......LongOperationWorkAsync(....)
return true; //return inmediately
}
else return false;
}
private void LongOperationWorkAsync(.....)
{
.... do work...
//send results when finished using the cached client call back proxy
this.clientCallBackProxy.SendResults(....);
//clear server busy state
}
....
}
In the client create a class that implements ILongOperationCallBack to receive results and add a method to start the long operation in the server (the start method and the event manager don't need to be in the same class)
public class LongOperationManager: ILongOperationCallBack
{
public busy StartLongOperation(ILongOperationService server, ....)
{
//here you can make the method async using a TaskCompletionSource
if(server.StartLongOperation(...)) Console.WriteLine("long oper started");
else Console.Writeline("Long Operation Server is busy")
}
public void OnResultsSend(.....)
{
... use long operation results..
//Complete the TaskCompletionSource if you used one
}
}
NOTES:
I use the bool return in the StartLongOperation method to indicate that the server is Busy as opposed to down, but it is only necessary when the long operation can't be concurrent as in my actual application, and maybe there are best ways in WCF to achieve non concurrency (to discover if the server is down, add a Try/Catch block as usual).
The important quote that I didn't see documented is the need to cache the call back client proxy in the StartLongOperation method. My problem was that I was trying to get the the proxy in the working method (yes, all the examples use the call back client proxy in the service method, but it isn't explicity stated in the documentation, and in the case of a long operation we must delay the call back until the operation ends).
Do not get and cache twice the call back Proxy after a service method has returned and before the next one.
Disclaimer: I haven't added code to control errors, etc.