We Keep Coding

How I can poll messages from Solace queue (instead of default pushing behavior)?

I'd like to write parallel execution module based on Solace. And I use request-reply schema for this.
I have:
Multiple message consumers, which publish messages into the same queue.
Multiple message producers, which read queue and create reply messages.
Message execution time is between 10 seconds to 10 minutes.
Queue access type is non-exclusive (e.g. it does round-robin between all consumers).
Each producer and consumer is asynchronous, e.g. Solace API blocks execution during the connection only.
What I'd like to have: if produces works on the message, it should not receive any other messages. This is extremely important, because some tasks blocks executor for several minutes, however other executors can be free after couple of seconds.
Scheme below can be workable (possible), however blocking code appears below. I'd like to avoid it.
while(true)
{
var inputMessage = flow.ReceiveMsg( /*timeout 1s*/1_000); // <--- blocking code, I'd like to avoid it
flow.Ack(inputMessage.ADMessageId);
var reply = await ProcessMessageAsync(inputMessage); // execute plus handle exceptions
session.SendReply(inputMessage, reply)
}

Messages are only pushed to the consuming applications.
That being said, your desired behavior can be obtained by setting the "max-delivered-unacked-msgs-per-flow" on your queue to 1.
This means that each consumer bound to the queue is only allowed to have 1 outstanding unacknowledged messages.
The next message will be only sent to the consumer after it has acknowledged the message.
Details about this feature can be found here.
Do note that your code snippet does not appear to be valid.
IFlow.ReceiveMsg is only used in transacted sessions, which makes use of ITransactedSession.Commit to acknowledge messages.

Accessing Service Fabric service state during cancellation

I have a Service Fabric cluster hosting an 'Orchestrator'-type service which spins up and shuts down other Stateful services to do work, using FabricClient.ServiceManagementClient's CreateServiceAsync and DeleteServiceAsync methods.
The work involves processing messages which are stored for a short time within a ReliableConcurrentQueue.
I'm trying to handle the graceful shutdown of these services via the CancellationToken by ensuring that the queue is completely drained of messages before the service is deleted, but have found that the service's access to the ReliableConcurrentQueue is revoked once the CancellationToken is cancelled.
For example, calling StateManager.GetOrAddAsync<T>() from a callback registered with the CancellationToken, results in a FabricNotReadableException, containing the message "Primary state manager is currently not readable".
Reading around, it seems this is expected behaviour:
"In Service Fabric, when a Primary is demoted, one of the first things
that happens is that write access to the underlying state is revoked."
https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-reliable-services-lifecycle
Also, the answers to this question suggest that FabricNotReadableException is often a transient issue, and affected calls can be retried. This doesn't seem to be the case in this example; multiple retries at various frequencies/delays all seem to fail the same way.
Is there a way to guarantee that everything in the queue is processed using the combination of Stateful services, Reliable Collections and CancellationTokens? Or should I be looking into storage outside of what Service Fabric can provide?

Consider performing the queue item processing inside RunAsync.
Stopping / changing the role of a service causes the CancellationToken passed to RunAsync to be cancelled.
Once that happens, you need to make sure that you only exit that method when the queue depth is 0.
Also, once this cancellation is requested, you should probably stop allowing new items to be enqueued.

Real time data storage with Azure Tables using ASP.NET

I am using a Lab View application to simulate a test running, which would post a JSON string to my ASP.NET application. Within the ASP.NET application I format the data with the proper partition and row keys, then send it to Azure Table Storage.
The problem that I am having is that after what seems like a random amount of time (i.e. 5 minutes, 2 hours, 5 hours), the data fails to be saved into Azure. I am try to catch any exceptions within the ASP.NET application and send the error message back to the Lab View app and the Lab View app is also catching any exceptions in may encounter so I can trouble shoot where the issue is occurring.
The only error that I am able to catch is a Timeout Error 56 in the Lab View program. My question is, does anyone have an idea of where I should be looking for the root cause of this? I do not know where to begin.
EDIT:
I am using a table storage writer that I found here to do batch operations with retries.

The constructor for exponential retry policy is below:
public ExponentialRetry(TimeSpan deltaBackoff, int maxAttempts)
when you (or the library you use to be exact) instantiate this as RetryPolicy = new ExponentialRetry(TimeSpan.FromMilliseconds(2),100) you are basically setting the max attempts as 100 which means you may end up waiting up to around 2^100 milliseconds (there is some more math behind this but just simplifying) for each of your individual batch requests to fail on the client side until the sdk gives up retrying.
The other issue with that code is it executes batch requests sequentially and synchronously, that has multiple bad effects, first, all subsequent batch requests are blocked by the current batch request, second your cores are blocked waiting on I/O operations, third it has no exception handling so if one of the batch operations throw an exception, the method bails out and would not continue any further processing other batch requests.
My recommendation, do not use that library, batch operations are fairly straight forward. The default retry policy if you do not explicitly define is the exponential retry policy anyways with sensible default parameters (does 3 retries) so you do not even need to define your own retry object. For best scalability and throughput run your batch operations async (and concurrently).
As to why things fail, when you write your own api, catch the StorageException and check the http status code on the exception itself. You could be getting throttled by azure as one of the possibilities but it is hard to say without further debugging or you providing the http status code for the failed batch operations to us.

You need to check whether an exception is transient or not. As Peter said on his comment, Azure Storage client already implements a retry policy. You can also wrap your code with another retry code (e.g using polly) or you should change the default policy associated to Azure Storage Client.

In what cases can the result of SubscriptionClient.GetMessageAsync be null?

I'm using the Azure service bus broker with the WindowsAzure.ServiceBus Nuget package version 3.4.4
When I call ReceiveAsyncon a SubscriptionClient that is connected to a subscription on which no messages are published, the returned task completes after a minute or so with a null result.
I expected the task never to complete with null
Under what conditions can the Task returned by this method complete with null?

You can pass a TimeSpan to ReceiveAsync to specify how long you would like to wait for a message. By default it probably is a minute.
So you could do something like this if you expect to receive at least one message every two weeks:
client.ReceiveAsync(TimeSpan.FromDays(14));

The API that you are using assumes that you want to poll Service Bus for messages; meaning that you will have to regularly make calls to the Service Bus endpoint in order to accept messages. When there are no messages, you will receive a null response. I would not recommend calling client.ReceiveAsync(TimeSpan.FromDays(14), because that will leave a single request open for that amount of time. Instead, a null result is the expected behavior, and you should call ReceiveAsync in a continuous loop.
However, there is also an API that will do this loop for you: OnMessage
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicebus.messaging.subscriptionclient?redirectedfrom=MSDN#Microsoft_ServiceBus_Messaging_SubscriptionClient_OnMessageAsync_System_Func_Microsoft_ServiceBus_Messaging_BrokeredMessage_System_Threading_Tasks_Task__
Just make sure that the main thread of your program does not exit.

Microsoft ServiceBus Receiving BrokeredMessages Out of Order

I have a javascript logging utility that sends requests in bulk to my server which then relays them to a Queue Client (Microsoft.ServiceBus.Messaging.QueueClient). I want to send them in batch asynchronously to the ServiceBus and still have them processed in the order they are placed into the batch I am sending. The documentation for SendBatchAsync shows that the method is for "batch" processing. This makes me think I can send it a batch of requests and have them processed as a single unit (i.e.: sequentially). Although, it appears that the messages are getting processed out of order. I'm using OnMessage to receive the messages; I'm not sure if this is a limitation or what am I missing?
I get that async doesn't guarantee order vs. other async requests, but this is a single request. I don't want to have to wait for a response before responding to the javascript client as I'm just trying to send them off, but I still need to ensure they stay in order since they are sequential events.
Here is how I send them to the queue:
MyQueueClient.SendBatchAsync(MyListOfBrokerMessages);
Then I process them:
ServiceBus.TrackerClient.OnMessage((m) =>
{
try
{
ProcessMessage(m);
}
I don't get the point of the batch processing if it doesn't process as a batch other than maybe making a single request. There must be some way to send a batch and have it process in order??
EDIT:
I've tried using Send instead of SendBatchAsync and I've set MaxConcurrentCalls to 1 and yet the messages are still not in order.

Taken from MSDN:
SessionId: If a message has the
Microsoft.ServiceBus.Messaging.BrokeredMessage.SessionId property set, then Service Bus
uses the SessionId property as the partition key. This way, all messages that belong to
the same session are handled by the same message broker. This enables Service Bus to
guarantee message ordering as well as the consistency of session states.
For a coding sample employing SessionId and AcceptSessionReceiver see.

What you can do is to use Sessions here,
Set the same sessions id to all the messages in the batch
Receiving side, AcceptMessageSession() will give you a session
Call receive on the session (ReceiveBatch). This session will give you all the messages in that batch alone.