I have successfully implemented a connection to ServiceBus with MessageReceiver using RegisterMessageHandler that starts a pump (from this example) and all seems to work just fine.
But in case of exception like e.g. when I turn off network connection the pump throws exceptions continuously to the ExceptionHandler. Every second or even faster. I am wondering if this is supposed default behavior and more importantly if it's possible to change, so that e.g. connection retries can happen every 1 minute. Or am I supposed to do Thread.Sleep or something to achieve that?
receiver.RegisterMessageHandler(
async (message, cancellationToken1) => await HandleMessage(receiver, message),
new MessageHandlerOptions(HandleException)
{
AutoComplete = false,
MaxConcurrentCalls = 1
});
P.S. This is how I solved it now, but not sure if it's a proper way:
private Task HandleException(ExceptionReceivedEventArgs args)
{
_logger.Error(...);
return Task.Delay(60000);
}
P.S Here is the RetryPolicy.Default dump:
Azure Service Bus has a default retry policy (RetryPolicy.Default), but given the transport is trying to receive messages and the broker is not available, will raise exceptions.
ExceptionReceivedContext provides a context, ExceptionReceivedContext which has an action that has failed, and the original exception. You can evaluate the action and decide what needs to be done. You could also check if the exception is transient or not. For transient errors, based on the action, you could just wait for the message to be retried again later (Receive action). In other cases you could either log an error or take a more specific action.
Try to configure the "RetryExponential" on your "SubscriptionClient" like this:
var receiver = new Microsoft.Azure.ServiceBus.SubscriptionClient(_serviceBusConnString, _topic, _subscription, this._receiveMode, new RetryExponential(TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(10), _retryPolicyMaximumRetryCount));
This is the parameters descriptions:
https://learn.microsoft.com/en-us/dotnet/api/microsoft.servicebus.retryexponential?view=azure-dotnet
Here other post about what the properties means:
ServiceBus RetryExponential Property Meanings
Related
I am using the MS Graph API from our web app to create an MS Teams Team in clients' systems and setup a few folders. But I will randomly get errors if I don't impose a hard-coded wait after creating the team. I call the following endpoints in the order shown:
//Create new Team and get basic info
POST teams
GET teams/{team-id}/primaryChannel
GET teams/{team-id}
GET teams/{team-id}/channels/{channel-id}/filesFolder
//Sometimes unknown users must be invited to join org as guest
POST invitations
//Everyone but the owner is added as a guest
POST teams/{team-id}/members
//This is done in a batch, because there is one folder per team guest + one for owner
POST groups/{team-id}/drive/items/{channel-folder-id}/children
//Team members' folders are permitted to them only. So all permissions are deleted and a single user added back
GET groups/{folder-id}/drive/items/{folder-id}/permissions
DELETE groups/{team-id}/drive/items/{folder-id}/permissions/{permission-id}
POST groups/{folder-id}/drive/items/{item-id}/invite
I will sporadically get Forbidden and/or Bad Request responses from:
POST teams/{team-id}/members
DELETE - groups/{team-id}/drive/items/{item-id}/permissions/{permission-id}
Obviously the return statuses of 403 are bugs, because the app definitely has permission to perform the action.
Imposing a 60 second wait after creating the Team seems to resolve this. However, I am currently testing on our Teams environment and am concerned that clients with larger Teams setups will require a longer wait period. I've seen other areas where the documentation says you should wait up to 15 minutes before using a Team that was created from a Group (I am not sure if this applies to creating a normal Team though).
Does anyone know what kind of latency I should be prepared for generally, and if there is any endpoint I can ping to see if the Team is ready for use?
Azure AD, Teams and Exchange are all different systems and need some kind of synchronization that sometimes needs some time.
Whenever you're going to create something in one of these systems, be prepared that it takes some time to access it.
One of the most awkward behaviour I came across is, when you create a group through Exchange Remote Powershell you'll get instantly the group object back. This object has an Azure Object ID. But if you immediately go to Graph and make a request for that group you'll get a 404. Also a look into Azure Portal shows nothing. But if you wait some time (minimum 30 secs, but up to 20!! minutes) the group suddenly appears.
The same also applies if you create a user in Azure through Graph. If you do this, you'll get back an object with the azure id. If you immediately try to add this user to a group or a directory role, it can also happen to get an error, but the timeout here is normally somewhere below 2 sec and I've never seen something above 10 secs.
So for everything, where I'm going to create something in Graph and immediately try to use it, I build some helper method, that tries it multiple times with some smaller timeout between each call:
internal static class Multiple
{
public static Task Try<TException>(int maxRetries, TimeSpan interval, Func<Task> task)
where TException : Exception
{
return Try<TException>(maxRetries, interval, task, exception => true);
}
public static async Task Try<TException>(int maxRetries, TimeSpan interval, Func<Task> task, Func<TException, bool> isExpectedException)
where TException : Exception
{
do
{
try
{
await task().ConfigureAwait(false);
return;
}
catch (Exception ex) when (ex.GetType() == typeof(TException) && isExpectedException((TException)ex))
{
maxRetries--;
if (maxRetries <= 0)
throw;
await Task.Delay(interval);
}
} while (true);
}
}
The usage of the class is as follows:
await Multiple.Try<ServiceException>(20, TimeSpan.FromSeconds(1), async () =>
{
educationClass = await serviceClient.Education.Classes[groupId.ToString()].Request().GetAsync();
}, ex => ex.Error.Code == "Request_ResourceNotFound");
This helper will call the inner method up to 20 times with a timeout of one second. Also the thrown exception must have the given error code. If the number of retries is exceeded or a different error is thrown, the call will rethrow the original exception and must be handled on a higher level.
Simply be aware that behind the Graph interface a highly distributed system works and it sometimes needs some time to get everything in sync.
I test it in my side and met same issues with yours. The 403 error should be a bug as you mentioned because I also have the permission to do the operation. But you mentioned that add guest user to owner, I test it with bad request response, I think it is by design.
Since you can request success after waiting 60 seconds, I think the solution is add a while loop in your code to request the graph api multiple times. In the while loop, if request fail, wait 10 seconds then request again(as Flydog57 mentioned in comments). But you also need to add a mechanism to break loop when request always fail in your code to avoid infinite loops.
I'm using the latest version of SignalR with jQuery and getting some odd behavior where a user disconnects, but still hangs around in my "UserQueue" despite having disconnected.
I think this may be related to the fact that a page refresh appears to trigger the OnDisconnected and OnConnected events almost simultaneously. When I set a break point in one of these methods and step through, my pointer bounces back and forth between the two methods with each step (F10).
I'd like to run a check with each OnConnected event to find out who is actually connected to my app. I want to fire a JS event from my C# code in the OnConnected event, and then allow the client/front end to fire back a confirmation of the user being present:
public override Task OnConnected()
{
// Check for other users:
Clients.All.checkForOtherUsers();
// Do stuff with my user queue
var curmodelId = GetModelId(Context);
var curUserConnection = GetCurrentUser(Context);
// First create the ledger is missing, setting the modelId as the first key
if (_ledger == null)
{
_ledger = new ConnectionLedger(curmodelId, curUserConnection);
}
else
{
// key not found, create new connection pool this model
if (!_ledger.PoolIds.Contains(curmodelId))
{
_ledger.AddConnectionPool(curmodelId, curUserConnection);
}
else
{
var curModelPool = _ledger.ConnectionPools.First(x => x.Id == curmodelId);
curModelPool.UserQueue.Enqueue(curUserConnection);
}
}
return base.OnConnected();
}
Now in the client JS, if I have this code:
modelHub.client.checkForOtherUsers = function () {
// I can see logging here if I add console.log("checking for other users...")
modelHub.server.send();
}
...I'm expecting my C# Send method to receive back the context (if the user is actually present at the client for the JS to execute) and update my UserQueue:
public void Send()
{
var curmodelId = GetModelId(Context);
var curModelPool = _ledger.ConnectionPools.First(x => x.Id == curmodelId);
var curUserConnection = GetCurrentUser(Context);
curModelPool.UserQueue.Enqueue(curUserConnection);
}
... but my Send method never gets fired, even when I fire it from the JS Console directly $.connection.modelingHub.server.send().
Why doesn't the Send method fire? Is there something about the order of events / async nature of the SignalR library that causes this?
Comments on the overall design are welcome too (since I suspect it could be stupid), but note that I cannot access Context.User or Clients.User because of my customized Authentication config.
Seems to be two causes below.
Server-side
Not good to call Clients.All.checkForOtherUsers(); within OnConnected().
I would recommend calling it after connected.
Please refer to SignalR - Send message OnConnected
Client-side
Might need to regist modelHub.client.checkForOtherUsers = function () { before calling start method.
Normally you register event handlers before calling the start method
to establish the connection. If you want to register some event
handlers after establishing the connection, you can do that, but you
must register at least one of your event handler(s) before calling the
start method. - How to establish a connection
For each queue masstransit has consumers, it automatically creates a [queuename]_error queue, and moves messages that could not be processed there (after retrials, etc.)
I´m trying to create a consumer, that takes errors from that queue, and writes it to a database.
In order to consume those messages, I had to create a handler/consumer for the error queue, receiving the original message.
cfg.ReceiveEndpoint(host, "myqueuename", e =>
{
e.Handler<MyMessage>(ctx =>
{
throw new Exception ("Not expected");
});
});
cfg.ReceiveEndpoint(host, "myqueuename_error", e =>
{
e.BindMessageExchanges = false;
e.Handler<MyMessage>(ctx =>
{
Console.WriteLine("Handled");
// do whatever
return ctx.CompleteTask;
});
});
All that works fine, the problem to retrieve the actual exception that occurred.
I was actually able to do that, with some serious hack....
e.Handler<MyMessage>(m =>
{
var buffer = m.ReceiveContext.TransportHeaders
.GetAll().Single(s => s.Key == "MT-Fault-Message").Value as byte[];
var errorText = new StreamReader(new MemoryStream(buffer)).ReadToEnd();
Console.WriteLine($"Handled, Error={errorText}");
return m.CompleteTask;
});
That just fells wrong though.
PS: I Know i could subscribe to a Fault event, but in this particular case, it is a RequestClient (request-response) pattern, and MT redirects FaultAddress back to the client, and I can´t garantee it is still running.
Request/reply should only be used for getting the data. It means that if the requestor goes down - there are no more reasons to reply with data or with fault and you do not have interest in consuming faults.
So, the reason for the request client to use a temporary (non-durable) queue instead of the receive endpoint queue is by design. It encourages you not to understand that the scope of your replies is only within the request waiting time.
If you send commands and need to be informed if the command has been processed - you should publish events to inform about the outcome of the command processing. Using message metadata (initiator id and conversation id) allows you to find out, how events correlate with commands.
So, only use request/reply for requesting information (queries) using decoupled invocation SOA pattern, where the reply only have a meaning in correlation with request and if the requestor goes down, the reply is no longer needed, no matter if it was a success of failure.
I'm trying to send email from a Controller asynchronous and getting the following error:
I don't want to wait email be sent to complete the action.
An asynchronous module or handler completed while an asynchronous
operation was still pending.
This is my implementation: I also tried using void instead async Task
public async Task<ActionResult> Login(LoginViewModel model, string returnUrl)
{
var user = await _userManager.FindByNameAsync(model.Email);
if (user != null)
{
if (!await _userManager.IsEmailConfirmedAsync(user.Id))
{
//dont want to await result
//just send email
_mailService.SendAccountConfirmationEmailAsync(user);
...
}
...
}
return View();
}
.
public async Task SendAccountConfirmationEmailAsync(ApplicationUser user)
{
dynamic email = GetAccountConfirmationEmailTemplate();
email.Username = user.Email;
email.Name = user.UserName;
email.To = user.Email;
email.AccountConfirmationUrl = await GetAccountConfirmationUrl(user);
await _mailService.SendAsync(email);
}
What am I missing?
Since the runtime can recycle your appdomain when it knows there are no more pending requests, starting tasks that you don't wait for is specifically not recommended, hence that message.
In particular, without this check you would have no guarantee this email would ever be sent since the entire appdomain could be yanked down around that task after your controller action returns.
Now in your case you might say that the loss of the email is acceptable, but the runtime doesn't know that it is only an email, it could very well be your database call that you've forgotten to wait for that would be prematurely aborted. Hence it just informs you that this is bad form.
Instead you got two options:
Wait for that task to complete, which you have specifically said you don't want to do
Inform the runtime that you're firing off a task that you specifically don't want to wait for, but please oh please, don't yank out the appdomain before task has completed
This last part is what HostingEnvironment.QueueBackgroundWorkItem is for.
In your case you would call it like this:
HostingEnvironment.QueueBackgroundWorkItem(ct =>
_mailService.SendAccountConfirmationEmailAsync(user));
Additionally, the method provides a CancellationToken, you should check if there is an overload of SendAccountConfirmationEmailAsync that accepts it. If not, and if this is your API, you should consider adding support for such tokens.
Please be aware that this mechanism is not meant for long-running threads or tasks, for that there are other more appropriate trees to bark up, but in this case it should be enough.
You need to await your async call in order to avoid the error message, but then you'll need to wait for the email to finish sending before your action method returns (and a failure sending the email will cause your action method to error out).
await _mailService.SendAccountConfirmationEmailAsync(user);
If you ever see a method name that ends in -Async, check if that method returns a Task or Task. If it does, you need to await it.
If you have an async call that you don't want to wait for it to complete in this context, you'll get that error. If you want to continue processing, you'll need some method of sending this email in a background task. That might be that you have a separate application that handles emails, and you communicate with it via a messaging system like RabbitMQ. Or there are ways to run code in the background directly in a website such as Hangfire.
I've been working on an application that starts some worker roles based on messaging.
This is the way I want the application to work:
Client sends a request for work (RPC).
One of the worker roles accepts the work, generates a random id, and responds to the RPC with the new id.
The worker will post its debug logs on a log channel with the id.
The client will subscribe to this channel so users can see what's going on.
The RPC is working fine, but I can't seem to figure out how to implement the log-sending.
This is the code that accepts work (simplified)
var bus = Bus.Factory.CreateUsingRabbitMq(sbc =>
{
var host = sbc.Host(new Uri("rabbitmq://xxxxxx.nl"), h =>
{
h.Username("xxx");
h.Password("xxxx");
});
sbc.ReceiveEndpoint(host, "post_work_item", e =>
{
e.Consumer<CreateWorkItemCommand>();
});
sbc.ReceiveEndpoint(host, "list_work_items", e =>
{
e.Consumer<ListWorkItemsCommand>();
});
});
The CreateWorkItemCommand will create the thread, do the work, etc. Now, how would I implement the log-sending with Masstransit? I was thinking something like:
bus.Publish(
obj: WorkUpdate{ Message = "Hello world!" },
channel: $"work/{work_id}"
)
And the client will do something this:
bus.ReceiveFromEvented($"work/{rpc.work_id}").OnMessage += { more_psuedo_code() }
I can't seem to find out how to do this.
Can anyone help me out?
Thanks!
It looks both like a saga and turnout. Current Turnout implementation is monitoring the job itself and I doubt you can really subscribe to that message flow. And it is still not really done.
You might solve this using the saga. Some external trigger (a command) will start the first saga, which will use Request/Response to start the process, which will do the work, and get its correlation id (job id). The long job can publish progress reports using the same correlation id and the saga will consume them, doing what it needs to do.
The "work/{rpc.work_id}" will be then replaced by the correlation.