We Keep Coding

Logging Hangfire jobs to Application Insights and correlating activity to an Operation Id

I feel like this should be a lot simpler than it's turning out to be, or I am just over thinking it too much.
I have a .NET Core 3.1 Web API application, which is using HangFire to process some jobs in the background. I have also configured Application Insights to log Telemetry from the .NET Core API.
I can see logging events and dependency telemetry data logged in Application Insights. However, each event/log/dependency is recorded against a unique OperationId and Parent Id.
I am trying to determine how to ensure that any activity which is logged, or any dependencies which are used in the context of the background job are logged against the OperationId and/or Parent Id of the original request which queued the background job.
When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job. When the job is then performed, I can get back that OperationId. What I then need to do is make that OperationID available throughout the context/lifetime of the job execution, so that it is attached to any Telemetry sent to Application Insightd.
I thought I could create a IJobContext interface, which could be injected into the class which performs the job. Within that context I could push the OperationID. I could then create a ITelemetryInitializer which would also take the IJobContext as a dependency. In the ITelemetryInitializer I could then set the OperationID and ParentId of the telemetry being sent to Application Insights. Here's some simple code:
public class HangFirePanelMessageQueue : IMessageQueue
{
private readonly MessageProcessor _messageProcessor;
private readonly IHangFireJobContext _jobContext;
private readonly TelemetryClient _telemetryClient;
public HangFirePanelMessageQueue(MessageProcessor panelMessageProcessor,
IIoTMessageSerializer iotHubMessageSerialiser,
IHangFireJobContext jobContext, TelemetryClient telemetryClient)
{
_messageProcessor = panelMessageProcessor;
_jobContext = jobContext;
_telemetryClient = telemetryClient;
}
public async Task ProcessQueuedMessage(string message, string operationId)
{
var iotMessage = _iotHubMessageSerialiser.GetMessage(message);
_jobContext?.Set(iotMessage.CorrelationID, iotMessage.MessageID);
await _messageProcessor.ProcessMessage(iotMessage);
}
public Task QueueMessageForProcessing(string message)
{
var dummyTrace = new TraceTelemetry("Queuing message for processing", SeverityLevel.Information);
_telemetryClient.TrackTrace(dummyTrace);
string opId = dummyTrace.Context.Operation.Id;
BackgroundJob.Enqueue(() =>
ProcessQueuedMessage(message, opId));
return Task.CompletedTask;
}
}
The IJobContext would look something like this:
public interface IHangFireJobContext
{
bool Initialised { get; }
string OperationId { get; }
string JobId { get; }
void Set(string operationId, string jobId);
}
And then I would have an ITelemetryInitializer which enriches any ITelemetry:
public class EnrichBackgroundJobTelemetry : ITelemetryInitializer
{
private readonly IHangFireJobContext jobContext;
public EnrichBackgroundJobTelemetry(IHangFireJobContext jobContext)
{
this.jobContext = jobContext;
}
public void Initialize(ITelemetry telemetry)
{
if (!jobContext.Initialised)
{
return;
}
telemetry.Context.Operation.Id = jobContext.OperationId;
}
}
The problem I have however is that the ITelemetryInitializer is a singleton, and so it would be instantiated once with a IHangFireJobContext which would then never update for any subsequent HangFire job.
I did find the https://github.com/skwasjer/Hangfire.Correlate project, which extends https://github.com/skwasjer/Correlate. Correlate creates a correlation context which can be accessed via a ICorrelationContextAccessor which is similar to the IHttpContextAccessor.
However, the footnotes for Correlate state "Please consider that .NET Core 3 now has built-in support for W3C TraceContext (blog) and that there are other distributed tracing libraries with more functionality than Correlate." which lists Application Insights as one of the alternatives for more Advanced distributed tracing.
So can anyone help me understand how I can enrich any Telemetry going to Application Insights when it is created within the context of a HangFire job? I feel the correct answer is to use an ITelemetryInitializer and populate the OperationId on that ITelemetry item, however, I am not sure what dependancy to inject into the ITelemetryInitialzer in order to get access to the HangFire Job Context.

When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job.
So, am I correct to say that you have a controller action that pushes work to hangfire? If so What you can do is inside the controller method get the operation id and pass it to the job. Use that operation id to start a new operation using the operation Id. That operation, together with all the telemetry generated during that operation, will be linked to the original request.
I have no hangfire integration but the code below shows the general idea: some work is queued to be done in the background and should be linked to the request regarding the telemetry:
[HttpGet("/api/demo5")]
public ActionResult TrackWorker()
{
var requestTelemetry = HttpContext.Features.Get<RequestTelemetry>();
_taskQueue.QueueBackgroundWorkItem(async ct =>
{
using(var op = _telemetryClient.StartOperation<DependencyTelemetry>("QueuedWork", requestTelemetry.Context.Operation.Id))
{
_ = await new HttpClient().GetStringAsync("http://blank.org");
await Task.Delay(250);
op.Telemetry.ResultCode = "200";
op.Telemetry.Success = true;
}
});
return Accepted();
}
The full example can be found here.

Working from Peter Bons' example I did it like this:
Code originally triggered from a controller action:
// Get the current ApplicationInsights Id. Could use .RootId if
// you only want the OperationId, but I want the ParentId too
var activityId = System.Diagnostics.Activity.Current?.Id;
_backgroundJobClient.Enqueue<JobDefinition>(x =>
x.MyMethod(queueName, otherMethodParams, activityId));
In my JobDefinition class:
// I use different queues, but you don't need to.
// otherMethodParams is just an example. Have as many as you need, like normal.
[AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete, Attempts = 10)]
[QueueNameFromFirstParameter]
public async Task MyMethod(string queueName, string otherMethodParams,
string activityId)
{
var (operationId, parentId) = SplitCorrelationIdIntoOperationIdAndParentId(
activityId);
// Starting this new operation will initialise
// System.Diagnostics.Activity.Current.
using (var operation = _telemetryClient.StartOperation<DependencyTelemetry>(
"JobDefinition.MyMethod", operationId, parentId))
{
try
{
operation.Telemetry.Data = $"something useful here";
// If you have other state you'd like in App Insights logs,
// call AddBaggage and they show up as a customDimension,
// e.g. in any trace logs.
System.Diagnostics.Activity.Current.AddBaggage("QueueName", queueName);
// ... do the real background work here...
operation.Telemetry.Success = true;
}
catch (Exception)
{
operation.Telemetry.Success = false;
throw;
}
}
}
// Splits full value from System.Diagnostics.Current.Activity.Id
// like "00-12994526f1cb134bbddd0f256e8bc3f0-872b3bd78c345a46-00"
// into values ( "12994526f1cb134bbddd0f256e8bc3f0", "872b3bd78c345a46" )
private static (string, string) SplitCorrelationIdIntoOperationIdAndParentId(string activityId)
{
if (string.IsNullOrEmpty(activityId))
return (null, null);
var splits = activityId.Split('-');
// This is what should happen
if (splits.Length >= 3)
return (splits[1], splits[2]);
// Must be in a weird format. Try to return something useful.
if (splits.Length == 2)
return (splits[0], splits[1]);
return (activityId, null);
}
I'm not sure using the OperationId and ParentId is quite right here, e.g. it does tie the background job to the originating request's OperationId, but if the originating Request has a ParentId then this background job should really have its ParentId set as the Request, not as the Request's ParentId. Anyone know?

How can I check how many connections are in a Group in my Azure SignalR service?

I am trying to implement Azure SignalR Service to facilitate two way messaging between a desktop, asp.net-core and a xamarin.ios apps.
I have created a Hub as per Microsoft's documentation here:
https://learn.microsoft.com/en-us/aspnet/core/signalr/hubs?view=aspnetcore-2.2
Hub:
public class ChatHub : Hub
{
public Task SendMessage(string user, string message)
{
return Clients.All.SendAsync("ReceiveMessage", user, message);
}
}
I add users's connections to a group when they connect to the hub as shown here: https://learn.microsoft.com/en-us/aspnet/core/signalr/groups?view=aspnetcore-2.2
Add to Group:
public async Task AddToGroup(string groupName)
{
await Groups.AddToGroupAsync(Context.ConnectionId, groupName);
await Clients.Group(groupName).SendAsync("Send", $"{Context.ConnectionId} has joined the group {groupName}.");
}
public async Task RemoveFromGroup(string groupName)
{
await Groups.RemoveFromGroupAsync(Context.ConnectionId, groupName);
await Clients.Group(groupName).SendAsync("Send", $"{Context.ConnectionId} has left the group {groupName}.");
}
When a message is sent it has a group name as a parameter, I would like to check if the provided group has any connections before sending the message to any registered clients, if there are no connections, I would like to send a push notification (already have working code for the push notification)
Send Message with Notification Fallback:
public class ChatHub : Hub
{
public Task SendMessage(string groupName, string user, string message)
{
var group = Clients.Group(groupName);
// todo: how to check if we have any open connections in this group?
if(group.Conections.Count > 0)
{
return group.SendAsync("ReceiveMessage", user, message);
}
else
{
// todo: run code to send push notification or anything else you might want to do
}
}
}
Problem: I don't see any way to check for the number of connections currently in a group via the available api (correct me if I'm wrong)
I see the group coming back as Microsoft.AspNetCore.SignalR.Internal.GroupProxy<ChatHub> at runtime with no public methods. Internal private variables do include _groupName and _lifeTimeManager and within lifetime manager there is a _clientConnectionManager which i can see has client connections when they are connected but I have no access to any of these, I'm using the Microsoft.Azure.SignalR (1.0.4) Nuget package. Does anyone know if what I'm trying to do possible with this SDK and if so how I can do this?

SignalR Client Authentication with Host

I came here reaching for your help. I am fairly new when it comes to SignalR and after several days of reading and retrying stuff again and again I have reached a dead end. Straight to the point.
I have a solution which includes several projects in it. The ones that are giving me trouble are the Web and a class library.
I have SignalR installed in the Web project and the SignalR Client in the Class Library.
My target is to track and monitor the progress of Long running processes that are running in the Class Library. I have SignalR working properly from the Web.
This is my Hub method that I am trying to call:
public void SendProgress(string progressMessage, int progressCount, int totalItems)
{
var percentage = (progressCount * 100) / totalItems;
Clients.All.sendProgress(progressMessage, percentage + "%");
}
Moreover I have a method in the Class Library (ImportProcess) which contains a foreach loop. Inside that foreach loop I am calling:
public void SendProgress(string message, int currentItem, int totalItems)
The body of the SendProgress is this:
public void SendProgress(string message, int currentItem, int totalItems)
{
var hubConnection = new HubConnection("http://localhost:13310/");
IHubProxy statementsHubProxy = hubConnection.CreateHubProxy("statementsHub");
hubConnection.Credentials = CredentialCache.DefaultNetworkCredentials;
hubConnection.Start().Wait();
statementsHubProxy.Invoke("SendProgress", message, currentItem, totalItems);
}
The problem is that since I am calling the SendProgress inside a foreach, I consider it wrong to define the connection everytime that SendProgress is called.
I tried including the Connection properties to the constructor of the class. It worked but it only returns the first iteration of the foreach and ignored all the others.
I have to mention though, that if I have the connection properties inside the SendProgress method, the code works and I am receiving the information I want in the cosnole, but again, I think it shouldn't be like that.
This is the MVC controller which calls a rest service which in turn calls the class library
public JsonResult AProcess()
{
_dataImportService.DataImportProcess("2018-02-27", "2018-02-28", "This is another parameter");
return Json("", JsonRequestBehavior.AllowGet);
}
Please ignore the parameters of the DataImportProcess as they are irrelevant at this point.
This is my Startup.cs
public class Startup
{
public void Configuration(IAppBuilder app)
{
app.MapSignalR();
}
}
I am using SignalR 2.2.2
Edit #1
I included the connection properties to the constructor of the class again like this:
static IHubProxy _proxy { get; set; }
public ImportClass()
{
var hubConnection = new HubConnection("http://localhost:13310/");
hubConnection.Credentials = CredentialCache.DefaultNetworkCredentials;
IHubProxy statementsHubProxy = hubConnection.CreateHubProxy("statementsHub");
_proxy = statementsHubProxy;
hubConnection.Start().Wait();
}
But no luck. SendProgress method is not sending any updates at all.
Any help is much, much appreciated.
Best regards,
Konstantinos.

Client not receiving SignalR message from controller

I am trying to send a message from a controller to client(s) group using SignalR.
When an event occurs in one of my controllers, I need to push a message using a signalr hub to be displayed like an alert on my clients screens.
i know a lot of questions have been asked here, and I've read and tried a lot of them. Since I am new to SignalR some of them even already helped me to put things in place.
At the moment everything seems in place. Clients can connect to the hub and join groups and controller can call methods from the hub. But the client never receive the message and I can't figure out why. I suspect that hub's method called by controller don't "see" the clients but I can't understand what's wrong.
Hub's code :
public static class UserHandler
{
public static HashSet<string> ConnectedIds = new HashSet<string>();
}
[HubName("myHub")]
public class MyHub : Hub
{
private static IHubContext hubContext = GlobalHost.ConnectionManager.GetHubContext<MyHub>();
public void Notify(string groupName, string message)
{
Clients.Group(groupName).displayNotification(message);
}
public static void Static_Notify(string groupName, string message)
{
var toto = UserHandler.ConnectedIds.Count();
hubContext.Clients.Group(groupName).displayNotification(message);
hubContext.Clients.All.displayNotification(message);//for testing purpose
}
public Task JoinGroup(string groupName)
{
return Groups.Add(Context.ConnectionId, groupName);
}
public Task LeaveGroup(string groupName)
{
return Groups.Remove(Context.ConnectionId, groupName);
}
public override Task OnConnected()
{
UserHandler.ConnectedIds.Add(Context.ConnectionId);
return base.OnConnected();
}
public override Task OnDisconnected(bool StopCalled)
{
UserHandler.ConnectedIds.Remove(Context.ConnectionId);
return base.OnDisconnected(StopCalled);
}
}
Here is the call from my controller :
//(For simplification and readability I define here variables actually obtained by treating some data )
//I already checked that problem did not come from missing data here
string groupName = "theGroupName";
string message = "My beautifull message.";
//prepare signalR call
var context = GlobalHost.ConnectionManager.GetHubContext<MyHub>();
//Following commented lines are different attempts made based on exemples and answers I found here and on others sites.
//The uncommented one is the one in use at the moment and is also based on an answer (from SO i think)
//context.Clients.Group(_userEmail).displayNotification(message);
//context.Clients.Group(_userEmail).Notify(_userEmail,message);
MyHub.Static_Notify(_userEmail, message);
And here is the client-side code :
$(document).ready(function () {
var userGroup = 'theGroupName';
$.connection.hub.url = 'http://localhost/SignalRHost/signalr';
var theHub = $.connection.myHub;
console.log($.connection)
console.log($.connection.myHub)
theHub.client.displayNotification = function (message) {
console.log('display message');
alert(message);
};
$.connection.hub.start()
.done(function () {
theHub.server.joinGroup(userGroup);
console.log("myHub hub started : " + $.connection.hub.id)
console.log(theHub)
})
.fail(function () {
console.log('myHub hub failed to connect')
});
});
Please help me understand what logic I failed to understand or what error I am missing.
EDIT :
To answer Alisson's comment:
The Startup.cs I forgot to show
public void Configuration(IAppBuilder app)
{
app.Map("/signalr", map =>
{
map.UseCors(CorsOptions.AllowAll);
var hubConfiguration = new HubConfiguration { };
map.RunSignalR();
});
}
Important stuff I forgot to mention too :
The Controller, the hub, and the client are 3 different projects (because of the global application architecture I had to separate hub logic)
All are on my localhost IIS but on different ports
I've set breakpoints on "OnConnected" and "onDiconnected" events and client connects and disconnects normally

You just need one app to act as server, in your case it should be the SIgnalRHost project. Your controller project should be a client of the server, therefore it just needs this package:
Install-Package Microsoft.AspNet.SignalR.Client
Your controller project doesn't actually need to reference the project containing the hub class. In your controller, you'll use C# SignalR client to connect to the server (as you would do in javascript client), join the group and invoke the hub method:
var hubConnection = new HubConnection("http://localhost/SignalRHost/signalr");
IHubProxy myHub = hubConnection.CreateHubProxy("MyHub");
await hubConnection.Start();
myHub.Invoke("JoinGroup", "theGroupName");
myHub.Invoke("Notify", "theGroupName", "My beautifull message.");
...finally, you don't need that Static_Notify at all.
Since you are passing the group name as a parameter on the Notify
method, you don't really need to join the group from your controller.
You'd need only if you were trying to send the message to the same
group the controller was connected (then you wouldn't need to pass the
group name as parameter anymore).
SignalR C# Client Reference.

MVVM Light Messeging issue

Im using MVVM Light Messenger in my WPF application, and something is not working as expected.
my view-model registered with a token. im using 'long' objects as tokens.
my code registered for example with the token 5, then sends request to a service.
when the service replies it handles in my second view-model, which will then send the message with the same token.
When i debug and print the registration and sending of the messages it seems OK, but for some reason not all the messenger are received by the registered.
My registration and handling looks as follows:
private void registerTest()
{
long tokenId = getNextToken();
ExtraData data = new ExtraData();
Messenger.Default.Register<MyMsg>(this, tokenId, (m) => recieve(m,data));
}
private void receive(MyMsg m,ExtraData data)
{
Messenger.Default.Unregister<MyMsg>(this, m.tokenId);
}
My sending looks as follows:
private void sendTest(long tokenId)
{
Messenger.Default.Send(new MyMsg(tokenId), tokenId);
}
I always register with token X before its received in my sendTest,
but for some reason, sometimes when sendTest(X) is called, its not received.
Anyone has any idea whats going on?

You should have your ExtraData as a class property on your message to be able to interact with it from different sources.
public class MyMsg {
public int TokenId {get;set;}
public ExtraData Data {get;set;}
}
public void registerTest()
{
Messenger.Default.Register<MyMsg>(this, recieve);
}
public void recieve(MyMsg myMsg)
{
Messenger.Default.Unregister<MyMsg>(this, myMsg.TokenId);
//Note that you can also access m.Data here if you need to
}
public void sendTest()
{
var myMsg = new MyMsg {TokenId = GetNextToken(), Data = new ExtraData()};
Messenger.Default.Send(myMsg);
}