We Keep Coding

Detect dead connections with SignalR

SignalR version: SignalR 2.4.1
.Net Framework version: 4.8 (I am not using .Net Core)
SignalR transport: websockets
I am developing a background service for SignalR (PresenceMonitor) where I need to detect whether a connection with specific clientid is alive or not.
I am using the following code for Presence Monitor to start with what I want to achieve:
using System;
using System.Data.Entity.SqlServer;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using Microsoft.AspNet.SignalR.Transports;
namespace UserPresence
{
/// <summary>
/// This class keeps track of connections that the <see cref="UserTrackingHub"/>
/// has seen. It uses a time based system to verify if connections are *actually* still online.
/// Using this class combined with the connection events SignalR raises will ensure
/// that your database will always be in sync with what SignalR is seeing.
/// </summary>
public class PresenceMonitor
{
private readonly ITransportHeartbeat _heartbeat;
private Timer _timer;
// How often we plan to check if the connections in our store are valid
private readonly TimeSpan _presenceCheckInterval = TimeSpan.FromSeconds(10);
// How many periods need pass without an update to consider a connection invalid
private const int periodsBeforeConsideringZombie = 3;
// The number of seconds that have to pass to consider a connection invalid.
private readonly int _zombieThreshold;
public PresenceMonitor(ITransportHeartbeat heartbeat)
{
_heartbeat = heartbeat;
_zombieThreshold = (int)_presenceCheckInterval.TotalSeconds * periodsBeforeConsideringZombie;
}
public void StartMonitoring()
{
if (_timer == null)
{
_timer = new Timer(_ =>
{
try
{
Check();
}
catch (Exception ex)
{
// Don't throw on background threads, it'll kill the entire process
Trace.TraceError(ex.Message);
}
},
null,
TimeSpan.Zero,
_presenceCheckInterval);
}
}
private void Check()
{
using (var db = new UserContext())
{
// Get all connections on this node and update the activity
foreach (var trackedConnection in _heartbeat.GetConnections())
{
if (!trackedConnection.IsAlive)
{
continue;
}
Connection connection = db.Connections.Find(trackedConnection.ConnectionId);
// Update the client's last activity
if (connection != null)
{
connection.LastActivity = DateTimeOffset.UtcNow;
}
else
{
// We have a connection that isn't tracked in our DB!
// This should *NEVER* happen
// Debugger.Launch();
}
}
// Now check all db connections to see if there's any zombies
// Remove all connections that haven't been updated based on our threshold
var zombies = db.Connections.Where(c =>
SqlFunctions.DateDiff("ss", c.LastActivity, DateTimeOffset.UtcNow) >= _zombieThreshold);
// We're doing ToList() since there's no MARS support on azure
foreach (var connection in zombies.ToList())
{
db.Connections.Remove(connection);
}
db.SaveChanges();
}
}
}
}
The issue I am facing is here:
// Get all connections on this node and update the activity
foreach (var trackedConnection in _heartbeat.GetConnections())
{
Scanning all the connections when there are large number of connections is deeply affecting the performance of my application and is giving lot of CPU spikes.
In my database, I already have the mapping for connection ids per user. Based on that I already a have field in my cache per user whether that user has any connection in db or not. Those mappings are are already cached. I would scan each of those mappings and would check whether the connection (connection id) for that specific user is is alive or not. I tried looking for ITransportHeartbeat Interface for the same but unfortunately, that interface gives us just these four methods:
//
// Summary:
// Manages tracking the state of connections.
public interface ITransportHeartbeat
{
//
// Summary:
// Adds a new connection to the list of tracked connections.
//
// Parameters:
// connection:
// The connection to be added.
//
// Returns:
// The connection it replaced, if any.
ITrackingConnection AddOrUpdateConnection(ITrackingConnection connection);
//
// Summary:
// Gets a list of connections being tracked.
//
// Returns:
// A list of connections.
IList<ITrackingConnection> GetConnections();
//
// Summary:
// Marks an existing connection as active.
//
// Parameters:
// connection:
// The connection to mark.
void MarkConnection(ITrackingConnection connection);
//
// Summary:
// Removes a connection from the list of tracked connections.
//
// Parameters:
// connection:
// The connection to remove.
void RemoveConnection(ITrackingConnection connection);
}
Ther is no method where I can get the state of connection by connectionid. Is there any way where I can get a specific connection information without scannig all the connetcions. I am aware of the traditional way to get that which could be using this: _heartbeat.GetConnections().Select(b => b.ConnectionId). But that code also will scan all the connections.
I am aware of OnDisconnected event also which we could use on a Hub itself but the OnDisconnected even doesn't guarantee to fire always (browser can close, internet shut down, site restart).
Is there any code which I could hook in my Hub itself to detect the ping done by the Heartbeat API? I could store the last pings per connection (kind of denormalize the way of detecting last ping) and can detect whether that connection is dead or not?
SignalR for .Net Core has something like that:
var heartbeat = Context.Features.Get<IConnectionHeartbeatFeature>();
heartbeat.OnHeartBeat(MyAction,
but I am looking for a similar feature like that in SignalR for .NET Framework.

Azure Cosmos db throwing Socket Exceptions

I am using azure cosmos db with .net core 2.1 application. I am using gremlin driver with this. It's working fine but after every few days it start throwing socket exception on server and we have to recycle IIS pool. Average per day hits are 10000.
Now we are using default gateway mode. Should we have to switch to direct mode as it might be a firewall issue ?
Here is the implementation:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy());
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
and in startup.cs:
services.AddSingleton(x => GetDocumentClient(cosmosDBConfig));
and here is how we are communicating with cosmos db:
private DocumentClient _documentClient;
private DocumentCollection _documentCollection;
private CosmosDbConnectionOptions _cosmosDBConfig;
public DocumentCollectionFactory(DocumentClient documentClient, CosmosDbConnectionOptions cosmosDBConfig)
{
_documentClient = documentClient;
_cosmosDBConfig = cosmosDBConfig;
}
public async Task<DocumentCollection> GetProfileCollectionAsync()
{
if (_documentCollection == null)
{
_documentCollection = await _documentClient.CreateDocumentCollectionIfNotExistsAsync(
UriFactory.CreateDatabaseUri(_cosmosDBConfig.Database),
new DocumentCollection { Id = _cosmosDBConfig.Collection },
new RequestOptions { OfferThroughput = _cosmosDBConfig.Throughput });
return _documentCollection;
}
return _documentCollection;
}
and then:
public async Task CreateProfile(Profile profile)
{
var graphCollection = await _graphCollection.GetProfileCollectionAsync();
var createQuery = GetCreateQuery(profile);
IDocumentQuery<dynamic> query = _documentClient.CreateGremlinQuery<dynamic>(graphCollection, createQuery);
if(query.HasMoreResults)
{
await query.ExecuteNextAsync();
}
}

I'm assuming that for communication with CosmosDB you are using HttpClient. The application should share a single instance of HttpClient.
Every time you make a connection after HttpClient disposal there are still a bunch of connections in the state of TIME_WAIT. This means that the connection was closed on one side ( OS ) but it is in "waiting for additional packets" state.
By default, Windows may hold this connection in this state for 240 seconds. There is a limit to how quickly OS can open new sockets. All this may lead to System.Net.Sockets.SocketException exception.
Very good article that explains in details why and how this problem appears digging into TCP diagram and explaining with more details.
UPDATED
Possible solution.
You are using the default ConnectionPolicy object. That object has a property called IdleTcpConnectionTimeout which controls the amount of idle time after which unused connections are closed. By default, idle connections are kept open indefinitely. The value must be greater than or equal to 10 minutes.
So the code could look like:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy() {
IdleTcpConnectionTimeout = new TimeSpan(0,0,10,0)
});
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
Here is a link to ConnectionPolicy Class documentation

HangFire Server Enable - Disable manually

During development of HangFire application with C# ASP.NET, and I decided to implement functionally where Admin can manage state of Server, jobs.
List item
Server Enable Disable state. Using Enable Button click event Admin
can start JOB server so all the Fire and Forget and Recurrent job can
performed. And Disable button stop all the activities of JOB.
Retrieve the current state of Server
I want to retrieve current state of JOB server, So I can show is
server is on or Off.
Retrieve state and enable / disable state of Jobs (Only recurrent).

If you want to manage Server/Job created by Hangfire, you can use MonitoringApi or JobStorage to get there statuses.
Sample Codes :
var _jobStorage = JobStorage.Current;
// How to get recurringjobs
using (var connection = _jobStorage.GetConnection())
{
var storageConnection = connection as JobStorageConnection;
if (storageConnection != null)
{
var recurringJob = storageConnection.GetRecurringJobs();
foreach(var job in recurringJob)
{
// do you stuff
}
}
}
// How to get Servers
var monitoringApi = _jobStorage.GetMonitoringApi();
var serverList = monitoringApi.Servers();
foreach( var server in serverList)
{
// do you stuff with the server
// you can use var connection = _jobStorage.GetConnection()
// to remove server
}
From here you can play around with Hangfire.

EWS - Renew subscription after timeout?

This may be a very basic problem, but I haven't found any answer for it yet. I'm using Exchange Web Services in a Windows service to monitor new mails sent to our Exchange 2010 server with a pull subscription. It's working all fine and dandy, but the problems is if the server is not available (such as after a power outage), then the subscription times out, and the Windows service needs to be restarted. Is there a way to renew the subscription after a timeout, or to pull EvenType.Status events?
Here's my code so far:
ExchangeService service;
PullSubscription subscriptionInbox;
private void SetService()
{
service = new ExchangeService(ExchangeVersion.Exchange2010);
service.Url = new Uri("myurl");
service.Credentials = new WebCredentials(emailAddress, pass);
}
private void SetSubscription()
{
if (service == null)
{
SetService();
}
subscriptionInbox = service.SubscribeToPullNotifications(
new FolderId[] { WellKnownFolderName.Inbox },
5,
null,
EventType.NewMail, EventType.Modified);
}
private void DoStuff(object sender, EventArgs e)
{
GetEventsResults eventsInbox = subscriptionInbox.GetEvents();
EmailMessage message;
foreach (ItemEvent itemEvent in eventsInbox.ItemEvents)
{
//Do Stuff
}
}
Any ideas how I could go on with this?

When you lose a subscription, it's best to create a new subscription - and not try to recover the interim data. You can resubscribe with the old watermark, but it's cost prohibitive. This link provides some additional context about recovering notifications related to lost subscriptions: http://msdn.microsoft.com/en-us/library/office/dn458788(v=exchg.150).aspx#bk_recover. You may also want to view this Channel 9 video, which discusses recovery from lost subscriptions: http://channel9.msdn.com/Events/Open-Specifications-Plugfests/Windows-Identity-and-Exchange-Protocols-Plugfest-2012/Exchange-Web-Services-Best-Practices-Part-2.

Listening to Events in the calendar from more than one person using EWS API

Simply I would like to receive a notification every time someone added a new appointment or made any changes on what he/she has.
The only way I know how to do it , is by using
service.SubscribeToStreamingNotifications
but the problem here that it only listens to the account that the service is bound to like in this way
var service = new ExchangeService(ExchangeVersion.Exchange2010_SP2)
{
Credentials = new WebCredentials(userName, password)
};
service.SubscribeToStreamingNotifications(new FolderId[]
{
WellKnownFolderName.Calendar
}, EventType.FreeBusyChanged, EventType.Deleted);
I have solved this problem by creating a list of services each service is bounded to different user and the application should listen to each of them.
The problem with this way is that I need to have the password of each account I wont to listen to its events, which is not possible in real world.
so is there any way to deal with that ?

I have solved this problem, by creating a list of services, all the services are a clone of the main ExchangeService, with the same credentials for the admin account, but they are impersonated to the other accounts.
NOTE: You need to setup the server so it allows impersonation.
private void ImpersonateUsers(ICollection<string> userSmtps)
{
if (userSmtps != null)
if (userSmtps.Count > 0)
{
foreach (var userSmtp in userSmtps)
{
if (_services.ContainsKey(userSmtp)) continue;
var newService = new ExchangeService(ExchangeVersion.Exchange2010_SP2);
try
{
var serviceCred = ((System.Net.NetworkCredential)(((WebCredentials)(_services.First().Value.Credentials)).Credentials));
newService.Credentials = new WebCredentials(serviceCred.UserName, serviceCred.Password, serviceCred.Domain);
newService.AutodiscoverUrl(serviceCred.UserName + "#" + serviceCred.Domain, RedirectionUrlValidationCallback);
newService.ImpersonatedUserId = new ImpersonatedUserId(ConnectingIdType.SmtpAddress, userSmtp);
}
catch (Exception ex)
{
System.Diagnostics.Debug.WriteLine(ex.Message);
}
_services.Add(userSmtp, newService);
}
}
}
Where userSmtps is a list of the email addresses I want to impersonate and _services is the dictionary of services where the first member is the main service.

you will have to create a service instance per user. There is no way to subscribe to other users folder.
But instead of StreamingNotifications you can use Pull and Push-Subscriptions too.
Something like this:
List<FolderId> folders = new List<FolderId>();
folders.Add(new FolderId(WellKnownFolderName.Calendar));
PullSubscription subscription = = service.SubscribeToPullNotifications(folders, 1440, watermark, EventType.Created, EventType.Deleted, EventType.Modified, EventType.Moved, EventType.NewMail);
Some time later....
GetEventsResults currentevents = m_subscription .GetEvents();