We Keep Coding

Azure Cosmos db throwing Socket Exceptions

I am using azure cosmos db with .net core 2.1 application. I am using gremlin driver with this. It's working fine but after every few days it start throwing socket exception on server and we have to recycle IIS pool. Average per day hits are 10000.
Now we are using default gateway mode. Should we have to switch to direct mode as it might be a firewall issue ?
Here is the implementation:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy());
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
and in startup.cs:
services.AddSingleton(x => GetDocumentClient(cosmosDBConfig));
and here is how we are communicating with cosmos db:
private DocumentClient _documentClient;
private DocumentCollection _documentCollection;
private CosmosDbConnectionOptions _cosmosDBConfig;
public DocumentCollectionFactory(DocumentClient documentClient, CosmosDbConnectionOptions cosmosDBConfig)
{
_documentClient = documentClient;
_cosmosDBConfig = cosmosDBConfig;
}
public async Task<DocumentCollection> GetProfileCollectionAsync()
{
if (_documentCollection == null)
{
_documentCollection = await _documentClient.CreateDocumentCollectionIfNotExistsAsync(
UriFactory.CreateDatabaseUri(_cosmosDBConfig.Database),
new DocumentCollection { Id = _cosmosDBConfig.Collection },
new RequestOptions { OfferThroughput = _cosmosDBConfig.Throughput });
return _documentCollection;
}
return _documentCollection;
}
and then:
public async Task CreateProfile(Profile profile)
{
var graphCollection = await _graphCollection.GetProfileCollectionAsync();
var createQuery = GetCreateQuery(profile);
IDocumentQuery<dynamic> query = _documentClient.CreateGremlinQuery<dynamic>(graphCollection, createQuery);
if(query.HasMoreResults)
{
await query.ExecuteNextAsync();
}
}

I'm assuming that for communication with CosmosDB you are using HttpClient. The application should share a single instance of HttpClient.
Every time you make a connection after HttpClient disposal there are still a bunch of connections in the state of TIME_WAIT. This means that the connection was closed on one side ( OS ) but it is in "waiting for additional packets" state.
By default, Windows may hold this connection in this state for 240 seconds. There is a limit to how quickly OS can open new sockets. All this may lead to System.Net.Sockets.SocketException exception.
Very good article that explains in details why and how this problem appears digging into TCP diagram and explaining with more details.
UPDATED
Possible solution.
You are using the default ConnectionPolicy object. That object has a property called IdleTcpConnectionTimeout which controls the amount of idle time after which unused connections are closed. By default, idle connections are kept open indefinitely. The value must be greater than or equal to 10 minutes.
So the code could look like:
private DocumentClient GetDocumentClient( CosmosDbConnectionOptions configuration)
{
_documentClient = new DocumentClient(
new Uri(configuration.Endpoint),
configuration.AuthKey,
new ConnectionPolicy() {
IdleTcpConnectionTimeout = new TimeSpan(0,0,10,0)
});
//create database if not exists
_documentClient.CreateDatabaseIfNotExistsAsync(new Database { Id = configuration.Database });
return _documentClient;
}
Here is a link to ConnectionPolicy Class documentation

Quickfix/n - No event for connection timeout or host not found?

So I have a project where I need to create failover between two FIX quote hosts in case of failure.
The FixApplication (IApplication) OnLogout() is nice to hook an event to when a socket is dropped or you are logged out. This is simple. But this only works if the socket connection was successful to start off with.
If you start up the application and the host is down, no method is actually called for this anywhere. Not in IApplication, nor in the socket IInitiator. No error is thrown anywhere either, the QuickFix initiator will simply just silently retry.
I am using a Timer with a callback right now to manually poll every so often and check if Initiator IsLoggedOn() is false. Then triggering a failover.
But this is really clunky and not elegant.
Is there any other event or method I can hook into to receive the notice for socket network failures BEFORE a network connection and session is established successfully?
Thanks!
Timer t = new Timer(5000)
t.Elapsed += CheckSocketConnected;
private void CheckSocketConnected(object source, ElapsedEventArgs e)
{
var connected = socketInitiator.IsLoggedOn;
if (!connected)
{
SwitchToAlternateProvider();
}
}

Well, after realising the limitation of the QuickFix/N component, I would never receive any feedback from this library if the host was down.
My resolution was to just simply use the following piece of code to check if socket was open before starting the connection in QuickFix/n
bool IsPortOpen(string host, int port, TimeSpan timeout)
{
try
{
using(var client = new TcpClient())
{
var result = client.BeginConnect(host, port, null, null);
var success = result.AsyncWaitHandle.WaitOne(timeout);
if (!success)
{
return false;
}
client.EndConnect(result);
}
}
catch
{
return false;
}
return true;
}

NetMQ - messages only work on reply to incoming message, not to other port

On initialize of my router application I call the following code. It binds fine, receives messages fine but refuses to work for the On_ReceiveXXX methods unless it's a direct response. I want to know why
public void Initialize(string frontEndAddress, string backEndAddress)
{
_poller = new Poller();
_timeAllowedBetweenPings = TimeSpan.FromMinutes(1);
_lastPingResponse = DateTime.Now;
using (var ctx = NetMQContext.Create())
{
_frontEnd = ctx.CreateRouterSocket();
_backEnd = ctx.CreateRouterSocket();
_frontEnd.Bind(frontEndAddress);
Console.WriteLine(string.Format("[Router]: Connected to {0}", frontEndAddress));
_backEnd.Bind(backEndAddress);
Console.WriteLine(string.Format("[Router]: Connected to {0}", backEndAddress));
_frontEnd.ReceiveReady += On_ReceiveFrontEnd;
_backEnd.ReceiveReady += On_ReceiveBackEnd;
_poller.AddSocket(_frontEnd);
_poller.AddSocket(_backEnd);
var timer = new NetMQTimer(TimeSpan.FromSeconds(1));
timer.Elapsed += On_Ping;
_poller.AddTimer(timer);
_poller.PollTillCancelled();
}
}
This fails to call the dealer ReceiveReady event:
private void On_ReceiveFrontEnd(object sender, NetMQSocketEventArgs e)
{
_lastPingResponse = DateTime.Now;
var frontEndMsg = e.Socket.ReceiveMultipartBytes();
var streamData = frontEndMsg.Last();
ApplicationMessage msg = PackageHelper.DeserializeOutgoing(streamData);
Console.WriteLine(string.Format("Command received: {0}", msg.CO));
_backEnd.SendMultipartBytes(frontEndMsg);
}
BUT if I change the line
_backEnd.SendMultipartBytes(frontEndMsg);
to
_frontEnd.SendMultipartBytes(frontEndMsg);
It suddenly works... so messages coming from my front end application can only be responded to, not passed on to the back end application. The same is true the other way round, for the back end messages.

When working with router the first frame is the routing id and it specific to the socket. So you can't pass the entire message from router to router. Change the backend to dealer and it will work, or prefix the message with routing id of the backend socket.

Azure Service Bus SubscriptionClient.OnMessage always completes message when it shouldnt

I am trying to receive all messages for a given subscription to a Service Bus Topic, but for the context of this app I do not want them dead lettered at this time, I just want to view them and leave them on the subscription. Despite instantiating the Client as
SubscriptionClient sc = SubscriptionClient.CreateFromConnectionString(connectionString, sub.topicName, sub.subscriptionName, ReceiveMode.PeekLock);
and making sure that I am using message.Abandon() rather than message.Complete() the message always gets Dead-lettered after accessing the message. I also have options.AutoComplete set to false
full method code below:
public List<ServiceBusMessage> RetrieveSubscriptionMessages(Subscription sub) {
ServiceBusMessage sbm;
List<ServiceBusMessage> list = new List<ServiceBusMessage>();
String connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"].ToString();
SubscriptionClient sc = SubscriptionClient.CreateFromConnectionString(connectionString, sub.topicName, sub.subscriptionName, ReceiveMode.PeekLock);
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
sc.OnMessage((message) => {
try {
sbm = new ServiceBusMessage() {
topicName = sub.topicName,
messageText = message.GetBody<String>()
};
list.Add(sbm);
message.Abandon();
}
catch (Exception) {
message.Abandon();
throw;
}
}, options);
return list;
}
Am I missing something ? Or is there an issue with auto dead-lettering with the onMessage() method?
Thanks !

When a message is abandoned the service bus will immediately make it available for re-delivery to any subscriber of the topic.
If you are trying to configure a multicast mechanism in which multiple listeners all receive the same message, then understand that all listeners on a given subscription will be competing for the same message. In order for every listener to receive its own copy of the message, then simply create a unique subscription to the topic for each listener.
If your intent is to delay re-delivery of the abandoned message, you might look at the SO question: What's the proper way to abandon an Azure SB Message so that it becomes visible again in the future in a way I can control?

Detect socket disconnect in WCF

We're building a WCF server (.NET 4.0). It will only use net.tcp transport.
When a client closes the TCP connection, the server gets unhandled CommunicationException, and terminates.
Q1. How do I handle the CommunicationException so the server does not terminate and continues serving other clients?
Q2. In the handler, how do I get SessionId of the session that was aborted? I need this to do clean up some session-specific data.
Thanks in advance!
P.S. The connection is over the Internet, so the socket may be closed anytime, regardless on whether the client disconnects gracefully, or not.

Any WCF channel implements ICommunicationObject , which provides events for the channel lifetime.
You should listen to the Faulted event
The sessionId can be accessed as always from the OperationContext.Current property.
When your client open the channel (on the first operation), register to the adequate events:
OperationContext.Current.Channel.Faulted += new EventHandler(Channel_Faulted);
OperationContext.Current.Channel.Closed += new EventHandler(Channel_Faulted);
and:
void Channel_Faulted(object sender, EventArgs e)
{
Logout((IContextChannel)sender);
}
protected void Logout(IContextChannel channel)
{
string sessionId = null;
if (channel != null)
{
sessionId = channel.SessionId;
}
[...]
}

ICommunicationObject obj = (ICommunicationObject)callback;
obj.Closed += new EventHandler((a, b) =>
{
if (list.Exists(cli => cli.CallbackService == (ITecnobelRemoteServiceCallback)a))
{
var query = (from cc in list where cc.CallbackService == (ITecnobelRemoteServiceCallback)a select cc).ToList();
query.ForEach(
delegate (Client ccc)
{
ccc.CallbackService = null;
});
}
});