Is there a common way to recover from a connection error in MongoDB with the C# driver?
Currently, my Windows service shuts down if MongoDB is turned off. I currently have my app structured like this at the start of my Windows service:
//Set up connections for Mongo
var con = new MongoConnectionStringBuilder(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString);
var client = new MongoClient(con.ToString());
var server = client.GetServer();
var db = server.GetDatabase(con.DatabaseName);
I then inject the db object into my repositories.
I'm trying to find something like an event handler or a condition I could listen to in my whole application to prevent from crashing the entire service should mongo go down for some reason.
As suggested by the driver document, the MongoClient is added to manage Replica Set stuff, which earlier or later you will need. To avoid mass code refactoring then, you need to make better use of it now.
The MongoClient, which is thread-safe, have implemented the failover logic among replica set nodes already. It's supposed to be singleton along with your application domain. Thus you can inject the MongoClient, other than db (which is not even thread safe).
So always retry the GetServer() and GetDatabase() from MongoClient, and try/catch the exceptions produced by them would finally give you the available db object when MongoDB is online again.
The point is, MongoDB will not notify the clients about its online, so there's no such event to notify you, either. You'll have to keep trying in your client side until it's ok. And to avoid the exceptions to bring down your service, you'll have to catch them.
EDIT: I am wrong about the thread-safety according to the document. However, it doesn't change the fact you shouldn't store MongoDatabase for future migration to replica set.
In addition to yaoxing answer, wanted to do show code piece to solve this issue.
var client = new MongoClient(connString);
var server = client.GetServer();
while (server.State == MongoServerState.Disconnected)
{
Thread.Sleep(1000);
try
{
server.Reconnect();
}
catch (Exception ex)
{
Debug.WriteLine("Failed to connect mongodb {0} Attempt Count: {1}",
ex, server.ConnectionAttempt);
}
}
Related
To Datastax C# driver engineers:
C# driver 3.3.0 is deadlocking while calling to Connect(). The following code snippet on Windows Forms will deadlock trying to connect:
public void SimpleConnectTest()
{
const string ip = "127.0.0.1";
const string keyspace = "somekeyspace";
QueryOptions queryOptions = new QueryOptions();
queryOptions.SetConsistencyLevel(ConsistencyLevel.One);
Cluster cluster = Cluster.Builder()
.AddContactPoints(ip)
.WithQueryOptions(queryOptions)
.Build();
var cassandraSession = cluster.Connect(keyspace);
Assert.AreNotEqual(null, cassandraSession);
cluster.Dispose();
}
Deadlocking happens here:
Cluster.cs ->
private void Init()
{
...
TaskHelper.WaitToComplete(_controlConnection.Init(), initialAbortTimeout);
...
}
I have tested this on Cassandra 3.9.0, CQL spec 3.4.2 on local machine.
Everything deadlocks on calling this method _controlConnection.Init() here:
task = Id = 11, Status = WaitingForActivation, Method = "{null}", Result = "{Not yet computed}"
This then just runs for 30000ms and throws this:
throw new TimeoutException(
"Cluster initialization was aborted after timing out. This mechanism is put in place to" +
" avoid blocking the calling thread forever. This usually caused by a networking issue" +
" between the client driver instance and the cluster.", ex);
Running same test on 3.2.0 has no such problems. Can anyone else test this? Maybe this just happens to me.
Edit:
Here is the screenshot for the deadlock:
Thanks to the details in your comments, we were able to identify the underlying issue.
Similar to what was proposed by Luke, there were some missing ConfigureAwait() calls.
This issue impacts users that are calling Cluster.Connect() on environments with SynchonizationContext which is not a common use case:
For Windows Forms, its unlikely to communicate directly to a database (without a service in the middle). Furthermore, users should call Connect() before creating a form (where there is no SynchonizationContext) to share the same Session instance across all forms.
For ASP.NET, users should call Connect() outside of any endpoint action, before the HttpContext is created (where there is no SynchonizationContext).
Note that this issue affects only Connect() calls. Other blocking calls like Execute() don't have this issue.
In any case, this issue could be a showstopper for users getting started with the driver, for example, users creating a simple windows forms app to try a concept.
I've submitted a pull request with the fix, which also contains a test that looks into the source code for the usage of await without ConfigureAwait() calls to avoid having this issue in the future:
https://github.com/datastax/csharp-driver/pull/309
You can expect the fix to land in the next patch release.
I can't reproduce the problem, but I suspect the problem might be with a recent change to make the connection process asynchronous internally. I don't know for sure, but tracing through the Connect code, I suspect it might be a missing ConfigureAwait(false). In particular, it looks like the Reconnect method (which could definitely get hit as part of that Init code path) is missing one after that commit. It's possible that I'm not able to reproduce it because I'm not hitting the Reconnect code path while for some reason you are in your environment.
I'm not 100% sure that's the culprit, but I opened a PR to fix it. Stephen Cleary wrote a great explanation on why this can happen in Forms/Web apps. You could try building the driver from my fork to see if that change fixes the problem, or wait and see what happens with the PR and a new release. If it's still happening, I'd suggest opening an issue on the JIRA.
Hope that helps!
Issue has been opened here with workaround:
https://datastax-oss.atlassian.net/projects/CSHARP/issues/CSHARP-579
For anyone experiencing the same - just wrap your connection code into a new task.
Task.Run(() =>
{
SimpleConnectTest();
});
I am having a very strange problem and am hoping someone out there has had a similar experience.
My companies application for one client is getting "banned" from the SQL Server at the beginning of our application. The behavior is strange. I'll write it out in point form.
SQL Connections are created, data is retrieved, the connections are closed, talk to another datasource and then denied access to SQL Server.
Here's the long winded version:
.NET application connects to database multiple times. Gets some data, does some work. It then goes to get some more data and then gets an error that the "SQL Server cannot be found or access is denied". If the process is started over again without re-starting the app then no more connections are able to be made to SQL Server. All new connections result in "SQL Server cannot be found or access is denied". If the application is restarted then it will repeat the above process.
This is the first in 5 years of my experience with the software to have this problem. The application does have code written in Delphi 7. The dephi 7 / VBA code has not issues. My .NET code that performs the actual query looks like:
protected abstract DbConnection GetConnection();
protected abstract DbDataAdapter GetDataAdapter(DbCommand cmd);
protected abstract DbCommand GetCommand(DbConnection conn, String sql);
protected abstract DbCommandBuilder GetCommandBuilder(DbDataAdapter adapter);
public virtual DataTable Query(string sql)
{
var dt = new DataTable();
using (var conn = GetConnection())
{
try
{
using (var cmd = GetCommand(conn, sql))
{
using (var adapter = GetDataAdapter(cmd))
{
adapter.Fill(dt);
}
}
}
catch (Exception ex)
{
throw new SqlStatementException(sql, ex);
}
}
return dt;
}
It is my own quite and dirty DAL. When it is used it is using an OleDbConnection.
Note: Due to legacy code the connection string is configured for OleDbConnection. After taking a moment to review my code I do have the ability to change the connection type to SqlConnection. I haven't tried that yet.
On the client's machine I have not been able to reproduce the issue outside of the main application. I tried creating a little app that would make 100 calls back to back using the format above with an OleDbConnection but it executed successfully.
The failure in the main app happens in the same spot. That should give me a clue except I cannot make sense of it since it is making duplicate query, getting the same data. But I will say that the application talks to two data sources and transfers data from one to the other. Before it does the transfer it does some validation on the sources. So it talks to another database (proprietary file based) via ODBC and comes back successfully and then fails when trying to talk to SQL Server through OleDbConnection.
My suspicion is something is happening in the connection pool. That is causing a failure which in turns causes a denial of access.
Other interesting points. All worked fine for about a year, client got a new machine a couple of months ago, all work fine and then suddenly stopped. I put the application on another machine at the client's site and all worked well for a week and then the same issue appeared. We turned everything off on the client's machine but the issue persisted. I thought firewall but no luck there.
Any assistance is greatly appreciated.
Was gonna put this in a comment, but it got too big :-)
I see your connection-creating methods are abstract. This of course means that derivatives can do all sorts of bad things when they create the connection. I'd look there first.
One thing I found in a similar situation...if you're doing something in the code that creates the connection that makes the connection string unique, you won't be reusing those pooled connections. So...doing something like adding an "App=MyApp" + an incrementing number, date/time, or guid, it will destroy your ability to use pooled connections. When this happened to me, it took me forever to figure it out.
If your application was "slow enough" in the past, such that "old" pooled connections fall out of the pool, you might never see a problem...but then, say a customer gets hot new hardware...and blam...weird errors from nowhere! This might not be what's happening to you, but maybe it will give you some ideas about where to look. Good luck!
With previous version of C# drivers (1.x) I could do :
var client = new MongoClient(settings);
var server = client.GetServer();
server.Shutdown();
How can I do this with driver version 2.2.3 ?
Update
Well the best I could find is something like this :
try
{
var client = new MongoClient(settings);
var adminDatabase = client.GetDatabase("admin");
var cmd = new BsonDocument("shutdown", 1);
adminDatabase.RunCommand<BsonDocument>(cmd);
}
catch (MongoConnectionException e)
{
if (!(e.InnerException is EndOfStreamException))
{
throw;
}
}
but I dont really like this, the Try/catch etc ...
They told me at the Google Groups Page that is because it should never be used from most applications.
Craig Wilson mentioned that shutdown is simply a command that can be send using
db.RunCommand("{shutdown: 1}")
So it isn't anymore available in the API .net Version 2.0.0 and above.
This is the best that I could find after some intensive searching today. I am using the MongoDB C# driver 2.2. There are no special credentials to my mongod instance, it is all default settings. I would imagine this code would change a bit if there are special login credentials for the admin database.
// Connecting. 1 DB for actual usage, 1 for running the shutdown command
Client = new MongoClient("mongodb://127.0.0.1:27017");
Database = Client.GetDatabase(DBName);
AdminDatabase = Client.GetDatabase("admin");
// Shutting down the DB "cleanly"
AdminDatabase.RunCommandAsync<BsonDocument>(new JsonCommand<BsonDocument>("{shutdown: 1}"));
From what I can tell by watching the mongod instance in a command prompt my application successfully connects, writes, reads, and then shuts down the mongod instance with dbexit: rc: 0 which from what I can tell means it shutdown correctly, I faintly remember seeing dbexit: rc: 12 when shutting down that way I was before (don't even ask).
I'm currently using the NEST ElasticSearch C# Library for interacting with ElasticSearch. My project is an MVC 4 WebAPI project that basically builds a RESTful webservice for accessing directory assistance information.
We've only just started working with NEST, and have been stumbling over the lack of documentation. What's there is useful, but it's got some very large holes. Currently, everything we need works, however, we're running into an issue with connections sometimes taking up to a full second. What we'd like to do is use some sort of connection pooling, similar to how you'd interact with SQL Server.
Here is the documentation on how to connect using nest: http://mpdreamz.github.com/NEST/concepts/connecting.html
Here is the relevant code snippet from our project:
public class EOCategoryProvider : IProvider
{
public DNList ExecuteQuery(Query query)
{
//Configure the elastic client and it's settings
ConnectionSettings elasticSettings = new ConnectionSettings(Config.server, Config.port).SetDefaultIndex(Config.index);
ElasticClient client = new ElasticClient(elasticSettings);
//Connect to Elastic
ConnectionStatus connectionStatus;
if (client.TryConnect(out connectionStatus))
{
// Elastic Search Code here ...
} // end if
} // end ExecuteQuery
} // end EOCategoryProvider
From looking at the documentation, I can't see any provisions for a connection pool. I've been thinking about implementing my own (having, say 3 or 4 ElasticClient objects stored, and selecting them round-robin style), but I was wondering if anyone had a better solution. If not, does anyone have advice on the best way to implement a connection pool by hand? Any articles to point to?
Thanks for anything you guys come up with.
Update: This seems to have been related to calling TryConnect on every request, and the particular network setup. The problem completely disappeared when using a machine on the same network as the Elastic box; My development machine (which averages 350ms to the Elastic box) seemed to fail to make http connections sometimes, which caused the long times in TryConnect.
You don't have to call TryConnect() each time you do a call to Elasticsearch. It's basically a sanity check call for when your application starts.
NEST is the C# REST client for Elasticsearch and the default IConnection uses WebRequest.Create which already pools TCP connections.
Review the actual implementation: https://github.com/elastic/elasticsearch-net/blob/master/src/Elasticsearch.Net/Connection/HttpConnection.cs
Reusing ElasticClient won't offer any performance gains since each call already gets its own HttpWebRequest. The whole client is built stateless on purpose.
I am however very interested in why calls are taking 1 second for you. Could you post the actual NEST code, how you are are measuring the calls and describe your data.
Disclaimer: I'm the author of NEST.
I've got a bunch of reports deployed as RDL's to a SSRS. Because of high security requirements, the db passwords change very frequently. It's become a huge task to keep up with the changes and having to modify dozens upon dozens of reports. This leads to my question...
Is it possible to programmatically set a data source or a connection string for a deployed report?
Could this be done using an app that modifies something in the report itself as it sits on the server?
Could this be done by modifying a shared data source from an app as the DS sits on the server?
Could it be done by embedding a script inside the report itself which retrieves the connection from say a web service?
Thanks
This can be done in multiple ways, I think one of the simplest is using the API of SSRS web service. The web service lets you manipulate all the reporting entities including the Data Sources.
As a solution to this issue it's possible to update the data sources credentials using calls to the web service every time the DB password changes (even automate it using some sort of trigger?).
I did something similar in a project where RDLs and data sources had to be generated, deployed and manipulated in runtime and it worked fine so updating data source must be feasible.
Add a Service Reference to your project for your Reporting Service endpoint(http://ReportServerHost.mydomain.tld/ReportServer/ReportService2005.asmx). Use the following code to modify the Data Source password:
public static void ChangeDataSourcePassword(string dataSourcePath, string password)
{
using (ReportingService2005SoapClient reportingService = new ReportingService2005SoapClient())
{
reportingService.ClientCredentials.Windows.AllowedImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Impersonation;
try
{
ServerInfoHeader serverInfo = null;
DataSourceDefinition dataSourceDefinition = null;
serverInfo = reportingService.GetDataSourceContents(dataSourcePath, out dataSourceDefinition);
dataSourceDefinition.Password = password;
serverInfo = reportingService.SetDataSourceContents(null, dataSourcePath, dataSourceDefinition);
}
catch (FaultException ex)
{
// Do something with the exception. Rethrow it and/or show it to the user.
Console.WriteLine(string.Format("Failed to change the password on {0}: {1}", dataSourcePath, ex.Message));
throw new ApplicationException("Failed to change the password on the Data Source.", ex);
}
}
}