I need to setup sharding over replica set as recommended in MongoDB reference for high availability & scalability. I have few questions about connection string and its behavior for C# driver in that scenario (code snippet below):
Is the connection string below looks right for connecting to mongos instances: mongos1, mongos2 & mongos3?
What happens to client if one of the mongos instance crashes? Will the failed call handled gracefully by retrying to second mongos instance? Does the client blacklist the failed mongos instance and try after sometime?
If I want to set readpreference, will the driver be aware of replica set existence and honor setting ReadPreference?
Code snippet:
MongoUrlBuilder bldr = new MongoUrlBuilder();
List<MongoServerAddress> servers = new List<MongoServerAddress>();
servers.Add(new MongoServerAddress("mongos1:27016"));
servers.Add(new MongoServerAddress("mongos2:27016"));
servers.Add(new MongoServerAddress("mongos3:27016"));
bldr.Username = "myuser";
bldr.Password = "mypwd";
bldr.Servers = servers;
bldr.DatabaseName = "mydb";
bldr.ReadPreference = ReadPreference.Primary;
var server = MongoServer.Create(bldr.ToMongoUrl());
1) Yes, this is just fine. Note that all of this could be put in an actual connection string as well. mongodb://myuser:mypwd#mongos1:27016,mongos2:27016,mongos3:27016/mydb/?readPreference=primary
2) The way your connection string is built, you'll be load balancing across the 3 mongos. If one goes down, then the other two will simply begin to receive more traffic. Errors, however, will happen and nothing gets retried automatically. You'll need to handle the errors and make decisions based on each query/write whether it is safe to retry.
3) The driver, when talking to a sharded system, will simply forward the read preference to mongos. Note that mongos version 2.2 had some difficulty with read preferences. I'd advise you to be on the 2.4 line.
Related
I've been using this C# library wrapper for the darksky API:
https://github.com/amweiss/dark-sky-core
In my implementation I poll once every 3 minutes to get the forecast, which I use in my home thermostat network:
async void GetForecast()
{
// https://darksky.net/dev/docs#forecast-request
float Temp, DewPoint, WindSpeed, WindChill, Humidity, HeatIndex;
var client = new DarkSkyService("user-api-key");
try
{
Forecast fc = await client.GetWeatherDataAsync(38.329444, -87.412778);
Temp = (float)Math.Floor(fc.Currently.Temperature);
PublishTemp(Temp);
// for database, get temp, dewpoint, calculate windchill, calculate heatindex
DewPoint = (float)fc.Currently.DewPoint;
WindSpeed = (float)fc.Currently.WindSpeed;
Humidity = (float)fc.Currently.Humidity; // range: 0-1
WindChill = (float)CalculateWindChill(Temp, WindSpeed);
HeatIndex = (float)CalculateHeatIndex(Temp, Humidity);
SaveToDatabase(Temp, DewPoint, WindChill, HeatIndex);
RxForecast = true;
if (DateTime.Now.Hour != LastForecastHour)
{
LatestForecast = fc;
LastForecastHour = DateTime.Now.Hour;
PublishForecasts();
}
}
catch (Exception s) {
RxForecast = false;
}
ForecastWaitTime = RxForecast ? FAST_FORECAST_CYCLE : SLOW_FORECAST_CYCLE;
}
This has worked fine for about 4 months before it abruptly stopped working a week ago. Darksky support said that they have recently implemented security updates and no longer support most common TLS ciphers (quoting):
- TLS 1.0
- TLS 1.1
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
- TLS_RSA_WITH_AES_128_GCM_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA256
- TLS_RSA_WITH_AES_128_CBC_SHA
- TLS_RSA_WITH_AES_256_GCM_SHA384
- TLS_RSA_WITH_AES_256_CBC_SHA256
- TLS_RSA_WITH_AES_256_CBC_SHA
You can definitively determine whether your app works with the new SSL permissions by testing against
https://api.darksky.net:4433/. If you decide to update SSL on your end, you can test the API by sending a request here: https://api.darksky.net:4433/v1/status.txt.
Note that we will be making additional security-related updates in the coming weeks so there will be more changes in the near future. We don't have a notification system for alerting users to changes made on our backend but we do offer a feed for our status page, which often includes information about updates that have been or will be made (https://status.darksky.net/). We'll do our very best to make sure we communicate them as we're able to. Additionally, to avoid future disruptions we strongly recommend switching to one of the following, which should carry you through any of the additional security updates that will be applied in the near future:
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
I have no idea what changes I need to make to this code to 'update TLS', and I can't seem to get any more information from darksky. In the meantime, my alarm system is at a standstill.
One thing I don't understand is that, if I type this URL in a browser:
https://api.darksky.net/forecast/my-api-key/38.329444, -87.412778
It works fine, and immediately returns a huge JSON forecast string. Trying this with HttpWebRequest, HttpClient, or WebClient, in code results in different "errors occurred" exceptions. Overall, I'd rather use the library for the returned Forecast object that is easy to interpret.
Is this TLS update something I do at the system level, outside the devlopment environment?
Or, are there any alternatives to darksky that I could switch to?
You have two options:
1: update the library you are using and recompile. This issue was reported on its github page:
https://github.com/jcheng31/DarkSkyApi/issues/28
2: It's a bit of work but you could move the forecast module to Linux/Raspberry Pi, where TLS12 is already configured. You will have to rewrite the routine in Python to do this. I verified this approach would work on my own PI network.
I'm going through the mongoDB Driver Documentation Quick Tour for the first time. Specifically the 2.4 version.
I've created a fresh mongodb instance at the 192.168.1.50 address, and it appears to be running correctly.
The MongoDB documentation gives the following example:
var client = new MongoClient("mongodb://192.168.1.50:27017");
#It's ok if the database doesn't yet exist. It will be created upon first use
var database = client.GetDatabase("testDB");
#It’s ok if the collection doesn’t yet exist. It will be created upon first use.
var collection = database.GetCollection<BsonDocument>("testCollection");
However, when I go on my server, and I enter the mongo console
mongo
And I list the databases using
show dbs
The output is only
admin 0.000GB
local 0.000GB
Is there anything else I should have done to make this work? I'm getting no errors on try/catch, and it appears to be running fine.
Troubleshooting
So far I've confirmed that mongodb is running by using the following:
netstat -plntu
Shows mongod running on 27017 in the LISTEN state.
I'd also be interested in knowing if there's a way on the mongodb server to view live connections, so I could see if it were actually successfully connecting.
Well the problem is that you need to create almost one collection in order to persist the created database (weird right?) i tested it with robomongo and works in that way.
The problem is that GetCollection method is not creating the target collection, you can try with this code:
static void Main(string[] args)
{
var client = new MongoClient("mongodb://192.168.1.50:27017");
//# It's ok if the database doesn't yet exist. It will be created upon first use
var database = client.GetDatabase("test");
//# It’s ok if the collection doesn’t yet exist. It will be created upon first use.
string targetCollection = "testCollection";
bool alreadyExists = database.ListCollections().ToList().Any(x => x.GetElement("name").Value.ToString() == targetCollection);
if (!alreadyExists)
{
database.CreateCollection(targetCollection);
}
var collection = database.GetCollection<BsonDocument>(targetCollection);
}
It turns out that a method I had found on how to set multiple bindIp's was incorrect. The problem wasn't with the C# at all. I found the solution here
In case that ever goes away, here's the current settings I had to follow for multiple ip's
edit file /etc/mongod.conf
Wrap the comma-separated-Ips with brackets
bindIp = [127.0.0.1, 192.168.184.155, 96.88.169.145]
My original code worked fine, I just didn't have the brackets on the bindIp.
I am currently going through the process of upgrading our product CRM SDK and the main change I have encountered is that instead of connecting to the Xrm service and creating my IOrganizationService using the tried and trusted method of:
var connection = CrmConnection.Parse(connectionString);
var service = new OrganizationService(connection);
I am now having to utilise the CrmServiceClient from the Tooling namespace:
CrmServiceClient conn = new Microsoft.Xrm.Tooling.Connector.CrmServiceClient(connectionString).OrganizationServiceProxy;
Now this is all fine except from one major issue...memory.
Using the older Xrm.Client method you were able to specify the Service Configuration Instance Mode (which defaulted to ServiceConfigurationInstanceMode.PerName). What this meant is that the service was reused if the same application called the create multiple times. This kept the memory footprint low. The below image show the amount of allocated memory after calling to create a service instance 100 times
However, using the newer method you cannot set this instance mode anywhere and it seems that a brand new connection is created every time whether you want it or not. Here are the results of the same test:
As you can see, with every new connection, more and more memory is allocated. I can;t see anywhere that i can tell it to reuse the service.
So what I'm basically asking is, am I going about this in the wrong way? Should I be creating and caching everything myself? Are there hidden classes/methods that I am missing? Any help would be greatly appreciated.
The latest SDK (8.2.0.1) caches and resuses the connection as long as the connectionstring does not inclue RequireNewInstance=true.
One thing worth noting is even if you new up another CrmServiceClientwith a unique connection string (pointing to a different CRM organization), but the connection string does not include RequireNewInstance=true, the CrmServiceClient will reuse the previous cached connection.
So
var connectionString = $#"Url=https://ORG1.crm.dynamics.com;AuthType=Office365;UserName=USER#DOMAIN.com;Password=PASSWORD";
var connectionString2 = $#"Url=https://ORG2.crm.dynamics.com;AuthType=Office365;UserName=USER#DOMAIN.com;Password=PASSWORD";
var crmSvcClient = new CrmServiceClient(connectionString);
((WhoAmIResponse)crmSvcClient.Execute(new WhoAmIRequest())).OrganizationId.Dump();
crmSvcClient.ConnectedOrgFriendlyName.Dump();
var crmSvcClient2 = new CrmServiceClient(connectionString2);
((WhoAmIResponse)crmSvcClient2.Execute(new WhoAmIRequest())).OrganizationId.Dump();
crmSvcClient2.ConnectedOrgFriendlyName.Dump();
Prints out the guid and ORG1 friendly name both times. If you pass RequireNewInstance=true in connectionstring2 then you will see ORG2 printed out.
I am using the C# driver to work with MongoDB. I need to use my program to update the set config during execution, and then allow other parts of the program to continue using the driver.
My current issue is reconfiguring the driver to use the new set.
try
{
var res = Database.Admin.RunCommand(new CommandDocument("replSetReconfig", replicaSetConfig));
}
catch (EndOfStreamException){}
catch (Exception e)
{
Log.Exc("Problem updating replica set", e);
ranCommand = false;
}
if (ranCommand)
{
HERE - I need to update the MongoClient or MongoServer to have the new servers
return;
}
I assume when an EndOfStreamException occurs, that the command has been successful, as the server has shutdown/started a reconfig.
I'd like to update the driver to have the new config to ensure that it will definitely reconnect to at least one of the servers, in any edge cases where changing the config will prevent it connecting back with its original connection string.
Is there any way to achieve this?
As an extra question, is there any way from C# to determine the current replica set config version?
Thanks
I can answer part 2 of your question - you can query the local database to get the current replica set configuration:
MongoServer server = client.GetServer();
var local = server.GetDatabase("local");
var collection = local.GetCollection("system.replset");
var cfg = collection.FindOne();
Console.WriteLine(cfg["version"]);
I would imagine that if the original connection string is no longer valid, you need to create a new MongoClient object with a new connection string.
[Later]
You can also get everything else about the replica set, including its status, by running the replSetGetStatus command on the admin database.
I've investigated the possibilities of creating database backups through SMO with C#.
The task is quite easy and code straightforward. I've got only one question: how can I check if the backup was really created?
SqlBackup.SqlBackup method returns no parameters and I don't even know if it throws any exceptions. (the only thing that I know is that it is blocking, because there's also SqlBackupAsync method)
I would appreciate any help.
you can and its very possible to do what you asked for,
but doing the backup it self using SMO its not very hard, but the hard part is managing the backup and the restore.
it would be hard to put all the code here, but its wont fit. so I will try my best to put the lines you need.
SqlBackup.SqlBackup doesn't return any value, its a void function.
but it takes one parameter which is "Server", try out the following code:
Server srvSql;
//Connect to Server using your authentication method and load the databases in srvSql
// THEN
Backup bkpDatabase = new Backup();
bkpDatabase.Action = BackupActionType.Database;
bkpDatabase.Incremental = true; // will take an incemental backup
bkpDatabase.Incremental = false; // will take a Full backup
bkpDatabase.Database = "your DB name";
BackupDeviceItem bDevice = new BackupDeviceItem("Backup.bak", DeviceType.File);
bkpDatabase.Devices.Add(bDevice );
bkpDatabase.PercentCompleteNotification = 1;// this for progress
bkpDatabase.SqlBackup(srvSql);
bkpDatabase.Devices.Clear();
I've investigated the problem using Reflector.NET (I suppose this is legal since RedGate is Ms Gold Certified Partner and Reflector.NET opens .NET libraries out of the box). As I found out the method throws two types of exceptions:
FailedOperationException - in most cases, other exceptions are "translated" (I suppose translating means creating new FailedOperationException and setting InnerException to what was actually thrown)
UnsupportedVersionException - in one case when log truncation is set to TruncateOnly and server major version is more or equal to 10 (which is sql server 2008?)
This solves my problem partially, because I'm not 100% sure that if something goes wrong those exceptions will actually be thrown.