We Keep Coding

Avoiding ElasticSearch error 503 Server Unavailable: Use of WaitForStatus

When I start my program, I run the ElasticSearch Service and check if an Index exists and if there is any documents, let's say I just run the ES service and I have these two functions:
public ElasticClient getElasticSearchClient()
{
ConnectionSettings connectionSettings = new Nest.ConnectionSettings(new Uri("http://localhost:9200"))
.DefaultIndex("myindex")
.DisableDirectStreaming();
ElasticClient client = new ElasticClient(connectionSettings);
//var health = client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)).Timeout(50));
return client;
}
public void checkElasticsearchIndex()
{
var client = getElasticSearchClient();
var health = this.client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)));
CountResponse count = client.Count<myobject>();
if (!client.Indices.Exists("myindex").IsValid || count.Count == 0)
{
BulkWriteAllToIndexES(client);
}
}
Inside the checkElasticsearchIndex function,
The count operation fails with the following error message:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: GET /myindex/_count. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
The Health fails as well:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Unable to connect to the remote server. Call: Status code unknown from: GET /_cluster/health/myindex?wait_for_status=yellow ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:9200
As you can see, I have tried the Cluster WaitForStatus, but it didn't work.
My question: is there any way to wait until client/cluster/nodes are ready and not get any exception?

It sounds like you're starting the Elasticsearch process at the same time as starting your program, but Elasticsearch takes longer than your program to be ready.
If that's the case, you may be interested in using the same abstractions that the .NET client uses for integration tests against Elasticsearch. The abstractions read output from the Elasticsearch process to know when it is ready, and block until this happens. They're available on an AppVeyor CI package feed (with plans to release them to Nuget in the future).
There are some examples of how to spin up a cluster with the abstractions. For single node, it would be something like
using System;
using Elastic.Managed.Configuration;
using Elastic.Managed.ConsoleWriters;
using Elastic.Managed.FileSystem;
namespace Elastic.Managed.Example
{
class Program
{
static void Main(string[] args)
{
var version = "7.5.1";
var esHome = Environment.ExpandEnvironmentVariables($#"%LOCALAPPDATA%\ElasticManaged\{version}\elasticsearch-{version}");
using (var node = new ElasticsearchNode(version, esHome))
{
node.SubscribeLines(new LineHighlightWriter());
if (!node.WaitForStarted(TimeSpan.FromMinutes(2))) throw new Exception();
// do your work here
}
}
}
}
This assumes that Elasticsearch 7.5.1 zip has been downloaded already, and exists at %LOCALAPPDATA%\ElasticManaged\7.5.1\elasticsearch-7.5.1. There are more complex examples of how to integrate this into tests with xUnit.
You can use the EphemeralCluster components to download, configure and run Elasticsearch
var plugins = new ElasticsearchPlugins(ElasticsearchPlugin.RepositoryAzure, ElasticsearchPlugin.IngestAttachment);
var config = new EphemeralClusterConfiguration("7.5.1", ClusterFeatures.XPack, plugins, numberOfNodes: 1);
using (var cluster = new EphemeralCluster(config))
{
cluster.Start();
var nodes = cluster.NodesUris();
var connectionPool = new StaticConnectionPool(nodes);
var settings = new ConnectionSettings(connectionPool).EnableDebugMode();
var client = new ElasticClient(settings);
Console.Write(client.CatPlugins().DebugInformation);
}

How to check database connection in MongoDB [duplicate]

I use MongoDB drivers to connect to the database. When my form loads, I want to set up connection and to check whether it is ok or not. I do it like this:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
var database = server.GetDatabase("reestr");
But I do not know how to check connection. I tried to overlap this code with try-catch, but to no avail. Even if I make an incorrect connectionString, I still can not get any error message.

To ping the server with the new 3.0 driver its:
var database = client.GetDatabase("YourDbHere");
database.RunCommandAsync((Command<BsonDocument>)"{ping:1}")
.Wait();

There's a ping method for that:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
server.Ping();

full example for 2.4.3 - where "client.GetServer()" isn't available.
based on "Paul Keister" answer.
client = new MongoClient("mongodb://localhost");
database = client.GetDatabase(mongoDbStr);
bool isMongoLive = database.RunCommandAsync((Command<BsonDocument>)"{ping:1}").Wait(1000);
if(isMongoLive)
{
// connected
}
else
{
// couldn't connect
}

I've had the same question as the OP, and tried every and each solution I was able to find on Internet...
Well, none of them worked to my true satisfaction, so I've opted for a research to find a reliable and responsive way of checking if connection to a MongoDB Database Server is alive. And this without to block the application's synchronous execution for too long time period...
So here are my prerequisites:
Synchronous processing of the connection check
Short to very short time slice for the connection check
Reliability of the connection check
If possible, not throwing exceptions and not triggering timeouts
I've provided a fresh MongoDB Installation (version 3.6) on the default localhost URL: mongodb://localhost:27017. I've also written down another URL, where there was no MongoDB Database Server: mongodb://localhost:27071.
I'm also using the C# Driver 2.4.4 and do not use the legacy implementation (MongoDB.Driver.Legacy assembly).
So my expectations are, when I'm checking the connection to the first URL, it should give to me the Ok for a alive connection to an existing MongoDB server, when I'm checking the connection to the second URL it should give to me the Fail for a non-existing MongoDB server...
Using the IMongoDatabase.RunCommand method, queries the server and causes the server response timeout to elapse, thus not qualifying against the prerequisites. Furthermore after the timeout, it breaks with a TimeoutException, which requires additional exception handling.
This actual SO question and also this SO question have delivered the most of the start information I needed for my solution... So guys, many thanks for this!
Now my solution:
private static bool ProbeForMongoDbConnection(string connectionString, string dbName)
{
var probeTask =
Task.Run(() =>
{
var isAlive = false;
var client = new MongoDB.Driver.MongoClient(connectionString);
for (var k = 0; k < 6; k++)
{
client.GetDatabase(dbName);
var server = client.Cluster.Description.Servers.FirstOrDefault();
isAlive = (server != null &&
server.HeartbeatException == null &&
server.State == MongoDB.Driver.Core.Servers.ServerState.Connected);
if (isAlive)
{
break;
}
System.Threading.Thread.Sleep(300);
}
return isAlive;
});
probeTask.Wait();
return probeTask.Result;
}
The idea behind this is the MongoDB Server does not react (and seems to be non-existing) until a real attempt is made to access some resource on the server (for example a database). But retrieving some resource alone is not enough, as the server still has no updates to its state in the server's Cluster Description. This update comes first, when the resource is retrieved again. From this time point, the server has valid Cluster Description and valid data inside it...
Generally it seems to me, the MongoDB Server does not proactivelly propagate its Cluster Description to all connected clients. Rather then, each client receives the description, when a request to the server has been made. If some of you fellows have more information on this, please either confirm or deny my understandings on the topic...
Now when we target an invalid MongoDB Server URL, then the Cluster Description remains invalid and we can catch and deliver an usable signal for this case...
So the following statements (for the valid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
var isAlive = ProbeForMongoDbConnection("mongodb://localhost:27017", "admin");
Console.WriteLine("Connection to mongodb://localhost:27017 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27017 was successful!
and the statements (for the invalid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
isAlive = ProbeForMongoDbConnection("mongodb://localhost:27071", "admin");
Console.WriteLine("Connection to mongodb://localhost:27071 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27071 was NOT successful!

Here a simple extension method to ping mongodb server
public static class MongoDbExt
{
public static bool Ping(this IMongoDatabase db, int secondToWait = 1)
{
if (secondToWait <= 0)
throw new ArgumentOutOfRangeException("secondToWait", secondToWait, "Must be at least 1 second");
return db.RunCommandAsync((Command<MongoDB.Bson.BsonDocument>)"{ping:1}").Wait(secondToWait * 1000);
}
}
You can use it like so:
var client = new MongoClient("yourConnectionString");
var database = client.GetDatabase("yourDatabase");
if (!database.Ping())
throw new Exception("Could not connect to MongoDb");

This is a solution by using the try-catch approach,
var database = client.GetDatabase("YourDbHere");
bool isMongoConnected;
try
{
await database.RunCommandAsync((Command<BsonDocument>)"{ping:1}");
isMongoConnected = true;
}
catch(Exception)
{
isMongoConnected = false;
}
so when it fails to connect to the database, it will throw an exception and we can handle our bool flag there.

If you want to handle connection issues in your program you can use the ICluster.Description event.
When the MongoClient is created, it will continue to attempt connections in the background until it succeeds.
using MongoDB.Driver;
using MongoDB.Driver.Core.Clusters;
var mongoClient = new MongoClient("localhost")
mongoClient.Cluster.DescriptionChanged += Cluster_DescriptionChanged;
public void Cluster_DescriptionChanged(object sender, ClusterDescriptionChangedEventArgs e)
{
switch (e.NewClusterDescription.State)
{
case ClusterState.Disconnected:
break;
case ClusterState.Connected:
break;
}
}

ZeroMQ performance issue

I'm having an issue with ZeroMQ, which I believe is because I'm not very familiar with it.
I'm trying to build a very simple service where multiple clients connect to a server and sends a query. The server responds to this query.
When I use REQ-REP socket combination (client using REQ, server binding to a REP socket) I'm able to get close to 60,000 messages per second at server side (when client and server are on the same machine). When distributed across machines, each new instance of client on a different machine linearly increases the messages per second at the server and easily reaches 40,000+ with enough client instances.
Now REP socket is blocking, so I followed ZeroMQ guide and used the rrbroker pattern (http://zguide.zeromq.org/cs:rrbroker):
REQ (client) <----> [server ROUTER -- DEALER --- REP (workers running on different threads)]
However, this completely screws up the performance. I'm getting only around 4000 messages per second at the server when running across machines. Not only that, each new client started on a different machine reduces the throughput of every other client.
I'm pretty sure I'm doing something stupid. I'm wondering if ZeroMQ experts here can point out any obvious mistakes. Thanks!
Edit: Adding code as per advice. I'm using the clrzmq nuget package (https://www.nuget.org/packages/clrzmq-x64/)
Here's the client code. A timer counts how many responses are received every second.
for (int i = 0; i < numTasks; i++) { Task.Factory.StartNew(() => Client(), TaskCreationOptions.LongRunning); }
void Client()
{
using (var ctx = new Context())
{
Socket socket = ctx.Socket(SocketType.REQ);
socket.Connect("tcp://192.168.1.10:1234");
while (true)
{
socket.Send("ping", Encoding.Unicode);
string res = socket.Recv(Encoding.Unicode);
}
}
}
Server - case 1: The server keeps track of how many requests are received per second
using (var zmqContext = new Context())
{
Socket socket = zmqContext.Socket(SocketType.REP);
socket.Bind("tcp://*:1234");
while (true)
{
string q = socket.Recv(Encoding.Unicode);
if (q.CompareTo("ping") == 0) {
socket.Send("pong", Encoding.Unicode);
}
}
}
With this setup, at server side, I can see around 60,000 requests received per second (when client is on the same machine). When on different machines, each new client increases number of requests received at server as expected.
Server Case 2: This is essentially rrbroker from ZMQ guide.
void ReceiveMessages(Context zmqContext, string zmqConnectionString, int numWorkers)
{
List<PollItem> pollItemsList = new List<PollItem>();
routerSocket = zmqContext.Socket(SocketType.ROUTER);
try
{
routerSocket.Bind(zmqConnectionString);
PollItem pollItem = routerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += RouterSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
dealerSocket = zmqContext.Socket(SocketType.DEALER);
try
{
dealerSocket.Bind("inproc://workers");
PollItem pollItem = dealerSocket.CreatePollItem(IOMultiPlex.POLLIN);
pollItem.PollInHandler += DealerSocket_PollInHandler;
pollItemsList.Add(pollItem);
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("{0}", ze.Message);
return;
}
// Start the worker pool; cant connect
// to inproc socket before binding.
workerPool.Start(numWorkers);
while (true)
{
zmqContext.Poll(pollItemsList.ToArray());
}
}
void RouterSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(routerSocket, dealerSocket);
}
void DealerSocket_PollInHandler(Socket socket, IOMultiPlex revents)
{
RelayMessage(dealerSocket, routerSocket);
}
void RelayMessage(Socket source, Socket destination)
{
bool hasMore = true;
while (hasMore)
{
byte[] message = source.Recv();
hasMore = source.RcvMore;
destination.Send(message, message.Length, hasMore ? SendRecvOpt.SNDMORE : SendRecvOpt.NONE);
}
}
Where the worker pool's start method is:
public void Start(int numWorkerTasks=8)
{
for (int i = 0; i < numWorkerTasks; i++)
{
QueryWorker worker = new QueryWorker(this.zmqContext);
Task task = Task.Factory.StartNew(() =>
worker.Start(),
TaskCreationOptions.LongRunning);
}
Console.WriteLine("Started {0} with {1} workers.", this.GetType().Name, numWorkerTasks);
}
public class QueryWorker
{
Context zmqContext;
public QueryWorker(Context zmqContext)
{
this.zmqContext = zmqContext;
}
public void Start()
{
Socket socket = this.zmqContext.Socket(SocketType.REP);
try
{
socket.Connect("inproc://workers");
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not create worker, error: {0}", ze.Message);
return;
}
while (true)
{
try
{
string message = socket.Recv(Encoding.Unicode);
if (message.CompareTo("ping") == 0)
{
socket.Send("pong", Encoding.Unicode);
}
}
catch (ZMQ.Exception ze)
{
Console.WriteLine("Could not receive message, error: " + ze.ToString());
}
}
}
}

Could you post some source code or at least a more detailed explanation of your test case? In general the way to build out your design is to make one change at a time, and measure at each change. You can always move stepwise from a known working design to more complex ones.

Most probably the 'ROUTER' is the bottleneck.
Check out these related questions on this:
Client maintenance in ZMQ ROUTER
Load testing ZeroMQ (ZMQ_STREAM) for finding the maximum simultaneous users it can handle
ROUTER (and ZMQ_STREAM, which is just a variant of ROUTER) internally has to maintain the client mapping, hence IMO it can accept limited connections from a particular client. It looks like ROUTER can multiplex multiple clients, only as long as, each client has only one active connection.
I could be wrong here - but I am not seeing much proof to the contrary (simple working code that scales to multi-clients with multi-connections with ROUTER or STREAM).
There certainly is a very severe restriction on concurrent connections with ZeroMQ, though it looks like no one know what is causing it.

I have done done performance testing on calling a native unmanaged DLL function with various methods from C#:
1. C++/CLI wrapper
2. PInvoke
3. ZeroMQ/clrzmq
The last might be interesting for you.
My finding at the end of my performance test was that using the ZMQ binding clrzmq was not useful and produced a factor of 100 performance overhead after I tried to optimize the PInvoke calls within the source code of the binding. Therefore I have used the ZMQ without a binding but with PInvoke calls.these calls must be done with the cdecl convention and with the option "SuppressUnmanagedCodeSecurity" to get most speed.
I had to import just 5 functions which was fairly easy.
At the end the speed was a bit slower than a PInvoke call but with the ZMQ-in my case over "inproc".
This may give you the hint to try it without the binding, if speed is interesting for you.
This is not a direct answer for your question but may help you to increase performance in general.

IPGlobalProperties.GetIPGlobalProperties

I wrote a test app to get all active ports on my network. Did some searching and found this was the easiest way. So I tried it and it work just fine. I then wrote another socket app with a sever and client side. It's pretty basic, has a create sever, join server and refresh button to get the active servers. The only time this method gets called is when you press the refresh button. If I open up the application 3 or more times and create a server with connected clients by the 4th one this method starts giving me this (Unknown error (0xc0000001)) error. Any idea why this could happen? Funny thing is I never get this on the initial application, the one I opened first. I don't know if somehow it get's a lock on this or something.
The exception gets thrown at this line:
IPEndPoint[] endPoints = properties.GetActiveTcpListeners();
Here's the method, it returns an object of List for all ports within a min and max range.
public static List<UserLocalSettings> ShowActiveTcpListeners(int min, int max)
{
List<UserLocalSettings> res = new List<UserLocalSettings>();
try
{
IPGlobalProperties properties = IPGlobalProperties.GetIPGlobalProperties();
IPEndPoint[] endPoints = properties.GetActiveTcpListeners();
foreach (IPEndPoint e in endPoints)
{
if (e.Port > (min - 1) && e.Port < (max + 1))
{
UserLocalSettings tmpClnt = new UserLocalSettings();
tmpClnt.player_ip = e.Address.ToString();
tmpClnt.player_port = e.Port;
tmpClnt.computer_name = Dns.GetHostEntry(e.Address).HostName;
res.Add(tmpClnt);
}
}
}
catch (Exception ex1)
{
}
return res;
}
Here's a screen print of the exception:

PooledRedisClientManager not releasing connections

I am storing lists of json data in redis and accessing it using the ServiceStack c# client. I am essentially managing my own foreign keys, where I store a zrange of ids and I use an interface internal to my application to pull the id's from the zrange and then fetch the underlying json objects from Redis and package them as a list to return to other parts of my application.
I am using the PooledRedisClientManager as I anticipate Redis to be hosted on a different server from the server executing the code.
I am doing all my development work locally on Windows 8, using the MSOpenTech Redis server. Currently my biggest challenge is that client connections are not being closed.
My Redis persister is being injected with an instance of IRedisClientManager (IoC is CastleWindsor). This code executes in the context of an azure worker role.
This is how I am fetching items from a zrange:
public class MyRedisPersister<T> : IResourcePersister<T>
{
IRedisClientManager _mgr;
public MyRedisPersister(IRedisClientManager mgr)
{
_mgr = mgr;
}
public IResourceList<T> Get<T>(string key, int offset, int count) where T
{
using (var redis = _clientManager.GetClient())
{
var itemKeys = redis.GetRangeFromSortedSet(key, offset, offset + count - 1).ToList();
var totalItems = redis.GetSortedSetCount(key);
if (itemKeys.Count == 0)
{
return new ResourceList<T>
{
Items = new List<T>(),
Offset = 0,
PageSize = 0,
TotalItems = 0,
TotalPages = 0
};
}
else
{
return new ResourceList<T>
{
Items = itemKeys.Select(k => redis.Get<T>(k)).ToList(),
Offset = offset,
PageSize = count,
TotalItems = totalItems,
TotalPages = (int) Math.Ceiling((float) totalItems/count)
};
}
}
}
}
This is the code I use to register the IRedisClientManager
var mgr = new PooledRedisClientManager(100, 10, "localhost:6379");
container.Register(Component.For<IRedisClientsManager>().Instance(mgr).LifeStyle.Singleton);
Any help would be greatly appreciated.

Currently my biggest challenge is that client connections are not being closed.
You are using the 'PooledRedisClientManager' so my understanding is that the client connections should not be closed, just put into the pool for reuse. It looks like your pool size is 100 connections.
You can try using
var mgr = new BasicRedisClientManager("localhost:6379")
which should dispose of the client.

edit The below approach is not recommended - you should take a dependency on the IRedisClientsManager and wrap all redis client calls inside a using() block, otherwise you will be bitten by gremlins.
I've been having similar problems getting Windsor to play nicely with the PooledRedisClientsManager, in the end this seemed to work:
container.Register(
Component.For<IRedisClientsManager>()
.Instance(redisClients)
.LifestyleSingleton(),
Component.For<IRedisClient>()
.UsingFactoryMethod(c => c.Resolve<IRedisClientsManager>().GetClient(),
managedExternally: true));
}
The managedExternally parameter tells Windsor to not try to apply decommissioning concerns to the IRedisClients and let the PooledRedisClientsManager handle recycling.