We Keep Coding

Avoiding ElasticSearch error 503 Server Unavailable: Use of WaitForStatus

When I start my program, I run the ElasticSearch Service and check if an Index exists and if there is any documents, let's say I just run the ES service and I have these two functions:
public ElasticClient getElasticSearchClient()
{
ConnectionSettings connectionSettings = new Nest.ConnectionSettings(new Uri("http://localhost:9200"))
.DefaultIndex("myindex")
.DisableDirectStreaming();
ElasticClient client = new ElasticClient(connectionSettings);
//var health = client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)).Timeout(50));
return client;
}
public void checkElasticsearchIndex()
{
var client = getElasticSearchClient();
var health = this.client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)));
CountResponse count = client.Count<myobject>();
if (!client.Indices.Exists("myindex").IsValid || count.Count == 0)
{
BulkWriteAllToIndexES(client);
}
}
Inside the checkElasticsearchIndex function,
The count operation fails with the following error message:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: GET /myindex/_count. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
The Health fails as well:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Unable to connect to the remote server. Call: Status code unknown from: GET /_cluster/health/myindex?wait_for_status=yellow ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:9200
As you can see, I have tried the Cluster WaitForStatus, but it didn't work.
My question: is there any way to wait until client/cluster/nodes are ready and not get any exception?

It sounds like you're starting the Elasticsearch process at the same time as starting your program, but Elasticsearch takes longer than your program to be ready.
If that's the case, you may be interested in using the same abstractions that the .NET client uses for integration tests against Elasticsearch. The abstractions read output from the Elasticsearch process to know when it is ready, and block until this happens. They're available on an AppVeyor CI package feed (with plans to release them to Nuget in the future).
There are some examples of how to spin up a cluster with the abstractions. For single node, it would be something like
using System;
using Elastic.Managed.Configuration;
using Elastic.Managed.ConsoleWriters;
using Elastic.Managed.FileSystem;
namespace Elastic.Managed.Example
{
class Program
{
static void Main(string[] args)
{
var version = "7.5.1";
var esHome = Environment.ExpandEnvironmentVariables($#"%LOCALAPPDATA%\ElasticManaged\{version}\elasticsearch-{version}");
using (var node = new ElasticsearchNode(version, esHome))
{
node.SubscribeLines(new LineHighlightWriter());
if (!node.WaitForStarted(TimeSpan.FromMinutes(2))) throw new Exception();
// do your work here
}
}
}
}
This assumes that Elasticsearch 7.5.1 zip has been downloaded already, and exists at %LOCALAPPDATA%\ElasticManaged\7.5.1\elasticsearch-7.5.1. There are more complex examples of how to integrate this into tests with xUnit.
You can use the EphemeralCluster components to download, configure and run Elasticsearch
var plugins = new ElasticsearchPlugins(ElasticsearchPlugin.RepositoryAzure, ElasticsearchPlugin.IngestAttachment);
var config = new EphemeralClusterConfiguration("7.5.1", ClusterFeatures.XPack, plugins, numberOfNodes: 1);
using (var cluster = new EphemeralCluster(config))
{
cluster.Start();
var nodes = cluster.NodesUris();
var connectionPool = new StaticConnectionPool(nodes);
var settings = new ConnectionSettings(connectionPool).EnableDebugMode();
var client = new ElasticClient(settings);
Console.Write(client.CatPlugins().DebugInformation);
}

non-seed node fails to join. Lighthouse error Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient

I am using Lighthouse as my seed node and I am writing my first cluster node. I am very new to akka cluster so don't know what I am doing wrong. I have lighthouse running and a non-seed node trying to join and get the error message
Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient.
Tried various configurations with no luck.
I made sure the Actor System name was the same in both lighthouse and non-seed node. See code and HOCON's
non-seed node Actor System creation:
public ClusterActorSystem()
{
system = ActorSystem.Create("ThisIsMyName");
Console.WriteLine($"The Actor System Name is {system.Name}");
}
Using this HOCON on Lighthouse
<pre>
lighthouse{
actorsystem: "ThisIsMyName" #POPULATE NAME OF YOUR ACTOR SYSTEM HERE
}
# See petabridge.cmd configuration options here:
https://cmd.petabridge.com/articles/install/host-configuration.html
petabridge.cmd{
# default IP address used to listen for incoming petabridge.cmd client connections
# should be a safe default as it listens on "all network interfaces".
host = "0.0.0.0"
# default port number used to listen for incoming petabridge.cmd client
connections
port = 9110
}
akka {
actor {
provider = cluster
}
remote {
log-remote-lifecycle-events = DEBUG
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
#will be populated with a dynamic host-name at runtime if left uncommented
#public-hostname = ""
hostname = "0.0.0.0"
port = 19621
}
}
cluster {
#will inject this node as a self-seed node at run-time
seed-nodes = []
roles = [lighthouse]
}
}
</pre>
Using this HOCON on Non-See Node
<akka>
<hocon>
<![CDATA[
akka {
log-remote-lifecycle-events = DEBUG
actor.provider = cluster
remote {
dot-netty.tcp {
transport-class = "Akka.Remote.Transport.DotNetty.TcpTransport, Akka.Remote"
applied-adapters = []
transport-protocol = tcp
port = 15245
hostname = localhost
}
}
cluster {
seed-nodes = ["akka.tcp://ThisIsMyName#localhost:19621"]
#roles = ["MyFirstRole"]
}
}
]]>
</hocon>
</akka>
Lighthouse Log Messages:
[ERROR][6/7/2019 4:42:50 PM][Thread 0011][[akka://ThisIsMyName/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FThisIsMyName%40localhost%3A51281-2/endpointWriter#1992505106]] Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient [[akka.tcp://ThisIsMyName#localhost:19621/]] arriving at [akka.tcp://ThisIsMyName#localhost:19621] inbound addresses [akka.tcp://ThisIsMyName#7815-pc1:19621]
Cause: Unknown
[ERROR][6/7/2019 4:42:55 PM][Thread 0009][[akka://ThisIsMyName/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FThisIsMyName%40localhost%3A51281-2/endpointWriter#1992505106]] Dropping message [Akka.Actor.ActorSelectionMessage] for non-local recipient [[akka.tcp://ThisIsMyName#localhost:19621/]] arriving at [akka.tcp://ThisIsMyName#localhost:19621] inbound addresses [akka.tcp://ThisIsMyName#7815-pc1:19621]
Cause: Unknown
non-seed node log messages:
INFO][6/7/2019 5:13:10 PM][Thread 0001][remoting] Starting remoting
[INFO][6/7/2019 5:13:11 PM][Thread 0001][remoting] Remoting started; listening on addresses : [akka.tcp://ThisIsMyName#localhost:15245]
[INFO][6/7/2019 5:13:11 PM][Thread 0001][remoting] Remoting now listens on addresses: [akka.tcp://ThisIsMyName#localhost:15245]
[INFO][6/7/2019 5:13:11 PM][Thread 0001][Cluster] Cluster Node [akka.tcp://ThisIsMyName#localhost:15245] - Starting up...
[INFO][6/7/2019 5:13:11 PM][Thread 0001][Cluster] Cluster Node [akka.tcp://ThisIsMyName#localhost:15245] - Started up successfully
The Actor System Name is ThisIsMyName
Actor System Started.
[WARNING][6/7/2019 5:13:21 PM][Thread 0003][[akka://ThisIsMyName/system/cluster/core/daemon/joinSeedNodeProcess-1#2129151453]] Couldn't join seed nodes after [2] attempts, will try again. seed-nodes=[akka.tcp://ThisIsMyName#localhost:19621]

Client IP address returns the same internal network address

I'm trying to get the user's IP address from ASP.NET MVC 5. I've looked up various examples, such as these:
https://stackoverflow.com/a/740431/177416
https://stackoverflow.com/a/20194511/177416
https://stackoverflow.com/a/3003254/177416
They've all produced the same result: the user is considered internal to the network. I've had friends try their phones (which are not on the network). Here's my latest attempt:
private static Logger _logger = LogManager.GetCurrentClassLogger();
public static bool IsIpInternal()
{
var ipAddress = HttpContext.Current.Request.UserHostAddress;
var logEvent = new LogEventInfo(LogLevel.Info, _logger.Name, ipAddress);
_logger.Log(logEvent);
try
{
if (ipAddress != null)
{
var ipParts = ipAddress.Split(new[] { "." }, StringSplitOptions.RemoveEmptyEntries)
.Select(int.Parse).ToArray();
var isDebug = System.Diagnostics.Debugger.IsAttached;
if (ipParts[0] == 10)
{
return true;
}
}
}
catch (Exception e)
{
logEvent = new LogEventInfo(LogLevel.Error, _logger.Name, e.Message);
_logger.Log(logEvent);
return false;
}
return false;
}
The log is showing 10.xxx.xx.xxx for all requests (based on the log). This is an internal address rather than the IP of the client connecting to the web app. The IsIpInternal() returns true always. What am I doing wrong?
Note that I'm ignoring 192.168.x.x and 172.16.xxx.xxx addresses as being internal.

If your web site is behind a load balancer, it is a common problem for the load balancer's IP address to appear when you are expecting the client's IP address. That is because in reality the load balancer is the only client that the web application knows is talking to it.
There are two ways to deal with this:
You can configure the load balancer to add an additional HTTP header (x-forwarded-for) that specifies the original IP address. You will need to modify your web site to look at this header instead of the UserHostAddress, like this:
//var clientIP = HttpContext.Current.Request.UserHostAddress;
var clientIP = HttpContext.Current.Request.Headers["x-forwarded-for"];
Note: The x-forwarded-for header can actually return a comma-delimited list of IP addresses in some cases. So to be compatible with such an occurence, you might write this instead:
var clientIP = HttpContext.Current.Request.Headers["x-forwarded-for"].Split(',')[0];
You can configure certain LBs to pass through the client IP by copying the IP header packet. For Citrix Netscaler, see this article for more information.

Grapevine REST server , I could not reach from other PC with IP or host name

I am using VS2012 and Grapevine 3.0.4 , when i use the Grapevine same machine with localhost
hostname , everything works well.
If I want to reach from other PC with client , Server could not be start listening with hostname ip address or Computername
If i try server pc set hostname to localhost , it starts listening but when reached from other PC with IP or name server returns bad request 400
Is it something wrong with my code or library.
My Server code is
public class embeddedHTTP
{
private RESTServer Server;
public void ServerStart()
{
try
{
Server = new RESTServer();
Server.Port = GlobalVars.HttpHostPort;
Server.Host = GlobalVars.HttpHostAdress; // THIS ONLY WORKS FOR LOCALHOST
Server.MaxThreads = 20;
Server.Start();
while (Server.IsListening)
{
Thread.Sleep(GlobalVars.HttpHostRespTime);
}
}
catch (Exception ex)
{
messenger.logque("embedded HTTP server not started, Error ID : 52", 3, null);
}
}
public void ServerStop()
{
Server.Stop();
}
public sealed class MyResource : RESTResource
{
//d+$^ [a-zA-Z]+
[RESTRoute(Method = Grapevine.HttpMethod.GET, PathInfo = #"/")]
public void HandleFooRequests(HttpListenerContext context)
{
//String RawuR = context.Request.RawUrl;
String URL = Convert.ToString(context.Request.Url);
String ResultXML = brain.HTTPCMD(URL);
this.SendTextResponse(context, ResultXML);
}
}
}

If you can't reach the server from a remote machine, you are likely running a firewall that is blocking inbound traffic to the port you are listening on. Try opening the port on your firewall, and see if that works for you.
How to Open a Port in the Windows 7 Firewall
Also, you can listen on all hosts by using the asterisk (*) as your hostname.

Unable to read data from the transport connection: the connection was closed

The exception is Remoting Exception - Authentication Failure. The detailed message says "Unable to read data from the transport connection: the connection was closed."
I'm having trouble with creating two simple servers that can comunicate as remote objects in C#. ServerInfo is just a class I created that holds the IP and Port and can give back the address. It works fine, as I used it before, and I've debugged it. Also the server is starting just fine, no exception is thrown, and the channel is registered without problems. I'm using Forms to do the interfaces, and call some of the methods on the server, but didn't find any problems in passing the parameters from the FormsApplication to the server when debugging. All seems fine in that chapter.
public ChordServerProgram()
{
RemotingServices.Marshal(this, "PADIBook");
nodeInt = 0;
}
public void startServer()
{
try
{
serverChannel = new TcpChannel(serverInfo.Port);
ChannelServices.RegisterChannel(serverChannel, true);
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
I run two instances of this program. Then startNode is called on one of the instances of the application. The port is fine, the address generated is fine as well. As you can see, I'm using the IP for localhost, since this server is just for testing purposes.
public void startNode(String portStr)
{
IPAddress address = IPAddress.Parse("127.0.0.1");
Int32 port = Int32.Parse(portStr);
serverInfo = new ServerInfo(address, port);
startServer();
//node = new ChordNode(serverInfo,this);
}
Then, in the other istance, through the interface again, I call another startNode method, giving it a seed server to get information from. This is where it goes wrong. When it calls the method on the seedServer proxy it just got, a RemotingException is thrown, due to an authentication failure. (The parameter I'll want to get is the node, I'm just using the int to make sure the ChordNode class has nothing to do with this error.)
public void startNode(String portStr, String seedStr)
{
IPAddress address = IPAddress.Parse("127.0.0.1");
Int32 port = Int32.Parse(portStr);
serverInfo = new ServerInfo(address, port);
IPAddress addressSeed = IPAddress.Parse("127.0.0.1");
Int32 portSeed = Int32.Parse(seedStr);
ServerInfo seedInfo = new ServerInfo(addressSeed, portSeed);
startServer();
ChordServerProgram seedServer = (ChordServerProgram)Activator.GetObject(typeof(ChordServerProgram), seedInfo.GetFullAddress());
// node = new ChordNode(serverInfo,this);
int seedNode = seedServer.nodeInt;
// node.chordJoin(seedNode.self);
}

Try setting the ensureSecurity to false, and it should start working.
ChannelServices.RegisterChannel(serverChannel, false);

You've specified that security is a must on your Remoting server in startServer() with:
ChannelServices.RegisterChannel(serverChannel, true);
Yet the 'client' end does not specify security, hence the authorisation error. You need to specify tcp channel security on both ends unless the server security setting is set to 'false'. In your second startNode method you need to do the following before using Activator.GetObject, note no port specified on the TcpChannel unlike the server end:
TcpChannel ClientChan = new TcpChannel();
ChannelServices.RegisterChannel(ClientChan, true);
Furthermore, unless you're doing it in some code you haven't given us, you also do not seem to have registered a well known service type server side, although you say it's been working in the debugger so maybe that's not necessary in the case. See MSDN on RegisterWellKnownServiceType.