I've been playing with StreamInsight v2.3 and the newer Rx capabilities it provides. I'm investigating the use of SI for an Event Sourcing implementation. I've tweaked some of the MSDN sample code to get the following:
code for the server process:
using (var server = Server.Create("Default"))
{
var host = new ServiceHost(server.CreateManagementService());
host.AddServiceEndpoint(typeof(IManagementService), new WSHttpBinding(SecurityMode.Message), "http://localhost/SIDemo");
host.Open();
var myApp = server.CreateApplication("SIDemoApp");
var mySource = myApp.DefineObservable(() => Observable.Interval(TimeSpan.FromSeconds(1))).ToPointStreamable(x => PointEvent.CreateInsert(DateTimeOffset.Now, x), AdvanceTimeSettings.StrictlyIncreasingStartTime);
mySource.Deploy("demoSource");
Console.WriteLine("Hit enter to stop.");
Console.ReadLine();
host.Close();
}
code for the client process:
using (var server = Server.Connect(new System.ServiceModel.EndpointAddress(#"http://localhost/SIDemo")))
{
var myApp = server.Applications["SIDemoApp"];
var mySource = myApp.GetObservable<long>("demoSource");
using (var mySink = mySource.Subscribe(x => Console.WriteLine("Output - {0}", x)))
{
Console.WriteLine("Hit enter to stop.");
Console.ReadLine();
}
}
Trying to run this produces the following error:
Reading from a remote
'System.Reactive.Linq.IQbservable`1[System.Int64]' is not supported.
Use the 'Microsoft.ComplexEventProcessing.Linq.RemoteProvider.Bind'
method to read from the source using a remote observer.
The sample code I started with defines an observer and sink and binds it in the StreamInsight server. I'm trying to keep the observer in the client process. Is there a way to set up an observer in the client app for a remote StreamInsight source? Does this have to be done through something like a WCF endpoint in the server that is observed by the client?
Actualy error is directing to the solution. You need 'bind' to the source.
Please check the snippet below:
//Get SOURCE from server
var serverSource = myApp.GetStreamable<long>("demoSource");
//SINK for 'demoSource'
var sinkToBind = myApp.DefineObserver<long>( ()=> Observer.Create<long>( value => Console.WriteLine( "From client : " + value)));
//BINDING
var processForSink = serverSource.Bind(sinkToBind).Run("processForSink");
Also note that, sink will run on server, not like I guessed at first that it will run on client. If you look to console apps for both server and client, console output is writing to server app.
If even there is a way to run sink on client, I don't know and I like to know that too.
Related
I'm building a WebAPI project. I have DynamoDB downloaded and running locally with -inMemory. After creating a couple of tables, I run this command locally: aws dynamodb list-tables --endpoint-url http://localhost:8000, which results in:
{
"TableNames": [
"Users",
"Tmp"
]
}
When I run my application, I create a client, and query for tables:
using (var client = DatabaseClientFactory.CreateClient())
{
// debugging
var temp = await client.ListTablesAsync();
return $"{temp.TableNames.Count} tables: " + string.Join(", ", temp.TableNames);
}
This returns 0 tables:
The DB client is a lightly-modified version of this starter code:
using System;
using System.Net;
using System.Net.NetworkInformation;
using Amazon;
using Amazon.DynamoDBv2;
namespace Database
{
// Adapted from https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GettingStarted.NET.01.html
public static class DatabaseClientFactory
{
/*-----------------------------------------------------------------------------------
* If you are creating a client for the Amazon DynamoDB service, make sure your credentials
* are set up first, as explained in:
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SettingUp.DynamoWebService.html,
*
* If you are creating a client for DynamoDBLocal (for testing purposes),
* DynamoDB-Local should be started first. For most simple testing, you can keep
* data in memory only, without writing anything to disk. To do this, use the
* following command line:
*
* java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -inMemory
*
* For information about DynamoDBLocal, see:
* https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html.
*-----------------------------------------------------------------------------------*/
private const int Port = 8000;
public static AmazonDynamoDBClient CreateClient()
{
// If this line throws, you need to download the AWS CLI, run `aws configure` and specify your access key.
var client = new AmazonDynamoDBClient();
var dynamoDbConfig = new AmazonDynamoDBConfig();
// First, check to see whether anyone is listening on the DynamoDB local port
// (by default, this is port 8000, so if you are using a different port, modify this accordingly)
var portAvailable = IsPortAvailable();
if (portAvailable)
{
Console.WriteLine(" -- The local instance of DynamoDB is NOT running. Using PROD.");
// TODO: this should come out of appSettings.Production.json
dynamoDbConfig.RegionEndpoint = RegionEndpoint.USEast2;
}
else
{
// Local address ignores AWS credentials
Console.WriteLine(" -- The local instance of DynamoDB is running!");
dynamoDbConfig.ServiceURL = $"http://localhost:{Port}";
}
client = new AmazonDynamoDBClient(dynamoDbConfig);
return client;
}
private static bool IsPortAvailable()
{
// Evaluate current system TCP connections. This is the same information provided
// by the netstat command line application, just in .Net strongly-typed object
// form. We will look through the list, and if our port we would like to use
// in our TcpClient is occupied, we will set isAvailable to false.
IPGlobalProperties ipGlobalProperties = IPGlobalProperties.GetIPGlobalProperties();
IPEndPoint[] tcpConnInfoArray = ipGlobalProperties.GetActiveTcpListeners();
foreach (IPEndPoint endpoint in tcpConnInfoArray)
{
if (endpoint.Port == Port)
{
return false;
}
}
return true;
}
}
}
That's odd - my two tables don't appear.
Even more strange: if I make calls like client.CreateTableAsync, they successfully create a table, and the return value of my API call becomes 1 tables: CREATED_IN_CODE
It seems like I have two versions of DynamoDB running locally on the same port - one the application accesses, and one that the CLI accesses.
Why is my app seeing a wrong/different list of tables?
Some other things I tried to no avail:
Nuking/reinstalling DynamoDB
Restarting my PC
Reverting to a previous version of my code from earlier today
Looking for the CREATED_IN_CODE table in AWS in different regions
The above code worked fine for me. Sometimes you need to be sure there is a region set, even though you are setting a ServiceUrl.
When I start my program, I run the ElasticSearch Service and check if an Index exists and if there is any documents, let's say I just run the ES service and I have these two functions:
public ElasticClient getElasticSearchClient()
{
ConnectionSettings connectionSettings = new Nest.ConnectionSettings(new Uri("http://localhost:9200"))
.DefaultIndex("myindex")
.DisableDirectStreaming();
ElasticClient client = new ElasticClient(connectionSettings);
//var health = client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)).Timeout(50));
return client;
}
public void checkElasticsearchIndex()
{
var client = getElasticSearchClient();
var health = this.client.Cluster.Health("myindex", a => (a.WaitForStatus(WaitForStatus.Yellow)));
CountResponse count = client.Count<myobject>();
if (!client.Indices.Exists("myindex").IsValid || count.Count == 0)
{
BulkWriteAllToIndexES(client);
}
}
Inside the checkElasticsearchIndex function,
The count operation fails with the following error message:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: GET /myindex/_count. ServerError: Type: search_phase_execution_exception Reason: "all shards failed" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
The Health fails as well:
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Unable to connect to the remote server. Call: Status code unknown from: GET /_cluster/health/myindex?wait_for_status=yellow ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:9200
As you can see, I have tried the Cluster WaitForStatus, but it didn't work.
My question: is there any way to wait until client/cluster/nodes are ready and not get any exception?
It sounds like you're starting the Elasticsearch process at the same time as starting your program, but Elasticsearch takes longer than your program to be ready.
If that's the case, you may be interested in using the same abstractions that the .NET client uses for integration tests against Elasticsearch. The abstractions read output from the Elasticsearch process to know when it is ready, and block until this happens. They're available on an AppVeyor CI package feed (with plans to release them to Nuget in the future).
There are some examples of how to spin up a cluster with the abstractions. For single node, it would be something like
using System;
using Elastic.Managed.Configuration;
using Elastic.Managed.ConsoleWriters;
using Elastic.Managed.FileSystem;
namespace Elastic.Managed.Example
{
class Program
{
static void Main(string[] args)
{
var version = "7.5.1";
var esHome = Environment.ExpandEnvironmentVariables($#"%LOCALAPPDATA%\ElasticManaged\{version}\elasticsearch-{version}");
using (var node = new ElasticsearchNode(version, esHome))
{
node.SubscribeLines(new LineHighlightWriter());
if (!node.WaitForStarted(TimeSpan.FromMinutes(2))) throw new Exception();
// do your work here
}
}
}
}
This assumes that Elasticsearch 7.5.1 zip has been downloaded already, and exists at %LOCALAPPDATA%\ElasticManaged\7.5.1\elasticsearch-7.5.1. There are more complex examples of how to integrate this into tests with xUnit.
You can use the EphemeralCluster components to download, configure and run Elasticsearch
var plugins = new ElasticsearchPlugins(ElasticsearchPlugin.RepositoryAzure, ElasticsearchPlugin.IngestAttachment);
var config = new EphemeralClusterConfiguration("7.5.1", ClusterFeatures.XPack, plugins, numberOfNodes: 1);
using (var cluster = new EphemeralCluster(config))
{
cluster.Start();
var nodes = cluster.NodesUris();
var connectionPool = new StaticConnectionPool(nodes);
var settings = new ConnectionSettings(connectionPool).EnableDebugMode();
var client = new ElasticClient(settings);
Console.Write(client.CatPlugins().DebugInformation);
}
I use MongoDB drivers to connect to the database. When my form loads, I want to set up connection and to check whether it is ok or not. I do it like this:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
var database = server.GetDatabase("reestr");
But I do not know how to check connection. I tried to overlap this code with try-catch, but to no avail. Even if I make an incorrect connectionString, I still can not get any error message.
To ping the server with the new 3.0 driver its:
var database = client.GetDatabase("YourDbHere");
database.RunCommandAsync((Command<BsonDocument>)"{ping:1}")
.Wait();
There's a ping method for that:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
server.Ping();
full example for 2.4.3 - where "client.GetServer()" isn't available.
based on "Paul Keister" answer.
client = new MongoClient("mongodb://localhost");
database = client.GetDatabase(mongoDbStr);
bool isMongoLive = database.RunCommandAsync((Command<BsonDocument>)"{ping:1}").Wait(1000);
if(isMongoLive)
{
// connected
}
else
{
// couldn't connect
}
I've had the same question as the OP, and tried every and each solution I was able to find on Internet...
Well, none of them worked to my true satisfaction, so I've opted for a research to find a reliable and responsive way of checking if connection to a MongoDB Database Server is alive. And this without to block the application's synchronous execution for too long time period...
So here are my prerequisites:
Synchronous processing of the connection check
Short to very short time slice for the connection check
Reliability of the connection check
If possible, not throwing exceptions and not triggering timeouts
I've provided a fresh MongoDB Installation (version 3.6) on the default localhost URL: mongodb://localhost:27017. I've also written down another URL, where there was no MongoDB Database Server: mongodb://localhost:27071.
I'm also using the C# Driver 2.4.4 and do not use the legacy implementation (MongoDB.Driver.Legacy assembly).
So my expectations are, when I'm checking the connection to the first URL, it should give to me the Ok for a alive connection to an existing MongoDB server, when I'm checking the connection to the second URL it should give to me the Fail for a non-existing MongoDB server...
Using the IMongoDatabase.RunCommand method, queries the server and causes the server response timeout to elapse, thus not qualifying against the prerequisites. Furthermore after the timeout, it breaks with a TimeoutException, which requires additional exception handling.
This actual SO question and also this SO question have delivered the most of the start information I needed for my solution... So guys, many thanks for this!
Now my solution:
private static bool ProbeForMongoDbConnection(string connectionString, string dbName)
{
var probeTask =
Task.Run(() =>
{
var isAlive = false;
var client = new MongoDB.Driver.MongoClient(connectionString);
for (var k = 0; k < 6; k++)
{
client.GetDatabase(dbName);
var server = client.Cluster.Description.Servers.FirstOrDefault();
isAlive = (server != null &&
server.HeartbeatException == null &&
server.State == MongoDB.Driver.Core.Servers.ServerState.Connected);
if (isAlive)
{
break;
}
System.Threading.Thread.Sleep(300);
}
return isAlive;
});
probeTask.Wait();
return probeTask.Result;
}
The idea behind this is the MongoDB Server does not react (and seems to be non-existing) until a real attempt is made to access some resource on the server (for example a database). But retrieving some resource alone is not enough, as the server still has no updates to its state in the server's Cluster Description. This update comes first, when the resource is retrieved again. From this time point, the server has valid Cluster Description and valid data inside it...
Generally it seems to me, the MongoDB Server does not proactivelly propagate its Cluster Description to all connected clients. Rather then, each client receives the description, when a request to the server has been made. If some of you fellows have more information on this, please either confirm or deny my understandings on the topic...
Now when we target an invalid MongoDB Server URL, then the Cluster Description remains invalid and we can catch and deliver an usable signal for this case...
So the following statements (for the valid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
var isAlive = ProbeForMongoDbConnection("mongodb://localhost:27017", "admin");
Console.WriteLine("Connection to mongodb://localhost:27017 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27017 was successful!
and the statements (for the invalid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
isAlive = ProbeForMongoDbConnection("mongodb://localhost:27071", "admin");
Console.WriteLine("Connection to mongodb://localhost:27071 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27071 was NOT successful!
Here a simple extension method to ping mongodb server
public static class MongoDbExt
{
public static bool Ping(this IMongoDatabase db, int secondToWait = 1)
{
if (secondToWait <= 0)
throw new ArgumentOutOfRangeException("secondToWait", secondToWait, "Must be at least 1 second");
return db.RunCommandAsync((Command<MongoDB.Bson.BsonDocument>)"{ping:1}").Wait(secondToWait * 1000);
}
}
You can use it like so:
var client = new MongoClient("yourConnectionString");
var database = client.GetDatabase("yourDatabase");
if (!database.Ping())
throw new Exception("Could not connect to MongoDb");
This is a solution by using the try-catch approach,
var database = client.GetDatabase("YourDbHere");
bool isMongoConnected;
try
{
await database.RunCommandAsync((Command<BsonDocument>)"{ping:1}");
isMongoConnected = true;
}
catch(Exception)
{
isMongoConnected = false;
}
so when it fails to connect to the database, it will throw an exception and we can handle our bool flag there.
If you want to handle connection issues in your program you can use the ICluster.Description event.
When the MongoClient is created, it will continue to attempt connections in the background until it succeeds.
using MongoDB.Driver;
using MongoDB.Driver.Core.Clusters;
var mongoClient = new MongoClient("localhost")
mongoClient.Cluster.DescriptionChanged += Cluster_DescriptionChanged;
public void Cluster_DescriptionChanged(object sender, ClusterDescriptionChangedEventArgs e)
{
switch (e.NewClusterDescription.State)
{
case ClusterState.Disconnected:
break;
case ClusterState.Connected:
break;
}
}
I'm using the Microsoft MapReduce SDK to start a Mapper only job.
The call to hadoop.MapReduceJob.ExecuteJob is throwing a "Response status code does not indicate success: 404 (not found)" exception immediately.
When inspecting the HDInsight Query Console the job successfully starts and finishes later. It also writes proper output files.
My guess is, ExecuteJob is trying to access output data before the job has finished.
What is the correct way to handle this situation?
using System;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using Microsoft.WindowsAzure.Management.HDInsight;
using Microsoft.Hadoop.MapReduce;
using AzureAnalyzer.MultiAnalyzer;
namespace AzureAnalyzer
{
class Program
{
static void Main(string[] args)
{
IHadoop hadoop = Hadoop.Connect(Constants.azureClusterUri, Constants.clusterUser,
Constants.hadoopUser, Constants.clusterPassword, Constants.storageAccount,
Constants.storageAccountKey, Constants.container, true);
try {
var output = hadoop.MapReduceJob.ExecuteJob<MultiAnalyzerJob>();
}
catch (Exception ex)
{
Console.WriteLine("\nException: " + ex.Message);
}
}
}
}
I got another way to do the same thing but takes a bit of effort since it requires the mapper and reducer files to be transferred to the hadoop cluster storage.
you need to add Microsoft.Hadoop.Client then Microsoft Azure HDInsight NuGet package as well.
var jobcred = new BasicAuthCredential();
jobcred.UserName = "clusteruserid";
jobcred.Password = "clusterpassword";
jobcred.Server = new Uri("https://clusterurl");
StreamingMapReduceJobCreateParameters jobpara = new StreamingMapReduceJobCreateParameters()
{
JobName="mapreduce",
Mapper = "Mapper.exe",
Reducer = "Reducer.exe",
Input= "wasb:///mydata/input",
Output = "wasb:///mydata/Output",
StatusFolder= "wasb:///mydata/sOutput"
};
jobpara.Files.Add("wasb:///mydata/Mapper.exe");
jobpara.Files.Add("wasb:///mydata/Reducer.exe");
// Create a Hadoop client to connect to HDInsight.
var jobClient = JobSubmissionClientFactory.Connect(jobcred);
// Run the MapReduce job.
JobCreationResults mrJobResults = jobClient.CreateStreamingJob(jobpara);
// Wait for the job to complete.
Console.Write("Job running...");
JobDetails jobInProgress = jobClient.GetJob(mrJobResults.JobId);
while (jobInProgress.StatusCode != JobStatusCode.Completed
&& jobInProgress.StatusCode != JobStatusCode.Failed)
{
Console.Write(".");
jobInProgress = jobClient.GetJob(jobInProgress.JobId);
Thread.Sleep(TimeSpan.FromSeconds(10));
}
// Job is complete.
Console.WriteLine("!");
Console.WriteLine("Job complete!");
Console.WriteLine("Press a key to end.");
Console.Read();
Hope this helps. I was able to run jobs without throwing any exceptions.
this in fact waits for the job to be complete.
Please verify if all required services to run the program are up and running. The 404 error indicates that some URL the program tries to access internally is not accessible.
i'm trying to write a small chat server using .Net Remoting in C#.
The connection to the Server from the client works nicely, but as soon as i try to "broadcast" a sent message to all other clients (or even the same client), the server throws an exception.
"Der Remoteproxy hat keine Channelsenke, d. h., der Server besitzt keine registrierten Serverchannel oder die Anwendung hat keinen passenden Clientchannel, um mit dem Server zu kommunizieren."
what translates to
"This remoting proxy has no channel sink which means either the server has no registered server channels that are listening, or this application has no suitable client channel to talk to the server."
How would i change this?
Here ( https://www.dropbox.com/s/y40cdv3lopsf6d7/Chatprojekt.zip ) you can find a copy of the full project, but to help others who have the same problem, i'll explain what i did.
On the server i opened a TcpChannel on a specific port, made an interface for both,the server and the client. I connect to the server and with a method,i pass the client instance to the server.
Both implementations of the interfaces implement MarshalByRefObj to use Proxies for method invokation.
Thanks already a lot for helping me
I gave up on doing real .NET remoting in favor of WCF a long time ago. Recently I've been using RemotingLite and my own variant of it which also supports named pipes called DuoVia.Net. You simply write your interface contract and DTO's, share those in an assembly on client and server and write your implementation on the server side.
No client to produce. These libraries produce their own through reflection and dynamic assembly generation by emitting IL. Cool stuff. Have a look at how easy the client side is:
class Program
{
static void Main(string[] args)
{
var sw = Stopwatch.StartNew();
var ipEndpoint = new IPEndPoint(IPAddress.Parse("127.0.0.1"), 8098);
var from = 0;
var to = 500;
Parallel.For(from, to, index =>
{
using (var client = new NetTcpTesterProxy(ipEndpoint))
{
var id = client.GetId("test1", 3.314, 42);
var response = client.Get(id, "mirror", 4.123, 42);
var list = client.GetItems(id);
}
});
sw.Stop();
var msperop = sw.ElapsedMilliseconds / 1500.0;
Console.WriteLine("tcp: {0}, {1}", sw.ElapsedMilliseconds, msperop);
Thread.Sleep(5000);
var pipeName = "DuoViaTestHost";
sw = Stopwatch.StartNew();
Parallel.For(from, to, index =>
{
using (var client = new NetNpTesterProxy(new NpEndPoint(pipeName)))
{
var id = client.GetId("test1", 3.314, 42);
var response = client.Get(id, "mirror", 4.123, 42);
var list = client.GetItems(id);
}
});
sw.Stop();
msperop = sw.ElapsedMilliseconds / 1500.0;
Console.WriteLine("pip: {0}, {1}", sw.ElapsedMilliseconds, msperop);
Console.ReadLine();
}
}
Have fun!