We Keep Coding

Best way to check if ubuntu service has stopped in c#

I want to get an alert when a service (grafana or influxdb) in an Azure virtual machine (Ubuntu 16.04) has stopped. I'd like to use c# to connect to the VM and check the status of grafana and influxdb services. Can anyone share a code sample that implements this?

Both services provide health endpoints that can be used to check their status from a remote server. There's no need to open a remote shell connection. In fact, it would be impossible to monitor large server farms if one had to SSH to each one.
In the simplest case, and ignoring networking issues, one can simply hit the health endpoints to check the status of both services. A rough implementation could look like this :
public async Task<bool> CheckBoth()
{
var client = new HttpClient
{
Timeout = TimeSpan.FromSeconds(30)
};
const string grafanaHealthUrl = "https://myGrafanaURL/api/health";
const string influxPingUrl = "https://myInfluxURL/ping";
var (grafanaOK, grafanaError) = await CheckAsync(client, grafanaHealthUrl,
HttpStatusCode.OK, "Grafana error");
var (influxOK, influxError) = await CheckAsync(client, influxPingUrl,
HttpStatusCode.NoContent,"InfluxDB error");
if (!influxOK || !grafanaOK)
{
//Do something with the errors
return false;
}
return true;
}
public async Task<(bool ok, string result)> CheckAsync(HttpClient client,
string healthUrl,
HttpStatusCode expected,
string errorMessage)
{
try
{
var status = await client.GetAsync(healthUrl);
if (status.StatusCode != expected)
{
//Failure message, get it and log it
var statusBody = await status.Content.ReadAsStringAsync();
//Possibly log it ....
return (ok: false, result: $"{errorMessage}: {statusBody}");
}
}
catch (TaskCanceledException)
{
return (ok: false, result: $"{errorMessage}: Timeout");
}
return (ok: true, "");
}
Perhaps a better solution would be to use Azure Monitor to ping the health URLs periodically and send an alert if they are down.

Here is something you can use to connect to Azure linux using SSH in c#
using (var client = new SshClient("my-vm.cloudapp.net", 22, "username", "password​"))
{
client.Connect();
Console.WriteLine("it worked!");
client.Disconnect();
Console.ReadLine();
}
Usually SSH server only allow public key auth or other two factor auth.
Change your /etc/ssh/sshd_configuncomment #PasswordAuthentication yes
# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes
Later you can poll for installed services.
Also for an alternative solution, you can deploy a rest api in your linux VM to check the status of your service and the call it from C# httpclient for the status.
Hope it helps

How to check database connection in MongoDB [duplicate]

I use MongoDB drivers to connect to the database. When my form loads, I want to set up connection and to check whether it is ok or not. I do it like this:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
var database = server.GetDatabase("reestr");
But I do not know how to check connection. I tried to overlap this code with try-catch, but to no avail. Even if I make an incorrect connectionString, I still can not get any error message.

To ping the server with the new 3.0 driver its:
var database = client.GetDatabase("YourDbHere");
database.RunCommandAsync((Command<BsonDocument>)"{ping:1}")
.Wait();

There's a ping method for that:
var connectionString = "mongodb://localhost";
var client = new MongoClient(connectionString);
var server = client.GetServer();
server.Ping();

full example for 2.4.3 - where "client.GetServer()" isn't available.
based on "Paul Keister" answer.
client = new MongoClient("mongodb://localhost");
database = client.GetDatabase(mongoDbStr);
bool isMongoLive = database.RunCommandAsync((Command<BsonDocument>)"{ping:1}").Wait(1000);
if(isMongoLive)
{
// connected
}
else
{
// couldn't connect
}

I've had the same question as the OP, and tried every and each solution I was able to find on Internet...
Well, none of them worked to my true satisfaction, so I've opted for a research to find a reliable and responsive way of checking if connection to a MongoDB Database Server is alive. And this without to block the application's synchronous execution for too long time period...
So here are my prerequisites:
Synchronous processing of the connection check
Short to very short time slice for the connection check
Reliability of the connection check
If possible, not throwing exceptions and not triggering timeouts
I've provided a fresh MongoDB Installation (version 3.6) on the default localhost URL: mongodb://localhost:27017. I've also written down another URL, where there was no MongoDB Database Server: mongodb://localhost:27071.
I'm also using the C# Driver 2.4.4 and do not use the legacy implementation (MongoDB.Driver.Legacy assembly).
So my expectations are, when I'm checking the connection to the first URL, it should give to me the Ok for a alive connection to an existing MongoDB server, when I'm checking the connection to the second URL it should give to me the Fail for a non-existing MongoDB server...
Using the IMongoDatabase.RunCommand method, queries the server and causes the server response timeout to elapse, thus not qualifying against the prerequisites. Furthermore after the timeout, it breaks with a TimeoutException, which requires additional exception handling.
This actual SO question and also this SO question have delivered the most of the start information I needed for my solution... So guys, many thanks for this!
Now my solution:
private static bool ProbeForMongoDbConnection(string connectionString, string dbName)
{
var probeTask =
Task.Run(() =>
{
var isAlive = false;
var client = new MongoDB.Driver.MongoClient(connectionString);
for (var k = 0; k < 6; k++)
{
client.GetDatabase(dbName);
var server = client.Cluster.Description.Servers.FirstOrDefault();
isAlive = (server != null &&
server.HeartbeatException == null &&
server.State == MongoDB.Driver.Core.Servers.ServerState.Connected);
if (isAlive)
{
break;
}
System.Threading.Thread.Sleep(300);
}
return isAlive;
});
probeTask.Wait();
return probeTask.Result;
}
The idea behind this is the MongoDB Server does not react (and seems to be non-existing) until a real attempt is made to access some resource on the server (for example a database). But retrieving some resource alone is not enough, as the server still has no updates to its state in the server's Cluster Description. This update comes first, when the resource is retrieved again. From this time point, the server has valid Cluster Description and valid data inside it...
Generally it seems to me, the MongoDB Server does not proactivelly propagate its Cluster Description to all connected clients. Rather then, each client receives the description, when a request to the server has been made. If some of you fellows have more information on this, please either confirm or deny my understandings on the topic...
Now when we target an invalid MongoDB Server URL, then the Cluster Description remains invalid and we can catch and deliver an usable signal for this case...
So the following statements (for the valid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
var isAlive = ProbeForMongoDbConnection("mongodb://localhost:27017", "admin");
Console.WriteLine("Connection to mongodb://localhost:27017 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27017 was successful!
and the statements (for the invalid URL)
// The admin database should exist on each MongoDB 3.6 Installation, if not explicitly deleted!
isAlive = ProbeForMongoDbConnection("mongodb://localhost:27071", "admin");
Console.WriteLine("Connection to mongodb://localhost:27071 was " + (isAlive ? "successful!" : "NOT successful!"));
will print out
Connection to mongodb://localhost:27071 was NOT successful!

Here a simple extension method to ping mongodb server
public static class MongoDbExt
{
public static bool Ping(this IMongoDatabase db, int secondToWait = 1)
{
if (secondToWait <= 0)
throw new ArgumentOutOfRangeException("secondToWait", secondToWait, "Must be at least 1 second");
return db.RunCommandAsync((Command<MongoDB.Bson.BsonDocument>)"{ping:1}").Wait(secondToWait * 1000);
}
}
You can use it like so:
var client = new MongoClient("yourConnectionString");
var database = client.GetDatabase("yourDatabase");
if (!database.Ping())
throw new Exception("Could not connect to MongoDb");

This is a solution by using the try-catch approach,
var database = client.GetDatabase("YourDbHere");
bool isMongoConnected;
try
{
await database.RunCommandAsync((Command<BsonDocument>)"{ping:1}");
isMongoConnected = true;
}
catch(Exception)
{
isMongoConnected = false;
}
so when it fails to connect to the database, it will throw an exception and we can handle our bool flag there.

If you want to handle connection issues in your program you can use the ICluster.Description event.
When the MongoClient is created, it will continue to attempt connections in the background until it succeeds.
using MongoDB.Driver;
using MongoDB.Driver.Core.Clusters;
var mongoClient = new MongoClient("localhost")
mongoClient.Cluster.DescriptionChanged += Cluster_DescriptionChanged;
public void Cluster_DescriptionChanged(object sender, ClusterDescriptionChangedEventArgs e)
{
switch (e.NewClusterDescription.State)
{
case ClusterState.Disconnected:
break;
case ClusterState.Connected:
break;
}
}

How to read from multiple EventHub partitions simultaneously with high throughput?

My one role instance needs to read data from 20-40 EventHub partitions at the same time (context: this is our internal virtual partitioning scheme - 20-40 partitions represent scale out unit).
In my prototype I use below code. By I get throughput 8 MBPS max. Since if I run the same console multiple times I get throughput (perfmon counter) multiplied accordingly then I think this is not neither VM network limit nor EventHub service side limit.
I wonder whether I create clients correctly here...
Thank you!
Zaki
const string EventHubName = "...";
const string ConsumerGroupName = "...";
var connectionStringBuilder = new ServiceBusConnectionStringBuilder();
connectionStringBuilder.SharedAccessKeyName = "...";
connectionStringBuilder.SharedAccessKey = "...";
connectionStringBuilder.Endpoints.Add(new Uri("sb://....servicebus.windows.net/"));
connectionStringBuilder.TransportType = TransportType.Amqp;
var clientConnectionString = connectionStringBuilder.ToString();
var eventHubClient = EventHubClient.CreateFromConnectionString(clientConnectionString, EventHubName);
var runtimeInformation = await eventHubClient.GetRuntimeInformationAsync().ConfigureAwait(false);
var consumerGroup = eventHubClient.GetConsumerGroup(ConsumerGroupName);
var offStart = DateTime.UtcNow.AddMinutes(-10);
var offEnd = DateTime.UtcNow.AddMinutes(-8);
var workUnitManager = new WorkUnitManager(runtimeInformation.PartitionCount);
var readers = new List<PartitionReader>();
for (int i = 0; i < runtimeInformation.PartitionCount; i++)
{
var reader = new PartitionReader(
consumerGroup,
runtimeInformation.PartitionIds[i],
i,
offStart,
offEnd,
workUnitManager);
readers.Add(reader);
}
internal async Task Read()
{
try
{
Console.WriteLine("Creating a receiver for '{0}' with offset {1}", this.partitionId, this.startOffset);
EventHubReceiver receiver = await this.consumerGroup.CreateReceiverAsync(this.partitionId, this.startOffset).ConfigureAwait(false);
Console.WriteLine("Receiver for '{0}' has been created.", this.partitionId);
var stopWatch = new Stopwatch();
stopWatch.Start();
while (true)
{
var message =
(await receiver.ReceiveAsync(1, TimeSpan.FromSeconds(10)).ConfigureAwait(false)).FirstOrDefault();
if (message == null)
{
continue;
}
if (message.EnqueuedTimeUtc >= this.endOffset)
{
break;
}
this.processor.Push(this.partitionIndex, message);
}
this.Duration = TimeSpan.FromMilliseconds(stopWatch.ElapsedMilliseconds);
}
catch (Exception ex)
{
Console.WriteLine(ex);
throw;
}
}

The above code snippet you provided is effectively: creating 1 Connection to ServiceBus Service and then running all receivers on one single connection (at protocl level, essentially, creating multiple Amqp Links on that same connection).
Alternately - to achieve high throughput for receive operations, You will need to create multiple connections and map your receivers to connection ratio to fine-tune your throughput. That's what happens when you run the above code in multiple processes.
Here's how:
You will need to go one layer down the .Net client SDK API and code at MessagingFactory level - you can start with 1 MessagingFactory per EventHubClient. MessagingFactory is the one - which represents 1 Connection to EventHubs service. Code to create a dedicated connection per EventHubClient:
var connStr = new ServiceBusConnectionStringBuilder("Endpoint=sb://servicebusnamespacename.servicebus.windows.net/;SharedAccessKeyName=saskeyname;SharedAccessKey=sakKey");
connStr.TransportType = TransportType.Amqp;
var msgFactory = MessagingFactory.CreateFromConnectionString(connStr.ToString());
var ehClient = msgFactory.CreateEventHubClient("teststream");
I just added connStr in my sample to emphasize assigning TransportType to Amqp.
You will end up with multiple connections with outgoing port 5671:
If you rewrite your code with 1 MessagingFactory per EventHubClient (or a reasonable ratio) - you are all set (in your code - you will need to move EventHubClient creation to Reader)!
The only extra criteria one need to consider while creating multiple connections is the Bill - only 100 connections are included (including senders and receivers) in basic sku. I guess you are already on standard (as you have >1 TUs) - which gives 1000 connections included in the package - so no need to worry - but mentioning just-in-case.
~Sree

A good option is to create a Task for each partition.
This a copy of my implementation which is able to process a rate of 2.5k messages per second per partition. This rate will be also related to your downstream speed.
static void EventReceiver()
{
for (int i = 0; i <= EventHubPartitionCount; i++)
{
Task.Factory.StartNew((state) =>
{
Console.WriteLine("Starting worker to process partition: {0}", state);
var factory = MessagingFactory.Create(ServiceBusEnvironment.CreateServiceUri("sb", "tests-eventhub", ""), new MessagingFactorySettings()
{
TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider("Listen", "PGSVA7L="),
TransportType = TransportType.Amqp
});
var client = factory.CreateEventHubClient("eventHubName");
var group = client.GetConsumerGroup("customConsumer");
Console.WriteLine("Group: {0}", group.GroupName);
var receiver = group.CreateReceiver(state.ToString(), DateTime.Now);
while (true)
{
if (cts.IsCancellationRequested)
{
receiver.Close();
break;
}
var messages = receiver.Receive(20);
messages.ToList().ForEach(aMessage =>
{
// Process your event
});
Console.WriteLine(counter);
}
}, i);
}
}

How to reuse TCP connections with .NET BigQuery API?

I'm streaming data into BQ with .NET API. And I noticed in Process Explorer that new TCP/IP connections are created and ended over and over again. I'm wondering if it's possible to reuse the connection and avoid big overhead of connection creation and end?
public async Task InsertAsync(BaseBigQueryTable table, IList<IDictionary<string, object>> rowList, GetBqInsertIdFunction getInsert,CancellationToken ct)
{
if (rowList.Count == 0)
{
return;
}
string tableId = table.TableId;
IList<TableDataInsertAllRequest.RowsData> requestRows = rowList.Select(row => new TableDataInsertAllRequest.RowsData {Json = row,InsertId = getInsert(row)}).ToList();
TableDataInsertAllRequest request = new TableDataInsertAllRequest { Rows = requestRows };
bool needCreateTable = false;
BigqueryService bqService = null;
try
{
bqService = GetBigQueryService();
TableDataInsertAllResponse response =
await
bqService.Tabledata.InsertAll(request, _account.ProjectId, table.DataSetId, tableId)
.ExecuteAsync(ct);
IList<TableDataInsertAllResponse.InsertErrorsData> insertErrors = response.InsertErrors;
if (insertErrors != null && insertErrors.Count > 0)
{
//handling errors, removed for easier reading..
}
}catch{
//... removed for easier reading
}
finally
{
if (bqService != null)
bqService.Dispose();
}
}
private BigqueryService GetBigQueryService()
{
return new BigqueryService(new BaseClientService.Initializer
{
HttpClientInitializer = _credential,
ApplicationName = _applicationName,
});
}
** Follow up **
The answer given below seems to be the only solution to reduce http connections. however, I found using batch request on large mount of live data streaming could have some limitation. see my another questions on this: Google API BatchRequest: An established connection was aborted by the software in your host machine

Below link documents how to batch API calls together to reduce the number of HTTP connections your client has to make
https://cloud.google.com/bigquery/batch
After batch request is issued, you can get response and parse out all involved jobids. As an alternative you can preset jobids in batch request for each and every inner request. Note: you need to make sure those jobids are unique
After that you can check what is going on with each of these jobs via jobs.get https://cloud.google.com/bigquery/docs/reference/v2/jobs/get

How to change the properties of a service bus queue?

I am using service bus queues to communicate between web role and worker role. Sometimes web role messages are not being accepted by worker role. But it immediately accepts the next message i send. So i was thinking maybe its happening because the Batched Operations is enabled. I have been trying to put it to false but i havent been successful. This is my code.
public static QueueClient GetServiceBusQueueClient(string queuename)
{
string connectionString;
if (RoleEnvironment.IsAvailable)
connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
else
connectionString = ConfigurationManager.AppSettings["Microsoft.ServiceBus.ConnectionString"];
var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
QueueDescription queue = null;
if (!namespaceManager.QueueExists(queuename))
{
queue = namespaceManager.CreateQueue(queuename);
queue.EnableBatchedOperations = false;
queue.MaxDeliveryCount = 1000;
}
else
{
queue = namespaceManager.GetQueue(queuename);
queue.EnableBatchedOperations = false;
queue.MaxDeliveryCount = 1000;
}
MessagingFactorySettings mfs = new MessagingFactorySettings();
mfs.NetMessagingTransportSettings.BatchFlushInterval = TimeSpan.Zero;
string issuer;
string accessKey;
if (RoleEnvironment.IsAvailable)
issuer = RoleEnvironment.GetConfigurationSettingValue("AZURE_SERVICEBUS_ISSUER");
else
issuer = ConfigurationManager.AppSettings["AZURE_SERVICEBUS_ISSUER"];
if (RoleEnvironment.IsAvailable)
accessKey = RoleEnvironment.GetConfigurationSettingValue("AZURE_SERVICEBUS_ACCESS_KEY");
else
accessKey = ConfigurationManager.AppSettings["AZURE_SERVICEBUS_ACCESS_KEY"];
mfs.TokenProvider = TokenProvider.CreateSharedSecretTokenProvider(issuer, accessKey);
MessagingFactory messagingFactory = MessagingFactory.Create(namespaceManager.Address, mfs);
QueueClient Client = messagingFactory.CreateQueueClient(queue.Path);
return Client;
}
But the EnableBatchedOperations is always true and the MaxDeliveryCount is always 10 by default.
Let me know if you know what's the issue
Thanks

If you want to set the EnabledBatchedOperations, you have to do that before you create the queue. you do that by creating a QueueDescription object then pass that to the CreateQueue method. For example:
QueueDescription orderQueueDescription =
new QueueDescription(queuename)
{
RequiresDuplicateDetection = true,
MaxDeliveryCount = 1000,
};
namespaceMgr.CreateQueue(orderQueueDescription);
Update:
The documentation is pretty clear on this:
Since metadata cannot be changed once a messaging entity is created, modifying the duplicate detection behavior requires deleting and recreating the queue. The same principle applies to any other metadata. [1]
QueueDescription Represents the metadata description of the queue.
[1] http://msdn.microsoft.com/en-us/library/windowsazure/hh532012.aspx

Update Azure SDK 2.3
UpdateQueue method on the NamespaceManager still doesn't let you update any properties apart from suspending or resuming the queue.
If you need to change MaxDeliveryCount on an existing queue and you don't want to delete and recreate the queue, your only option is to change it in the Azure portal.