Amazon Kinesis GetRecords Api Best approach - c#

I have setup Kinesis stream in Amazon WebServices. I Also want to accomplish the following tasks:
Put Records into Single Stream with Single Shard (C# Api) - SUCCESS
Also I wrote Sample App in which multiple Producers are working on different Stream - SUCCESS
Also I setup Sample App to Perform Mutiple Workers put the Data into Single Stream - SUCCESS
Also I want to be able to enforce the SequenceNumberOrdering in the Reacords.
But the real pain is the GetRecords Consumer Operation using Kinesis C# Api.
I created a sample App for the Records. The problem is that it doesn't stop the Iteration even if there are no Records present in the Kinesis Stream. Also keeping the SequenceNumber in the DB or some file and retrieving the file again is time consuming - what is the advantage of using Kinesis Stream for GetRecords?
Why does it keep on iterating even when there is no data in the Stream?
I used following piece of code for the REFERENCE;
private static void GetFilesKinesisStream()
{
IAmazonKinesis kinesis = AWSClientFactory.CreateAmazonKinesisClient();
try
{
ListStreamsResponse listStreams = kinesis.ListStreams();
int numBuckets = 0;
if (listStreams.StreamNames != null &&
listStreams.StreamNames.Count > 0)
{
numBuckets = listStreams.StreamNames.Count;
Console.WriteLine("You have " + numBuckets + " Amazon Kinesis Streams.");
Console.WriteLine(string.Join(",\n", listStreams.StreamNames.ToArray()));
DescribeStreamRequest describeRequest = new DescribeStreamRequest();
describeRequest.StreamName = "******************";
DescribeStreamResponse describeResponse = kinesis.DescribeStream(describeRequest);
List<Shard> shards = describeResponse.StreamDescription.Shards;
foreach (Shard s in shards)
{
Console.WriteLine("shard: " + s.ShardId);
}
string primaryShardId = shards[0].ShardId;
GetShardIteratorRequest iteratorRequest = new GetShardIteratorRequest();
iteratorRequest.StreamName = "*********************";
iteratorRequest.ShardId = primaryShardId;
iteratorRequest.ShardIteratorType = ShardIteratorType.AT_SEQUENCE_NUMBER;
iteratorRequest.StartingSequenceNumber = "49544005271533118105145368110776211536226129690186743810";
GetShardIteratorResponse iteratorResponse = kinesis.GetShardIterator(iteratorRequest);
string iterator = iteratorResponse.ShardIterator;
Console.WriteLine("Iterator: " + iterator);
//Step #3 - get records in this iterator
GetShardRecords(kinesis, iterator);
Console.WriteLine("All records read.");
Console.ReadLine();
}
// sr.WriteLine("You have " + numBuckets + " Amazon S3 bucket(s).");
}
catch (AmazonKinesisException ex)
{
if (ex.ErrorCode != null && ex.ErrorCode.Equals("AuthFailure"))
{
Console.WriteLine("The account you are using is not signed up for Amazon EC2.");
Console.WriteLine("You can sign up for Amazon EC2 at http://aws.amazon.com/ec2");
}
else
{
Console.WriteLine("Caught Exception: " + ex.Message);
Console.WriteLine("Response Status Code: " + ex.StatusCode);
Console.WriteLine("Error Code: " + ex.ErrorCode);
Console.WriteLine("Error Type: " + ex.ErrorType);
Console.WriteLine("Request ID: " + ex.RequestId);
}
}
}
private static void GetShardRecords(IAmazonKinesis client, string iteratorId)
{
//create reqest
GetRecordsRequest getRequest = new GetRecordsRequest();
getRequest.Limit = 100;
getRequest.ShardIterator = iteratorId;
//call "get" operation and get everything in this shard range
GetRecordsResponse getResponse = client.GetRecords(getRequest);
//get reference to next iterator for this shard
string nextIterator = getResponse.NextShardIterator;
//retrieve records
List<Record> records = getResponse.Records;
//print out each record's data value
foreach (Record r in records)
{
//pull out (JSON) data in this record
string s = Encoding.UTF8.GetString(r.Data.ToArray());
Console.WriteLine("Record: " + s);
Console.WriteLine("Partition Key: " + r.PartitionKey);
}
if (null != nextIterator)
{
//if there's another iterator, call operation again
GetShardRecords(client, nextIterator);
}
}

Why does a kinesis consumer keep iterating after the "end" of the data?
Because there is no "end". Kinesis is sort of like a queue, but not exactly. Think of it like a moving time window of recorded events. You don't consume records, you examine records passively that are currently in the window (which amazon hardcodes to 24 hours). Because the window is always moving, once you reach the "last" record, it keeps watching in real time. New records could come at any time; the consumer doesn't know that there aren't any producers.
If you want to stop based on some condition, that condition has to be contained in your payload. For example, if you wanted to stop when you got to "now", part of your payload could be a timestamp, which the consumer checks for proximity to its current time.

Related

Nested for loop resets after incrementing normally

Hi I'm trying to create a simple program that pings everything on our network over a minute and changes the UI based on the results of that ping.
I have a background worker that sets the variables and pings the appropriate IP address.
This is contained within a while loop that waits for ping to be true. Within this loop is a For Loop that takes in user input to ping that specific amount of times.
The for loop worked fine, and in debugging I can see i incrementing as usual.
I have a break point on the for loop line itself as well as the curly brace after it. If the user input is 2, it will run through twice, then the i variable explicitly changes to 0 and then reaches the if() statement
This was all working perfectly until I added some code within the nested for loops. But I cannot see anything that would be changing i
Here are some pictures showing what I mean
The loop has ran twice, and as such i is equal to 2. I then continue once more to the curly brace and i is now 0 and so doesn't break the loops.
Here's the entire method:
...
private async void backgroundWorker1_DoWork(object sender, DoWorkEventArgs e)
{
// variables
bool ping = false;
int i;
// Create ping objects
// TODO read these variables in from the config screen and create a class to handle it (populatePing)
// Ping 2
ping0.SetFqdn("##");
ping0.SetIpAddress("##");
ping0.SetFriendlyName("##");
// Ping 1
ping1.SetFqdn("##");
ping1.SetIpAddress("##");
ping1.SetFriendlyName("##");
// Create file (needs updating to utilise wildcards)
using StreamWriter file = new("C:/Users/##/AppData/Roaming/log.txt", append: true);
try
{
// Create background worker
BackgroundWorker worker = (BackgroundWorker)sender;
while (!ping)
{
// Write to log file
await file.WriteAsync("Starting job...\n");
await file.WriteAsync("Requested amount of pings: " + count + "\n");
// Create date object for logs
DateTime localDate = DateTime.Now;
for (i = 0; i < count; i++)
{
// if count has reached the user input limit, break the loop
if (i == count)
{
ping = true;
}
// Create ping objects
System.Net.NetworkInformation.Ping pinger = new System.Net.NetworkInformation.Ping();
PingReply ip0_ping, ip1_ping;
try
{
if(!ping0.Equals(null))
{
try
{
// Send ping to specified IP address
ip0_ping = pinger.Send(ping0.GetIpAddress());
// Write log file
await file.WriteLineAsync(localDate.TimeOfDay + " | Friendly Name" + ping0.GetFriendlyName() + ": Ping: " + ip0_ping.Address + " status: " + ip0_ping.Status + ". time: " + ip0_ping.RoundtripTime + "ms");
// Successful ping has been sent
if(ip0_ping.Status.ToString().Contains("Success"))
{
ping0.SetSuccessfulPings(1);
} else // Unsuccessful ping has been sent
{
ping0.SetFailedPings(1);
}
}
catch(Exception d)
{
Debug.WriteLine(d.ToString());
}
} else
{
Debug.WriteLine("ERROR: ping0 is not being populated correctly");
}
if(!ping1.Equals(null))
{
try
{
// Send ping to specified IP address
ip1_ping = pinger.Send(ping1.GetIpAddress());
// Write log file
await file.WriteLineAsync(localDate.TimeOfDay + " | Friendly Name" + ping1.GetFriendlyName() + ": Ping: " + ip1_ping.Address + " status: " + ip1_ping.Status + ". time: " + ip1_ping.RoundtripTime + "ms");
// Successful ping has been sent
if (ip1_ping.Status.ToString().Contains("Success"))
{
ping1.SetSuccessfulPings(1);
}
else // Unsuccessful ping has been sent
{
ping1.SetFailedPings(1);
}
}
catch(Exception d)
{
Debug.WriteLine(d.ToString());
}
}
// wait one second
wait(1000);
}
catch (Exception b)
{
Debug.WriteLine(b.ToString());
}
}
}
} catch (Exception a)
{
Debug.WriteLine(a.ToString());
}
}
...
Sorry if this is really obvious but I'm really stumped
The method will not exit since the for loop is not entered if i == count since the condition section of your for loop evaluates if i < count.
See the following link under the for loop section: https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/statements/iteration-statements
The condition section that determines if the next iteration in the loop should be executed. If it evaluates to true or is not present, the next iteration is executed; otherwise, the loop is exited. The condition section must be a Boolean expression.
and also
The iterator section that defines what happens after each execution of the body of the loop.

Com object method is blocking in asp.net core but not unit tests

I have a class I have used to wrap up a com object (MLApp) that is used to call into matlab. I have a unit test that initializes 5 matlab sessions and runs a command that pauses for 10 seconds in each individual session. My unit test takes ~10 seconds to run which is expected since they should be executing in parallel.
[Test]
public void ParallelExecution()
{
List<MatlabRunner> runners = new List<MatlabRunner>();
for (int i = 0; i < 5; i++)
{
runners.Add(new MatlabRunner());
}
try
{
List<Task<string>> t = new List<Task<string>>();
foreach (MatlabRunner r in runners)
t.Add(Task.Run<string>(()=>r.Execute( "pause(10)" )));
Task<string>.WaitAll(t.ToArray());
t.ForEach((task) => System.Console.WriteLine(task.Result));
}
finally
{
runners.ForEach((r)=>r.Quit());
}
}
However, now when I put what amounts to the same code in my asp.net core project, I can make two web requests and they are both received in parallel (I know this because I log that the request is received) but then the call into runner.Execute() is serialized.
[HttpGet("Verify")]
public JsonResult Verify(String lot = null,String lotStatus = null,int? NumPorts = null,int? wafer = null)
{
System.IO.File.AppendAllText(#"C:\Logs\log.txt", "\n\Received Request! " + System.DateTime.Now + "\n");
String command = "pause(10)";
MatlabRunner runner = null;
String result = "";
try
{
runner = new MatlabRunner();
result = runner.Execute(command);
}
catch(Exception e)
{
System.IO.File.AppendAllText(#"C:\Logs\log.txt", e.StackTrace + " " + e.InnerException + " " + e.GetType() + e.Source + " " + e.GetBaseException());
}
finally
{
runner.Quit();
System.IO.File.AppendAllText(#"C:\Logs\log.txt", "\nQuit! " + System.DateTime.Now + "\n");
}
return new JsonResult(result);
}
The problem is that I make two, parallel GET requests to this Verify method. While one request is calling runner.Execute("pause(10)") the other request is blocked from calling this method. I would expect to make two calls to this Verify method in parallel and them both to execute in parallel, finishing in a total of ~10 seconds. However, for both requests to complete takes 20 seconds (10 for the first, 10 for the second). In my unit test, I call this same method in parallel using Tasks and the entire unit test executes in 10 seconds, despite having 5 runners launched. If I were to do the same in the web service, it would take 50 seconds. Hopefully this is enough info. I can attach a log with time stamps to show how things are executing, if it would be valuable.
Am I missing something from a configuration perspective in IIS? Potentially something when I build or publish?

What is the best approach while polling database?

Currently I am doing an assignment in which I need to visit database of n number of servers to fetch some results.I have achieved this by iterating through the list of servers and raising tasks each one for each server in the collection. The task calls a function which basically makes an connection with the database,run query and disconnect from database.
My question is I am doing right by making a new connection on each polling with the database and closing it everytime or is this would be the best approach to keep a db connection open and fetch the result and then keep it open on next polling iteration.
PollingServerTimer() is being called by timer everytime.My polling timer is 3 sec.
Something like this :
private void PollingServerTimer(object sender, ElapsedEventArgs e)
{
foreach (var item in ServerOperationCollHandler)
{
if (item.RebootStatus != true)
{
PushItemIntoQueue(item);
}
}
}
public void PollingServerQueue()
{
while (isRunning)
{
this.waitHandle.WaitOne();
lock (syncRoot)
{
if (ServerQueue.Count > 0)
{
ServerOperationDataModel obj;
try
{
ServerQueue.TryDequeue(out obj);
Task GetCountFromDbTask = new Task(() => GetCountFromDb(obj));
GetCountFromDbTask.Start();
this.waitHandle.Reset();
}
catch (Exception ex)
{
MessageBox.Show("Problem encountered while finding iterim and recovery count");
isRunning = false;
break;
}
}
}
}
}
public void GetCountFromDb(ServerOperationDataModel obj)
{
ServerOperationDataModel serverObject = (ServerOperationDataModel)obj;
DataBaseHandler dbHandler = new DataBaseHandler(serverObject.DataBaseIP, serverObject.DataBasePort, serverObject.DataBaseName, serverObject.DataUserName, serverObject.DataUserPassword);
int attempts = 0;
do
{
try
{
dbHandler.connect();
}
catch (Exception ex)
{
break;
serverObject.DataBaseConnectionStatus = false;
log.Error("Connection attempt " + attempts + " failed.Retrying connection. Exception details :" + ex.ToString());
attempts++;
}
} while (attempts < _connectiontRetryAttempts && !dbHandler.isConnected());
if (dbHandler.isConnected())
{
/*Fetch Result and then get disconnect*/
dbHandler.disConnect();
}
else
{
//string msgLog = "Server : " + obj.ServerComponentIdentifier + " | " + obj.IPstring + "Connection cannot be established with the DB: " + obj.DataBaseIP + " | "+ obj.DataBasePort + " | " + obj.DataBaseName + " after a series of retries";
//LoggerUpdate.LogMessage(msgLog, LOGTYPE.POLLINGDATABASE, LoggerUpdate.ReturnLogDisplayObject(DateTime.Now, obj.ServerComponentIdentifier + "|" + obj.IPstring, Convert.ToInt16(LOGTYPE.POLLINGDATABASE), obj, msgLog));
}
}
I would not be concerned at all. Assuming you connect to the SQL Server (or a similar, enterprise DBMS), database connections are pooled at the client side which means that establishing the connection is costly only the first time the client connects to a particular db (formally: to a new, previously unseen connection string) and then each connection to the same database costs almost nothing.
If it hadn't been for pooling, applications servers would not be able to handle hundreds of concurrent browser connections that query the same data source. You would need much more than a connection every 3 seconds to cause any risk of depleting server or client resources.
You can read more on how pooling works
https://learn.microsoft.com/en-us/dotnet/framework/data/adonet/sql-server-connection-pooling
A side note: You should polish your code a little bit.
For example, you have
GetCountFromDb(ServerOperationDataModel obj)
but then
ServerOperationDataModel serverObject = (ServerOperationDataModel)obj;
Why would you need to cast the obj to another variable of the very same type?
In the catch clause, you have break and some code below it which looks unreachable.
Take a look at the .NET SqlDependency object. This allows you to register a query with a database and, using an OnChange handler, receive notification whenever the result of the query changes.

Changing Textbox.text within a method; For loop not looping due to Return

Below is a button, when pressed it calls a function that pings a bunch of IP addresses. If the IP address returns a response, it adds the IP address to the output_networkSearch.Text.
private void button_networkSearch_Click(object sender, RoutedEventArgs e)
{
output_networkSearch.Text = networkSearch(Convert.ToInt32(input_searchLimit.Text));
}
Below isn't the whole method, just the part that I can't get to work. The for loop starts at whatever the last digit on the users default gateway IP address is, and stops at whatever limit they have inputed (1 - 255).
// i is equal to the last digit in the default gateway IP, if it was 192.168.0.1 then i = 1.
for (int i = Convert.ToInt32(splitGatewayIP[3]); i <= searchLimit; i = i + 1)
{
// If the method receieves a ping reply...
if (PingHostSweep(gatewayIPRebuild + i))
{
// Returns 192.168.0. + i + ACTIVE
string response = gatewayIPRebuild + i + " ACTIVE";
return response;
}
else
{
string response = gatewayIPRebuild + i + " CLOSED";
return response;
}
}
This worked on a console application but for a WPF application it seems to run through the loop once and stop due to the return statement.
My idea to work around this would be to remove the Return Response statements and try and access the TextBox (output_networkSearch) directly.
So I would do something like:
for (int i = Convert.ToInt32(splitGatewayIP[3]); i <= searchLimit; i = i + 1)
{
// If the method receieves a ping reply...
if (PingHostSweep(gatewayIPRebuild + i))
{
// Returns 192.168.0. + i + ACTIVE
string response = gatewayIPRebuild + i + " ACTIVE";
output_networkSearch.Text = reponse;
}
else
{
string response = gatewayIPRebuild + i + " CLOSED";
output_networkSearch.Text = reponse;
}
}
HOWEVER, I can't access the textbox within the method for some reason. I've only just started learning C# so I'm not entirely familiar with how it works.
Here's an image of a partially working concept. As you can see the limit is set at 10, so it should ping IP address 1 through 10 and give an ACTIVE or CLOSED response. This did work in my console application version.
WPF version
Console version
This might do the trick for you
List<string> responses = new List<string>();
string response;
for (int i = Convert.ToInt32(splitGatewayIP[3]); i <= searchLimit; i = i + 1)
{
if (PingHostSweep(gatewayIPRebuild + i))
{
response = gatewayIPRebuild + i + " ACTIVE";
}
else
{
response = gatewayIPRebuild + i + " CLOSED";
}
responses.Add(response)
}
Now after the loop the list which is responses would have the list of all the IPs which are active and closed. Like the way you do had in the console Application.
i think you need use threading, there are need many child threading work in backend to scan, when they finish them work then response the result to MainForm, so i write some code hope can help you!
using System.Threading;
using System.Threading.Tasks;
public void Start(string ip)
{
Task.Factory.StartNew(() =>
{
// If the method receieves a ping reply...
string response;
if (PingHostSweep(ip))
{
// Returns 192.168.0. + i + ACTIVE
response = ip + " ACTIVE";
}
else
{
response = ip + " CLOSED";
}
this.Invoke((MethodInvoker)(() => { textBox1.AppendText("\r\n" + response); }));
});
}
private void button1_Click(object sender, EventArgs e)
{
for (int i = 1; i <= 255; i++)
{
Start(String.Format("192.168.100.{0}", i));
}
}
The previous answer was correct (though it didn't touch on a more advanced point that you will ultimately need to learn ... delegation and invocation ... long story ... won't bore you now).
What you wrote distills to this:
// SIDE NOTE: you cannot actually treat an IPv4 address as four "pure" quads (but that's not your question)
var notNecessarilyAHost = splitGatewayIP[3];
var searchStart = Convert.ToInt32(notNecessarilyAHost);
for (var i = searchStart; i <= searchLimit; ++i)
{
if (PingHostSweep(gatewayIPRebuild + i))
{
return $"{gatewayIPRebuild}{i} ACTIVE";
}
else
{
return $"{gatewayIPRebuild}{i} CLOSED";
}
}
...and if you (mentally) step through what you wrote it's fairly straightforward to see that the loop will only ever cycle once. Upon entry to the loop i will be equal to whatever searchStart is. Then you enter the if test. If that test is true, you fall into the true side of the branch (i.e., "...ACTIVE"). Otherwise, you'll drop into the else branch (i.e., "...CLOSED". FROM THERE...
You ALWAYS return. That will exit the loop (and the function that contains it). You will never cycle the loop again. "break" and "return" (and plausibly goto ... but that's for a different day) will ALWAYS exit the current scope (scope being a block of code wrapped by '{' and '}' (be they explicitly or implicitly written).
Following?
The previous answer was correct. It adjusts your code so that the loop adds the string you're composing with each iteration to a list of strings. Then, when you exit the loop (because i reaches searchLimit) that list of strings will contain N many, well, strings. You probably want to return or continue working that.
All that said, you can't (technically you can but you SHOULDN'T) do any of this inside a UI thread. If you do, the UI will block (and become 100% unresponsive to the user) while the loop runs (and the network calls that it makes run), etc.

How to get update object with webhook in C#?

I use C# for programming Telegram Bot, but when I set webhook I can't fill Update object? I use ashx handler.
public void ProcessRequest(HttpContext context)
{
Fwk_Log.Insert("before","before");
var update = context.Request.QueryString["Update"];
Fwk_Log.Insert(update, "update = ");
long offset = 0;
int whilecount = 0;
int updateId = 0;
whilecount += 1;
string updates = Fwk_HttpRequest.ExecuteUrlRequestJSONString("https://api.telegram.org/bot" + Token + "/getUpdates");
Shp_Telegram_GetUpdate list = new JavaScriptSerializer().Deserialize<Shp_Telegram_GetUpdate>(updates);
if (list != null)
{
foreach (var r in list.result)
{
//offset = list.result.First().update_id;
if (r.message.text == "/start")
{
Fwk_HttpRequest.ExecuteUrlRequestJSONString("https://api.telegram.org/bot" + Token +
"/sendMessage?chat_id=" + r.message.chat.id + "&text=" + "Hello World");
Fwk_Log.Insert("sendMessage", "");
}
}
}
}
You are getting things mixed up. See here
There are two mutually exclusive ways of receiving updates for your
bot — the getUpdates method on one hand and Webhooks on the other.
Incoming updates are stored on the server until the bot receives them
either way, but they will not be kept longer than 24 hours.
Regardless of which option you choose, you will receive
JSON-serialized Update objects as a result.
If you are using webhooks, you do not need to call getUpdate method.

Categories