We Keep Coding

Saving images via producer-consumer pattern using BlockingCollection

I'm facing a producer-consumer problem: I have a camera that sends images very quickly and I have to save them to disk. The images are in the form of ushort[]. The camera always overrides the same variable of type ushort[]. So between one acquisition and another I have to copy the array and when possible, save it in order to free up the memory of that image. The important thing is not to lose any images from the camera, even if it means increasing the memory used: it is entirely acceptable that the consumer (saving images with freeing of memory) is slower than the producer; however, it is not acceptable not to copy the image into memory in time.
I've written sample code that should simulate the problem:
immage_ushort: is the image generated by the camera that must be copied to the BlockingCollection before the next image arrives
producerTask: has a cycle that should simulate the arrival of the image every time_wait; within this time the producer should copy the image in the BlockingCollection.
consumerTask: must work on the BlockingCollection by saving the images to disk and thus freeing up the memory; it doesn't matter if the consumer works slower than the producer.
I put a time_wait of 1 millisecond, to test the performance (actually the camera will not be able to reach that speed). The times are respected (with a maximum delay of 1-2 ms, therefore acceptable) if there is no saving to disk in the code (commenting image1.ImWrite (file_name)). But with saving to disk on, I instead get delays that sometimes exceed 100ms.
This is my code:
private void Execute_test_producer_consumer1()
{
//Images are stored as ushort array, so we create a BlockingCollection<ushort[]>
//to keep images when they arrive from camera
BlockingCollection<ushort[]> imglist = new BlockingCollection<ushort[]>();
string lod_date = "";
/*producerTask simulates a camera that returns an image every time_wait
milliseconds. The image is copied and inserted in the BlockingCollection
to be then saved on disk in the consumerTask*/
Task producerTask = Task.Factory.StartNew(() =>
{
//Number of images to process
int num_img = 3000;
//Time between one image and the next
long time_wait = 1;
//Time log variables
var watch1 = System.Diagnostics.Stopwatch.StartNew();
long watch_log = 0;
long delta_time = 0;
long timer1 = 0;
List<long> timer_delta_log = new List<long>();
List<long> timer_delta_log_time = new List<long>();
int ii = 0;
Console.WriteLine("-----START producer");
watch1.Restart();
//Here I expect every wait_time (or a little more) an image will be inserted
//into imglist
while (ii < num_img)
{
timer1 = watch1.ElapsedMilliseconds;
delta_time = timer1 - watch_log;
if (delta_time >= time_wait || ii == 0)
{
//Add image
imglist.Add((ushort[])immage_ushort.Clone());
//Inserting data for time log
timer_delta_log.Add(delta_time);
timer_delta_log_time.Add(timer1);
watch_log = timer1;
ii++;
}
}
imglist.CompleteAdding();
watch1.Stop();
lod_date = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.fff");
Console.WriteLine("-----END producer: " + lod_date);
// We only print images that are not inserted on schedule
int gg = 0;
foreach (long timer_delta_log_t in timer_delta_log)
{
if (timer_delta_log_t > time_wait)
{
Console.WriteLine("-- Image " + (gg + 1) + ", delta: "
+ timer_delta_log_t + ", time: " + timer_delta_log_time[gg]);
}
gg++;
}
});
Task consumerTask = Task.Factory.StartNew(() =>
{
string file_name = "";
int yy = 0;
// saving images and removing data
foreach (ushort[] imm in imglist.GetConsumingEnumerable())
{
file_name = #"output/" + yy + ".png";
Mat image1 = new Mat(row, col, MatType.CV_16UC1, imm);
//By commenting on this line, the timing of the producer is respected
image1.ImWrite(file_name);
image1.Dispose();
yy++;
}
imglist.Dispose();
lod_date = DateTime.Now.ToString("yyyy-MM-dd HH:mm:ss.fff");
Console.WriteLine("-----END consumer: " + lod_date);
});
}
I thought, also, that the BlockingCollection could remain blocked for the entire duration of the foreach and therefore of saving the image to disk. So I also tried replacing the foreach with this:
while(!imglist.IsCompleted)
{
ushort[] elem = imglist.Take();
file_name = #"output/" + yy + ".png";
Mat image1 = new Mat(row, col, MatType.CV_16UC1, elem);
//By commenting on this line, the timing of the producer is respected
image1.ImWrite(file_name);
image1.Dispose();
yy++;
}
But the result doesn't change.
What am I doing wrong?

You migth want to start your tasks with the "LongRunning" option:
LongRunning
Specifies that a task will be a long-running, coarse-grained operation involving fewer, larger components than fine-grained systems. It provides a hint to the TaskScheduler that oversubscription may be warranted. Oversubscription lets you create more threads than the available number of hardware threads. It also provides a hint to the task scheduler that an additional thread might be required for the task so that it does not block the forward progress of other threads or work items on the local thread-pool queue.

Replace Thread.Sleep with System.Threading.Timer?

I'm trying to replace Thread.Sleep with System.Threading.Timer, and I'm trying to implement it in the following code. The main reason for changing is that, even though it works when testing locally, it's not working properly in the server. Plus, I've read that using it here is bad practice.
I've seen several examples (including the one below), but I'm not certain how I could use it in my case: System.Threading.Timer in C# it seems to be not working. It runs very fast every 3 second
In my console app, I need to copy files to our server every 15 minutes. So at :20, :35, :50, :05, I begin reading files for that quarter. In the case below, the files will be available at :45, and I add 5 minutes just in case.
This is my code. I had previously tried to copy files from several quarters in parallel, but the server where the source files reside is having trouble with that. So I'm going back to this.
My question is, how can I replace Thread.Sleep with System.Threading.Timer in this example?
I wanted to try await Task.Delay(Int32), but I have VS2010:
DateTime lastTimeRead = new DateTime(2014, 9, 9, 8, 35, 0); //Last read at 8:35AM
DateTime nextTimeRead;
for (; ; )
{
now = DateTime.Now; //It's currently 8:43AM
nextTimeRead = LastTimeRead.AddMinutes(15); // nextTimeRead = 8:50AM.
if (nextTimeRead > now) //Yes, so wait 7 minutes for files to be available
{
TimeSpan span = nextTimeRead.Subtract(now);
Double milliseconds = span.TotalMilliseconds;
Console.WriteLine("Sleep for milliseconds: " + milliseconds.ToString());
Thread.Sleep(Convert.ToInt32(milliseconds));
Console.WriteLine("Download files after sleep of: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}
else // Files are available. Read.
{
Console.WriteLine("Download files no sleep: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}
LastTimeRead = nextTimeRead;
}

The idea is to have a timer and Enable it when you want to set the delay in your program:
System.Timers.Timer Delay = new System.Timers.Timer();
Delay.Elapsed += new System.Timers.ElapsedEventHandler(Delay_Elapsed);
Delay.Interval=Convert.ToInt32(milliseconds);
Delay.Enabled = false;
void Delay_Elapsed(object sender, System.Timers.ElapsedEventArgs e)
{
Delay.Enabled = false;
}
......
.....
if (nextTimeRead > now) //Yes, so wait 7 minutes for files to be available
{
TimeSpan span = nextTimeRead.Subtract(now);
Double milliseconds = span.TotalMilliseconds;
Console.WriteLine("Sleep for milliseconds: " + milliseconds.ToString());
Delay.Enabled = true;
while (Delay.Enabled)
{
////Wait until time passes
}
Console.WriteLine("Download files after sleep of: " + nextTimeRead.ToString());
DownloadFilesByPeriod(nextTimeRead);
}

How to get accurate download/upload speed in C#.NET?

I want to get accurate download/upload speed through a Network Interface using C# .NET
I know that it can be calculated using GetIPv4Statistics().BytesReceived and putting the Thread to sleep for sometime. But it's not giving the output what I am getting in my browser.

Here is a quick snippet of code from LINQPad. It uses a very simple moving average. It shows "accurate speeds" using "Speedtest.net". Things to keep in mind are Kbps is in bits and HTTP data is often compressed so the "downloaded bytes" will be significantly smaller for highly compressible data. Also, don't forget that any old process might be doing any old thing on the internet these days (without stricter firewall settings) ..
I like flindenberg's answer (don't change the accept), and I noticed that some polling periods would return "0" that aligns with his/her conclusions.
Use at your own peril.
void Main()
{
var nics = System.Net.NetworkInformation.NetworkInterface.GetAllNetworkInterfaces();
// Select desired NIC
var nic = nics.Single(n => n.Name == "Local Area Connection");
var reads = Enumerable.Empty<double>();
var sw = new Stopwatch();
var lastBr = nic.GetIPv4Statistics().BytesReceived;
for (var i = 0; i < 1000; i++) {
sw.Restart();
Thread.Sleep(100);
var elapsed = sw.Elapsed.TotalSeconds;
var br = nic.GetIPv4Statistics().BytesReceived;
var local = (br - lastBr) / elapsed;
lastBr = br;
// Keep last 20, ~2 seconds
reads = new [] { local }.Concat(reads).Take(20);
if (i % 10 == 0) { // ~1 second
var bSec = reads.Sum() / reads.Count();
var kbs = (bSec * 8) / 1024;
Console.WriteLine("Kb/s ~ " + kbs);
}
}
}

Please try this. To check internet connection speed.
public double CheckInternetSpeed()
{
// Create Object Of WebClient
System.Net.WebClient wc = new System.Net.WebClient();
//DateTime Variable To Store Download Start Time.
DateTime dt1 = DateTime.UtcNow;
//Number Of Bytes Downloaded Are Stored In ‘data’
byte[] data = wc.DownloadData("http://google.com");
//DateTime Variable To Store Download End Time.
DateTime dt2 = DateTime.UtcNow;
//To Calculate Speed in Kb Divide Value Of data by 1024 And Then by End Time Subtract Start Time To Know Download Per Second.
return Math.Round((data.Length / 1024) / (dt2 - dt1).TotalSeconds, 2);
}
It gives you the speed in Kb/Sec and share the result.

By looking at another answer to a question you posted in NetworkInterface.GetIPv4Statistics().BytesReceived - What does it return? I believe the issue might be that you are using to small intervals. I believe the counter only counts whole packages, and if you for example are downloading a file the packages might get as big as 64 KB (65,535 bytes, IPv4 max package size) which is quite a lot if your maximum download throughput is 60 KB/s and you are measuring 200 ms intervals.
Given that your speed is 60 KB/s I would have set the running time to 10 seconds to get at least 9 packages per average. If you are writing it for all kinds of connections I would recommend you make the solution dynamic, ie if the speed is high you can easily decrease the averaging interval but in the case of slow connections you must increase the averaging interval.
Either do as #pst recommends by having a moving average or simply increase the sleep up to maybe 1 second.
And be sure to divide by the actual time taken rather than the time passed to Thread.Sleep().
Additional thought on intervals
My process would be something like this, measure for 5 second and gather data, ie bytes recieved as well as the number of packets.
var timePerPacket = 5000 / nrOfPackets; // Time per package in ms
var intervalTime = Math.Max(d, Math.Pow(2,(Math.Log10(timePerPacket)))*100);
This will cause the interval to increase slowly from about several tens of ms up to the time per packet. That way we always get at least (on average) one package per interval and we will not go nuts if we are on a 10 Gbps connection. The important part is that the measuring time should not be linear to the amount of data received.

The SSL handshake takes some time as a result modified #sandeep answer. I first created a request and then measure the time to download the content. I believe this is a little more accurate but still not 100%. It is an approximation.
public async Task<int> GetInternetSpeedAsync(CancellationToken ct = default)
{
const double kb = 1024;
// do not use compression
using var client = new HttpClient();
int numberOfBytesRead = 0;
var buffer = new byte[10240].AsMemory();
// create request
var stream = await client.GetStreamAsync("https://www.google.com", ct);
// start timer
DateTime dt1 = DateTime.UtcNow;
// download stuff
while (true)
{
var i = await stream.ReadAsync(buffer, ct);
if (i < 1)
break;
numberOfBytesRead += i;
}
// end timer
DateTime dt2 = DateTime.UtcNow;
double kilobytes = numberOfBytesRead / kb;
double time = (dt2 - dt1).TotalSeconds;
// speed in Kb per Second.
return (int)(kilobytes / time);
}

Fastest Multi-Threading Method of Serial Port Data Parsing C#

I am currently writing an application that communicates with an integrated servo via a serial connection.
The motor sends out position data at a rate of up to 1000 times/second. What I'm trying to achieve is to be able to format the data coming back (by stripping it of white spaces, new lines, etc) and parsing it to extract the relevant data from the received strings.
Currently, I have the data received event handler read the data, format it using a series of string.replace method calls, and append it to a string that acts as a buffer. Then, using threads, I constantly check the buffer as it fills for a particular delimiter (in my case "\r") which signifies the end of one message from the motor, then remove that message from the buffer and print it to a rich text field.
There are two problems with this approach. One is that because the motor streams position data at such a high rate, the buffer fills faster than the data can be processed by the threads. Thus when I send a command to the motor, it acts immediately but the response is delayed by a few seconds because all of the preceding data in the buffer must be processed first. Second, having two threads running a method that implements a while(true) structure means processor utilization skyrockets and within a few seconds the fans in the pc are on max.
Is there a better way of handling the data?
Here is my event handler code:
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
}
}
Here is the method that the threads execute:
private void parseBuffer()
{
while (true)
{
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}

Your parseBuffer will go wild spinning even if there is no new data since last try.
You can mitigate this with signalling.
private AutoResetEvent waitHandle = new AutoResetEvent(false);
Trigger the signal in dataReceived
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
waitHandle.Set(); // <-- tell parseBuffer that new data is available
}
}
wait for the signal in parseBuffer
private void parseBuffer()
{
while (true)
{
waitHandle.WaitOne(); // <-- waits until there is more data to parse
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}

There are a couple of things you can do to improve this dramatically.
1) Build a state-machine parser that parses the incomming data one character at a time. When it has built a complete "message", add it to a List<MyMessage> structure.
2) Use a Virtualized ListView or DataGridView to display the List<MyMessage>.

In the data received event read the incoming data as raw bytes and store in a queue. Don't process the data in the event handler. Then use something similar to what albin said to process the data in another method. The important thing is to allow the eventhandler to fire as often as possible and do no more that required.

In general I would use a blocking collection between a reader thread which solely reads from the socket and a parsing thread.
In terms of performance look at using split for parsing - this is a lot faster than replacing inside a string. You should look at using a regular expression and/or the StringBuilder class - you should use 1 expression with alternatives
The regex code would look like this:
string pattern = " |\\n|\\r";
string replacement = " ";
regex rgx = new Regex(pattern);
string result = rgx.Replace(input, replacement);

C# -calculate download/upload time using network bandwidth

As a feature in the application which Im developing, I need to show the total estimated time left to upload/download a file to/from server.
how would it possible to get the download/upload speed to the server from client machine.
i think if im able to get speed then i can calculate time by -->
for example ---for a 200 Mb file = 200(1024 kb) = 204800 kb and
divide it by 204800 Mb / speed Kb/s = "x" seconds

The upload/download speed is no static property of a server, it depends on your specific connection and may also vary over time. Most application I've seen do an estimation over a short time window. That means they start downloading/uploading and measure the amount of data over, lets say 10 seconds. This is then taken as the current transfer speed and used to calculate the remaining time (e.g. 2500kB / 10s -> 250Kb/s). The time window is moved on and recalculated continuously to keep the calculation accurate to the current speed.
Although this is a quite basic approach, it will serve well in most cases.

Try something like this:
int chunkSize = 1024;
int sent = 0
int total = reader.Length;
DateTime started = DateTime.Now;
while (reader.Position < reader.Length)
{
byte[] buffer = new byte[
Math.Min(chunkSize, reader.Length - reader.Position)];
readBytes = reader.Read(buffer, 0, buffer.Length);
// send data packet
sent += readBytes;
TimeSpan elapsedTime = DateTime.Now - started;
TimeSpan estimatedTime =
TimeSpan.FromSeconds(
(total - sent) /
((double)sent / elapsedTime.TotalSeconds));
}

This is only tangentially related, but I assume if you're trying to calculate total time remaining, you're probably also going to be showing it as some kind of progress bar. If so, you should read this paper by Chris Harrison about perceptual differences. Here's the conclusion straight from his paper (emphasis mine).
Different progress bar behaviors appear to have a significant effect on user perception of process duration. By minimizing negative behaviors and incorporating positive behaviors, one can effectively make progress bars and their associated processes appear faster. Additionally, if elements of a multistage operation can be rearranged, it may be possible to reorder the stages in a more pleasing and seemingly faster sequence.
http://www.chrisharrison.net/projects/progressbars/ProgBarHarrison.pdf

I don't know why do you need this but i would go simpliest way possible and ask user what connection type he has. Then take file size divide it by speed and then by 8 to get number of seconds.
Point is you won't need processing power to calculate speeds. Microsoft on their website use function that calculates a speed for most default connections based on file size which you can get while uploading the file or to enter it manually.
Again, maybe you have other needs and you must calculate upload on fly...

The following code computes the remaining time in minute.
long totalRecieved = 0;
DateTime lastProgressChange = DateTime.Now;
Stack<int> timeSatck = new Stack<int>(5);
Stack<long> byteSatck = new Stack<long>(5);
using (WebClient c = new WebClient())
{
c.DownloadProgressChanged += delegate(object s, DownloadProgressChangedEventArgs args)
{
long bytes;
if (totalRecieved == 0)
{
totalRecieved = args.BytesReceived;
bytes = args.BytesReceived;
}
else
{
bytes = args.BytesReceived - totalRecieved;
}
timeSatck.Push(DateTime.Now.Subtract(lastProgressChange).Seconds);
byteSatck.Push(bytes);
double r = timeSatck.Average() * ((args.TotalBytesToReceive - args.BytesReceived) / byteSatck.Average());
this.textBox1.Text = (r / 60).ToString();
totalRecieved = args.BytesReceived;
lastProgressChange = DateTime.Now;
};
c.DownloadFileAsync(new Uri("http://www.visualsvn.com/files/VisualSVN-1.7.6.msi"), #"C:\SVN.msi");
}

I think I ve got the estimated time to download.
double timeToDownload = ((((totalFileSize/1024)-((fileStream.Length)/1024)) / Math.Round(currentSpeed, 2))/60);
this.Invoke(new UpdateProgessCallback(this.UpdateProgress), new object[] {
Math.Round(currentSpeed, 2), Math.Round(timeToDownload,2) });
where
private void UpdateProgress(double currentSpeed, double timeToDownload)
{
lblTimeUpdate.Text = string.Empty;
lblTimeUpdate.Text = " At Speed of " + currentSpeed + " it takes " + timeToDownload +" minute to complete download";
}
and current speed is calculated like
TimeSpan dElapsed = DateTime.Now - dStart;
if (dElapsed.Seconds > 0) {currentSpeed = (fileStream.Length / 1024) / dElapsed.Seconds;
}