We Keep Coding

C# .NET - Buffer messages w/Timer

I need to implement a message buffering system that is also timed based.
What I need to do is store instances of my class and then send them forward either when I reach 100 instances or when 1 minute has passed.
Basically:
List<Message> messages;
public void GotNewMessage(Message msg)
{
messages.add(msg);
if (messages.count() == 100 || timer.elapsed(1 minute))
{
SendMessages(messages);
messages.clear()
}
}
I just can't seem to figure out how to implement this without an excessive use of locks which will slow down the process considerably. Does anyone know of a good way to implement such a system? Thanks in advance.

There is a fantastic library for these kind of requirements (combine time with sequences), it is Reactive Extensions. See https://github.com/Reactive-Extensions/Rx.NET
You could then write something like
void Main()
{
messages
.Buffer(TimeSpan.FromMinutes(1), 100) // Buffer until 100 items or 1 minute has elapsed, whatever comes first.
.Subscribe(msgs => SendMessages(msgs));
}
Subject<Message> messages = new Subject<Message>();
public void GotNewMessage(Message msg)
{
messages.OnNext(msg);
}
Note: this is not production ready but it shows the basic of how to do it. Depending on where you het the messages from there are better ways to create an Observable to subscribe to.
More references:
http://www.introtorx.com/
https://msdn.microsoft.com/en-us/library/hh242985(v=vs.103).aspx
If your message are received using an event you can link the event to a RX stream, see https://msdn.microsoft.com/en-us/library/hh242978(v=vs.103).aspx and https://msdn.microsoft.com/en-us/library/system.reactive.linq.observable.fromeventpattern(v=vs.103).aspx

First of all, you should consider using a ConcurrentQueue<> insted of a List<>. ConcurrentQueue<> is all the way thread safe and needs no additional locks. With this, you have already spared yourself a lock for the message queue.
Interlocked provides atomicity, when it's not available.
According to the C# language specification, independent reads/writes are atomic (but only for some data types and long is not always atomic - that's why I shifted the DateTime.Now.Ticks to get an int32 without losing any bits that would influence the elapsed time) and read-modify-write (eg. ++i) is never atomic.
Shifting (eg. <<) is on its own atomic and doesn't need any additional locking.
private ConcurrentQueue<Message> Queue = new ConcurrentQueue<Message>();
private int QueueSize = 0;
private int LastSend = (int)(DateTime.Now.Ticks >> 23);
private int LastMessage = (int)(DateTime.Now.Ticks >> 23);
public void GotNewMessage(Message Message)
{
Queue.Enqueue(Message);
Interlocked.Increment(ref QueueSize);
Interlocked.Exchange(ref LastMessage, (int)(DateTime.Now.Ticks >> 23));
if (Interlocked.CompareExchange(ref QueueSize, 0, 100) >= 100 ||
LastMessage - LastSend >= 60)
{
Message Dummy;
while (!Queue.IsEmpty)
if (Queue.TryDequeue(out Dummy))
SendMessage(Dummy);
Interlocked.Exchange(ref LastSend, (int)(DateTime.Now.Ticks >> 23));
}
}
public void SendMessage(Message Message)
{
// ...
}
Edit: It may occur, that more than 100 messages are sent out. If you wish to send out strictly 100 messages, you can implement an another atomic incrementation in the cycle.

BinaryFormatter.Deserialize hangs the whole thread

I have two simple applications connected via named pipes. In the client side I have a method that checks incoming messages every n ms:
private void timer_Elapsed(Object sender, ElapsedEventArgs e)
{
IFormatter f = new BinaryFormatter();
try
{
object temp = f.Deserialize(pipeClient); //hangs here
result = (Func<T>)temp;
}
catch
{
}
}
In the beginning the pipe is empty, and f.Deserialize method hangs the whole application. And I can't even check that pipe's empty? Is there any solution to this problem?
UPD: tried XmlSerializer, everything's the same.

The thing that is hanging on you is the pipeClient.Read( call both formatters are making internally.
This is the expected behavior of a Stream, when you call Read:
Return Value
Type: System.Int32
The total number of bytes that are read into buffer. This might be less than the number of bytes
requested if that number of bytes is not currently available, or 0 if
the end of the stream is reached.
So the stream will block till data shows up or throw a timeout exception if it is the type of stream that supports timeouts. It will never just return without reading anything unless you are "at the end of the stream" which for a PipeStream (or similarly a NetworkStream) only happens when the connection is closed.
The way you solve the problem is don't use a timer to check if a new message arrives, just start up a background thread and have it sitting in a loop, it will block itself until a message shows up.
class YourClass
{
public YourClass(PipeStream pipeClient)
{
_pipeClient = pipeClient;
var task = new Task(MessageHandler, TaskCreationOptions.LongRunning);
task.Start();
}
//SNIP...
private void MessageHandler()
{
while(_pipeClient.IsConnected)
{
IFormatter f = new BinaryFormatter();
try
{
object temp = f.Deserialize(_pipeClient);
result = (Func<T>)temp;
}
catch
{
//You really should do some kind of logging.
}
}
}
}

Why UploadProgressChanged in WebClient.UploadFileAsync work not correctly?

I'm upload file and get upload progress like this:
using (var wc = new WebClient())
{
wc.UploadProgressChanged += FileUploadProgressChanged;
wc.Headers.Add(HttpRequestHeader.ContentType, "image/png");
wc.UploadFileAsync(new Uri(url), filePath);
}
...
private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
{
ProgressBarUpload.Value = e.ProgressPercentage;
}
But after 50% e.ProgressPercentage return -441850 and then immediately returns 100. Why is this happening?

My solution:
private void FileUploadProgressChanged(object sender, UploadProgressChangedEventArgs e)
{
ProgressBarUpload.Value = e.BytesSent * 100 / e.TotalBytesToSend;
}
I also found two questions similar to this, but I have not managed to solve the problem. But it can be useful to others:
WebClient UploadFileAsync strange behaviour in progress reporting (cause of the problem - problems with authorization)
Uploading HTTP progress tracking (cause of the problem - the third-party application)
Note. Аfter downloading the file we receive a response from the server, it would be better to display the download file is 95% and the remaining 5% leave to display the response from the server. And in the end after a successful download and response from the server, we will be 100%.
PS: In the code, I did not show it, just say to those who might need it.

Multiple SerialPort returns in real-time

I have made a data logging application in C#. It connects to 4 USB sensors with the SerialPort class. I have the data received event threshold triggered on every byte. When data is received, the program checks to see if that byte is the end of the line. If it isn't, the input is added to a buffer. If it is a line end, the program adds a timestamp and writes the data and timestamp to a file (each input source gets a dedicated file).
Issues arise when using more than one COM port inputs. What I see in the output files is:
Any of the 4 Files:
...
timestamp1 value1
timestamp2 value2
timestamp3 value3
value4
value5
value6
timestamp7 value7
...
So, what it looks like is the computer isn't fast enough to get to all 4 interrupts before the next values arrive. I have good reason to believe that this is the culprit because sometimes I'll see output like this:
...
timestamp value
timestamp value
value
val
timestamp ue
timestamp value
...
It might be due to the fact that I changed the processor affinity to run only on Core 2. I did this because the timestamps I'm using are counted with processor cycles, so I can't have multiple time references depending on which core is running. I've put some of the code snippets below; any suggestions that might help with the dropped timestamps would be greatly appreciated!
public mainLoggerIO()
{
//bind to one cpu
Process proc = Process.GetCurrentProcess();
long AffinityMask = (long)proc.ProcessorAffinity;
AffinityMask &= 0x0002; //use only the 2nd processor
proc.ProcessorAffinity = (IntPtr)AffinityMask;
//prevent any other threads from using core 2
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;
Thread.CurrentThread.Priority = ThreadPriority.Highest;
long frequency = Stopwatch.Frequency;
Console.WriteLine(" Timer frequency in ticks per second = {0}",
frequency);
long nanosecPerTick = (1000L * 1000L * 1000L) / frequency;
Console.WriteLine(" Timer is accurate within {0} nanoseconds",
nanosecPerTick);
if (Stopwatch.IsHighResolution)
MessageBox.Show("High Resolution Timer Available");
else
MessageBox.Show("No High Resolution Timer Available on this Machine");
InitializeComponent();
}
And so on. Each data return interrupt looks like this:
private void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
//serialPort1.DataReceived = serialPort1_DataReceived;
rawPort1Data = "";
rawPort1Data = serialPort1.ReadExisting();
this.Invoke((MethodInvoker)delegate { returnTextPort1(); });
}
The method returnTextPort#() is:
private void returnTextPort1()
{
if (string.Compare(saveFileName, "") == 0)//no savefile specified
return;
bufferedString += rawPort1Data;
if(bufferedString.Contains('\r')){
long timeStamp = DateTime.Now.Ticks;
textBox2.AppendText(nicknames[0] + " " + timeStamp / 10000 + ", " + rawPort1Data);
//write to file
using (System.IO.StreamWriter file = new System.IO.StreamWriter(#saveFileName, true))
{
file.WriteLine(nicknames[0] + " " + timeStamp / 10000 + ", " + rawPort1Data);//in Ms
}
bufferedString = "";
}
}

A cleaner approach would be to use a ConcurrentQueue<T> between the data received event handler and a separate Thread that will deal with the resulting data. That way the event handler can return immediately AND instead of modifying rawPort1 data in a totally non-thread-safe manner you could move to a thread-safe solution.
Create a Thread that reads from the concurrent queue, writes to the file and Invokes the UI changes. Note that writing to the file should NOT be on the UI thread.
Your ConcurrentQueue<T> can capture in the class T that you will implement: the port number, the data received and the timestamp at which it was received.
Note also that DateTime.Now is rarely ever the right answer, for most locations it jumps by an hour twice every year when daylight savings time starts or ends, instead of DateTime.UtcNow. Note however that neither has the accuracy you seem to be trying to obtain with your StopWatch code.
You should not need to manipulate processes or thread priorities to do this: the serial port has a buffer, you'll not miss data provided you handle it efficiently.

Fastest Multi-Threading Method of Serial Port Data Parsing C#

I am currently writing an application that communicates with an integrated servo via a serial connection.
The motor sends out position data at a rate of up to 1000 times/second. What I'm trying to achieve is to be able to format the data coming back (by stripping it of white spaces, new lines, etc) and parsing it to extract the relevant data from the received strings.
Currently, I have the data received event handler read the data, format it using a series of string.replace method calls, and append it to a string that acts as a buffer. Then, using threads, I constantly check the buffer as it fills for a particular delimiter (in my case "\r") which signifies the end of one message from the motor, then remove that message from the buffer and print it to a rich text field.
There are two problems with this approach. One is that because the motor streams position data at such a high rate, the buffer fills faster than the data can be processed by the threads. Thus when I send a command to the motor, it acts immediately but the response is delayed by a few seconds because all of the preceding data in the buffer must be processed first. Second, having two threads running a method that implements a while(true) structure means processor utilization skyrockets and within a few seconds the fans in the pc are on max.
Is there a better way of handling the data?
Here is my event handler code:
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
}
}
Here is the method that the threads execute:
private void parseBuffer()
{
while (true)
{
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}

Your parseBuffer will go wild spinning even if there is no new data since last try.
You can mitigate this with signalling.
private AutoResetEvent waitHandle = new AutoResetEvent(false);
Trigger the signal in dataReceived
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
waitHandle.Set(); // <-- tell parseBuffer that new data is available
}
}
wait for the signal in parseBuffer
private void parseBuffer()
{
while (true)
{
waitHandle.WaitOne(); // <-- waits until there is more data to parse
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}

There are a couple of things you can do to improve this dramatically.
1) Build a state-machine parser that parses the incomming data one character at a time. When it has built a complete "message", add it to a List<MyMessage> structure.
2) Use a Virtualized ListView or DataGridView to display the List<MyMessage>.

In the data received event read the incoming data as raw bytes and store in a queue. Don't process the data in the event handler. Then use something similar to what albin said to process the data in another method. The important thing is to allow the eventhandler to fire as often as possible and do no more that required.

In general I would use a blocking collection between a reader thread which solely reads from the socket and a parsing thread.
In terms of performance look at using split for parsing - this is a lot faster than replacing inside a string. You should look at using a regular expression and/or the StringBuilder class - you should use 1 expression with alternatives
The regex code would look like this:
string pattern = " |\\n|\\r";
string replacement = " ";
regex rgx = new Regex(pattern);
string result = rgx.Replace(input, replacement);