We Keep Coding

Parsing data from serial port thread-safe

I read data from the serial port and parse it in a separate class. However data is incorrectly parsed and some samples are repeated while others are missing.
Here is an example of the parsed packet. It starts with the packetIndex (shoudl start from 1 and incrementing). You can see how the packetIdx repeats and some of the other values repeat as well. I think that's due to multithreading but I'm not sure how to fix it.
2 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
5 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
6 -124801.747477263 -68045.7062179692 -164221.105184687 -163297.46404856
6 -124832.8387538 -68041.9287731563 -164214.936103217 -163294.983004926
This is what I should receive:
1 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
4 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
...
This is the SerialPort_DataReceived
public void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
lock (_lock)
{
byte[] buffer = new byte[_serialPort1.BytesToRead];
_serialPort1.Read(buffer, 0, buffer.Length);
for (int i = 0; i < buffer.Length; i++)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
And the class that parses the data:
public class DataParser
{
private int packetSampleCounter = 0;
private int localByteCounter = 0;
private int packetState = 0;
private byte[] tmpBuffer = new byte[3];
private double[] ParsedData = new double[5]; //[0] packetIdx (0-255), [1-4] signal
public double[] interpretBinaryStream(byte actbyte)
{
bool returnDataFlag = false;
switch (packetState)
{
case 0: // end packet indicator
if (actbyte == 0xC0)
packetState++;
break;
case 1: // start packet indicator
if (actbyte == 0xA0)
packetState++;
else
packetState = 0;
break;
case 2: // packet Index
packetSampleCounter = 0;
ParsedData[packetSampleCounter] = actbyte;
packetSampleCounter++;
localByteCounter = 0;
packetState++;
break;
case 3: //channel data (4 channels x 3byte/channel)
// 3 bytes
tmpBuffer[localByteCounter] = actbyte;
localByteCounter++;
if (localByteCounter == 3)
{
ParsedData[packetSampleCounter] = Bit24ToInt32(tmpBuffer);
if (packetSampleCounter == 5)
packetState++; //move to next state, end of packet
else
localByteCounter = 0;
}
break;
case 4: // end packet
if (actbyte == 0xC0)
{
returnDataFlag = true;
packetState = 1;
}
else
packetState = 0;
break;
default:
packetState = 0;
break;
}
if (returnDataFlag)
return ParsedData;
else
return null;
}
}

Get rid of the DataReceived event and instead use await serialPort.BaseStream.ReadAsync(....) to get notified when data comes in. async/await is much cleaner and doesn't force you into multithreaded data processing. For high speed networking, parallel processing is great. But serial ports are slow, so extra threads have no benefit.
Also, BytesToRead is buggy (it does return the number of queued bytes, but it destroys other state) and you should never call it.
Finally, do NOT ignore the return value from Read (or BaseStream.ReadAsync). You need to know how bytes were actually placed into your buffer, because it is not guaranteed to be the same number you asked for.
private async void ReadTheSerialData()
{
var buffer = new byte[200];
while (serialPort.IsOpen) {
var valid = await serialPort.BaseStream.ReadAsync(buffer, 0, buffer.Length);
for (int i = 0; i < valid; ++i)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
Just call this function after opening the port and setting your flow control, timeouts, etc. You may find that you no longer need the blocking queue, but can just handle the contents of samplesAtTimeT directly.

Looping through stream data

I'm writing a program that uses a fingerprint reader. I have stored the fingerprint data in an array [arr]. Unfortunately, only the first value is read i.e [0]. So only one finger is detected and the rest are ignored but if I place a specific number in the array e.g 2. It works fine for that value alone:
Here's my code:
for (int x = 0; x < (arr.Length - 1); x++)
{
byte[] fpbyte = GetStringToBytes(arr[x]);
Stream stream = new MemoryStream(fpbyte);
Data.Templates[x] = new DPFP.Template(stream);
}
foreach (DPFP.Template template in Data.Templates)
{
// Get template from storage.
if (template != null)
{
// Compare feature set with particular template.
ver.Verify(FeatureSet, template, ref res);
Data.IsFeatureSetMatched = res.Verified;
Data.FalseAcceptRate = res.FARAchieved;
if (res.Verified)
MessageBox.Show("Yes");
break; // success
}
}
if (!res.Verified)
Status = DPFP.Gui.EventHandlerStatus.Failure;
MessageBox.Show("No");
Data.Update();

You unconditionally break from your loop, whether verified or not.
Your code should read :
if (res.Verified) {
MessageBox.Show("Yes");
break; // success
}
This is a good example why good coding practice suggests always having the brackets, even for a one line conditional effect, as the error would have been much more obvious.
Similarly you should have written
if (!res.Verified) {
Status = DPFP.Gui.EventHandlerStatus.Failure;
MessageBox.Show("No");
}
at the end of your snippet.

Thanks to Dragonthoughts, I made the following changes and the code works just fine:
for (int x = 0; x < (arr.Length - 1); x++)
{
byte[] fpbyte = GetStringToBytes(arr[x]);
using (Stream stream = new MemoryStream(fpbyte))
{
Data.Templates[x] = new DPFP.Template(stream);
// Get template from storage.
if (Data.Templates[x] != null)
{
// Compare feature set with particular template.
ver.Verify(FeatureSet, Data.Templates[x], ref res);
Data.IsFeatureSetMatched = res.Verified;
Data.FalseAcceptRate = res.FARAchieved;
if (res.Verified)
{
status.Text = "Verified";
break; // success
}
}
}
}
if (!res.Verified)
{
Status = DPFP.Gui.EventHandlerStatus.Failure;
status.Text = "Unverified";
}
Data.Update();

C# Wait until data received and go to next iteration

I'm using Serial Port to get data from device in for loop iteration.
The problem is in loop iteration i need to get data from serial port, validate it and going to the next iteration.
How i can achieved this?
Here are my code :
private void processData()
{
// Loop Procedure
int x = Int32.Parse(master["Cycle"].ToString());
int y = Int32.Parse(master["MinWeight"].ToString());
// Loop for each line
for (int i = this.CLine; i < 2; i++)
{
this.CLine = i;
if (i == 0)
label15.Text = master["LLINE"].ToString();
else
label15.Text = master["RLINE"].ToString();
IDictionary<string, string> dic = (Dictionary<String, String>)master[i.ToString()];
label18.Text = this.CProcess = dic["PROCESSID"];
int z = Int32.Parse(dic["PRODLANE"].ToString());
// Loop for each sampling session (Cycle)
for (int j = this.CCycle; j <= x; j++)
{
this.CCycle = j;
// Loop for production lane
for (int k = this.CLane; k <= z; k++)
{
this.CLane = k;
label16.Text = k.ToString();
// In this section i want to send command over serial port
// get value from my device
// validate it if current weight bellow standard weight
// do it again (get data from device)
// else we can go to next iteration
while (this.CWeight < y)
{
XApi.l("xxx2 " + this.CWeight + " vs " + y + " " + k.ToString() + " " + this.isDataReady);
SendData("Q");
}
// Commit Transaction
// XDb.CommitTrans(this.CCycle.ToString(), dic["LINEID"].ToString(), this.CLane.ToString(), weight.ToString(), this.isTrialStage == true ? "1" : "0");
}
}
}
}
I've tried this
while (this.CWeight < y)
{
XApi.l("xxx2 " + this.CWeight + " vs " + y + " " + k.ToString() + " " + this.isDataReady);
SendData("Q");
}
but it seems blocked UI thread and make my application solaggy.
Anyone can give me some idea? Thanks in advance.
private void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
if (e.EventType == System.IO.Ports.SerialData.Eof)
return;
// If the com port has been closed, do nothing
if (!comport.IsOpen)
return;
// Update flag data received
this.isDataReady = true;
// Determain which mode (string or binary) the user is in
if (CurrentDataMode == DataMode.Text)
{
// Read all the data waiting in the buffer
string data = comport.ReadExisting();
// Update result
result += data;
if (result.Length > 16)
{
SetText(result.ToString());
}
// Display the text to the user in the terminal
Log(LogMsgType.Incoming, data);
}
else
{
// Obtain the number of bytes waiting in the port's buffer
int bytes = comport.BytesToRead;
// Create a byte array buffer to hold the incoming data
byte[] buffer = new byte[bytes];
// Read the data from the port and store it in our buffer
comport.Read(buffer, 0, bytes);
// Show the user the incoming data in hex format
Log(LogMsgType.Incoming, ByteArrayToHexString(buffer));
}
}
private void SendData(String msg)
{
this.isDataReady = false;
result = "";
if (CurrentDataMode == DataMode.Text)
{
// Send the user's text straight out the port
comport.Write(msg + "\r\n");
// Show in the terminal window the user's text
Log(LogMsgType.Outgoing, msg + "\n");
}
else
{
try
{
// Convert the user's string of hex digits (ex: B4 CA E2) to a byte array
byte[] data = HexStringToByteArray(txtSendData.Text);
// Send the binary data out the port
comport.Write(data, 0, data.Length);
// Show the hex digits on in the terminal window
Log(LogMsgType.Outgoing, ByteArrayToHexString(data) + "\n");
}
catch (FormatException)
{
// Inform the user if the hex string was not properly formatted
Log(LogMsgType.Error, "Not properly formatted hex string: " + txtSendData.Text + "\n");
}
}
}

Anyone can give me some idea?
You can use async/await in your code not to block your UI by writing an extension method like below. Usage would be:
async void SomeMethod()
{
SerialPort serialPort = .......
while (true)
{
serialPort.Write(.....);
var retval = await serialPort.ReadAsync();
}
}
The keyword here is using TaskCompletionSource class with your events...
static public class SerialPortExtensions
{
public static Task<byte[]> ReadAsync(this SerialPort serialPort)
{
var tcs = new TaskCompletionSource<byte[]>();
SerialDataReceivedEventHandler dataReceived = null;
dataReceived = (s, e) =>
{
serialPort.DataReceived -= dataReceived;
var buf = new byte[serialPort.BytesToRead];
serialPort.Read(buf, 0, buf.Length);
tcs.TrySetResult(buf);
};
serialPort.DataReceived += dataReceived;
return tcs.Task;
}
}

Why sometimes I received some random data from serial port? using c#

Data supposed to be received should be something like"-31.12345 167.12345", but sometimes I received data like "2.378999E11 3.593719E"? I have checked my port setting they are using same parameter. My code as following:
private void DataReceivedHandler(object sender, SerialDataReceivedEventArgs e)
{
byte rxChar=0;
byte[] rxDataBuffer = new byte[240];
int rxdata = 0;
rxNumberDataBytes = (byte)serialPort.BytesToRead;
//SerialPort sp = (SerialPort)sender;
//rxdata = sp.Read(rxDataBuffer, 0, 8);
while (serialPort.BytesToRead > 0)
{
rxChar = (byte)serialPort.ReadByte(); // read data
// to display bytes received in hex,check was all data received
Invoke(new Action(() =>
{
textBox3.Text += " " + rxChar.ToString("X") + "\r\n";
}));
switch (rxState)
{
case RxIdleState:
if (rxChar == 0x5A)
{
rxState = RxInstructionState;
}
break;
case RxInstructionState:
rxInstruction = rxChar;
rxState = RxNumberofbytesState;
break;
case RxNumberofbytesState:
rxChar = rxNumberDataBytes;
rxState = RxDataState;
rxdata = 0;
break;
case RxDataState:
// count number of rxdata until 8 bytes real data
rxDataBuffer[rxdata] = rxChar;
rxdata++;
if (rxdata == 8)
{
float f11 = BitConverter.ToSingle(rxDataBuffer, 0);
float f22 = BitConverter.ToSingle(rxDataBuffer, 4);
Invoke(new Action(() =>
{
textBox3.Text += f11.ToString() + " " + f22.ToString() + "\r\n";
}));
rxState = RxStopState;
}
break;
case RxStopState:
if (rxChar == 0x2C)
{
rxState = RxIdleState;
}
break;
default:
rxState = RxIdleState;
break;
}
}
}
}

I think you're grabbing wrong data due to a wrong set operation.
In the following, you are setting the number of bytes you want and then setting the rxChar to the first available byte in the stream. That seems fine.
rxNumberDataBytes = (byte)serialPort.BytesToRead;
while (serialPort.BytesToRead > 0)
{
rxChar = (byte)serialPort.ReadByte(); // read data
....
}
However, in the following, you are then ignoring that rxChar value and setting it to the number of bytes that were available for reading.
case RxNumberofbytesState:
rxChar = rxNumberDataBytes; // <--- This is correct?
rxState = RxDataState;
rxdata = 0;
break;
You are then using the new rxChar value to add to your rxDataBuffer instead of the first byte that was available in the serial port.
Looking at the flow of the code, the rest of the 7 bytes look like they will be correct, but the first one will always be wrong.
May also just be floating point precision issue ;)

Circular log TextBox for raw serial characters

The problem
From the serial port I receive a stream of characters. The stream I receive will be terminated by \n. I receive the serial port data in parts like this:
Serial port Event 1: "123\n456\n789"
Serial port Event 2: "\n0123\n456\n789\n"
Serial port Event 3: "0123\n456\n789\n"
As you can see my stream fluctuates, and this is very normal since I read the serial port with "what is available currently".
My problem is that I want to log this to the UI with a RichTextBox. I cannot use ListBox because I need color and font formatting.
First approach
Before I tried this below and that works very well till the time the messages exceed around 30.000 lines of text. The UI gets unresponsive. This was the code:
uiTextActivity.SelectionFont = new Font(uiTextActivity.SelectionFont, FontStyle.Bold);
uiTextActivity.SelectionColor = LogMsgTypeColor[(int)msgtype];
uiTextActivity.AppendText(msg);
uiTextActivity.ScrollToCaret();
Later on I used this as a quick fix that clears the box after 2000 lines:
if ((int.Parse(uiLabelCounterActivity.Text) + 1) > 2000)
uiTextBoxResetActivity();
I want to keep a history of around 500lines.
But with this quick fix above I lose my log completely when the counter hits 2000.
I think what I need is a circular FIFO TextBox. So I can remove after 500 lines the oldest logs and append the new one.
Second approach
I also tried this (note that in this case below the newest logs entries are on top and the oldest below in the textbox)
msg = msgToPasteWhenComplete + msg;
int c = 0; // c=zero if no newline, 1 for single newline, 2 for 2times newline and so on
int latestTermination = 0;// store the point where \n is found
for (int e = 0; e < msg.Length; e++)
{
if (msg[e] == '\n')
{
c++;
latestTermination = e+1;
}
}
// contains strings all terminated with \n
string msgToPaste = msg.Substring(0, latestTermination);
// contains string that is not complete (not yet terminated with \n)
msgToPasteWhenComplete = msg.Substring(latestTermination, msg.Length - latestTermination);
string previous = msgToPaste + uiTextActivity.Text;
if (previous.Length > maxDisplayTextLength)
{
string debugInfo3 = previous.Substring(0, maxDisplayTextLength);
uiTextActivity.Text = debugInfo3;
}
else
uiTextActivity.Text = previous;
This works almost very well. The problem with this approach is that the line that comes from the serial port will not be pasted to the UI until the \n is received. This means that whenever communication is slow, I will have to wait for the serial stream to complete the whole line including \n before I can see it... What I want is to see every character directly.
info about serialport
The serial data am I getting from the SerialDataReceivedEvent, in that event I use comport.ReadExisting() to have a non-blocking event.
The serial data comes from my embedded programming board that has analog readings. The analog readings supply me with 20 lines per second, each line containing 20 chars.
I need the raw data to be read in a user interface where it has to be filtered to other textboxes depending on the prefix of the serial message (like err goes to error , warn goes to warning textbox, but they all go to the activity log. The activity log is what this question is about.

I wish I could give you a specific chunk of code to work off of, but I haven't had to do this kind of input processing in a while.
That being said, you might get better performance by buffering your inputs into a Queue (MSDN reference) object, and then polling the queue on a timer or by responding to some other event (maybe OnChanged?).

I found a solution that works, here is my code:
private const int maxDisplayTextLength = 5000;
private string splitString = "";
private int activityCount = 0;
private string errorMessageBox = "";
private bool errorMessageBoxNeedUpdate = true;
private int errorMessageBoxCount = 0;
private string filteredMessageBox = "";
private int filteredMessageCount = 0;
private bool filteredMessageBoxNeedUpdate = true;
private string activityLogFilterKeyword = "Warning";
string logHelperStringMaxSizeLimiter(string input)
{
// check if our buffer isn't getting bigger than our specified max. length
if (input.Length > maxDisplayTextLength)
{
// discard the oldest data in our buffer (FIFO) so we have room for our newest values
input = input.Substring(input.Length - maxDisplayTextLength);
}
return input;
}
private void logHelperIncoming(string msg)
{
msg = msg.Replace('\0', '\n'); // remove \0 NULL characters as they have special meanings in C# RichTextBox
// add the new received string to our buffer
splitString += msg;
// filter out special strings
int searchIndexStart = splitString.Length - msg.Length;
for (int e = searchIndexStart; e < splitString.Length; e++)
{
if (splitString[e] == '\n')
{
activityCount++;
string subString = splitString.Substring(searchIndexStart, e - searchIndexStart) + "\n";
searchIndexStart += e - searchIndexStart + 1; // update searchindex for next search
string filterError = "error";
// filter messages that start with error
if (subString.Length > filterError.Length) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, filterError, 0, 4, true) == 0)
{
errorMessageBox += subString;
errorMessageBoxNeedUpdate = true;
errorMessageBoxCount++;
}
}
// filter messages that start with XXX
if (subString.Length > activityLogFilterKeyword.Length && activityLogFilterKeyword.Length != 0) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, activityLogFilterKeyword, 0, activityLogFilterKeyword.Length, true) == 0)
{
filteredMessageBox += subString;
filteredMessageBoxNeedUpdate = true;
filteredMessageCount++;
}
}
}
}
}
// outputs to main activity textbox
private void Log(LogMsgType msgtype, string msg)
{
if (msgtype == LogMsgType.Incoming || msgtype == LogMsgType.Normal)
{
logHelperIncoming(msg);
}
else if (msgtype == LogMsgType.Outgoing)
{
}
splitString = logHelperStringMaxSizeLimiter(splitString);
filteredMessageBox = logHelperStringMaxSizeLimiter(filteredMessageBox);
errorMessageBox = logHelperStringMaxSizeLimiter(errorMessageBox);
uiTextActivity.Invoke(new EventHandler(delegate
{
// time to post our updated buffer to the UI
uiTextActivity.Text = splitString;
uiTextActivity.Update();
// scroll to the newest data only if user has no focus on the
uiTextActivity.SelectionStart = uiTextActivity.TextLength; // make sure view is to the latest
uiTextActivity.ScrollToCaret();
uiLabelCounterActivity.Text = activityCount.ToString();
if (errorMessageBoxNeedUpdate)
{
errorMessageBoxNeedUpdate = false;
uiTextActivity.SelectionColor = Color.Red;
// time to post our updated buffer to the UI
uiTextboxErrorLog.Text = errorMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxErrorLog.SelectionStart = uiTextboxErrorLog.TextLength; // make sure view is to the latest
uiTextboxErrorLog.ScrollToCaret();
uiLabelCounterError.Text = errorMessageBoxCount.ToString();
}
if (filteredMessageBoxNeedUpdate)
{
filteredMessageBoxNeedUpdate = false;
// time to post our updated buffer to the UI
uiTextboxFilteredLog.Text = filteredMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxFilteredLog.SelectionStart = uiTextboxFilteredLog.TextLength; // make sure view is to the latest
uiTextboxFilteredLog.ScrollToCaret();
uiLabelCounterFiltered.Text = filteredMessageCount.ToString();
}
// extract value from filter and store to filter
activityLogFilterKeyword = uiTextboxFilterKeyword.Text;
}));
}