We Keep Coding

Parsing data from serial port thread-safe

I read data from the serial port and parse it in a separate class. However data is incorrectly parsed and some samples are repeated while others are missing.
Here is an example of the parsed packet. It starts with the packetIndex (shoudl start from 1 and incrementing). You can see how the packetIdx repeats and some of the other values repeat as well. I think that's due to multithreading but I'm not sure how to fix it.
2 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
5 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
6 -124801.747477263 -68045.7062179692 -164221.105184687 -163297.46404856
6 -124832.8387538 -68041.9287731563 -164214.936103217 -163294.983004926
This is what I should receive:
1 -124558.985180734 -67934.4168823262 -164223.049786454 -163322.386243628
2 -124619.580759952 -67962.535376851 -164191.757344217 -163305.68949052
3 -124685.719571795 -67995.8394760894 -164191.042088394 -163303.119039907
4 -124801.747477263 -68045.7062179692 -164195.288919841 -163299.140429394
...
This is the SerialPort_DataReceived
public void serialPort1_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
lock (_lock)
{
byte[] buffer = new byte[_serialPort1.BytesToRead];
_serialPort1.Read(buffer, 0, buffer.Length);
for (int i = 0; i < buffer.Length; i++)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
And the class that parses the data:
public class DataParser
{
private int packetSampleCounter = 0;
private int localByteCounter = 0;
private int packetState = 0;
private byte[] tmpBuffer = new byte[3];
private double[] ParsedData = new double[5]; //[0] packetIdx (0-255), [1-4] signal
public double[] interpretBinaryStream(byte actbyte)
{
bool returnDataFlag = false;
switch (packetState)
{
case 0: // end packet indicator
if (actbyte == 0xC0)
packetState++;
break;
case 1: // start packet indicator
if (actbyte == 0xA0)
packetState++;
else
packetState = 0;
break;
case 2: // packet Index
packetSampleCounter = 0;
ParsedData[packetSampleCounter] = actbyte;
packetSampleCounter++;
localByteCounter = 0;
packetState++;
break;
case 3: //channel data (4 channels x 3byte/channel)
// 3 bytes
tmpBuffer[localByteCounter] = actbyte;
localByteCounter++;
if (localByteCounter == 3)
{
ParsedData[packetSampleCounter] = Bit24ToInt32(tmpBuffer);
if (packetSampleCounter == 5)
packetState++; //move to next state, end of packet
else
localByteCounter = 0;
}
break;
case 4: // end packet
if (actbyte == 0xC0)
{
returnDataFlag = true;
packetState = 1;
}
else
packetState = 0;
break;
default:
packetState = 0;
break;
}
if (returnDataFlag)
return ParsedData;
else
return null;
}
}

Get rid of the DataReceived event and instead use await serialPort.BaseStream.ReadAsync(....) to get notified when data comes in. async/await is much cleaner and doesn't force you into multithreaded data processing. For high speed networking, parallel processing is great. But serial ports are slow, so extra threads have no benefit.
Also, BytesToRead is buggy (it does return the number of queued bytes, but it destroys other state) and you should never call it.
Finally, do NOT ignore the return value from Read (or BaseStream.ReadAsync). You need to know how bytes were actually placed into your buffer, because it is not guaranteed to be the same number you asked for.
private async void ReadTheSerialData()
{
var buffer = new byte[200];
while (serialPort.IsOpen) {
var valid = await serialPort.BaseStream.ReadAsync(buffer, 0, buffer.Length);
for (int i = 0; i < valid; ++i)
{
//Parse data
double[] samplesAtTimeT = DataParserObj.interpretBinaryStream(buffer[i]);
//Add data to BlockingCollection when parsed
if (samplesAtTimeT != null)
_bqBufferTimerSeriesData.Add(samplesAtTimeT);
}
}
}
Just call this function after opening the port and setting your flow control, timeouts, etc. You may find that you no longer need the blocking queue, but can just handle the contents of samplesAtTimeT directly.

Serial to send data to ardiuno from C# APP

I have trouble with application with sending data(two trackbars positions) to set PWM frequency and Voltage. I wrote this in arduino:
#include <PWM.h>
#include <Servo.h>
int led = 9;
int32_t frequency = 100;
void setup()
{
InitTimersSafe();
bool success = SetPinFrequencySafe(led, frequency);
}
void loop()
{
//pwmWrite(led, 124);
int channel;
int freq;
int Pwm;
pwmWrite(led, Pwm);
channel = Serial.readStringUntil(':').toInt();
if (channel = 1)
{
Pwm = Serial.readStringUntil('*').toInt();
pwmWrite(led,Pwm);
}
else if (channel = 2)
{
freq = Serial.readStringUntil('*').toInt();
frequency = freq;
pwmWrite(led,Pwm);
}
}
I've seacrhed milion websites, rode serial references and still it doesn't work.
The C# app has connection (COM3) and i see the data transfer when I moving the trackbar.
the C# code:
private void trackBar2_Scroll(object sender, EventArgs e)
{
string bar2 = trackBar2.Value.ToString();
label2.Text = bar2;
int freq = trackBar2.Value;
//port.WriteLine(trackBar2.Value.ToString());
SendFrequencyInfo(1, freq);
}
private void SendFrequencyInfo(int frequency, int trackBar2Pos)
{
string message = frequency.ToString() + ':' + trackBar2Pos.ToString() + '*';
try
{
port.Write(message);
}
catch
{
}
}
The question is:
How to send two values via Serial connection (two trackbar values) to setup frequency and PWM voltage?
Did I miss something, I don't really.
Please help me :)

I think, you should rethink your 'protocol'. Every message should be one line. Read the whole line and use sscanf (http://forum.arduino.cc/index.php?topic=43247.0) to extract your values out of the string. Something like sscanf(Serial.ReadLine(), "%d:%d", &channel, &frequency) should make your logic easier. Check if sscanf is returning 2 (2 variables parsed) and if all variables are set, call pwmWrite(..)
Also you mixed your parameters in the C# SendMethod

C# Wait until data received and go to next iteration

I'm using Serial Port to get data from device in for loop iteration.
The problem is in loop iteration i need to get data from serial port, validate it and going to the next iteration.
How i can achieved this?
Here are my code :
private void processData()
{
// Loop Procedure
int x = Int32.Parse(master["Cycle"].ToString());
int y = Int32.Parse(master["MinWeight"].ToString());
// Loop for each line
for (int i = this.CLine; i < 2; i++)
{
this.CLine = i;
if (i == 0)
label15.Text = master["LLINE"].ToString();
else
label15.Text = master["RLINE"].ToString();
IDictionary<string, string> dic = (Dictionary<String, String>)master[i.ToString()];
label18.Text = this.CProcess = dic["PROCESSID"];
int z = Int32.Parse(dic["PRODLANE"].ToString());
// Loop for each sampling session (Cycle)
for (int j = this.CCycle; j <= x; j++)
{
this.CCycle = j;
// Loop for production lane
for (int k = this.CLane; k <= z; k++)
{
this.CLane = k;
label16.Text = k.ToString();
// In this section i want to send command over serial port
// get value from my device
// validate it if current weight bellow standard weight
// do it again (get data from device)
// else we can go to next iteration
while (this.CWeight < y)
{
XApi.l("xxx2 " + this.CWeight + " vs " + y + " " + k.ToString() + " " + this.isDataReady);
SendData("Q");
}
// Commit Transaction
// XDb.CommitTrans(this.CCycle.ToString(), dic["LINEID"].ToString(), this.CLane.ToString(), weight.ToString(), this.isTrialStage == true ? "1" : "0");
}
}
}
}
I've tried this
while (this.CWeight < y)
{
XApi.l("xxx2 " + this.CWeight + " vs " + y + " " + k.ToString() + " " + this.isDataReady);
SendData("Q");
}
but it seems blocked UI thread and make my application solaggy.
Anyone can give me some idea? Thanks in advance.
private void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
if (e.EventType == System.IO.Ports.SerialData.Eof)
return;
// If the com port has been closed, do nothing
if (!comport.IsOpen)
return;
// Update flag data received
this.isDataReady = true;
// Determain which mode (string or binary) the user is in
if (CurrentDataMode == DataMode.Text)
{
// Read all the data waiting in the buffer
string data = comport.ReadExisting();
// Update result
result += data;
if (result.Length > 16)
{
SetText(result.ToString());
}
// Display the text to the user in the terminal
Log(LogMsgType.Incoming, data);
}
else
{
// Obtain the number of bytes waiting in the port's buffer
int bytes = comport.BytesToRead;
// Create a byte array buffer to hold the incoming data
byte[] buffer = new byte[bytes];
// Read the data from the port and store it in our buffer
comport.Read(buffer, 0, bytes);
// Show the user the incoming data in hex format
Log(LogMsgType.Incoming, ByteArrayToHexString(buffer));
}
}
private void SendData(String msg)
{
this.isDataReady = false;
result = "";
if (CurrentDataMode == DataMode.Text)
{
// Send the user's text straight out the port
comport.Write(msg + "\r\n");
// Show in the terminal window the user's text
Log(LogMsgType.Outgoing, msg + "\n");
}
else
{
try
{
// Convert the user's string of hex digits (ex: B4 CA E2) to a byte array
byte[] data = HexStringToByteArray(txtSendData.Text);
// Send the binary data out the port
comport.Write(data, 0, data.Length);
// Show the hex digits on in the terminal window
Log(LogMsgType.Outgoing, ByteArrayToHexString(data) + "\n");
}
catch (FormatException)
{
// Inform the user if the hex string was not properly formatted
Log(LogMsgType.Error, "Not properly formatted hex string: " + txtSendData.Text + "\n");
}
}
}

Anyone can give me some idea?
You can use async/await in your code not to block your UI by writing an extension method like below. Usage would be:
async void SomeMethod()
{
SerialPort serialPort = .......
while (true)
{
serialPort.Write(.....);
var retval = await serialPort.ReadAsync();
}
}
The keyword here is using TaskCompletionSource class with your events...
static public class SerialPortExtensions
{
public static Task<byte[]> ReadAsync(this SerialPort serialPort)
{
var tcs = new TaskCompletionSource<byte[]>();
SerialDataReceivedEventHandler dataReceived = null;
dataReceived = (s, e) =>
{
serialPort.DataReceived -= dataReceived;
var buf = new byte[serialPort.BytesToRead];
serialPort.Read(buf, 0, buf.Length);
tcs.TrySetResult(buf);
};
serialPort.DataReceived += dataReceived;
return tcs.Task;
}
}

Thread with while (true) loop exits somehow

I have a thread that is supposed to run continuously in my program and parse incoming serial data from a collection of sensors.
//initialized as a global variable
System.Threading.Thread processThread = new System.Threading.Thread(ProcessSerialData);
//this gets called when you press the start button
processThread.Start();
private void ProcessSerialData()
{
while (true)
{
//do a whole bunch of parsing stuff
}
int howDidYouGetHere = 0;
}
How is it possible that my program is reaching the "int howDidYouGetHere = 0" line??
Full code can be found here:
/// <summary>
/// ProcessSerialData thread will be used to continue testing the connection to the controller,
/// process messages in the incoming message queue (actually a linked list),
/// and sends new messages for updated data.
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void ProcessSerialData()
{
while (true)
{
UpdateAhTextbox(test.ampHoursOut.ToString());
if (!relayBoard.OpenConn())
{
MessageBox.Show("Connection to controller has been lost.");
testInterrupted = "Lost connection. Time = " + test.duration;
UpdateStatusText(false);
ShutErDown();
}
/////////////////////////////////////////////////////
if (incomingData.Count > 0)
{
string dataLine = "";
try
{
dataLine = incomingData.First();
UpdateParsedDataTextbox(dataLine + "\r\n");
}
catch (System.InvalidOperationException emai)
{
break; //data hasn't come in yet, it will by the time this runs next.
}
incomingData.RemoveFirst();
if (dataLine.Contains("GET")) // some sensor values (analog/temp/digital) has come in
{
if (dataLine.Contains("GETANALOG")) // an analog value has come in
{
int index = dataLine.IndexOf("CH");
int pin = (int)Char.GetNumericValue(dataLine[index + 2]);
double value = 0;
int dataLineLength = dataLine.Length;
if (dataLineLength > 13) // value is appended to end of line
{
try
{
value = Convert.ToDouble(dataLine.Substring(13));
}
catch // can't convert to double
{
int index2 = dataLine.IndexOf("CH", 3);
if (index2 != -1) // there happen to be two sets of commands stuck together into one
{
string secondHalf = dataLine.Substring(index2);
incomingData.AddFirst(secondHalf);
}
}
}
else // value is on the next line
{
try
{
value = Convert.ToDouble(incomingData.First());
incomingData.RemoveFirst();
}
catch // can't convert to double
{
MessageBox.Show("Error occured: " + dataLine);
}
}
switch (pin)
{
case 1:
ReadVoltage(value);
break;
case 2:
ReadAmperage(value);
break;
}
}
else if (dataLine.Contains("GETTEMP")) // received reply with temperature data
{
int index = dataLine.IndexOf("CH");
int pin = (int)Char.GetNumericValue(dataLine[index + 2]); // using index of CH, retrieve which pin this message is coming from
double value = 0;
int dataLineLength = dataLine.Length;
if (dataLineLength > 11) // value is appended to end of line
{
try
{
value = Convert.ToDouble(dataLine.Substring(11));
}
catch // can't convert to double
{
int index2 = dataLine.IndexOf("CH", 3);
if (index2 != -1) // there happen to be two sets of commands stuck together into one
{
string secondHalf = dataLine.Substring(index2);
incomingData.AddFirst(secondHalf);
}
}
}
else // value is on the next line
{
value = Convert.ToDouble(incomingData.First());
incomingData.RemoveFirst();
}
ReadTemperature(pin, value);
}
else // must be CH3.GET
{
int index = dataLine.IndexOf("CH");
int pin = (int)Char.GetNumericValue(dataLine[index + 2]); // using index of CH, retrieve which pin this message is coming from
if (pin == 3) // only care if it's pin 3 (BMS), otherwise it's a mistake
{
double value = 0;
int dataLineLength = dataLine.Length;
if (dataLineLength > 7) // value is appended to end of line
{
try
{
value = Convert.ToDouble(dataLine.Substring(7));
}
catch // can't convert to double
{
int index2 = dataLine.IndexOf("CH", 3);
if (index2 != -1) // there happen to be two sets of commands stuck together into one
{
string secondHalf = dataLine.Substring(index2);
incomingData.AddFirst(secondHalf);
}
}
}
else // value is on the next line
{
value = Convert.ToDouble(incomingData.First());
incomingData.RemoveFirst();
}
ReadBMS(value);
}
}
}
else if (dataLine.Contains("REL")) // received reply about relay turning on or off.
{
if (dataLine.Contains("RELS")) // all relays turning on/off
{
if (dataLine.Contains("ON"))
{
for (int pin = 1; pin <= 4; pin++)
{
test.contactors[pin] = true;
}
}
else // "OFF"
{
for (int pin = 1; pin <= 4; pin++)
{
test.contactors[pin] = false;
}
}
}
else // single relay is turning on/off
{
int index = dataLine.IndexOf("REL");
int pin = (int)Char.GetNumericValue(dataLine[index + 3]);
if (dataLine.Contains("ON"))
{
test.contactors[pin] = true;
}
else if (dataLine.Contains("OFF"))
{
test.contactors[pin] = false;
}
else if (dataLine.Contains("GET"))
{
if (Convert.ToInt32(incomingData.First()) == 1)
test.contactors[pin] = true;
else
test.contactors[pin] = false;
incomingData.RemoveFirst();
}
}
}
}
/////////////////////////////////////////////////////
// we only want more data if we're done processing the current data, otherwise we're stuck with too much and processing is heavily delayed.
if (isTestRunning && incomingData.Count < 3)
{
//read & output v, a, bms state
sendToController("CH1.GETANALOG"); // get voltage
sendToController("CH2.GETANALOG"); // get amperage
sendToController("CH3.GET"); // get BMS state
//read & output temperature
sendToController("CH4.GETTEMP"); // get temperature
sendToController("CH5.GETTEMP");
string lines = "Ah Out: " + test.ampHoursOut + ", Voltage: " + test.voltage +
", Amperage: " + test.amperage + ", Cell Temperature: " + test.tempBattery +
", Load Temperature: " + test.tempLoad;
WriteToLog(lines);
}
}
int howDidYouGetHere = 0;
}

The break (ignoring those inside the nested switch) breaks out of the while loop.

Perhaps you have tried to update your UI using UpdateParsedDataTextbox This will cause InvalidOperationException(then breaks the while loop) because you tries to access the UI control from thread that doesn't own the control.

Circular log TextBox for raw serial characters

The problem
From the serial port I receive a stream of characters. The stream I receive will be terminated by \n. I receive the serial port data in parts like this:
Serial port Event 1: "123\n456\n789"
Serial port Event 2: "\n0123\n456\n789\n"
Serial port Event 3: "0123\n456\n789\n"
As you can see my stream fluctuates, and this is very normal since I read the serial port with "what is available currently".
My problem is that I want to log this to the UI with a RichTextBox. I cannot use ListBox because I need color and font formatting.
First approach
Before I tried this below and that works very well till the time the messages exceed around 30.000 lines of text. The UI gets unresponsive. This was the code:
uiTextActivity.SelectionFont = new Font(uiTextActivity.SelectionFont, FontStyle.Bold);
uiTextActivity.SelectionColor = LogMsgTypeColor[(int)msgtype];
uiTextActivity.AppendText(msg);
uiTextActivity.ScrollToCaret();
Later on I used this as a quick fix that clears the box after 2000 lines:
if ((int.Parse(uiLabelCounterActivity.Text) + 1) > 2000)
uiTextBoxResetActivity();
I want to keep a history of around 500lines.
But with this quick fix above I lose my log completely when the counter hits 2000.
I think what I need is a circular FIFO TextBox. So I can remove after 500 lines the oldest logs and append the new one.
Second approach
I also tried this (note that in this case below the newest logs entries are on top and the oldest below in the textbox)
msg = msgToPasteWhenComplete + msg;
int c = 0; // c=zero if no newline, 1 for single newline, 2 for 2times newline and so on
int latestTermination = 0;// store the point where \n is found
for (int e = 0; e < msg.Length; e++)
{
if (msg[e] == '\n')
{
c++;
latestTermination = e+1;
}
}
// contains strings all terminated with \n
string msgToPaste = msg.Substring(0, latestTermination);
// contains string that is not complete (not yet terminated with \n)
msgToPasteWhenComplete = msg.Substring(latestTermination, msg.Length - latestTermination);
string previous = msgToPaste + uiTextActivity.Text;
if (previous.Length > maxDisplayTextLength)
{
string debugInfo3 = previous.Substring(0, maxDisplayTextLength);
uiTextActivity.Text = debugInfo3;
}
else
uiTextActivity.Text = previous;
This works almost very well. The problem with this approach is that the line that comes from the serial port will not be pasted to the UI until the \n is received. This means that whenever communication is slow, I will have to wait for the serial stream to complete the whole line including \n before I can see it... What I want is to see every character directly.
info about serialport
The serial data am I getting from the SerialDataReceivedEvent, in that event I use comport.ReadExisting() to have a non-blocking event.
The serial data comes from my embedded programming board that has analog readings. The analog readings supply me with 20 lines per second, each line containing 20 chars.
I need the raw data to be read in a user interface where it has to be filtered to other textboxes depending on the prefix of the serial message (like err goes to error , warn goes to warning textbox, but they all go to the activity log. The activity log is what this question is about.

I wish I could give you a specific chunk of code to work off of, but I haven't had to do this kind of input processing in a while.
That being said, you might get better performance by buffering your inputs into a Queue (MSDN reference) object, and then polling the queue on a timer or by responding to some other event (maybe OnChanged?).

I found a solution that works, here is my code:
private const int maxDisplayTextLength = 5000;
private string splitString = "";
private int activityCount = 0;
private string errorMessageBox = "";
private bool errorMessageBoxNeedUpdate = true;
private int errorMessageBoxCount = 0;
private string filteredMessageBox = "";
private int filteredMessageCount = 0;
private bool filteredMessageBoxNeedUpdate = true;
private string activityLogFilterKeyword = "Warning";
string logHelperStringMaxSizeLimiter(string input)
{
// check if our buffer isn't getting bigger than our specified max. length
if (input.Length > maxDisplayTextLength)
{
// discard the oldest data in our buffer (FIFO) so we have room for our newest values
input = input.Substring(input.Length - maxDisplayTextLength);
}
return input;
}
private void logHelperIncoming(string msg)
{
msg = msg.Replace('\0', '\n'); // remove \0 NULL characters as they have special meanings in C# RichTextBox
// add the new received string to our buffer
splitString += msg;
// filter out special strings
int searchIndexStart = splitString.Length - msg.Length;
for (int e = searchIndexStart; e < splitString.Length; e++)
{
if (splitString[e] == '\n')
{
activityCount++;
string subString = splitString.Substring(searchIndexStart, e - searchIndexStart) + "\n";
searchIndexStart += e - searchIndexStart + 1; // update searchindex for next search
string filterError = "error";
// filter messages that start with error
if (subString.Length > filterError.Length) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, filterError, 0, 4, true) == 0)
{
errorMessageBox += subString;
errorMessageBoxNeedUpdate = true;
errorMessageBoxCount++;
}
}
// filter messages that start with XXX
if (subString.Length > activityLogFilterKeyword.Length && activityLogFilterKeyword.Length != 0) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, activityLogFilterKeyword, 0, activityLogFilterKeyword.Length, true) == 0)
{
filteredMessageBox += subString;
filteredMessageBoxNeedUpdate = true;
filteredMessageCount++;
}
}
}
}
}
// outputs to main activity textbox
private void Log(LogMsgType msgtype, string msg)
{
if (msgtype == LogMsgType.Incoming || msgtype == LogMsgType.Normal)
{
logHelperIncoming(msg);
}
else if (msgtype == LogMsgType.Outgoing)
{
}
splitString = logHelperStringMaxSizeLimiter(splitString);
filteredMessageBox = logHelperStringMaxSizeLimiter(filteredMessageBox);
errorMessageBox = logHelperStringMaxSizeLimiter(errorMessageBox);
uiTextActivity.Invoke(new EventHandler(delegate
{
// time to post our updated buffer to the UI
uiTextActivity.Text = splitString;
uiTextActivity.Update();
// scroll to the newest data only if user has no focus on the
uiTextActivity.SelectionStart = uiTextActivity.TextLength; // make sure view is to the latest
uiTextActivity.ScrollToCaret();
uiLabelCounterActivity.Text = activityCount.ToString();
if (errorMessageBoxNeedUpdate)
{
errorMessageBoxNeedUpdate = false;
uiTextActivity.SelectionColor = Color.Red;
// time to post our updated buffer to the UI
uiTextboxErrorLog.Text = errorMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxErrorLog.SelectionStart = uiTextboxErrorLog.TextLength; // make sure view is to the latest
uiTextboxErrorLog.ScrollToCaret();
uiLabelCounterError.Text = errorMessageBoxCount.ToString();
}
if (filteredMessageBoxNeedUpdate)
{
filteredMessageBoxNeedUpdate = false;
// time to post our updated buffer to the UI
uiTextboxFilteredLog.Text = filteredMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxFilteredLog.SelectionStart = uiTextboxFilteredLog.TextLength; // make sure view is to the latest
uiTextboxFilteredLog.ScrollToCaret();
uiLabelCounterFiltered.Text = filteredMessageCount.ToString();
}
// extract value from filter and store to filter
activityLogFilterKeyword = uiTextboxFilterKeyword.Text;
}));
}