We Keep Coding

Data loss during Serial.Write method

Now I would like to use WinForms to develop a GUI to control a embedded board.
Between WinForm and the embedded board, serial communication is utilized.
Currently, result of reading data is well.
However, writing data with Serial.Write method did not perform well.
https://learn.microsoft.com/zh-tw/dotnet/api/system.io.ports.serialport.write?view=netframework-4.8
Some characters were not received by the board.
the result is show in the following:
private void ButtonSendCommand_Click(object sender, EventArgs e)
{
byte[] txdata = Encoding.UTF8.GetBytes(TXTBoxCommand.Text);
OBCSerialPort.Write(txdata, 0, txdata.Length);
OBCSerialPort.Write(Environment.NewLine);
}
in my embedded board, I use scanf to get command and decode them
scanf("%[^\n]%*c", cStr);
And decode function:
void decode_context(char *str)
{
uint8_t i = 0;
uint8_t argc = 0; //number of argument
char *p = strtok(str, " ");
char *cBuf[MAX_INPUT_LEN];
command_t* pCase = 0;
struct cmd_context ctx;
// Split input string with space
while (p != NULL){
cBuf[i++] = p;
p = strtok (NULL, " ");
argc++;
}
if(argc < Max_ARGC){
// Duplicate arg & argc to command context
memcpy(ctx.arg, cBuf, sizeof(cBuf[0])*argc);
memcpy(&ctx.argc, &argc, sizeof(uint8_t));
// Linear serach for key of instruction
for(pCase = command
; pCase != command + sizeof(command) / sizeof(command[0])
; pCase++)
{
if(0 == strcmp(pCase->name, ctx.arg[0])) {
(*pCase->handler)(&ctx);
break;
}
if(pCase == command-1 + sizeof(command)/sizeof(command[0]))
printf("\n\r\t Not defined command:'%s' Please refer instruction set!\n", ctx.arg[0]);
}
}
else
printf("\n\r\t Error: too many arguments ! \n\r");
}
P.S.Before the development of this GUI, I have used Tera Term to communicate with the board and both reading and writing performed well.
Configuration is :
Receiving: CR, encoding with UTF8
Transmitting: LF, encoding with UTF8
Does anyone figure out why it happened just like that?
Update
I have solved this issue after I modified writing process:
int index = 0;
byte[] txdata = Encoding.UTF8.GetBytes(TXTBoxCommand.Text);
while (index < txdata.Length)
{
OBCSerialPort.Write(txdata, index, 1);
index++;
if (index == txdata.Length)
OBCSerialPort.Write("\n");
}
It seems that for my embedded board using OBCSerialPort.Write to transmit multiple bytes will cause data loss.
However, it does not make sense...

Order reversed when echoing/debugging

I'm trying to send data from a VS Windows Form to an Arduino, but my data keeps getting mirrored. I'm not sure if it has to do with the buffer. If my understanding is correct, it should have nothing to do with the buffer nor to do with my code reading/writing data and it's a timing issue? The buffer is like the queue of the system. Since it's FIFO, the order in which I initialized, it'll have the same order.
I found this article, but I'm not sure if it applies. In this example about UART Ring Buffer the head and tail when declared share the same element. Does this apply to regular buffers? I assumed that since it's FIFO the head and tail wouldn't share the same element.
This Article on Double buffering seems to be what I'm talking about, but I don't think I'm technically using 2 buffers?
For example,
String a = "1";
String b = "2";
String c = "3";
String d = "4";
String e = "5";
String f = "6";
String g = "7";
String h = "8";
String[] sendchar = new String [] {a, b, c, d, e, f, g, h};
So when I send my data, the buffer stream should be, from the right being the first element and left being the last; "h,g,f,e,d,c,b,a" a would be sent first, then b and et cetera.
Currently, when I send data and it gets echoed back, I get in the reverse order, I'll send "a,b,c,d,e,f,g,h" but get "h,g,f,e,d,c,b,a" returned.
I'm receiving the data by reading it, and then storing it into an array, duplicating it, and then accessing elements in the duplicated array. This way the data order should be preserved.
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();
if (rb != endMarker)
{
receivedChar[ndx] = rb;
copyarray[ndx] = receivedChar[ndx];
ndx++;
How I get the data and send it on Arduino
void loop()
{
recvWithEndMarkers();//get Data
Serial.flush();//Clear Input buffer
delay(10);//delay
testblink();//Test blink
//blink();
echo();//echo data back
Serial.flush();
delay(2000);
}
void echo()
{
for (int b = 0; b <= 7; b++)
{
Serial.write(copyarray[b]);// Send b'th element of array
delay(50);
Serial.write("\n");//newline character terminates read
}
void recvWithEndMarkers() {
static boolean recvInProgress = false;
static byte ndx = 0;
char endMarker = '}';
byte rb;
byte comp;
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();//read data from input buffer
if (rb != endMarker)//not end marker
{
receivedChar[ndx] = rb;//store data into array index
copyarray[ndx] = receivedChar[ndx];//copy data
ndx++;//next index
if (ndx >= numBytes)
{
ndx = numBytes - 1;
}
}
else//endmarker
{
receivedChar[ndx] = '}'; // terminate the string
recvInProgress = false;
ndx = 0;//reset ndx
}
}
}
On the VS side
port = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
port.Open();
for (int a = 0; a <= 7; a++)
{
port.Write(sendchar[a]);
Thread.Sleep(50);
}
Thread.Sleep(50);
port.DiscardOutBuffer();
String[] recchar = new String[8];
while (port.BytesToRead != 0)
{
for (int a = 0; a <= 7; a++)
{
recchar[a] = port.ReadLine();
Thread.Sleep(50);
}
}
port.DiscardInBuffer();

I see at least a couple issues with your code. First, I assume that you reset the Arduino, then run your windows program, right?
ARDUINO:
Your recvWithEndMarkers() will probably see Serial.available() == 0, so exit the while loop right away.
When you put a character in the buffer, you increment the ndx (good). But when ndx == numBytes, you then set ndx to 7. It should probably be set to 0.
Also, you have ndx as a "static". So it will retain it's value when you run it a second time -- I'll bet that's not what you want.
When you exit the recvWithEndMarkers() function, you do a serial.flush(), which may cause you to lose characters, esp after the first time.
Then your echo() routine will "send" whatever is in copyarray[] at the time (not sure on Arduino if it's 0's or 255's, but probably not what you expect).
If your loop code had the flush and delay at the TOP of the loop (and maybe longer than 2 seconds), you could start Arduino, then start the VS program, and maybe get better results.
On the VS side, I don't see anything quite like that, but you did not share the code that prints the received data.
Good luck!

C# Serial port randomly return all 0 when overloaded

I am making a C# User Interface for a project using MCU communicate with the PC using UART. The string sent from the MCU in this format(3 numbers per group, 3 groups):
<number>,<number>,<number>,.<number>,<number>,<number>,.<number>,<number>,<number>,.\n
With each number can change depend on the MCU sending. I have written a program which work fine for this configuration. In general, it take in the data from COM port as a string, filter out any characters that is not a number, then save them separately into a List.
However, when we expand the format to use 7 numbers per group (21 in total):
<number>,<number>,<number>,<number>,<number>,<number>,<number>,.<number>,<number>,<number>,<number>,<number>,<number>,<number>,.<number>,<number>,<number>,<number>,<number>,<number>,<number>,.\n
When I try the same approach, there is some sort of "stagger" in the data in. When the data is stable, it is fine. When there is any rapid change in the number coming in, the graph of the data look like this:
The sharp drop in the middle of the graph is not how the data change; the UART port just "freeze up" and output 0 for all number in the receiving List, I have checked the data from Hercules myself.
I think the problem come from overload since there is no differences in any other aspect between the 3 and 7 numbers datastream, but can StackOverflow users think of any other solution?
EDIT: Should it helps, here is the code I used:
// Initialization
static SerialPort mySerialPort;
string receiveMessage = "";
public delegate void AddDataDelegate(string text);
public AddDataDelegate DELEGATE_WRITE;
DELEGATE_WRITE = new AddDataDelegate(writeConsole);
// com and baud are defined variables
mySerialPort = new SerialPort(com, baud);
mySerialPort.DataReceived += new SerialDataReceivedEventHandler(mySerialPort_datareceived);
mySerialPort.Open();
//Supporting functions
private void mySerialPort_datareceived(object sender, SerialDataReceivedEventArgs e)
{
string input = mySerialPort.ReadExisting();
textBox.BeginInvoke(method: DELEGATE_WRITE, args: new Object[] { input });
}
public void writeConsole(string text)
{
receiveMessage += text;
// check if the receiving data steam is valid: end with '\n' and not longer than 64 characters (21 value of maybe 2-3 digits each)
if (receiveMessage.Length > 64)
{
receiveMessage = "";
}
if (receiveMessage.Length > 0)
if (receiveMessage[receiveMessage.Length - 1] == '\n')
{
updateInterface(receiveMessage);
receiveMessage = "";
}
return;
}
void updateInterface(string input)
{
try
{
int[] result = calculate(input);
if (result == null)
{
MessageBox.Show("Error", caption: "Algorithm fail to extract data", buttons: MessageBoxButtons.OK, icon: MessageBoxIcon.Error);
return;
}
//Save the data to List
}
static int[] calculate(string input)
{
int[] outputs = new int[21];
int index = 0;
string s = "";
for (int i = 0; i < input.Length; i++)
{
char c = input[i];
if (c == '.')
{
int output = int.Parse(s);
outputs[index] = output;
index++;
s = "";
}
else if (c >= '0' && c <= '9'|| c == '-')
{
s += c;
}
}
return outputs;
}

Always the devil is in the details
In the writeConsole function of my code, I make a check for valid string by checking if they end in '\n' (since the data will be coming in real time) and if they are less than 64 characters long (21 value of maybe 2-3 digits each). Turn out, that is not a correct assumption at all when the data could easily reach twice as long when the data changed quickly enough (in my examples, the change could reach 100). So just delete the length check and the program work as usual.
PS: Come on guys, this is the second time I have to answer my own question already

Fast packet processing from stream

First background. I have a camera that my app is connected to via Ethernet cable and it performs operations based on commands I send and responds with commands back.
If I take the camera trigger command as an example.
The camera will take a picture if I send it 'T1'. All commands use a start and end char to mark the start and end of pack so the full packet I send will look like this
(char)0x02T1(char)0x03
with (char)0x02 as the start of packet and (char)0x03 as the end of pack.
The camera will then take a picture and send the same command back to say it is done. You can also set the camera to send some data back when it has taken the picture. In my case I have the camera inspecting a few areas and I want some values from the inspection.
So the return of the camera would be something like 'T1' and then say '1,1,500,20,300'
On to the problem. I am using a TcpClient and a NetworkStream for the communication but I am struggling with processing the packets on the receiving side. I have tried various methods but they seem slow or don't guarantee I get all the data.
Firstly I need something that will read all the data and make sure I have all the data.
Secondly I need something that can process the data and split up into packets and do this as fast as possible.
This is just one method I have tried and full credit goes to the person who made the StreamReaderExtensions. I'm sure I found it on here somewhere.
internal static class StreamReaderExtensions
{
public static IEnumerable<string> ReadUntil(this StreamReader reader, string delimiter)
{
List<char> buffer = new List<char>();
CircularBuffer<char> delim_buffer = new CircularBuffer<char>(delimiter.Length);
while (reader.Peek() >= 0)
{
char c = (char)reader.Read();
delim_buffer.Enqueue(c);
if (delim_buffer.ToString() == delimiter)
{
if (buffer.Count > 0)
{
yield return new String(buffer.ToArray());
buffer.Clear();
}
continue;
}
buffer.Add(c);
}
}
private class CircularBuffer<T> : Queue<T>
{
private int _capacity;
public CircularBuffer(int capacity)
: base(capacity)
{
_capacity = capacity;
}
new public void Enqueue(T item)
{
if (base.Count == _capacity)
{
base.Dequeue();
}
base.Enqueue(item);
}
public override string ToString()
{
List<String> items = new List<string>();
foreach (var x in this)
{
items.Add(x.ToString());
};
return String.Join("", items);
}
}
}
and my process method. _stream is TcpClient.GetStream()
public static class Constants
{
public const char StartOfPacket = (char)0x02;
public const char EndOfPacket = (char)0x03;
}
private IEnumerable<string> ProcessResponse()
{
var streamReader = new StreamReader(_stream);
var packets = streamReader.ReadUntil(new string(new char[] {Constants.EndOfPacket}));
return packets;
}

I made some pseudo code for, I didn't tested it, but it might give you a direction. This can't be static
char[] _buffer = new char[4 * 1024 * 1024];
int currentPosition;
public bool ReadNext(StreamReader reader, char delimiter, out string value)
{
// read as many chars from the stream.
int charsRead = reader.ReadBlock(_buffer, currentPosition, _buffer.Length - currentPosition);
// keep track of the current position.
currentPosition += charsRead;
// find the first delimiter.
for (int i = 0; i < currentPosition; i++)
{
if (_buffer[i] == delimiter)
{
// a complete packet was found, copy it to a string.
value = new string(_buffer, 0, i);
// move the rest of the data back to the start of the buffer.
Array.Copy(_buffer, i, _buffer, 0, _buffer.Length - i);
// decrease the current position also.
currentPosition -= i;
// return true because you have data.
return true;
}
}
// no complete packet was found.
value = String.Empty;
return false;
}
Usage:
string data;
while(true)
{
if(ReadNext(reader, (char)0x03, out data))
MessageBox.Show(data);
}
Like I said untested, this is just a brainfart... You might need to add the check of the start of packet.

Circular log TextBox for raw serial characters

The problem
From the serial port I receive a stream of characters. The stream I receive will be terminated by \n. I receive the serial port data in parts like this:
Serial port Event 1: "123\n456\n789"
Serial port Event 2: "\n0123\n456\n789\n"
Serial port Event 3: "0123\n456\n789\n"
As you can see my stream fluctuates, and this is very normal since I read the serial port with "what is available currently".
My problem is that I want to log this to the UI with a RichTextBox. I cannot use ListBox because I need color and font formatting.
First approach
Before I tried this below and that works very well till the time the messages exceed around 30.000 lines of text. The UI gets unresponsive. This was the code:
uiTextActivity.SelectionFont = new Font(uiTextActivity.SelectionFont, FontStyle.Bold);
uiTextActivity.SelectionColor = LogMsgTypeColor[(int)msgtype];
uiTextActivity.AppendText(msg);
uiTextActivity.ScrollToCaret();
Later on I used this as a quick fix that clears the box after 2000 lines:
if ((int.Parse(uiLabelCounterActivity.Text) + 1) > 2000)
uiTextBoxResetActivity();
I want to keep a history of around 500lines.
But with this quick fix above I lose my log completely when the counter hits 2000.
I think what I need is a circular FIFO TextBox. So I can remove after 500 lines the oldest logs and append the new one.
Second approach
I also tried this (note that in this case below the newest logs entries are on top and the oldest below in the textbox)
msg = msgToPasteWhenComplete + msg;
int c = 0; // c=zero if no newline, 1 for single newline, 2 for 2times newline and so on
int latestTermination = 0;// store the point where \n is found
for (int e = 0; e < msg.Length; e++)
{
if (msg[e] == '\n')
{
c++;
latestTermination = e+1;
}
}
// contains strings all terminated with \n
string msgToPaste = msg.Substring(0, latestTermination);
// contains string that is not complete (not yet terminated with \n)
msgToPasteWhenComplete = msg.Substring(latestTermination, msg.Length - latestTermination);
string previous = msgToPaste + uiTextActivity.Text;
if (previous.Length > maxDisplayTextLength)
{
string debugInfo3 = previous.Substring(0, maxDisplayTextLength);
uiTextActivity.Text = debugInfo3;
}
else
uiTextActivity.Text = previous;
This works almost very well. The problem with this approach is that the line that comes from the serial port will not be pasted to the UI until the \n is received. This means that whenever communication is slow, I will have to wait for the serial stream to complete the whole line including \n before I can see it... What I want is to see every character directly.
info about serialport
The serial data am I getting from the SerialDataReceivedEvent, in that event I use comport.ReadExisting() to have a non-blocking event.
The serial data comes from my embedded programming board that has analog readings. The analog readings supply me with 20 lines per second, each line containing 20 chars.
I need the raw data to be read in a user interface where it has to be filtered to other textboxes depending on the prefix of the serial message (like err goes to error , warn goes to warning textbox, but they all go to the activity log. The activity log is what this question is about.

I wish I could give you a specific chunk of code to work off of, but I haven't had to do this kind of input processing in a while.
That being said, you might get better performance by buffering your inputs into a Queue (MSDN reference) object, and then polling the queue on a timer or by responding to some other event (maybe OnChanged?).

I found a solution that works, here is my code:
private const int maxDisplayTextLength = 5000;
private string splitString = "";
private int activityCount = 0;
private string errorMessageBox = "";
private bool errorMessageBoxNeedUpdate = true;
private int errorMessageBoxCount = 0;
private string filteredMessageBox = "";
private int filteredMessageCount = 0;
private bool filteredMessageBoxNeedUpdate = true;
private string activityLogFilterKeyword = "Warning";
string logHelperStringMaxSizeLimiter(string input)
{
// check if our buffer isn't getting bigger than our specified max. length
if (input.Length > maxDisplayTextLength)
{
// discard the oldest data in our buffer (FIFO) so we have room for our newest values
input = input.Substring(input.Length - maxDisplayTextLength);
}
return input;
}
private void logHelperIncoming(string msg)
{
msg = msg.Replace('\0', '\n'); // remove \0 NULL characters as they have special meanings in C# RichTextBox
// add the new received string to our buffer
splitString += msg;
// filter out special strings
int searchIndexStart = splitString.Length - msg.Length;
for (int e = searchIndexStart; e < splitString.Length; e++)
{
if (splitString[e] == '\n')
{
activityCount++;
string subString = splitString.Substring(searchIndexStart, e - searchIndexStart) + "\n";
searchIndexStart += e - searchIndexStart + 1; // update searchindex for next search
string filterError = "error";
// filter messages that start with error
if (subString.Length > filterError.Length) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, filterError, 0, 4, true) == 0)
{
errorMessageBox += subString;
errorMessageBoxNeedUpdate = true;
errorMessageBoxCount++;
}
}
// filter messages that start with XXX
if (subString.Length > activityLogFilterKeyword.Length && activityLogFilterKeyword.Length != 0) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, activityLogFilterKeyword, 0, activityLogFilterKeyword.Length, true) == 0)
{
filteredMessageBox += subString;
filteredMessageBoxNeedUpdate = true;
filteredMessageCount++;
}
}
}
}
}
// outputs to main activity textbox
private void Log(LogMsgType msgtype, string msg)
{
if (msgtype == LogMsgType.Incoming || msgtype == LogMsgType.Normal)
{
logHelperIncoming(msg);
}
else if (msgtype == LogMsgType.Outgoing)
{
}
splitString = logHelperStringMaxSizeLimiter(splitString);
filteredMessageBox = logHelperStringMaxSizeLimiter(filteredMessageBox);
errorMessageBox = logHelperStringMaxSizeLimiter(errorMessageBox);
uiTextActivity.Invoke(new EventHandler(delegate
{
// time to post our updated buffer to the UI
uiTextActivity.Text = splitString;
uiTextActivity.Update();
// scroll to the newest data only if user has no focus on the
uiTextActivity.SelectionStart = uiTextActivity.TextLength; // make sure view is to the latest
uiTextActivity.ScrollToCaret();
uiLabelCounterActivity.Text = activityCount.ToString();
if (errorMessageBoxNeedUpdate)
{
errorMessageBoxNeedUpdate = false;
uiTextActivity.SelectionColor = Color.Red;
// time to post our updated buffer to the UI
uiTextboxErrorLog.Text = errorMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxErrorLog.SelectionStart = uiTextboxErrorLog.TextLength; // make sure view is to the latest
uiTextboxErrorLog.ScrollToCaret();
uiLabelCounterError.Text = errorMessageBoxCount.ToString();
}
if (filteredMessageBoxNeedUpdate)
{
filteredMessageBoxNeedUpdate = false;
// time to post our updated buffer to the UI
uiTextboxFilteredLog.Text = filteredMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxFilteredLog.SelectionStart = uiTextboxFilteredLog.TextLength; // make sure view is to the latest
uiTextboxFilteredLog.ScrollToCaret();
uiLabelCounterFiltered.Text = filteredMessageCount.ToString();
}
// extract value from filter and store to filter
activityLogFilterKeyword = uiTextboxFilterKeyword.Text;
}));
}