We Keep Coding

Order reversed when echoing/debugging

I'm trying to send data from a VS Windows Form to an Arduino, but my data keeps getting mirrored. I'm not sure if it has to do with the buffer. If my understanding is correct, it should have nothing to do with the buffer nor to do with my code reading/writing data and it's a timing issue? The buffer is like the queue of the system. Since it's FIFO, the order in which I initialized, it'll have the same order.
I found this article, but I'm not sure if it applies. In this example about UART Ring Buffer the head and tail when declared share the same element. Does this apply to regular buffers? I assumed that since it's FIFO the head and tail wouldn't share the same element.
This Article on Double buffering seems to be what I'm talking about, but I don't think I'm technically using 2 buffers?
For example,
String a = "1";
String b = "2";
String c = "3";
String d = "4";
String e = "5";
String f = "6";
String g = "7";
String h = "8";
String[] sendchar = new String [] {a, b, c, d, e, f, g, h};
So when I send my data, the buffer stream should be, from the right being the first element and left being the last; "h,g,f,e,d,c,b,a" a would be sent first, then b and et cetera.
Currently, when I send data and it gets echoed back, I get in the reverse order, I'll send "a,b,c,d,e,f,g,h" but get "h,g,f,e,d,c,b,a" returned.
I'm receiving the data by reading it, and then storing it into an array, duplicating it, and then accessing elements in the duplicated array. This way the data order should be preserved.
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();
if (rb != endMarker)
{
receivedChar[ndx] = rb;
copyarray[ndx] = receivedChar[ndx];
ndx++;
How I get the data and send it on Arduino
void loop()
{
recvWithEndMarkers();//get Data
Serial.flush();//Clear Input buffer
delay(10);//delay
testblink();//Test blink
//blink();
echo();//echo data back
Serial.flush();
delay(2000);
}
void echo()
{
for (int b = 0; b <= 7; b++)
{
Serial.write(copyarray[b]);// Send b'th element of array
delay(50);
Serial.write("\n");//newline character terminates read
}
void recvWithEndMarkers() {
static boolean recvInProgress = false;
static byte ndx = 0;
char endMarker = '}';
byte rb;
byte comp;
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();//read data from input buffer
if (rb != endMarker)//not end marker
{
receivedChar[ndx] = rb;//store data into array index
copyarray[ndx] = receivedChar[ndx];//copy data
ndx++;//next index
if (ndx >= numBytes)
{
ndx = numBytes - 1;
}
}
else//endmarker
{
receivedChar[ndx] = '}'; // terminate the string
recvInProgress = false;
ndx = 0;//reset ndx
}
}
}
On the VS side
port = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
port.Open();
for (int a = 0; a <= 7; a++)
{
port.Write(sendchar[a]);
Thread.Sleep(50);
}
Thread.Sleep(50);
port.DiscardOutBuffer();
String[] recchar = new String[8];
while (port.BytesToRead != 0)
{
for (int a = 0; a <= 7; a++)
{
recchar[a] = port.ReadLine();
Thread.Sleep(50);
}
}
port.DiscardInBuffer();

I see at least a couple issues with your code. First, I assume that you reset the Arduino, then run your windows program, right?
ARDUINO:
Your recvWithEndMarkers() will probably see Serial.available() == 0, so exit the while loop right away.
When you put a character in the buffer, you increment the ndx (good). But when ndx == numBytes, you then set ndx to 7. It should probably be set to 0.
Also, you have ndx as a "static". So it will retain it's value when you run it a second time -- I'll bet that's not what you want.
When you exit the recvWithEndMarkers() function, you do a serial.flush(), which may cause you to lose characters, esp after the first time.
Then your echo() routine will "send" whatever is in copyarray[] at the time (not sure on Arduino if it's 0's or 255's, but probably not what you expect).
If your loop code had the flush and delay at the TOP of the loop (and maybe longer than 2 seconds), you could start Arduino, then start the VS program, and maybe get better results.
On the VS side, I don't see anything quite like that, but you did not share the code that prints the received data.
Good luck!

Is there an equivalent to mmap.mmap.rfind in C#?

While looking at memory-mapped files in C#, there was some difficulty in identifying how to search a file quickly forward and in reverse. My goal is to rewrite the following function in the language, but nothing could be found like the find and rfind methods used below. Is there a way in C# to quickly search a memory-mapped file using a particular substring?
#! /usr/bin/env python3
import mmap
import pathlib
# noinspection PyUnboundLocalVariable
def drop_last_line(path):
with path.open('r+b') as file:
with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ) as search:
for next_line in b'\r\n', b'\r', b'\n':
if search.find(next_line) >= 0:
break
else:
raise ValueError('cannot find any line delimiters')
end_1st = search.rfind(next_line)
end_2nd = search.rfind(next_line, 0, end_1st - 1)
file.truncate(0 if end_2nd < 0 else end_2nd + len(next_line))

Is there a way in C# to quickly search a memory-mapped file using a particular substring?
Do you know of any way to memory-map an entire file in C# and then treat it as a byte array?
Yes, it's quite easy to map an entire file into a view then to read it into a single byte array as the following code shows:
static void Main(string[] args)
{
var sourceFile= new FileInfo(#"C:\Users\Micky\Downloads\20180112.zip");
int length = (int) sourceFile.Length; // length of target file
// Create the memory-mapped file.
using (var mmf = MemoryMappedFile.CreateFromFile(sourceFile.FullName,
FileMode.Open,
"ImgA"))
{
var buffer = new byte[length]; // allocate a buffer with the same size as the file
using (var accessor = mmf.CreateViewAccessor())
{
var read=accessor.ReadArray(0, buffer, 0, length); // read the whole thing
}
// let's try searching for a known byte sequence. Change this to suit your file
var target = new byte[] {71, 213, 62, 204,231};
var foundAt = IndexOf(buffer, target);
}
}
I couldn't seem to find any byte searching method in Marshal or Array but you can use this search algorithm courtesy of Social MSDN as a start:
private static int IndexOf2(byte[] input, byte[] pattern)
{
byte firstByte = pattern[0];
int index = -1;
if ((index = Array.IndexOf(input, firstByte)) >= 0)
{
for (int i = 0; i < pattern.Length; i++)
{
if (index + i >= input.Length ||
pattern[i] != input[index + i]) return -1;
}
}
return index;
}
...or even this more verbose example (also courtesy Social MSDN, same link)
public static int IndexOf(byte[] arrayToSearchThrough, byte[] patternToFind)
{
if (patternToFind.Length > arrayToSearchThrough.Length)
return -1;
for (int i = 0; i < arrayToSearchThrough.Length - patternToFind.Length; i++)
{
bool found = true;
for (int j = 0; j < patternToFind.Length; j++)
{
if (arrayToSearchThrough[i + j] != patternToFind[j])
{
found = false;
break;
}
}
if (found)
{
return i;
}
}
return -1;
}

C# Serial port randomly return all 0 when overloaded

I am making a C# User Interface for a project using MCU communicate with the PC using UART. The string sent from the MCU in this format(3 numbers per group, 3 groups):
<number>,<number>,<number>,.<number>,<number>,<number>,.<number>,<number>,<number>,.\n
With each number can change depend on the MCU sending. I have written a program which work fine for this configuration. In general, it take in the data from COM port as a string, filter out any characters that is not a number, then save them separately into a List.
However, when we expand the format to use 7 numbers per group (21 in total):
<number>,<number>,<number>,<number>,<number>,<number>,<number>,.<number>,<number>,<number>,<number>,<number>,<number>,<number>,.<number>,<number>,<number>,<number>,<number>,<number>,<number>,.\n
When I try the same approach, there is some sort of "stagger" in the data in. When the data is stable, it is fine. When there is any rapid change in the number coming in, the graph of the data look like this:
The sharp drop in the middle of the graph is not how the data change; the UART port just "freeze up" and output 0 for all number in the receiving List, I have checked the data from Hercules myself.
I think the problem come from overload since there is no differences in any other aspect between the 3 and 7 numbers datastream, but can StackOverflow users think of any other solution?
EDIT: Should it helps, here is the code I used:
// Initialization
static SerialPort mySerialPort;
string receiveMessage = "";
public delegate void AddDataDelegate(string text);
public AddDataDelegate DELEGATE_WRITE;
DELEGATE_WRITE = new AddDataDelegate(writeConsole);
// com and baud are defined variables
mySerialPort = new SerialPort(com, baud);
mySerialPort.DataReceived += new SerialDataReceivedEventHandler(mySerialPort_datareceived);
mySerialPort.Open();
//Supporting functions
private void mySerialPort_datareceived(object sender, SerialDataReceivedEventArgs e)
{
string input = mySerialPort.ReadExisting();
textBox.BeginInvoke(method: DELEGATE_WRITE, args: new Object[] { input });
}
public void writeConsole(string text)
{
receiveMessage += text;
// check if the receiving data steam is valid: end with '\n' and not longer than 64 characters (21 value of maybe 2-3 digits each)
if (receiveMessage.Length > 64)
{
receiveMessage = "";
}
if (receiveMessage.Length > 0)
if (receiveMessage[receiveMessage.Length - 1] == '\n')
{
updateInterface(receiveMessage);
receiveMessage = "";
}
return;
}
void updateInterface(string input)
{
try
{
int[] result = calculate(input);
if (result == null)
{
MessageBox.Show("Error", caption: "Algorithm fail to extract data", buttons: MessageBoxButtons.OK, icon: MessageBoxIcon.Error);
return;
}
//Save the data to List
}
static int[] calculate(string input)
{
int[] outputs = new int[21];
int index = 0;
string s = "";
for (int i = 0; i < input.Length; i++)
{
char c = input[i];
if (c == '.')
{
int output = int.Parse(s);
outputs[index] = output;
index++;
s = "";
}
else if (c >= '0' && c <= '9'|| c == '-')
{
s += c;
}
}
return outputs;
}

Always the devil is in the details
In the writeConsole function of my code, I make a check for valid string by checking if they end in '\n' (since the data will be coming in real time) and if they are less than 64 characters long (21 value of maybe 2-3 digits each). Turn out, that is not a correct assumption at all when the data could easily reach twice as long when the data changed quickly enough (in my examples, the change could reach 100). So just delete the length check and the program work as usual.
PS: Come on guys, this is the second time I have to answer my own question already

Method to receive all bytes from a message-framed stream freezes

Recently, I studied Message Framing in my own time and decided that I should use the common solution of prepending the message with the length of the actual message. I use the Tcp wrappers for sockets (TcpListener, TcpClient).
Context:
Both server and client communicate with base64 encoded serialized objects. On networks with low traffic, the complete data seems to reach its destination in one piece, and can be decoded and then deserialized. However, on networks with high traffic, or low resource availability, I always get a FormatException that the data is not in a proper Base64 format (or similar errors). Here is the method for sending a message:
public void Send(Packet packData)
{
try
{
StreamWriter w = new StreamWriter(clientObject.GetStream());
string s = SerializeData(packData);
w.WriteLine(s.Length.ToString() + s);
w.Flush();
}
catch (NullReferenceException) { }
catch (IOException) { CloseObject(); }
}
Now, here is the method I devised for reading the stream on the receiving end. It appears to freeze at the while loop:
private string ReceiveAllDataFromStream(StreamReader r)
{
int a = 0;
int b = 0;
string str = null;
try
{
str = r.ReadLine();
// find out how many chars are integers, then get the value in int. (this is the length of the message)
foreach (char f in str) { if (int.TryParse(f.ToString(), out a)) { b++; } else { break; } }
// reuse "a", it doesn't actually contain the result of the TryParse method called above
a = int.Parse(str.Substring(0, b));
// remove these chars (the ints) from the message
str = str.Remove(0, b);
// calculate how many chars need to be received to complete this message
int msgRemaining = a - str.Length;
int msgAcquired = str.Length;
if (msgAcquired != a)
{
// where i believe the freeze is caused
while (msgAcquired != a)
{
// make a call to supposedly retreieve these last few chars
string s = r.ReadLine();
// insert as much of the new string received as "msgRemaining" demands
str.Insert(0, s.Substring(0, msgRemaining));
// add the total length of the inserted string to the amount already aquired, continue with loop
msgAcquired = msgAcquired + s.Substring(0, msgRemaining).Length;
}
} return str;
}
catch (FormatException) { System.Windows.Forms.MessageBox.Show(b.ToString() + "|" + str.Substring(0, b)); return null; }
}
When I say it freezes, I mean the application goes into an idle state (UI isn't blocked, but I believe that's because this method is running on another thread). What is wrong with my method, and how do I go about fixing it?
Thank you.

Circular log TextBox for raw serial characters

The problem
From the serial port I receive a stream of characters. The stream I receive will be terminated by \n. I receive the serial port data in parts like this:
Serial port Event 1: "123\n456\n789"
Serial port Event 2: "\n0123\n456\n789\n"
Serial port Event 3: "0123\n456\n789\n"
As you can see my stream fluctuates, and this is very normal since I read the serial port with "what is available currently".
My problem is that I want to log this to the UI with a RichTextBox. I cannot use ListBox because I need color and font formatting.
First approach
Before I tried this below and that works very well till the time the messages exceed around 30.000 lines of text. The UI gets unresponsive. This was the code:
uiTextActivity.SelectionFont = new Font(uiTextActivity.SelectionFont, FontStyle.Bold);
uiTextActivity.SelectionColor = LogMsgTypeColor[(int)msgtype];
uiTextActivity.AppendText(msg);
uiTextActivity.ScrollToCaret();
Later on I used this as a quick fix that clears the box after 2000 lines:
if ((int.Parse(uiLabelCounterActivity.Text) + 1) > 2000)
uiTextBoxResetActivity();
I want to keep a history of around 500lines.
But with this quick fix above I lose my log completely when the counter hits 2000.
I think what I need is a circular FIFO TextBox. So I can remove after 500 lines the oldest logs and append the new one.
Second approach
I also tried this (note that in this case below the newest logs entries are on top and the oldest below in the textbox)
msg = msgToPasteWhenComplete + msg;
int c = 0; // c=zero if no newline, 1 for single newline, 2 for 2times newline and so on
int latestTermination = 0;// store the point where \n is found
for (int e = 0; e < msg.Length; e++)
{
if (msg[e] == '\n')
{
c++;
latestTermination = e+1;
}
}
// contains strings all terminated with \n
string msgToPaste = msg.Substring(0, latestTermination);
// contains string that is not complete (not yet terminated with \n)
msgToPasteWhenComplete = msg.Substring(latestTermination, msg.Length - latestTermination);
string previous = msgToPaste + uiTextActivity.Text;
if (previous.Length > maxDisplayTextLength)
{
string debugInfo3 = previous.Substring(0, maxDisplayTextLength);
uiTextActivity.Text = debugInfo3;
}
else
uiTextActivity.Text = previous;
This works almost very well. The problem with this approach is that the line that comes from the serial port will not be pasted to the UI until the \n is received. This means that whenever communication is slow, I will have to wait for the serial stream to complete the whole line including \n before I can see it... What I want is to see every character directly.
info about serialport
The serial data am I getting from the SerialDataReceivedEvent, in that event I use comport.ReadExisting() to have a non-blocking event.
The serial data comes from my embedded programming board that has analog readings. The analog readings supply me with 20 lines per second, each line containing 20 chars.
I need the raw data to be read in a user interface where it has to be filtered to other textboxes depending on the prefix of the serial message (like err goes to error , warn goes to warning textbox, but they all go to the activity log. The activity log is what this question is about.

I wish I could give you a specific chunk of code to work off of, but I haven't had to do this kind of input processing in a while.
That being said, you might get better performance by buffering your inputs into a Queue (MSDN reference) object, and then polling the queue on a timer or by responding to some other event (maybe OnChanged?).

I found a solution that works, here is my code:
private const int maxDisplayTextLength = 5000;
private string splitString = "";
private int activityCount = 0;
private string errorMessageBox = "";
private bool errorMessageBoxNeedUpdate = true;
private int errorMessageBoxCount = 0;
private string filteredMessageBox = "";
private int filteredMessageCount = 0;
private bool filteredMessageBoxNeedUpdate = true;
private string activityLogFilterKeyword = "Warning";
string logHelperStringMaxSizeLimiter(string input)
{
// check if our buffer isn't getting bigger than our specified max. length
if (input.Length > maxDisplayTextLength)
{
// discard the oldest data in our buffer (FIFO) so we have room for our newest values
input = input.Substring(input.Length - maxDisplayTextLength);
}
return input;
}
private void logHelperIncoming(string msg)
{
msg = msg.Replace('\0', '\n'); // remove \0 NULL characters as they have special meanings in C# RichTextBox
// add the new received string to our buffer
splitString += msg;
// filter out special strings
int searchIndexStart = splitString.Length - msg.Length;
for (int e = searchIndexStart; e < splitString.Length; e++)
{
if (splitString[e] == '\n')
{
activityCount++;
string subString = splitString.Substring(searchIndexStart, e - searchIndexStart) + "\n";
searchIndexStart += e - searchIndexStart + 1; // update searchindex for next search
string filterError = "error";
// filter messages that start with error
if (subString.Length > filterError.Length) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, filterError, 0, 4, true) == 0)
{
errorMessageBox += subString;
errorMessageBoxNeedUpdate = true;
errorMessageBoxCount++;
}
}
// filter messages that start with XXX
if (subString.Length > activityLogFilterKeyword.Length && activityLogFilterKeyword.Length != 0) // for this filter, the length should be at least length of error
{
if (String.Compare(subString, 0, activityLogFilterKeyword, 0, activityLogFilterKeyword.Length, true) == 0)
{
filteredMessageBox += subString;
filteredMessageBoxNeedUpdate = true;
filteredMessageCount++;
}
}
}
}
}
// outputs to main activity textbox
private void Log(LogMsgType msgtype, string msg)
{
if (msgtype == LogMsgType.Incoming || msgtype == LogMsgType.Normal)
{
logHelperIncoming(msg);
}
else if (msgtype == LogMsgType.Outgoing)
{
}
splitString = logHelperStringMaxSizeLimiter(splitString);
filteredMessageBox = logHelperStringMaxSizeLimiter(filteredMessageBox);
errorMessageBox = logHelperStringMaxSizeLimiter(errorMessageBox);
uiTextActivity.Invoke(new EventHandler(delegate
{
// time to post our updated buffer to the UI
uiTextActivity.Text = splitString;
uiTextActivity.Update();
// scroll to the newest data only if user has no focus on the
uiTextActivity.SelectionStart = uiTextActivity.TextLength; // make sure view is to the latest
uiTextActivity.ScrollToCaret();
uiLabelCounterActivity.Text = activityCount.ToString();
if (errorMessageBoxNeedUpdate)
{
errorMessageBoxNeedUpdate = false;
uiTextActivity.SelectionColor = Color.Red;
// time to post our updated buffer to the UI
uiTextboxErrorLog.Text = errorMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxErrorLog.SelectionStart = uiTextboxErrorLog.TextLength; // make sure view is to the latest
uiTextboxErrorLog.ScrollToCaret();
uiLabelCounterError.Text = errorMessageBoxCount.ToString();
}
if (filteredMessageBoxNeedUpdate)
{
filteredMessageBoxNeedUpdate = false;
// time to post our updated buffer to the UI
uiTextboxFilteredLog.Text = filteredMessageBox;
// scroll to the newest data only if user has no focus on the
uiTextboxFilteredLog.SelectionStart = uiTextboxFilteredLog.TextLength; // make sure view is to the latest
uiTextboxFilteredLog.ScrollToCaret();
uiLabelCounterFiltered.Text = filteredMessageCount.ToString();
}
// extract value from filter and store to filter
activityLogFilterKeyword = uiTextboxFilterKeyword.Text;
}));
}