We Keep Coding

Receiving data from a serial port using ReadAsync

I'm trying to communicate with a piece of hardware over a serial port and I've noticed that it's quite tricky. After reading up on some of the SerialPort caveats, I discovered this post by Ben Voigt on alternative approaches to handle serial communication in .NET with the SerialPort class.
However, this code is quite old (8 years or so as of today) and the Stream class now exposes asynchronous read/write support, so I was wondering how the code in that blog could be re-written to account for the following scenarios:
The device periodically sends a status update through the port (auto generated)
I can send a byte array that represents a command that doesn't need a response
I can send a byte array that represents a command that does need a response, and the device will send that reply data through the port
So far, I have a simple read loop that looks a bit like this:
public async IAsyncEnumerable<byte> ReadBytesAsync()
{
while (_port.IsOpen)
{
byte[]? buffer = new byte[1];
try
{
await _port.BaseStream.ReadAsync(buffer, 0, 1);
}
catch (Exception e)
{
// Do stuff
}
if (buffer is not null) yield return buffer[0];
await Task.Delay(1); // Wait a bit before reading again
}
}
I also have a GetRepliesAsync() method that await foreaches over the read bytes and constructs the right data types for the replies.
The autogenerated status update gets read correctly when received, but if I use _port.BaseStream.WriteAsync(commandBytes, 0, commandBytes.Length), the device receives the update but I somehow don't receive its reply through the base stream.
What could I be doing wrong, and how can I actually read the received bytes?
UPDATES
1. Modified read and added delay
I got the code to work somewhat with the following approach:
while (_port.IsOpen)
{
if (cancellationToken.IsCancellationRequested) yield break;
int bufferLength = 2048; // Arbitrary buffer length
byte[] buffer = new byte[bufferLength];
try
{
// Partially using Ben Voigt's approach
int bytesRead = await _port.BaseStream
.ReadAsync(buffer.AsMemory(0, bufferLength), cancellationToken);
nextBytes = new byte[bytesRead];
Buffer.BlockCopy(buffer, 0, nextBytes, 0, bytesRead);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
Console.WriteLine("Terminating serial read polling...");
yield break;
}
if (nextBytes is not null)
{
for (int i = 0; i < nextBytes.Length; i++)
{
yield return nextBytes[i];
}
}
// Introduce delay in case a write operation is going on
await Task.Delay(200, cancellationToken);
}
But I still don't see a way to guarantee that each write will happen during the 200 millisecond delay.

COM: Device only sends data when things are changing. How to deal with that?

So I want to connect to a device via Serial that only sends data when things are changing with the settings on the device (a measuring device).
I use C# and .Net's SerialPort.
I am able to read data and it looks kind of good. But there are a few problems I encountered.
I realized my reading- method (ReadExistingData()) so that it will constantly use the SerialDataReceivedEventHandler when there's new data.
Unfortunately, when I read it like that (probably because of varying package sizes) it will read very chaotically and thus I need to "catch" the first initiating byte (here it's 0xA5).
That means, I always check whether the byte I just received is a 0xA5 and if it is, I read the rest.
But I feel like that way I am missing some commands my device sends me and thats why I cant display the data consistently right, only from time to time.
On a side note: The device sends the device time and a value. The value is delayed and kind of unaccurate, but the time is always right on spot. The other parameters it sends are always weirded out and dont seem to be recognized and thus changed at all.
To display data I use the console for testing purposes, and the whole construct seems to be very reactive to Console outputs.
Here's a little code snippet:
class Device
{
private int stdMessageLengthInBytes = 5;
public DeviceData processedData;
byte[] dataBuffer;
int receivedMessages = 0;
public Device()
{
// Initialize BaseClass (SerialPort derivative)
this.port = new System.IO.Ports.SerialPort();
// Initialize Device
this.data = new byte[stdMessageLengthInBytes];
this.processedData = new P8005_Data();
dataBuffer = new byte[stdMessageLengthInBytes];
}
// Is supposed to read the data from serial port
protected override void ReadExistingData()
{
// Check whether buffer is empty -> Try to catch a 0xA5
if (dataBuffer[0] == 0x00)
{
port.Read(dataBuffer, 0, 1);
}
// If no 0xA5 was catched, restart
if (dataBuffer[0] != 0xA5)
{
dataBuffer = new byte[stdMessageLengthInBytes]; // Reset buffer
return;
}
// Read next byte -> Command byte
port.Read(dataBuffer, 1, 1);
// If command was empty, restart
if (dataBuffer[1] == 0x00)
{
dataBuffer = new byte[stdMessageLengthInBytes]; // Reset buffer
return;
}
// If its time that is communicated: Read 3 bytes
if (dataBuffer[1] == 0x06)
{
// 4 ms delay seems to be needed otherwise it wont function correctly somehow
System.Threading.Thread.Sleep(5);
port.Read(dataBuffer, 2, 3);
// Write buffer to actual raw data byte array
this.data = dataBuffer;
dataBuffer = new byte[stdMessageLengthInBytes]; // Reset buffer
}
// Otherwise: Just read 2 bytes
System.Threading.Thread.Sleep(5); // Needed delay
port.Read(dataBuffer, 2, 2);
// Write buffer to actual raw data byte array
this.data = dataBuffer;
dataBuffer = new byte[stdMessageLengthInBytes]; // Reset buffer
}
// Method called by SerialDataReceivedEventHandler
protected override void DataReceivedOnComPort(object sender, SerialDataReceivedEventArgs e)
{
bool valid = false;
ReadExistingData(); // Read data from COM- Port
lock (processedData)
{
switch (data[1]) // Check command byte
{
// Time (3 btyes)
case (0x06):
processedData.currentTime = String.Format("{0:D2}:{1:D2}:{2:D2}", DecodeBcd(data[2]), DecodeBcd(data[3]), DecodeBcd(data[4]));
valid = true;
receivedMessages++;
break;
// Value (2 bytes)
case (0x0D):
double val = 0;
val += DecodeBcd(data[2]) * 100;
val += DecodeBcd(data[3]);
val /= 10;
processedData.currentValue = val;
valid = true;
receivedMessages++;
break;
// ... here are various other hex- code that represent a command from the device (2 btyes)
default:
valid = false;
break;
}
}
// only to check when
if (valid)
{
Console.WriteLine("Received Valid Messages: {0}", receivedMessages);
ConsoleOutput();
}
}
}
On a note: The initialization of the port happens in another method from the base class and works fine.
Is there anything I am missing? How to deal with something like that? Are there any improvements that would help improving my performance? I thought about threading with locks, but I dont think that is the solution somehow... Or maybe everything is just a console problem?
EDIT:
I know changed my code (as #jdweng suggested) so that I put everything in a buffer (basically List<byte> mainBuffer. Then, I take all bytes in the buffer whenever its possbile and work with them, skimming it for 0xA5. When one is found, I read the command and determine how long the "message" has to be according to it (Time -> +3 bytes, Data -> +2 bytes, Other -> +1 byte). Then I can work off those messages (I put them into a List<byte[]>) and determine my output to my screen.
However, even after outsourcing the chopping up into messages and processing the messages, I still seem to either miss some messages, since some action are just not registered and have a big delay, or my processing is wrong. What I can think of is that because I lock my mainBuffer maybe some data isnt written to it.
Is this really this time critical? There is a software that comes with the device and it doesnt seem to have such big problems with delay and slightly wrong values...

Since you don't have the exact specs and/or an unreliable connection (which with serial data has to be expected) you need to sync to the 0xa5 at every message. I would just run every single byte you receive through a parser while keeping the state of the currently received message.
Make sure you validate your input since there are a bunch of things that can go wrong if you get messed up serial data. For example if there is an 0xa5 in the other message types, you might miss your next message. To prevent that I strongly recommend to either get to the specs if possible or code more logic based on data observations.
private const int MESSAGE_LENGTH = 5;
private const int VALUE_COMMAND = 0x0D;
private const int VALUE_SIZE = 4;
private const int TIME_COMMAND = 0x06;
private const int TIME_SIZE = 5;
private byte[] _message = new byte[MESSAGE_LENGTH];
private int _messagePos = 0;
private void port_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
var data = new byte[_serialPort.BytesToRead];
_serialPort.Read(data, 0, data.Length);
foreach (var b in data)
{
_message[_messagePos] = b;
if (_messagePos == 0 && b != 0xa5)
continue;
++_messagePos;
if (_messagePos > 2) // if command byte present, process command of any size
ProcessCommand(_message[1]);
}
}
private void ProcessCommand(byte command)
{
if (_messagePos == VALUE_SIZE && command == VALUE_COMMAND)
{
// parse value...
_messagePos = 0;
}
else if (_messagePos == TIME_SIZE && _message[1] == TIME_COMMAND)
{
// parse time...
_messagePos = 0;
}
}

.NET Native incredibly slower than Debug build with ReadAsync calls

so I just found a really weird issue in my app and it turns out it was caused by the .NET Native compiler for some reason.
I have a method that compares the content of two files, and it works fine. With two 400KBs files, it takes like 0.4 seconds to run on my Lumia 930 in Debug mode. But, when in Release mode, it takes up to 17 seconds for no apparent reason. Here's the code:
// Compares the content of the two streams
private static async Task<bool> ContentEquals(ulong size, [NotNull] Stream fileStream, [NotNull] Stream testStream)
{
// Initialization
const int bytes = 8;
int iterations = (int)Math.Ceiling((double)size / bytes);
byte[] one = new byte[bytes];
byte[] two = new byte[bytes];
// Read all the bytes and compare them 8 at a time
for (int i = 0; i < iterations; i++)
{
await fileStream.ReadAsync(one, 0, bytes);
await testStream.ReadAsync(two, 0, bytes);
if (BitConverter.ToUInt64(one, 0) != BitConverter.ToUInt64(two, 0)) return false;
}
return true;
}
/// <summary>
/// Checks if the content of two files is the same
/// </summary>
/// <param name="file">The source file</param>
/// <param name="test">The file to test</param>
public static async Task<bool> ContentEquals([NotNull] this StorageFile file, [NotNull] StorageFile test)
{
// If the two files have a different size, just stop here
ulong size = await file.GetFileSizeAsync();
if (size != await test.GetFileSizeAsync()) return false;
// Open the two files to read them
try
{
// Direct streams
using (Stream fileStream = await file.OpenStreamForReadAsync())
using (Stream testStream = await test.OpenStreamForReadAsync())
{
return await ContentEquals(size, fileStream, testStream);
}
}
catch (UnauthorizedAccessException)
{
// Copy streams
StorageFile fileCopy = await file.CreateCopyAsync(ApplicationData.Current.TemporaryFolder);
StorageFile testCopy = await file.CreateCopyAsync(ApplicationData.Current.TemporaryFolder);
using (Stream fileStream = await fileCopy.OpenStreamForReadAsync())
using (Stream testStream = await testCopy.OpenStreamForReadAsync())
{
// Compare the files
bool result = await ContentEquals(size, fileStream, testStream);
// Delete the temp files at the end of the operation
Task.Run(() =>
{
fileCopy.DeleteAsync(StorageDeleteOption.PermanentDelete).Forget();
testCopy.DeleteAsync(StorageDeleteOption.PermanentDelete).Forget();
}).Forget();
return result;
}
}
}
Now, I have absolutely no idea why this same exact method goes from 0.4 seconds all the way up to more than 15 seconds when compile with the .NET Native toolchain.
I fixed this issue using a single ReadAsync call to read the entire files, then I generated two MD5 hashes from the results and compared the two. This approach worked in around 0.4 seconds on my Lumia 930 even in Release mode.
Still, I'm curious about this issue and I'd like to know why it was happening.
Thank you in advance for your help!
EDIT: so I've tweaked my method in order to reduce the number of actual IO operations, this is the result and it looks like it's working fine so far.
private static async Task<bool> ContentEquals(ulong size, [NotNull] Stream fileStream, [NotNull] Stream testStream)
{
// Initialization
const int bytes = 102400;
int iterations = (int)Math.Ceiling((double)size / bytes);
byte[] first = new byte[bytes], second = new byte[bytes];
// Read all the bytes and compare them 8 at a time
for (int i = 0; i < iterations; i++)
{
// Read the next data chunk
int[] counts = await Task.WhenAll(fileStream.ReadAsync(first, 0, bytes), testStream.ReadAsync(second, 0, bytes));
if (counts[0] != counts[1]) return false;
int target = counts[0];
// Compare the first bytes 8 at a time
int j;
for (j = 0; j < target; j += 8)
{
if (BitConverter.ToUInt64(first, j) != BitConverter.ToUInt64(second, j)) return false;
}
// Compare the bytes in the last chunk if necessary
while (j < target)
{
if (first[j] != second[j]) return false;
j++;
}
}
return true;
}

Reading eight bytes at a time from an I/O device is a performance disaster. That's why we are using buffered reading (and writing) in the first place. It takes time for an I/O request to be submitted, processed, executed and finally returned.
OpenStreamForReadAsync appears to not be using a buffered stream. So your 8-byte requests are actually requesting 8 bytes at a time. Even with the solid-state drive, this is very slow.
You don't need to read the whole file at once, though. The usual approach is to find a reasonable buffer size to pre-read; something like reading 1 kiB at a time should fix your whole issue without requiring you to load the whole file in memory at once. You can use BufferedStream between the file and your reading to handle this for you. And if you're feeling adventurous, you could issue the next read request before the CPU processing is done - though it's very likely that this isn't going to help your performance much, given how much of the work is just I/O.
It also seems that .NET native has a lot bigger overhead than managed .NET for asynchronous I/O in the first place, which would make those tiny asynchronous calls all the more of a problem. Fewer requests of larger data will help.

Understanding the NetworkStream.EndRead()-example from MSDN

I tried to understand the MSDN example for NetworkStream.EndRead(). There are some parts that i do not understand.
So here is the example (copied from MSDN):
// Example of EndRead, DataAvailable and BeginRead.
public static void myReadCallBack(IAsyncResult ar ){
NetworkStream myNetworkStream = (NetworkStream)ar.AsyncState;
byte[] myReadBuffer = new byte[1024];
String myCompleteMessage = "";
int numberOfBytesRead;
numberOfBytesRead = myNetworkStream.EndRead(ar);
myCompleteMessage =
String.Concat(myCompleteMessage, Encoding.ASCII.GetString(myReadBuffer, 0, numberOfBytesRead));
// message received may be larger than buffer size so loop through until you have it all.
while(myNetworkStream.DataAvailable){
myNetworkStream.BeginRead(myReadBuffer, 0, myReadBuffer.Length,
new AsyncCallback(NetworkStream_ASync_Send_Receive.myReadCallBack),
myNetworkStream);
}
// Print out the received message to the console.
Console.WriteLine("You received the following message : " +
myCompleteMessage);
}
It uses BeginRead() and EndRead() to read asynchronously from the network stream.
The whole thing is invoked by calling
myNetworkStream.BeginRead(someBuffer, 0, someBuffer.Length, new AsyncCallback(NetworkStream_ASync_Send_Receive.myReadCallBack), myNetworkStream);
somewhere else (not displayed in the example).
What I think it should do is print the whole message received from the NetworkStream in a single WriteLine (the one at the end of the example). Notice that the string is called myCompleteMessage.
Now when I look at the implementation some problems arise for my understanding.
First of all: The example allocates a new method-local buffer myReadBuffer. Then EndStream() is called which writes the received message into the buffer that BeginRead() was supplied. This is NOT the myReadBuffer that was just allocated. How should the network stream know of it? So in the next line numberOfBytesRead-bytes from the empty buffer are appended to myCompleteMessage. Which has the current value "". In the last line this message consisting of a lot of '\0's is printed with Console.WriteLine.
This doesn't make any sense to me.
The second thing I do not understand is the while-loop.
BeginRead is an asynchronous call. So no data is immediately read. So as I understand it, the while loop should run quite a while until some asynchronous call is actually executed and reads from the stream so that there is no data available any more. The documentation doesn't say that BeginRead immediately marks some part of the available data as being read, so I do not expect it to do so.
This example does not improve my understanding of those methods. Is this example wrong or is my understanding wrong (I expect the latter)? How does this example work?

I think the while loop around the BeginRead shouldn't be there. You don't want to execute the BeginRead more than ones before the EndRead is done. Also the buffer needs to be specified outside the BeginRead, because you may use more than one reads per packet/buffer.
There are some things you need to think about, like how long are my messages/blocks (fixed size). Shall I prefix it with a length. (variable size) <datalength><data><datalength><data>
Don't forget it is a Streaming connection, so multiple/partial messages/packets can be read in one read.
Pseudo example:
int bytesNeeded;
int bytesRead;
public void Start()
{
bytesNeeded = 40; // u need to know how much bytes you're needing
bytesRead = 0;
BeginReading();
}
public void BeginReading()
{
myNetworkStream.BeginRead(
someBuffer, bytesRead, bytesNeeded - bytesRead,
new AsyncCallback(EndReading),
myNetworkStream);
}
public void EndReading(IAsyncResult ar)
{
numberOfBytesRead = myNetworkStream.EndRead(ar);
if(numberOfBytesRead == 0)
{
// disconnected
return;
}
bytesRead += numberOfBytesRead;
if(bytesRead == bytesNeeded)
{
// Handle buffer
Start();
}
else
BeginReading();
}

networkstream always empty!

hey I'm writing on an Server-Client program
but when my client sends something, it never reaches my server!
I'm sending like this:
public void Send(string s)
{
char[] chars = s.ToCharArray();
byte[] bytes = chars.CharToByte();
nstream.Write(bytes, 0, bytes.Length);
nstream.Flush();
}
and Receiving in a background thread like this
void CheckIncoming(object dd)
{
RecievedDelegate d = (RecievedDelegate)dd;
try
{
while (true)
{
List<byte> bytelist = new List<byte>();
System.Threading.Thread.Sleep(1000);
int ssss;
ssss = nstream.ReadByte();
if (ssss > 1)
{
System.Diagnostics.Debugger.Break();
}
if (bytelist.Count != 0)
{
d.Invoke(bytelist.ToArray());
}
}
}
catch (Exception exp)
{
MSGBOX("ERROR:\n" + exp.Message);
}
}
the ssss int is never > 1
whats happening here???

NetworkStream.Flush() actually has no effect:
The Flush method implements the Stream..::.Flush method; however, because NetworkStream is not buffered, it has no affect [sic] on network streams. Calling the Flush method does not throw an exception
How much data is being sent?
If you don't send enough data it may remain buffered at the network level, until you close the stream or write more data.
See the TcpClient.NoDelay property for a way to disable this, if you are only going to be sending small chunks of data and require low latency.

You should change the check of the return value to if (ssss >= 0).
ReadByte returns a value greater or equal 0 if it succeeds to read a byte (source).
To elaborate on Marc's comment: How is nstream created? Maybe there is an underlying class that does not flush.

well, Im creating a TcpClient, and use GetStream(); to get the NetworkStream