I am currently writing an application that communicates with an integrated servo via a serial connection.
The motor sends out position data at a rate of up to 1000 times/second. What I'm trying to achieve is to be able to format the data coming back (by stripping it of white spaces, new lines, etc) and parsing it to extract the relevant data from the received strings.
Currently, I have the data received event handler read the data, format it using a series of string.replace method calls, and append it to a string that acts as a buffer. Then, using threads, I constantly check the buffer as it fills for a particular delimiter (in my case "\r") which signifies the end of one message from the motor, then remove that message from the buffer and print it to a rich text field.
There are two problems with this approach. One is that because the motor streams position data at such a high rate, the buffer fills faster than the data can be processed by the threads. Thus when I send a command to the motor, it acts immediately but the response is delayed by a few seconds because all of the preceding data in the buffer must be processed first. Second, having two threads running a method that implements a while(true) structure means processor utilization skyrockets and within a few seconds the fans in the pc are on max.
Is there a better way of handling the data?
Here is my event handler code:
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
}
}
Here is the method that the threads execute:
private void parseBuffer()
{
while (true)
{
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}
Your parseBuffer will go wild spinning even if there is no new data since last try.
You can mitigate this with signalling.
private AutoResetEvent waitHandle = new AutoResetEvent(false);
Trigger the signal in dataReceived
//data recieved event handler
private void dataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string tmp;
tmp = sp.ReadExisting();
//cut out any unnecessary characters
tmp = tmp.Replace("\n", "");
tmp = tmp.Replace(",", "\r");
tmp = tmp.Replace(" ", "");
lock (this)
{
//put all received data into the read buffer
readBuffer += tmp;
waitHandle.Set(); // <-- tell parseBuffer that new data is available
}
}
wait for the signal in parseBuffer
private void parseBuffer()
{
while (true)
{
waitHandle.WaitOne(); // <-- waits until there is more data to parse
//obtain lock, parse one message from buffer
lock (this)
{
if (readBuffer.IndexOf("\r") > 0)
{
String t = readBuffer.Substring(0, readBuffer.IndexOf("\r") + 1);
readBuffer = readBuffer.Replace(t, "");
dataReady(this, new CustomEventArgs(t, null));
}
}
}
}
There are a couple of things you can do to improve this dramatically.
1) Build a state-machine parser that parses the incomming data one character at a time. When it has built a complete "message", add it to a List<MyMessage> structure.
2) Use a Virtualized ListView or DataGridView to display the List<MyMessage>.
In the data received event read the incoming data as raw bytes and store in a queue. Don't process the data in the event handler. Then use something similar to what albin said to process the data in another method. The important thing is to allow the eventhandler to fire as often as possible and do no more that required.
In general I would use a blocking collection between a reader thread which solely reads from the socket and a parsing thread.
In terms of performance look at using split for parsing - this is a lot faster than replacing inside a string. You should look at using a regular expression and/or the StringBuilder class - you should use 1 expression with alternatives
The regex code would look like this:
string pattern = " |\\n|\\r";
string replacement = " ";
regex rgx = new Regex(pattern);
string result = rgx.Replace(input, replacement);
Related
In my scenario there are one or two serial port. If only one port is configured, that port can be dedicated to a PLC or to a Scale.
The PLC communicate only when user push some button on the machine, the Scale instead constantly poll the measured that has weighted. If only one port is configured there are no issue.
When two port are configured the communication protocol change a bit, if user push the PLC's weight button PLC send a weight request. If that request is listened on DataReceived event the Scale port is opened and the polled value is readen.
Now i have to admit that the legacy code that i will paste here is not the best code possible, and that is bad that it use the same DataReceived event for both the ports; but that code had work for quite a lot (about six years). Recently the Scale has been changed and the program had stoped working properly (only in the two port configuration) raising a sistematic timeout exception. I have sniffed the traffic and nothing change with protocol or data format.
Here the problematic piece of code :
private void SerialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
lock (_myLockSerialPort)
{
string source = "No Data" ;
try
{
if (sender == null)
return;
SerialPort genericPort = (SerialPort)sender;
if (genericPort.Equals(ScalePort))
{
if (genericPort.BytesToRead > 0)
{
source = "Scale" ;
string datas = string.Empty;
datas = genericPort.ReadTo(_currentConfiguration.ReadToForPese);
ReadDataFromScale(genericPort, datas);
}
}
else
{
if (genericPort.BytesToRead > 0)
{
source = "PLC" ;
string datas = string.Empty;
datas = genericPort.ReadTo(_currentConfiguration.ReadToForPlc);
ReadDataFromPlc(genericPort, datas);
}
}
}
catch (TimeoutException timeoutException)
{
//Handle Timeout
}
catch (Exception err)
{
//Handle Other Errors
}
}
}
The question is if there is something in the ReadTo method or in the polling interval that can explain that behaviour.
I would underline that nothing in the data exchange format has changed, so the character used to terminate the communication are still in place (and present in the sniffed traffic).
NO, NO, NO!!! If you had a text file normally you would use ReadLine() method to read one line at a time. With ports or sockets data is coming at you at real time and the program is probably running faster than the data so you will not get a full line at one time. The serial uart in the PC has only a small read buffer (could be as small as 8 bytes). Windows is using timers to move the serial port data into a stream. Your program is reading the windows stream. So the actual data received is a combination of how fast the sender is sending data, the latency of the UART, the latency of Windows, and the latency of the Net Library. It is not a configuration issue. It is a real time issue.
I would recommend using two SerialPort classes, one for your scale and one for the PLC in which they would have their own DataReceived events. As others have suggested, you should append characters until you see your end character, then parse the string with your methods you wrote. Below is an example where Line Feed (LF) is my end character I'm looking for.
StringBuilder sbScale = new StringBuilder();
StringBuilder sbPLC = new StringBuilder();
char LF = (char)10;
private void serialPortScale_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string Data = serialPortScale.ReadExisting();
foreach (char c in Data)
{
if (c == LF)
{
sbScale.Append(c);
CurrentLine = sbScale.ToString();
sbScale.Clear();
ReadDataFromScale(CurrentLine);
}
else
{
sbScale.Append(c);
}
}
}
private void serialPortPLC_DataReceived(object sender, System.IO.Ports.SerialDataReceivedEventArgs e)
{
string Data = serialPortPLC.ReadExisting();
foreach (char c in Data)
{
if (c == LF)
{
sbPLC.Append(c);
CurrentLine = sbPLC.ToString();
sbPLC.Clear();
ReadDataFromPlc(CurrentLine);
}
else
{
sbPLC.Append(c);
}
}
}
You could try to implement that into one DataReceived event, but it can be messy. I think have dedicated classes to each device is much cleaner and easier to read, especially if someone has to take over the code some day after you. Also I clear the string builder immediately, because the DataReceived event is threaded, so it can fire while you're already in there.
I have script in Unity that is exchanging data with another Python app. It has a while loop that listens for UDP messages as a background thread. Also the script is asking for new data every frame via the Update function.
After I receive a message, the script parses it as a string and it needs to split the string by tabs in order to retrieve all the values. Currently, the string contains eyetracker and joystick data that Unity needs as player inputs.
UDPController.cs
private void init()
{
// define address to send data to
pythonEndPoint = new IPEndPoint(IPAddress.Parse(IP), pythonPort);
unityEndPoint = new IPEndPoint (IPAddress.Parse (IP), unityPort);
pythonSock = new Socket(AddressFamily.InterNetwork, SocketType.Dgram, ProtocolType.Udp);
//define client to receive data at
client = new UdpClient(unityPort);
client.Client.ReceiveTimeout = 1;
client.Client.SendTimeout = 1;
// start background thread to receive information
receiveThread = new Thread(new ThreadStart(ReceiveData));
receiveThread.IsBackground = true;
receiveThread.Start();
}
void Update(){
if (Calibration.calibrationFinished && startRequestNewFrame) {
RequestData();
}
}
private void RequestData() {
// Sends this to the UDP server written in Python
SendString("NEWFRAME");
}
// receive thread which listens for messages from Python UDP Server
private void ReceiveData()
{
while (true)
{
try
{
if(client.Available > 0) {
double unixRecvTimeStamp = DataManager.ConvertToUnixTimestamp(DateTime.Now);
byte[] data = client.Receive(ref pythonEndPoint);
string rawtext = Encoding.UTF8.GetString(data);
string[] msgs = rawtext.Split('\t');
string msgType = msgs[0];
double pythonSentTimeStamp = double.Parse(msgs[msgs.Length-1].Split(' ')[1]);
DataManager.UdpRecvBuffer += '"' + rawtext + '"' + "\t" + pythonSentTimeStamp + "\t" + unixRecvTimeStamp + "\t" + DataManager.ConvertToUnixTimestamp(DateTime.Now) + "\n";
if (String.Equals(msgType, "FRAMEDATA"))
{
DataManager.gazeAdcsPos = new Vector2(float.Parse(msgs[1].Split(' ')[1]), float.Parse(msgs[2].Split(' ')[1]));
float GazeTimeStamp = float.Parse(msgs[3].Split(' ')[1]);
DataManager.rawJoy = new Vector2(float.Parse(msgs[4].Split(' ')[1]), 255 - float.Parse(msgs[5].Split(' ')[1]));
float joyC = float.Parse(msgs[6].Split(' ')[1]);
float ArduinoTimeStamp = float.Parse(msgs[7].Split(' ')[1]);
}
}
}
catch (Exception err)
{
print(err.ToString());
}
}
}
So according to the Unity Profiler, it seems like there is a huge amount of time spent in Behaviour Update, especially inside UDPController.Update() and GC.Collect. My initial hypothesis is that perhaps I'm creating too many strings and arrays overtime and the garbage collector kicks in quite often to remove the unused memory space.
So my question is, is my hypothesis right? If so, how I can rewrite this code to increase my performance and reduce the drop in FPS and perceived lag. If not, where is the issue at because currently the game starts to lag right about 10 minutes in.
Moreover, is there a better way or format for data transferring? It seems like I can be using objects like JSON, Google Protocol Buffer or msgpack or would that be an overkill?
I can see a lot of local variables in your while loop (along with arrays). Local variables cause Garbage collector to run. You should declare all the variables outside of the method.
Moreover, avoid string operations in while/update() as strings are immutable. Thus your code create a new copy to store the result after every concatenation. Use StringBuilder in these situations to avoid GC.
Read More
I have a SerialPort DataReceived event handler in C# code:
void serPort1_DataReceived(object sender, SerialDataReceivedEventArgs e) {
SerialPort sp = (SerialPort)sender;
string indata = "";
while (sp.ReadBufferSize > 0) {
int bytes = sp.ReadBufferSize;
char[] result = new char[bytes];
sp.Read(result, 0, bytes);
indata += new string(result).Replace("\0", "");
}
Dispatcher.Invoke((Action)(() => port1out.Text += indata + "\n"));
}
Using breakpoints, I know that the funtion is called, while loop is being executed, even indata variable contains received data, but when the while loop ends, the whole listener ends. No text is added to port1out textbox. I tried to add nothing-doing code after(and before too) the Dispatcher.Invoke.... line, use breakpoint and it's not executed - code just ends after while loop.
Why? Why isn't the whole statement executed?
Your loop will never end because ReadBufferSize will never be zero. You are misunderstanding what ReadBufferSize means. It is not the amount of data waiting to be read, but the maximum amount of data the SerialPort class will hold (or buffer) while it waits for you to read out. What you should be looking at instead is the BytesToRead property.
You can also call the ReadExisting() method as is shown on the MSDN example instead.
Throw away the DataReceived event, and just do this instead:
void ReadFromPort(SerialPort p)
{
Stream s = p.BaseStream;
s.ReadTimeout = 20;
// 1/50th of a second and 10 serial bits per byte
byte[] buffer = new byte[p.BaudRate / 500];
ReadSerialStream(s, buffer);
}
void async ReadSerialStream(Stream s, byte[] buffer)
{
int bytesRead = await s.ReadAsync(buffer, 0, buffer.Length);
string indata = Encoding.ASCII.GetString(
buffer.Take(bytesRead).Where(b => b != 0).ToArray());
port1out.Text += indata + Environment.Newline;
ReadSerialStream(s, buffer);
// or use an infinite while loop. With async methods, recursion is safe
}
Actually having a read in progress will make virtual serial ports actually start transferring data. If you instead rely on DataReceived and BytesToRead, your data may get stuck in (for example) USB transfer buffers, and never reach the serial port object. Or you may end up with multiple USB transfers for each individual byte (first reporting the "buffer not empty" flag, then transferring the buffer content). ReadAsync is more efficient at all levels. Also, it works within the async/await paradigm to receive serial events directly on the UI thread, so you don't have to mess with cross-thread calls or synchronizing access to internal data structure.
I need to send commands to a serial port to communicate with a Enfora modem. For every command, I will get a reply but the length of the reply string may vary.
I need to know how write and then wait for the reply until it finishes...
So i thought of making a thread that reads from the serial port and the program only writes...
The thread function
private void thread_Handler()
{
while(true)
this.read();
}
private void read()
{
if (this.rtbMessages.InvokeRequired)
{
try
{
SetTextCallback d = new SetTextCallback(read);
this.Invoke(d);
}
catch{}
}
else
{
readBuffer = serialPort.ReadExisting();
rtbMessages.AppendText(readBuffer);
}
}
So this thread is always trying to read from the com PORT and I send the messages this way
writeBuffer = "COMMAND 1";
serialPort.Write(writeBuffer);
writeBuffer = "COMMAND 2";
serialPort.Write(writeBuffer);
However I don't get the reply from the second command I send with Write()...
I tried removing the thread and using ReadExisting() after every Write() but that also didn't work.
The only way I could get it to work was to add a
System.Threading.Thread.Sleep(1000);
after every call to Write, then I get all the replies from every Write() command...
But I don't want to use this, I want to know of another way to effectively write and get every reply from every command I send regardless of the length of the reply string and how long it takes for me to receive the reply message.
Sometimes I will keep getting messages forever until I send another command to stop generating messages.
Thank you!
.Net will do all this for you.
Just create a SerialPort and subscribe to its DataReceived event. (Note that under some circumstances you may need to stitch together several chunks of data that are received in this way in order to assemble a complete data packet, but if it's a short reply from a modem command you'll probably find you always/usually get the full packet each time the event is raised.
Use an Event to receive the data.
Here is an example from DreamInCode (you will have to customize it to your particular needs):
/// <summary>
/// This method will be called when there's data waiting in the comport buffer
/// </summary>
void comPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
//determine the mode the user selected (binary/string)
switch (CurrentTransmissionType)
{
//user chose string
case TransmissionType.Text:
//read data waiting in the buffer
string msg = comPort.ReadExisting();
//display the data to the user
DisplayData(MessageType.Incoming, msg + "\n");
break;
//user chose binary
case TransmissionType.Hex:
//retrieve number of bytes in the buffer
int bytes = comPort.BytesToRead;
//create a byte array to hold the awaiting data
byte[] comBuffer = new byte[bytes];
//read the data and store it
comPort.Read(comBuffer, 0, bytes);
//display the data to the user
DisplayData(MessageType.Incoming, ByteToHex(comBuffer) + "\n");
break;
default:
//read data waiting in the buffer
string str = comPort.ReadExisting();
//display the data to the user
DisplayData(MessageType.Incoming, str + "\n");
break;
}
}
http://www.dreamincode.net/forums/topic/35775-serial-port-communication-in-c%23/
I recently wrote a quick-and-dirty proof-of-concept proxy server in C# as part of an effort to get a Java web application to communicate with a legacy VB6 application residing on another server. It's ridiculously simple:
The proxy server and clients both use the same message format; in the code I use a ProxyMessage class to represent both requests from clients and responses generated by the server:
public class ProxyMessage
{
int Length; // message length (not including the length bytes themselves)
string Body; // an XML string containing a request/response
// writes this message instance in the proper network format to stream
// (helper for response messages)
WriteToStream(Stream stream) { ... }
}
The messages are as simple as could be: the length of the body + the message body.
I have a separate ProxyClient class that represents a connection to a client. It handles all the interaction between the proxy and a single client.
What I'm wondering is are they are design patterns or best practices for simplifying the boilerplate code associated with asynchronous socket programming? For example, you need to take some care to manage the read buffer so that you don't accidentally lose bytes, and you need to keep track of how far along you are in the processing of the current message. In my current code, I do all of this work in my callback function for TcpClient.BeginRead, and manage the state of the buffer and the current message processing state with the help of a few instance variables.
The code for my callback function that I'm passing to BeginRead is below, along with the relevant instance variables for context. The code seems to work fine "as-is", but I'm wondering if it can be refactored a bit to make it clearer (or maybe it already is?).
private enum BufferStates
{
GetMessageLength,
GetMessageBody
}
// The read buffer. Initially 4 bytes because we are initially
// waiting to receive the message length (a 32-bit int) from the client
// on first connecting. By constraining the buffer length to exactly 4 bytes,
// we make the buffer management a bit simpler, because
// we don't have to worry about cases where the buffer might contain
// the message length plus a few bytes of the message body.
// Additional bytes will simply be buffered by the OS until we request them.
byte[] _buffer = new byte[4];
// A count of how many bytes read so far in a particular BufferState.
int _totalBytesRead = 0;
// The state of the our buffer processing. Initially, we want
// to read in the message length, as it's the first thing
// a client will send
BufferStates _bufferState = BufferStates.GetMessageLength;
// ...ADDITIONAL CODE OMITTED FOR BREVITY...
// This is called every time we receive data from
// the client.
private void ReadCallback(IAsyncResult ar)
{
try
{
int bytesRead = _tcpClient.GetStream().EndRead(ar);
if (bytesRead == 0)
{
// No more data/socket was closed.
this.Dispose();
return;
}
// The state passed to BeginRead is used to hold a ProxyMessage
// instance that we use to build to up the message
// as it arrives.
ProxyMessage message = (ProxyMessage)ar.AsyncState;
if(message == null)
message = new ProxyMessage();
switch (_bufferState)
{
case BufferStates.GetMessageLength:
_totalBytesRead += bytesRead;
// if we have the message length (a 32-bit int)
// read it in from the buffer, grow the buffer
// to fit the incoming message, and change
// state so that the next read will start appending
// bytes to the message body
if (_totalBytesRead == 4)
{
int length = BitConverter.ToInt32(_buffer, 0);
message.Length = length;
_totalBytesRead = 0;
_buffer = new byte[message.Length];
_bufferState = BufferStates.GetMessageBody;
}
break;
case BufferStates.GetMessageBody:
string bodySegment = Encoding.ASCII.GetString(_buffer, _totalBytesRead, bytesRead);
_totalBytesRead += bytesRead;
message.Body += bodySegment;
if (_totalBytesRead >= message.Length)
{
// Got a complete message.
// Notify anyone interested.
// Pass a response ProxyMessage object to
// with the event so that receivers of OnReceiveMessage
// can send a response back to the client after processing
// the request.
ProxyMessage response = new ProxyMessage();
OnReceiveMessage(this, new ProxyMessageEventArgs(message, response));
// Send the response to the client
response.WriteToStream(_tcpClient.GetStream());
// Re-initialize our state so that we're
// ready to receive additional requests...
message = new ProxyMessage();
_totalBytesRead = 0;
_buffer = new byte[4]; //message length is 32-bit int (4 bytes)
_bufferState = BufferStates.GetMessageLength;
}
break;
}
// Wait for more data...
_tcpClient.GetStream().BeginRead(_buffer, 0, _buffer.Length, this.ReadCallback, message);
}
catch
{
// do nothing
}
}
So far, my only real thought is to extract the buffer-related stuff into a separate MessageBuffer class and simply have my read callback append new bytes to it as they arrive. The MessageBuffer would then worry about things like the current BufferState and fire an event when it received a complete message, which the ProxyClient could then propagate further up to the main proxy server code, where the request can be processed.
I've had to overcome similar problems. Here's my solution (modified to fit your own example).
We create a wrapper around Stream (a superclass of NetworkStream, which is a superclass of TcpClient or whatever). It monitors reads. When some data is read, it is buffered. When we receive a length indicator (4 bytes) we check if we have a full message (4 bytes + message body length). When we do, we raise a MessageReceived event with the message body, and remove the message from the buffer. This technique automatically handles fragmented messages and multiple-messages-per-packet situations.
public class MessageStream : IMessageStream, IDisposable
{
public MessageStream(Stream stream)
{
if(stream == null)
throw new ArgumentNullException("stream", "Stream must not be null");
if(!stream.CanWrite || !stream.CanRead)
throw new ArgumentException("Stream must be readable and writable", "stream");
this.Stream = stream;
this.readBuffer = new byte[512];
messageBuffer = new List<byte>();
stream.BeginRead(readBuffer, 0, readBuffer.Length, new AsyncCallback(ReadCallback), null);
}
// These belong to the ReadCallback thread only.
private byte[] readBuffer;
private List<byte> messageBuffer;
private void ReadCallback(IAsyncResult result)
{
int bytesRead = Stream.EndRead(result);
messageBuffer.AddRange(readBuffer.Take(bytesRead));
if(messageBuffer.Count >= 4)
{
int length = BitConverter.ToInt32(messageBuffer.Take(4).ToArray(), 0); // 4 bytes per int32
// Keep buffering until we get a full message.
if(messageBuffer.Count >= length + 4)
{
messageBuffer.Skip(4);
OnMessageReceived(new MessageEventArgs(messageBuffer.Take(length)));
messageBuffer.Skip(length);
}
}
// FIXME below is kinda hacky (I don't know the proper way of doing things...)
// Don't bother reading again. We don't have stream access.
if(disposed)
return;
try
{
Stream.BeginRead(readBuffer, 0, readBuffer.Length, new AsyncCallback(ReadCallback), null);
}
catch(ObjectDisposedException)
{
// DO NOTHING
// Ends read loop.
}
}
public Stream Stream
{
get;
private set;
}
public event EventHandler<MessageEventArgs> MessageReceived;
protected virtual void OnMessageReceived(MessageEventArgs e)
{
var messageReceived = MessageReceived;
if(messageReceived != null)
messageReceived(this, e);
}
public virtual void SendMessage(Message message)
{
// Have fun ...
}
// Dispose stuff here
}
I think the design you've used is fine that's roughly how I would and have done the same sort of thing. I don't think you'd gain much by refactoring into additional classes/structs and from what I've seen you'd actually make the solution more complex by doing so.
The only comment I'd have is as to whether the two reads where the first is always the messgae length and the second always being the body is robust enough. I'm always wary of approaches like that as if they somehow get out of sync due to an unforseen circumstance (such as the other end sending the wrong length) it's very difficult to recover. Instead I'd do a single read with a big buffer so that I always get all the available data from the network and then inspect the buffer to extract out complete messages. That way if things do go wrong the current buffer can just be thrown away to get things back to a clean state and only the current messages are lost rather than stopping the whole service.
Actually at the moment you would have a problem if you message body was big and arrived in two seperate receives and the next message in line sent it's length at the same time as the second half of the previous body. If that happened your message length would end up appended to the body of the previous message and you'd been in the situation as desecribed in the previous paragraph.
You can use yield return to automate the generation of a state machine for asynchronous callbacks. Jeffrey Richter promotes this technique through his AsyncEnumerator class, and I've played around with the idea here.
There's nothing wrong with the way you've done it. For me, though, I like to separate the receiving of the data from the processing of it, which is what you seem to be thinking with your proposed MessageBuffer class. I have discussed that in detail here.