We Keep Coding

Parsing continuous incoming byte array from serial port

I have an incoming byte array in real-time that has the following format:
Header : 3 bytes
message: 10 bytes
Header : 3 bytes
message: 10 bytes
and so on
The problem is it is a continuous stream that I can not predict when the header comes as I could start reading incoming bytes at a portion of the message or header. So I would appreciate it if someone could assist me in how to trap a header and then stay in sync using dataReceived event.
Edit: I tried using a counter for checking each incoming byte as a header, then set the threshold to trigger after the payload is received.
Set Incoming threshold trigger to 1 byte
Check first, second headers
Get the payload if headers are detected and set the incoming threshold trigger to the payload.
Parse payload, set threshold again to 1 byte.
Although it is not bad, it is not reliable with each incoming frames unlike software like serialplot
private void _serialPort_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
byte data = 0;
if (_syncCounter < 3)
{
data = (byte)_serialPort.ReadByte();
}
switch (_syncCounter)
{
case 0: // First Byte in Header
_syncCounter = data == (byte)0xAA ? (byte) 1 : (byte) 0;
break;
case 1: // Second Byte in Header
_syncCounter = data == (byte)0xBB ? (byte)2 : (byte)0;
break;
case 2:
// Payload
_syncCounter = data == (byte)data ? (byte)3 : (byte)0;
if (_syncCounter == 3)
{
_serialPort.ReceivedBytesThreshold = data;
_payload = data;
}
break;
case 3:
// Packet
var packet = Enumerable.Repeat((byte)0x00, _payload).ToArray();
byte[] fieldBytes;
float fieldFloat;
_serialPort.Read(packet, 0, _payload);
if (packet.Length < _payload)
{
_serialPort.ReceivedBytesThreshold = 1;
_syncCounter = 0;
_payload = 0;
}
else
{
if (_isFirstPayload)
{
_syncCounter = 0;
_payload = 0;
_isFirstPayload = false;
break;
}
for (uint i = 0; i <= (_payload-1); i += 4)
{
fieldBytes = Slice(packet, i, i+4);
fieldFloat = ReadSingleBigEndian(fieldBytes, 0);
if (fieldFloat < 0)
fieldFloat = 0;
_sb.Append(fieldFloat.ToString(CultureInfo.InvariantCulture));
_sb.Append(",");
}
IncomingFrame.BeginInvoke(() =>
{
IncomingFrame.Text = ByteArrayToString(packet);
if (_sb.Length > 0)
{
_sb.Remove(_sb.Length - 1, 1);
FloatFields.Text = _sb.ToString();
}
_sb.Clear();
_serialPort.ReceivedBytesThreshold = 1;
_syncCounter = 0;
});
}
break;
}
}
The problem is, sometimes I receive the header as part of the payload although, from the code I wrote, it should be filtered before reading the payload.
I checked the same response using serialplot and it is syncing meaning the hardware is sending data as expected and parsed correctly.

Order reversed when echoing/debugging

I'm trying to send data from a VS Windows Form to an Arduino, but my data keeps getting mirrored. I'm not sure if it has to do with the buffer. If my understanding is correct, it should have nothing to do with the buffer nor to do with my code reading/writing data and it's a timing issue? The buffer is like the queue of the system. Since it's FIFO, the order in which I initialized, it'll have the same order.
I found this article, but I'm not sure if it applies. In this example about UART Ring Buffer the head and tail when declared share the same element. Does this apply to regular buffers? I assumed that since it's FIFO the head and tail wouldn't share the same element.
This Article on Double buffering seems to be what I'm talking about, but I don't think I'm technically using 2 buffers?
For example,
String a = "1";
String b = "2";
String c = "3";
String d = "4";
String e = "5";
String f = "6";
String g = "7";
String h = "8";
String[] sendchar = new String [] {a, b, c, d, e, f, g, h};
So when I send my data, the buffer stream should be, from the right being the first element and left being the last; "h,g,f,e,d,c,b,a" a would be sent first, then b and et cetera.
Currently, when I send data and it gets echoed back, I get in the reverse order, I'll send "a,b,c,d,e,f,g,h" but get "h,g,f,e,d,c,b,a" returned.
I'm receiving the data by reading it, and then storing it into an array, duplicating it, and then accessing elements in the duplicated array. This way the data order should be preserved.
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();
if (rb != endMarker)
{
receivedChar[ndx] = rb;
copyarray[ndx] = receivedChar[ndx];
ndx++;
How I get the data and send it on Arduino
void loop()
{
recvWithEndMarkers();//get Data
Serial.flush();//Clear Input buffer
delay(10);//delay
testblink();//Test blink
//blink();
echo();//echo data back
Serial.flush();
delay(2000);
}
void echo()
{
for (int b = 0; b <= 7; b++)
{
Serial.write(copyarray[b]);// Send b'th element of array
delay(50);
Serial.write("\n");//newline character terminates read
}
void recvWithEndMarkers() {
static boolean recvInProgress = false;
static byte ndx = 0;
char endMarker = '}';
byte rb;
byte comp;
while (Serial.available() > 0 && newData == false)
{
rb = Serial.read();//read data from input buffer
if (rb != endMarker)//not end marker
{
receivedChar[ndx] = rb;//store data into array index
copyarray[ndx] = receivedChar[ndx];//copy data
ndx++;//next index
if (ndx >= numBytes)
{
ndx = numBytes - 1;
}
}
else//endmarker
{
receivedChar[ndx] = '}'; // terminate the string
recvInProgress = false;
ndx = 0;//reset ndx
}
}
}
On the VS side
port = new SerialPort("COM3", 9600, Parity.None, 8, StopBits.One);
port.Open();
for (int a = 0; a <= 7; a++)
{
port.Write(sendchar[a]);
Thread.Sleep(50);
}
Thread.Sleep(50);
port.DiscardOutBuffer();
String[] recchar = new String[8];
while (port.BytesToRead != 0)
{
for (int a = 0; a <= 7; a++)
{
recchar[a] = port.ReadLine();
Thread.Sleep(50);
}
}
port.DiscardInBuffer();

I see at least a couple issues with your code. First, I assume that you reset the Arduino, then run your windows program, right?
ARDUINO:
Your recvWithEndMarkers() will probably see Serial.available() == 0, so exit the while loop right away.
When you put a character in the buffer, you increment the ndx (good). But when ndx == numBytes, you then set ndx to 7. It should probably be set to 0.
Also, you have ndx as a "static". So it will retain it's value when you run it a second time -- I'll bet that's not what you want.
When you exit the recvWithEndMarkers() function, you do a serial.flush(), which may cause you to lose characters, esp after the first time.
Then your echo() routine will "send" whatever is in copyarray[] at the time (not sure on Arduino if it's 0's or 255's, but probably not what you expect).
If your loop code had the flush and delay at the TOP of the loop (and maybe longer than 2 seconds), you could start Arduino, then start the VS program, and maybe get better results.
On the VS side, I don't see anything quite like that, but you did not share the code that prints the received data.
Good luck!

CSCore: playback float array from pure data

I'm writing a program in C# which communicates with pure data using the libpd library (more information: http://libpd.cc/)
My pure data patch is simply creating a sine wave.
I wrote some test code to see if I get any data from my pure data patch. The LibPD.Process method writes samples into the outbuffer array and the contents are continuous values between -1 and 1:
// Initialise Pd with 0 input and 2 output channels and a samplerate of 44100. Project dependent.
int pdOpen = -1;
pdOpen = LibPD.OpenAudio(0, 2, 44100);
if (pdOpen != 0)
{
Console.WriteLine("Error opening audio... exiting");
return;
}
string patchFile = #"..\..\..\test.pd";
int patch = LibPD.OpenPatch(patchFile);
LibPD.ComputeAudio(true);
//Read from Process Function...
// The size of each buffer must be the product of the number of channels, the number of ticks,
// and the number of samples per tick. (The number of samples per tick is the value
// returned by libpd_blocksize(), i.e., 64.)
int ret;
float[] outbuffer = new float[2 * 1 * LibPD.BlockSize]; // Size = 128
for (int i = 0; i < 10; i++)
{
ret = LibPD.Process(1, new float[0], outbuffer);
Thread.Sleep(1000);
}
LibPD.ClosePatch(patch);
LibPD.Release();
So I am sure that I am getting processed data from my patch.
Now I would like to play back this float array using CSCore. I found the ISampleSource interface in the documentation and I think it's the right one to go for this task. I tried something like this in the Read method of my interface implementation:
...
public PureDataSource()
{
_WaveFormat = new WaveFormat(44100, 16, 2);
int pdOpen = -1;
pdOpen = LibPD.OpenAudio(0, 2, 44100);
if (pdOpen != 0)
{
Console.WriteLine("Error opening audio... exiting");
return;
}
string patch = #"..\..\..\test.pd";
_Patch = LibPD.OpenPatch(patch);
LibPD.ComputeAudio(true);
}
...
public int Read(float[] buffer, int offset, int count)
{
LibPD.Process(1, new float[0], buffer);
return count;
}
But it doesn't work out - I only get crackling sounds (guess what). I know it has something to do with the buffer size of the Read method, but where can I configure this? The Process function of LibPd works like this:
The size of each buffer must be the product of the number of channels, the number of ticks,
and the number of samples per tick. (The number of samples per tick is the value
returned by libpd_blocksize(), i.e., 64.)
in my case that's: 2 channels (output channels), 1 tick and number of samples per tick is 64 --> 128.
EDIT:
I wrote a PureDataSource class which implements the ISampleSource interface using the information above:
class Program
{
static void Main(string[] args)
{
PureDataSource pdSource = new PureDataSource();
WasapiOut soundOut = new WasapiOut();
soundOut.Initialize(pdSource.ToWaveSource());
soundOut.Play();
Thread.Sleep(5000);
soundOut.Stop();
}
}
class PureDataSource : ISampleSource
{
public long Length
{
get
{
return 0;
}
}
public long Position
{
get
{
return 0;
}
set
{
throw new NotImplementedException();
}
}
private WaveFormat _WaveFormat;
public WaveFormat WaveFormat
{
get
{
return _WaveFormat;
}
}
private int _Patch;
public int Patch
{
get { return _Patch; }
//set { _Patch = value; }
}
public PureDataSource()
{
_WaveFormat = new WaveFormat(44100, 16, 2);
// Initialise Pd with 2 ins and outs and a samplerate of 44100. Project dependent.
int pdOpen = -1;
pdOpen = LibPD.OpenAudio(0, 2, 44100);
if (pdOpen != 0)
{
Console.WriteLine("Error opening audio... exiting");
return;
}
string patch = #"..\..\..\test.pd";
_Patch = LibPD.OpenPatch(patch);
LibPD.ComputeAudio(true);
}
public void Dispose()
{
LibPD.ClosePatch(_Patch);
LibPD.Release();
}
public int Read(float[] buffer, int offset, int count)
{
int ticks = 1;
int pdBufferPos = 0;
float[] pdBuffer = new float[2 * ticks * LibPD.BlockSize];
LibPD.Process(ticks, new float[0], pdBuffer);
for (int i = offset; i < count; i++)
{
if (pdBufferPos >= pdBuffer.Length)
{
pdBufferPos = 0;
LibPD.Process(ticks, new float[0], pdBuffer);
}
buffer[i] = pdBuffer[pdBufferPos];
pdBufferPos++;
}
return count;
}
}
The Read method fills the whole buffer with the output provided by LibPD.Process (this is a float array with the size of 128 each time).
I can hear a sine wave now, but with a lot of crackling sound - it seems the samples are not processed continuously. Any ideas how to solve this?

But it doesn't work out - I only get crackling sounds (guess what). I know it has something to do with the buffer size of the Read method, but where can I configure this?
You can't "configure" it. If the WasapiOut requests a certain amount of data, it is in your hands, how much data you want to return. Fill the buffer, starting at the specified offset, with the data you want. Return the number of provided samples. That's how you "configure" it.

Listen to synchronous callbacks with C#

I have a program which write data to a USB HID device. When data is received from the USB device I get a callback from the usblib library through a delegate event DataRecievedEventHandler.
I am used to programming on firmware with interrupts, so that I can do
while(!flag); // Will continue when interrupt triggers and change the flag
I want to write an array element by element to the USB and wait for a return from the device after each array element
for (int i = 0; i > 16; i++)
{
sendUsbData(array[i]);
while(!receivedComplete);
// Wait for response from USB before transmitting next iteration
}
The problem is that the callback will not get triggered when I'm spooling in that while loop. Any suggestion on how to do these kind of operations?
The library I am using for USB communication is the same as this one. In SpecifiedDevice.cs there is a method called public void SendData(byte[] data) which I am using to send arrays of bytes.
Transmit method:
public void sendUsbData(byte _txData)
{
byte[] txData = new byte[this.usb.SpecifiedDevice.OutputReportLength];
txData[1] = 0x50; // 0x50 = loopback command. Element 0 is always 0x00
int pos = 2;
foreach (byte b in _flashData)
{
txData[pos] = b;
pos++;
}
this.usb.SpecifiedDevice.SendData(txData);
}
Upon received data from USB, the callback usb_OnDataRecieved is called.
private void usb_OnDataRecieved(object sender, DataRecievedEventArgs args)
{
this.ParseReceivePacket(args.data); // Format to string and print to textbox
/*public bool*/receiveComplete = true;
}

You could switch to using a AutoResetEvent wait handle:
public void sendUsbData(byte _txData)
{
byte[] txData = new byte[this.usb.SpecifiedDevice.OutputReportLength];
txData[1] = 0x50; // 0x50 = loopback command. Element 0 is always 0x00
int pos = 2;
foreach (byte b in _flashData)
{
txData[pos] = b;
pos++;
}
// reset member wait handle
waitHandle = new AutoResetEvent(false);
this.usb.SpecifiedDevice.SendData(txData);
}
private void usb_OnDataRecieved(object sender, DataRecievedEventArgs args)
{
this.ParseReceivePacket(args.data); // Format to string and print to textbox
// signal member wait handle
waitHandle.Set();
}
And then in your for loop:
for (int i = 0; i > 16; i++)
{
sendUsbData(array[i]);
// wait for member wait handle to be set
waitHandle.WaitOne();
}

Why array behaves different when different thread access it

I need to parse large text that is similar to XML. Because the text it is not in memory ( I have a StreamReader object) placing that stream on memory is where I take the most time. So on one thread I place that stream into an array (memory). And I have another thread that process that array. But I am having wierd behavieours. For example take a look at this image:
Note that listToProcess[counter] = buffer and right now that should be listToProcess[10] = buffer Note that the debugger says that listToProcess[10]=null why!? . the other thread just reads the items it does not modify them. At first I thought that maybe the other thread was making that item = null but that is not the case. why am I experiencing this behavior?
In case you want to see my code here it is:
Semaphore sem = new Semaphore(0, 1000000);
bool w;
bool done = false;
// this task is responsible for parsing text created by main thread. Main thread
// reads text from the stream and places chunks in listToProces[]
var task1 = Task.Factory.StartNew(() =>
{
sem.WaitOne(); // wait so there are items on list (listToProcess) to work with
// counter to identify which chunk of char[] in listToProcess we are ading to the dictionary
int indexOnList = 0;
while (true)
{
if (listToProcess[indexOnList] == null)
{
if (done)
break;
w = true;
sem.WaitOne();
w = false;
if (done)
break;
if (listToProcess[indexOnList] == null)
{
throw new NotFiniteNumberException();
}
}
// add chunk to dictionary
ProcessChunk(listToProcess[indexOnList]);
indexOnList++;
}
}); // close task1
bool releaseSem = false;
// this main thread is responsible for placing the streamreader into chunks of char[] so that
// task1 can start processing those chunks
int counter = 0;
while (true)
{
char[] buffer = new char[2048];
// unparsedDebugInfo is a streamReader object
var charsRead = unparsedDebugInfo.Read(buffer, 0, buffer.Length);
if (charsRead < 1)
{
listToProcess[counter] = pattern;
break;
}
listToProcess[counter] = buffer;
counter++;
if (releaseSem)
{
sem.Release();
releaseSem = false;
}
if (counter == 10 || w)
{
releaseSem = true;
}
}
done = true;
sem.Release();
task1.Wait();
Edit
Sorry in other words why do I hit this break point:
I thought that counter was the problem but maybe I am doing something wrong with the semaphore...

You have a counter++ so the one you updated before that was at index 9, not index 10.
Meaning : your claim that it set
listToProcess[10] = buffer:
Is incorrect: it set
listToProcess[9] = buffer: