We Keep Coding

C# Multicast UDP Packet Drops while storing objects

I'm currently subscribing to a multicast UDP. It streams multiple messages, each about 80 bytes, in a single max 1000 byte packet. As the packets come in, I parse them into objects and then store them in a dictionary.
Each packet I receive comes with a sequential number so that I know if I've dropped any packets.
After about 10k packets received, I start to drop packets here and there.
securityDefinition xyz = new securityDefinition(p1,p2,p3,p4,p5...etc);
if (!secDefs.ContainsKey(securityID))
{
secDefs.Add(securityID, xyz); //THIS WILL CAUSE DROPS EVENTUALLY
secDefs.Add(securityID, null); //THIS WORKS JUST FINE
}
else
{
//A repeat definition is received and assuming all
//sequence numbers in the packet line up sequentially, I know i am done
//However if there is a drop somewhere (gap in sequence number),
//I know I am missing something
}
securityDefinition is a class containing roughly 15 ints, 10 decimals and 5 strings (<10 characters each).
Is there a faster way to store these objects in real time that can keep up with the fast UDP feed? I have tried making securityDefinition a struct, I have tried storing the data in a datatable, I have tried adding the secDef to a list and queue. Same issue with all.
It seems the only bottleneck is putting the objects in the dictionary. Creating the object and checking the dictionary to see if it already exists seems fine.
EDIT:
To clarify a few things - the security definitions come in from a server in a loop. There are roughly 1,000,000 definitions. Once they all are sent, they are then sent again, over and over. When my program starts, I need to initialize all the definitions. Once I get a repeat, I know I am done and can close this connection. However, if i receive a packet at sequence number 1, and the next packet is sequence number 3, I know I have dropped packet 2 and have no way of recovering it.

ConcurrentQueue<byte[]> pkts = new ConcurrentQueue<byte[]>();
//IN THE RECEIVER THREAD...
void ProductDefinitionReceiver()
{
while (!secDefsComplete)
{
byte[] data = new byte[1000];
s.Receive(data);
pkts.Enqueue(data);
}
}
//IN A SEPARATE THREAD:
public void processPacketQueue()
{
int dumped = 0;
byte[] pkt;
while (!secDefsComplete)
{
while (pkts.TryDequeue(out pkt))
{
if (!secDefsComplete)
{
//processPkt includes the parsing and inserting the secDef object into the dictionary.
processPkt(pkt);
}
else
{
dumped++;
}
}
}
Console.WriteLine("Dumped: " + dumped);
}

What is the right way to have multiple threads polling the same serial port?

I am very new to interacting with hardware. I am currently building a GUI in C# using the Windows Forms. I am using a serial port/usb to interact with a hardware device. What I am trying to achieve is to have multiple threads poll the device at different times. Some data such as temperature, current, power, etc. should be retrieved regularly (at least every second) and be updated on the GUI for the user to see. While other data will only be retrieved when the user presses a button on the form. Would multi-threading be the right approach to tackle this problem? If not, what would be a better solution? Examples would be greatly appreciated. Thank you for your time guys!
Update: I am trying to use the SerialPort.DataReceived Event as many of you have suggested and a terminator character '\r' to parse out individual replies from the serial port. Then I am invoking a method called DisplayText to handle that string. My problem now is I have no idea how to figure out what the string represents. Does it represent a temperature, a current, etc.
private char terminator = '\r';
private void SerialPorts_DataReceived(object sender, SerialDataReceivedEventArgs e)
{
try
{
output += serialPort1.ReadExisting();
if (output.IndexOf((char)this.terminator) > -1)
{
string workingString = output.Substring(0, output.IndexOf(terminator));
output = output.Substring(output.IndexOf(terminator) + 1);
this.Invoke(new EventHandler((s, a)=>DisplayText(s, a, workingString)));
}
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
private void DisplayText(object sender, EventArgs e, string s)
{
Console.WriteLine(s); // for testing only
richTextBox1.AppendText(s); // for testing only
}

I think it is not a good idea to have multiple threads to poll single port and try to synchronize.
Better have one thread which does the polling and which stores all the values in some "middle" object (singleton, static fields, what you like) and than synchronize access to this storage.
So one threads polls the port, collects the values and stores them somewhere, than the other threads gets the values from there.

You can guard all serial port access functionality using a mutex and design your application in such a way that the serial port is closed after each discrete operation (e.g. reading the temperature). One possible way to do this would be to write a wrapper API of-sorts which would:
Check if the mutex is unlocked (and return with an error code if it is locked),
Open the serial port
Read the required value
Close the serial port and clear the mutex
Return the read value
However, you are opening yourself up to all sorts of deadlocks with this solution and you will have to implement additional application level checks to prevent this.
As suggested above, a better solution would be to have a single thread periodically poll the serial port and continuously update all values in a data structure of your choice.
For the values which only need to be read after user input, you will have to implement a MessageQueue to request this data.
Update in response to the OPs edit
I would recommend either of the following 2 ways to approach the communication in your device:
Implement a request/response mechanism where you send request for a
specific data value (e.g. current) via serial port to your device,
the device parses this request and returns the corresponding value
Have your device return all values in a "frame" e.g.
[SOF][D0...Dn][EOF], where "D0...Dn" is all your data in a
pre-determined sequence; SOF and EOF can be any values of your
choosing (you can also choose to skip these entirely and rely on the
number of bytes to delimit frames).
In case you don't have the option of changing the code in the device, there must be some sort of order in which the values are reported by the device so maintaining a sequence number of the received data would be the way to go. But without knowing more details about what your device is doing, it would be difficult to suggest a solution.

My problem now is I have no idea how to figure out what the string
represents. Does it represent a temperature, a current, etc.
Use simple protocol to send temperature, a current, ect.
SOH + DATA + ETX + BCC
SOH = 0x02;
ETX = 0x03;
BCC = 0x00-0xFF, XOR value of of all DATA + ETX.
DATA = use ';' as delimiter between Temperature(Temp) and Current(Curr)
Example:
TempCmd = 0x01
TempValue = 32 celcius = 0x20
CurrCmd = 0x02
CurrValue = 1,4 ampere = 0x0E -> result must be divide with 10 on receiver.
SOH + DATA + ETX + BCC
SOH + TempCmd:TempValue; CurrCmd:CurrValue + ETX + BCC
02 + 01:20;02:0E + 03 + BCC
BCC = 01 xor ':' xor 20 xor ';' xor 02 xor ':' xor 0E xor 03
Send data from microcontroller "0201:20;02:0E03;XX" (send as string)
Receive data in computer:
Detect SOH and ETX in package
Validate BCC is true (completed package) or false (broken package)
Parsing DATA with ; as delimiter // data1= 01:20 data2= 02:0E03
Parsing data1 and data2 with : as delimiter // data1= 20 data2= 0E03
Convert hex to decimal and divide data2 with 10 for get current with float format.

Visual Studio windows form application C# serial communication receive data?

I need a code that will receive serial data from arduino and display them in text boxes .
I am receiving int type data separated by commas .
Arduino serial data sample : 250,389,123,232,255,536,366,455,...
I need first six data to be displayed in six separate text boxes and then the consecutive data must replace the already existing values in those text boxes.
I tried several times but everything went in vain. Some one help me.

I assume you're using System.IO.Ports namespace and SerialPort class for communication. I also hope that BaudRate and other communication settings are as expected by the device.
If yes, then further if you receive the data repeatedly, you may capture it by using ReadTo method and giving it a comma as parameter. Such reading loop may look like this:
while(true) // replace it with some wiser condition
{
string textRead = serialPort.ReadTo(",");
// do the rest here
}
Now, you may also want to capture some larger amounts of data at once. Of course you can (say, capture 100 characters), but then you should:
- split the string with comma;
- leave the last item of string array (which will be the result of splitting method) and insert at 0 position of next received pack of characters;
- repeat those steps in loop like above.
Now, to the TextBox'es. If there are N of them and we take the first capturing method into account, you may do it like this:
TextBox[] tboxes = new TextBox[N]; // your number is 6, I know
int boxIndex = 0;
while(true) // again, use a reasonable condition
{
string textRead = serialPort.ReadTo(",");
tboxes[boxIndex].Text = textRead.Trim(',');
if(++boxIndex >= tboxes.Length)
boxIndex = 0;
}
This should do the job.

correct way to parse serial port data?

I'm testing out a program right now that is supposed to receive data from a serial port and parse it, and continually update certain variables tied to that parsed data. I'm having trouble with the actual parsing itself.
The data string output of the port I'm using is: 8 data bits, 1 start bit, 1 stop bit, and no parity. A message starts with the colon symbol : and ends with . An example - : 3.00 20.45 2355 1000 554
First question: Is there any method for setting the start bit? I know there's a .StopBits which I have equal to StopBits.One, but I haven't found anything for a start bit.
In regards to the parsing-
In the code method below:
serial.DataReceived += new System.IO.Ports.SerialDataReceivedEventHandler(Receive);
private void Receive(object sender, SerialDataReceivedEventArgs e)
{
// Collecting the characters received to our 'buffer' (string).
received_Data = serial.ReadExisting();
}
Would the best way be to set up a string array = received_Data.Split, then use a for loop to go through each "split" and assign it to the correct variable? I've never worked with ports before so this is all new to me. After looking around there seems to be several ways to do this, not sure what the best way would be though.

Improper format reading the serial port in C#

I am reading data from an Arduino board using C#.
In C#, I have the following:
// Write a string
port.Write("x");
Thread.Sleep(50);
SerialPortRead();
... and in SerialPortRead() I have:
private static void SerialPortRead()
{
port.DataReceived += new SerialDataReceivedEventHandler(port_DataReceived);
}
private static void port_DataReceived(object sender,SerialDataReceivedEventArgs e)
{
Console.WriteLine(port.ReadExisting());
}
and, the output would look something like:
329327
32
7
327
3
26
327
3
26
32
7
What did I do wrong? The output should be a around 326-329, where this value is coming from a compass which is hooked up to one of the pins that I am reading from Arduino.
Note that:
In Arduino, I have a Serial read method that watch for the input character x and return the value of the compass.

I'd guess that you are reading faster that the compass is writing. So that way your program believes that it is receiving two direction, while if it has delayed reading a little bit it would have received only one.
This hypothesis is supported by that if you group the read data into numbers of three, we get directions in the range you expect.
Try adding a delay before reading after sending the trigger character, or even better - add a control character to signal that the compass has written the entire direction and then read until you receive that char.
Edit: I now noticed that you already have a sleep - so the option remaining would be a separation/termination char.