We Keep Coding

Data overlapping while buffering message stream from Bluetooth

I am trying to process JPS (Javad GREIS) live data stream from Bluetooth Serial. Here is the description about the stream.
Stream Host sends data in various lengths but Plugin.BLE only caches 20 bytes (max) long packets.
Each packet may contain part of a message, single message or multiple messages.
Packets never padded with zeroes.
Each message starts with header consists of 2 letter message identifier,3 letter message length in hex, leads with actual message and ends with 0x0A.
A message looks like this :~~005XXXXX\n
"~~": Message Type,
"005": Message Length,
"X" represents actual message.
And packet may look like "X\n~~005XXXXX\nSE001X\n". To process stream in real time, my app has to be in sync with message stream.
Problem is Bluetooth ValueUpdated event. It doesn't wait for GREIS.Parse() method to end before firing up the next event. buffer.AddRange() writes over old data. Due to this error, buffer looks like this ~~005XXSE002XX\n. But it should be ~~005XXXXX\nSE002XX\n.
Is there any way to hold the events or cache incoming data while Parser does it's job without losing any data or packet order?
//GREIS.Parse
public static string Parse(byte[] data)
{
if (!sync)
{
if (isSync(data))
{
Wait = false;
int MessageLength = decode_length("" + (char)data[2] + (char)data[3] + (char)data[4]);
if (MessageLength != 0 && MessageLength < 8192)
{
buffer.AddRange(data);
return Process();
}
else
return "Length Error";
}
else
return "Not SYNC!";
}
buffer.AddRange(data);
return Process();
}
Bluetooth ValueUpdated event handler:
private async void OnReceiveGREIS_ValueUpdated(object sender, Plugin.BLE.Abstractions.EventArgs.CharacteristicUpdatedEventArgs e)
{
string res = "";
res = GREIS.Parse(e.Characteristic.Value);
await Xamarin.Essentials.MainThread.InvokeOnMainThreadAsync(() =>
{
Output.Text = res;
});
}

How to validate the message that is receive in socket in C#

First 4 characters represent the length of message.
I want to validate by getting first 4 bit of the received message to find its length and verify whether it matches with the first 4 bit.
For example
First four bit give me 45 and message length is 49 then this is true (45 body + first 4 bit length)
else first four bit give 45 but message length 35 . drop this message. This where the problem is.
Class:
internal static void BeginReceive(this Client client)
{
client.outBuffer.Clear();
client.KindOfMessage = KindMessage.Unknown;
client.MessageLength = int.MaxValue;
using (client.mreBeginReceive = new ManualResetEvent(false))
{
try
{
while (!client.closed)
{
client.mreBeginReceive.Reset();
client.socket.BeginReceive(client.socketBuffer, 0, Const.BufferSize, SocketFlags.None, EndReceive, client);
client.mreInit.SetIfNotNull();
client.mreBeginReceive.WaitOne();
client.mreIsConnected.WaitOne();
}
}
catch (Exception e)
{
client.HandleError(e);
}
}
}
private static void EndReceive(IAsyncResult result)
{
var client = (Client)result.AsyncState;
if (client.closed)
{
return;
}
try
{
var receive = client.socket.EndReceive(result);
if (receive == 0)
{
client.Disconnect();
return;
}
client.ProcessNewData(receive);
}
catch (Exception e)
{
client.HandleError(e);
}
client.mreBeginReceive.SetIfNotNull();
}
internal static void ProcessNewData(this Client client, int receive)
{
lock (client.outBuffer)
{
client.outBuffer.AddRange(client.socketBuffer.Take(receive));
do
{
client.EnvelopeRead();
if (client.outBuffer.Count >= client.MessageLength)
{
var msg = client.outBuffer.GetRange(0, client.MessageLength).ToArray();
client.outBuffer.RemoveRange(0, client.MessageLength);
client.RaiseMessageReceived(msg, client.KindOfMessage);
client.KindOfMessage = KindMessage.Unknown;
client.MessageLength = client.outBuffer.Count >= Const.TotalSizeOfEnvelope ? 0 : int.MaxValue;
}
} while (client.outBuffer.Count >= client.MessageLength);
}
}
and process data as following
internal static void ProcessNewData(this Client client, int receive)
{
lock (client.outBuffer)
{
client.outBuffer.AddRange(client.socketBuffer.Take(receive));
do
{
client.EnvelopeRead();
if (client.outBuffer.Count >= client.MessageLength)
{
var msg = client.outBuffer.GetRange(0, client.MessageLength).ToArray();
client.outBuffer.RemoveRange(0, client.MessageLength);
client.RaiseMessageReceived(msg, client.KindOfMessage);
client.KindOfMessage = KindMessage.Unknown;
client.MessageLength = client.outBuffer.Count >= Const.TotalSizeOfEnvelope ? 0 : int.MaxValue;
}
} while (client.outBuffer.Count >= client.MessageLength);
}
**i change it as **
internal static void ProcessNewData(this Client client, int receive)
{
lock (client.outBuffer)
{
client.outBuffer.AddRange(client.socketBuffer.Take(receive));
List<Byte> a = new List<byte>();
a.AddRange(client.socketBuffer.Take(receive));
totmsglen2 = ((int.Parse(a.GetRange(0, 2)[0].ToString()) * 256) + int.Parse(a.GetRange(0, 2)[1].ToString()) + 2);
if (a.Count != totmsglen2)
{
// this is not valid messge discared it
a.RemoveRange(0,totmsglen2);
}
else
{// valid message process it
client.outBuffer.AddRange(a.GetRange(0,totmsglen2));
a.RemoveRange(0,totmsglen2);
}
do
{
client.EnvelopeRead();
if (client.outBuffer.Count >= client.MessageLength)
{
var msg = client.outBuffer.GetRange(0, client.MessageLength).ToArray();
client.outBuffer.RemoveRange(0, client.MessageLength);
client.RaiseMessageReceived(msg, client.KindOfMessage);
client.KindOfMessage = KindMessage.Unknown;
client.MessageLength = client.outBuffer.Count >= Const.TotalSizeOfEnvelope ? 0 : int.MaxValue;
}
} while (client.outBuffer.Count >= client.MessageLength);
}
}
The code works only for one message and not when continuous messages are received.
Cases:
0010aaaaaaaaa valid
0007asd invalid
0005iiiii valid

For example First four bit give me 45 and message length is 49 then this is true (45 body + first 4 bit length) else first four bit give 45 but message length 35 . drop this message.
The first 4 bytes are the "message length". There's no other way for a server to know when a message completes. So it's not possible to detect a mismatch.
it also not secure for examples if they send the length 500 and message is 200 long then it fail, in this situation
Your protocol will just be waiting for the completion of that message. This is not a failure; it is by design. For example, if a client sends a length of 500 and a message that is 500, but if the packets got broken up, your sever could get a length of 500 with a message that is 200... and then seconds later get the other 300 of the message. That's the way TCP/IP works.
However:
it also not secure
This is true. By following this simple approach, you allow two trivial denial-of-service attacks:
The client sends some huge message length, causing your server to allocate a huge buffer expecting some huge message.
The client only sends a partial message, causing your server to keep that buffer and socket allocated. A distributed attack can successfully consume a lot of your server resources.
To mitigate these attacks, you should do two things:
Have a reasonable maximum message size, and reject any client that tries to send a larger one.
Add an "idle timer" to each socket. Each time you receive data, reset the timer. When the timer goes off, kill the connection. Also, if the client is not sending enough data (i.e., the rate of data is too low for a period of time), then kill the connection.

A simple approach would be to make sure that the receiving logic only passes full messages back. Also you cannot assume that a full message (or even size info) is received always. So you would do something like:
Beginreceive(4 bytes length info) to start things off, then loop through EndReceive and new BeginReceives as long as the 4 bytes are not in.
Then BeginReceive(number of bytes expected) and again loop through EndReceive/BeginReceieve until you have all bytes in. That would be the moment to pass on the message to the decoding logic.

Transferring multiple commands/decisions through tcp socket

Good afternoon all!
The goal of the project:
Build a notification program with client, console, and server executable. Only selected users should get the notification.
The problem:
Sometimes the code works great, and everything works (20% of runs). The rest of the time, it will mess up the order the data is being sent in.
The code:
Server (console = TCPclient):
private void Connect()
{
string username = ReadFromConsole();
if (IsUserAllowed(username)) // Receive username
SendToConsole(bool.TrueString); // Send confirmation
else
{
SendToConsole(bool.FalseString); // Send denial
console.Close();
return;
}
string messageID = ReadFromConsole(); // Receive MessageID
string recipientCount = ReadFromConsole();
int numOfRecipients = int.Parse(recipientCount); // Receive and parse number of recipients
List<string> recipients = new List<string>();
for (int i = 0; i < numOfRecipients; i++)
{
string recipient = ReadFromConsole();
recipients.Add(recipient); // Receive recipient, add to list (required for Message)
}
string department = ReadFromConsole(); // Receive department string
string visibleTime = ReadFromConsole(); // Receive visibility timespan
string expiration = ReadFromConsole(); // Receive expiration datetime
StoreRTF(messageID); // Receive and store RTF file
console.Close(); // Connection is done, close
Message message = new Message(messageID, department, recipients, visibleTime, expiration);
}
Console (server = TCPclient):
private void SendMessage()
{
SendToServer(Environment.UserName);
if (bool.Parse(ReadFromServer()))
{
// User is allowed, continue
string messageID = DateTime.Now.ToUniversalTime().Ticks.ToString();
SendToServer(messageID); // MessageID
string recipientCount = lvRecipients.Items.Count.ToString();
SendToServer(lvRecipients.Items.Count.ToString()); // Amount of recipients
foreach (string item in lvRecipients.Items) // Loop to send each recipient
{
SendToServer(item);
}
string department = TB_Department.Text;
SendToServer(department); // Send department string
string visibleTime = TimeSpan.FromSeconds(SLIDER_VisibleTime.Value).Ticks.ToString();
SendToServer(visibleTime); // Send message visibility time
string expiration = DateTime.Now.ToUniversalTime().AddMinutes(2).ToString();
SendToServer(expiration); //TODO add UI control for this
SendRTFToServer(); // Send RTF file
MessageBox.Show(
"Your designated MessageID is: " + messageID + Environment.NewLine +
"Message upload is succesful.",
"Complete",
MessageBoxButton.OK);
}
else
{
// User is not allowed. Report to user. Disconnect (will be managed by the finally block)
MessageBox.Show("You are not allowed to upload messages to the server.", "Access denied", MessageBoxButton.OK, MessageBoxImage.Stop);
return;
}
}
Send and receive parts (same between console/server/client):
private void SendToServer(string toSend)
{
while (server.GetStream().DataAvailable)
{
// Should wait
}
StreamWriter writer = new StreamWriter(server.GetStream());
writer.WriteLine(toSend);
writer.Flush();
}
private void SendRTFToServer()
{
while (server.GetStream().DataAvailable)
{
// Should wait
}
File.Open(RTFLocation, FileMode.Open, FileAccess.Read).CopyTo(server.GetStream());
server.GetStream().Flush();
server.GetStream().Close();
}
private string ReadFromServer()
{
server.GetStream().Flush();
StreamReader reader = new StreamReader(server.GetStream());
return reader.ReadLine();
}
I have also tried different loops, implementations, switching to byte[]...
After lots of debugging I am getting nowhere. I have checked which information is leaving the console, and that all checks out and is in the right order. However, at the server end it seems to be receiving it in an entirely different order.
Anyone have an idea what is causing this?

I've found how to properly handle this. Code for the relevant parts can be found here:
C# Sockets send/receive problems and questions
Hope this may help someone in the future!

Socket programming- client server principles

I'm embarrassed to even ask this question, but after an exhausting search in google (starting to have MSDN...), I've decided to post it:
Just now started learning client-server programming (using C#), and trying to write my first code using tcpClient. I'm writing both the server side and the client side.
here's my question: Constantly, the client sends one String to the server, and then the server sends a String back to the client and so forth.
Can't the server send 2 Strings in a row? he has to wait for the client response? is this the principle of client-server??
Once again, sorry for the lousy question.
Thanks...
<>
I'll try to post some of my code (a long one...).
I tryed to cut the redandent parts, so hope the code makes any sense... (i marked the main problam with //*********)
public class MServer2
{
Dictionary<String, String> nameAndPass = new Dictionary<String, String>();
Dictionary<String, List<String> > nameAndMail = new Dictionary<String, List<String>>();
public static void Main()
{
new MServer2();
}
public MServer2()
{
TcpListener server = new TcpListener(8500);
try
{
server.Start();
Console.WriteLine("started " + server);
while (true)
{
TcpClient client = server.AcceptTcpClient();
Console.WriteLine("connection accepted " + client);
new Server1(client, nameAndPass, nameAndMail);
}
}
catch (Exception e)
{
Console.WriteLine("exception" + e);
}
finally
{
server.Stop();
}
}
class Server1
{
TcpClient client;
NetworkStream netStream;
Dictionary<String, String> nameAndPass1 = new Dictionary<String, String>();
Dictionary<String, List<String>> nameAndMail1 = new Dictionary<String, List<String>>();
internal Server1(TcpClient client, Dictionary<String, String> nameandPassFromFile, Dictionary<String, List<String> > nameAndMailsFromFile)
{
nameAndPass1 = nameandPassFromFile;
nameAndMail1 = nameAndMailsFromFile;
this.client = client;
Thread thr = new Thread(new ThreadStart(Run));
thr.Start();
}
public void Run()
{
try
{
netStream = client.GetStream();
StreamReader reader = new StreamReader(netStream);
StreamWriter writer = new StreamWriter(netStream);
writer.AutoFlush = true;
Console.WriteLine("beginning to receive loop");
writer.WriteLine("Choose your user name.");
strFromClient = reader.ReadLine();
userName = strFromClient;
writer.WriteLine("Choose your user password.");
strFromClient = reader.ReadLine();
password = strFromClient;
writer.WriteLine("Do you want to see the list of email addresses? (y/n)");
strFromClient = reader.ReadLine();
//***********************************************************************************
//HERE'S MY PROBLAM:
//HERE THE CLIENT WILL GET A STRING SHOWING HIS EMAILS, AND I WANT HIM TO GET ANOTHER STRING ASKING "Do you want to add an email address? (y/n)", BUT IT LOOKS LIKE THE SERVER "WAITS" FOR A RESPONSE FROM THE CLIENT TO SHOW THE NEXT STRING...
if (strFromClient == "y")
{
String tmpStr = null;
List<String> tmp = nameAndMail1[userName];
for(int i=0; i<nameAndMail1[userName].Count; i++)
{
tmpStr += tmp[i] + " ";
}
writer.WriteLine(tmpStr);
}
writer.WriteLine("Do you want to add an email address? (y/n)");
strFromClient = reader.ReadLine();
}
}
catch (Exception e)
{
Console.WriteLine("{0} Exception caught.", e);
}
EDIT VOL 2
OK! After 2 hours of misery, I think I found the problam thanks to Phil Frost (the genius!) --- the problam is probably in the client... (Im souch an a-hole!).
The server does send 2 string in a row, but my stupid implementation of the client side doesn't show a message (which recived from the server) that doesn't follow a message sent by the client...
So once again I need your help. Here's a view of how I designed the client form:
My lack of experience led me to connect to the server when the "connect to server" button is pressed, and only when the "send Message" button is pressed, a message from the server is desplayed. The problam is when 2 (or more) messages from the server are recived without sending a message from the client to the server- the client doesn't know that a new message is recived!
Where do I suppose to recive the messages from the server? ('where' means under which function, for example- right now it happens in the sendMessage_click function).
thanks again for all the help so far!!

This question is neither lousy nor trivial. You are touching an important point in protocol design.
AFAIK, there are 2 ways to skin this particular cat:
"Dialog" (your way) ... Request is followed by reply, is followed by next request and so on. This has the big advantage of being easy, but the big disadvantage of only one command per connection being processed at any point in time
"Async" (Both parties can send without waiting for an answer) ... This has the implementation difficulty, that you need some sort of request-ID to make it possible to know, which reply belongs to which request: Imagine the client sending two requests, the first taking longer to process than the second: The replies would be in a different order than the requests.
The choice between these variants is not allways an easy one, but the trend goes towards the async model.
Additional difficulties in the async flavour include a mechanism to make sure, only one message is sent at a time (interweaving messages will most likely result in an unparseable stream) and timeout considerations.

Yes, the server can send two strings in a row. But probably not if your server is blocked on read(). Perhaps if you included some more details about your program, a more specific answer could be provided.

In TCP the client-to-server and server-to-client channels are two independent streams. Either party can send at any time.

TCP data occasionally received in wrong order and incomplete

I have created TCP Server application in Java, and a client application in C#. When i am sending data, the client sometimes receives data out of order, and sometimes parts miss entirely. Basically, the code i use in the server (java) looks like this (stripped):
ServerSocket welcomeSocket = new ServerSocket(port);
Socket connectionSocket = welcomeSocket.accept();
outputStream = new DataOutputStream(socket.getOutputStream()); //Create stream
outputStream.writeBytes(message + "\n");
outputStream.flush();
I use "\n" as a delimiter. On the client side (C#) i use the following code:
private const char Delimiter = '\n';
tcpclnt = new TcpClient();
tcpclnt.NoDelay = true;
tcpclnt.Client.DontFragment = true;
tcpclnt.Connect(ip, port);
//This function is executed in a separate thread
public void Receive()
{
try
{
stream = tcpclnt.GetStream();
streamreader = new StreamReader(stream);
this.Connected = true;
while (Connected)
{
string line = ReadLine(streamreader);
Console.WriteLine("Received data: " + line);
}
}
}
private string ReadLine(StreamReader reader)
{
bool finished = false;
string line = "";
while (finished == false)
{
int asciiNumber = reader.Read();
char character = Convert.ToChar(asciiNumber);
if (!character.Equals(Delimiter))
line += character;
else finished = true;
}
return line;
}
The code is not very complicated. However, the data sent from the server is not always received correctly in the client. As an example, I should receive the following two strings:
"5_8_1" and "6_LEVELDATA"
What i get (sometimes) however, is this: "5_8_61" and "_LEVELDATA"
Another example: "5_4_1" and "6_LEVELDATA" result in one single string: "5_6_LEVELDATA"
This seems like some small problem, but it does in fact pretty much ruin my application. I have read a lot of posts, but the only answers i have read are either "this shouldnt happen with TCP" or "send the length of the tcp message first" which would not help in any way in this case, because the problem isn't the data being split up in multiple packages, it simply isn't arriving in the right order, which is something TCP should do.
I am 100% sure the string is always complete before it is being sent by the Java application.
I really wonder what i'm doing wrong here. Is something messed up bad in my code? I would appreciate any help with this problem. Thanks in advance.

After trying Wireshark, it appears my problem existed in the server. Apparently every TCP-message was sent in a seperate thread. Thank you for all of your comments! My problem is solved now.