We Keep Coding

How to deal with network data packets in a client/server application code?

I'm creating a client/server application using C# and I would like to know the best approach to deal with data packets in C# code.
From what I read I could A) use the Marshal class to construct a safe class from a data packet raw buffer into or B) use raw pointers in unsafe context to directly cast the data packet buffer into a struct and use it on my code.
I can see big problems in both approaches. For instance using Marshal seems very cumbersome and hard to maintain (compared to the pointer solution) and on the other hand the use of pointers in C# is very limited (e.g. I can't even declare a fixed size array of a non-primitive type which is a deal breaker in such application).
That said I would like to know which of these two approaches is better to deal with network data packets and if there is any other better approach to this. I'm open to any other possible solution.
PS.: I'm actually creating a custom client to an already existing client/server application so the communication protocol between client and server is already created and can not be modified. So my C# client need to adhere to this already existing protocol to its lower level details (eg. binary offsets of each information in the data packets are fixed and need to be respected).

the best approach to deal with network packets. you have to create the Payload attached to the header of actual packet and you receiving client always first read the header and convert it to the actual length. then set the buffer length that you received in the header. it will perfectly work without any loss of packets. it will also improve the memory leakage issue. you don't need to create the hard coded array to get the buffer bytes. just append the actual byte length to the header of packet. it will dynamically set the buffer bytes.
like.
public void SendPackettoNetwork(AgentTransportBO ObjCommscollection, Socket Socket)
{
try
{
byte[] ActualBufferMessage = PeersSerialization(ObjCommscollection);
byte[] ActualBufferMessagepayload = BitConverter.GetBytes(ActualBufferMessage.Length);
byte[] actualbuffer = new byte[ActualBufferMessage.Length + 4];
Buffer.BlockCopy(ActualBufferMessagepayload, 0, actualbuffer, 0, ActualBufferMessagepayload.Length);
Buffer.BlockCopy(ActualBufferMessage, 0, actualbuffer, ActualBufferMessagepayload.Length, ActualBufferMessage.Length);
Logger.WriteLog("Byte to Send :" + actualbuffer.Length, LogLevel.GENERALLOG);
Socket.Send(actualbuffer);
}
catch (Exception ex)
{
Logger.WriteException(ex);
}
}
Just pass your transport class object. use you serialization approach here i used binary formatter serialization of class object

Reading Protobuf TCP Packets with existing C# classes

This problem seems simple and do-able enough, but I cannot for the life of me get it to work.
I have:
A PCAP file with a few packets I know are some type of ProtoBuf data (probably created with protobuf-csharp-port)
All the possible C# classes from an assembly decorated with:
[DebuggerNonUserCode, CompilerGenerated, GeneratedCode("ProtoGen", "2.4.1.473")]
public sealed class thing : GeneratedMessageLite<thing, thing.Builder>
All I want to do is parse those packets using what I know from the assembly file. Simple? Probably, but no matter what I do, nothing actually is getting parsed.
Here's an example of one of the many possible classes:
[DebuggerNonUserCode, CompilerGenerated, GeneratedCode("ProtoGen", "2.4.1.473")]
public sealed class Thing: GeneratedMessageLite<Thing, Thing.Builder>
{
// Fields
private static readonly string[] _thingFieldNames = new string[] { "list" };
private static readonly uint[] _thingFieldTags = new uint[] { 10 };
...
public static Builder CreateBuilder()
{
return new Builder();
}
...
public static thing ParseFrom(ByteString data)
{
return CreateBuilder().MergeFrom(data).BuildParsed();
}
...
public override void WriteTo(ICodedOutputStream output)
{
int serializedSize = this.SerializedSize;
string[] strArray = _thingFieldNames;
if (this.list_.Count > 0)
{
output.WriteMessageArray<thingData>(1, strArray[0], this.list_);
}
}
...
[DebuggerNonUserCode, GeneratedCode("ProtoGen", "2.4.1.473"), CompilerGenerated]
public static class Types
{
// Nested Types
[CompilerGenerated, GeneratedCode("ProtoGen", "2.4.1.473")]
public enum PacketID
{
ID = 19
}
}
}
There are many others like that. I've tried doing something like this with each packet (using protobuf-csharp-port):
Console.WriteLine(Thing.ParseFrom(packet.Buffer).ToString());
I'm expecting to see the actual text data. But I either get nothing, an error about invalid packet tags, or an error about it being a "0".
I've also tried using protobuf-net, but it just gives me random errors about incompatibility, unexpected types, etc.:
Console.WriteLine(ProtoBuf.Serializer.Deserialize<Thing>(ms));
What on Earth am I doing wrong here? Is there a better way to, using all the known types in an assembly, simply decode the Protobuf message and see what's inside? Ideally without having to know beforehand what type of message it is?
Thank you so much if you can figure this out!

Guessing from the failed attempts outlined in the question, I believe that you have some misconceptions about the content of your pcap file.
This line in particular
Console.WriteLine(Thing.ParseFrom(packet.Buffer).ToString());
makes me think that you are working under the wrong assumption that a single pcap packet contains the serialized bytes of one single object. Unfortunately, this is not the case.
As you might know, TCP/IP networks use a layered protocol stack, where each layer adds functionality and isolates upper layer protocols from the details of lower layer protocols (and vice versa). This is done by encapsulating the data sent from the upper layers down to the network and de-encapsulating the data as it travels up the stack on the receiving side.
Now, your pcap file contains the raw data as seen by your network interface, i.e. the serialized payload plus all the data added by the application, transport, internet, and link layer.
Now, if you want to de-serialize the objects contained in your dump, you will need to write some code that removes all the headers of the link layer and internet protocols, (un-)does the work of the transport protocol and reassembles the stream of bytes that was sent over the network.*
Next, you will need to analyze the resulting byte dump and make some sophisticated guesses about the design of the application level protocol. Does it implement a handshake when it starts communicating? Does it send a checksum together with the actual payload? Was the data compressed before it was sent over the network? Does the application encrypt the data prior to sending it? If TCP was used as the transport protocol, how is message framing implemented etc. Of course, if you have access to the source code of the application that generated the data (or at least the application binaries), then you can just read the code (or reverse engineer the binaries) to figure this part out.
Once you are at this point you are in a position to interpret the raw data. All that is left is to write some code that extracts the relevant bytes, feeds it to the protocol-buffer deserializer and voilà, you have your objects back!
(* And there are other minor issues like fragmented IP packets, TCP segments that arrived out of order, and TCP retransmissions, of course.)
To sum it all up:
It is theoretically possible to write a tool that de-serializes objects that were serialized using protocol-buffers from a pcap dump, provided that the dump contains the full communication between both peers, i.e. packets were not truncated by the tool that generated the dump etc.
In practice however, there are multiple hurdles to overcome that are anything but trivial even for a seasoned practitioner of the art, as such a tool must:
be able to deal with all the complex issues of the lower level protocols of TCP/IP to rebuild the data flow between the peers.
be able to understand the application level protocol that was used to transmit the serialized objects.
Note that point 1 above alone results in the requirement to implement the functionality of a TCP/IP stack at least in part. The easiest way to accomplish this would probably consist in reusing code of an open source TCP/IP implementation such as the one found in the Linux or *BSD kernel. Many tools that do similar things, like reconstructing HTTP traffic from capture files, do exactly this. (See e.g. Justsniffer.)

How to wait for right moment to write?

I'm trying to send my rs232 device multiple SerialPort.Write commands right after each other. However, I don't think it can handle multiple WRITE commands at once.
Currently, I'm just using Thread.Sleep(500) to delay between WRITEs, but is there a way to detect when the perfect time to send is? Or with buffers?
example code
Interface
private void btn_infuse_Click(object sender, RoutedEventArgs e) {
cmd.SetTargetInfusionVolume(spmanager, double.Parse(tbox_targetvolume.Text));
cmd.StartInfuse(spmanager);//cmd object of Command class
}
Command Class
public void StartInfuse(SPManager spm){
spm.Write("RUN");//spm object of serialportmanager class
}
public void SetTargetInfusionVolume(SerialPortManager spm, double num) {
spm.Write("MLT " + num.ToString());
}
SerialPortManager class
public void Write(string sData) {
if (oSerialPort.IsOpen) {//oSerialPort object of SerialPort class
try {
oSerialPort.Write(sData + "\r");
}
catch { MessageBox.Show("error"); }
}
}

If your serial port settings (especially, as Hans Passsant mentioned, flow control) are correct, then the problem with speed is most likely that your device can't handle messages fast enough to keep up with you if you send them too fast, or that it expects "silent" gaps between messages in order to delineate them.
In this case, a Sleep() to introduce a transmission delay is a very reasonable approach. You will need to find a sensible delay that guarantees the device handles your messages successfully, ideally without stalling your application for too long.
All too often this involves trial and error, but consult the documentation for the device, as quite a few devices use gaps in transmission to indicate the end of a message packet (e.g. often they may expect you to be silent for a short time after a message, e.g. if they specified 10 bits worth of "silent time" on a 2400 bps link, this would correspond to 10/2400ths or just over 4 milliseconds). This can all be compromised a bit by windows, though, as it tends to buffer data (i.e. it will hang on to it for a few milliseconds to see if you are going to ask it to transmit anything more) - you may therefore need a significantly longer delay than should strictly be required to get this to work - maybe 15-20ms. And of course, I could be barking up the wrong tree and you may find you need something as large as 500ms to get it to work.

Google Protocol Buffers Serialization hangs writing 1GB+ data

I am serializing a large data set using protocol buffer serialization. When my data set contains 400000 custom objects of combined size around 1 GB, serialization returns in 3~4 seconds. But when my data set contains 450000 objects of combined size around 1.2 GB, serialization call never returns and CPU is constantly consumed.
I am using .NET port of Protocol Buffers.

Looking at the new comments, this appears to be (as the OP notes) MemoryStream capacity limited. A slight annoyance in the protobuf spec is that since sub-message lengths are variable and must prefix the sub-message, it is often necessary to buffer portions until the length is known. This is fine for most reasonable graphs, but if there is an exceptionally large graph (except for the "root object has millions of direct children" scenario, which doesn't suffer) it can end up doing quite a bit in-memory.
If you aren't tied to a particular layout (perhaps due to .proto interop with an existing client), then a simple fix is as follows: on child (sub-object) properties (including lists / arrays of sub-objects), tell it to use "group" serialization. This is not the default layout, but it says "instead of using a length-prefix, use a start/end pair of tokens". The downside of this is that if your deserialization code doesn't know about a particular object, it takes longer to skip the field, as it can't just say "seek forwards 231413 bytes" - it instead has to walk the tokens to know when the object is finished. In most cases this isn't an issue at all, since your deserialization code fully expects that data.
To do this:
[ProtoMember(1, DataFormat = DataFormat.Group)]
public SomeType SomeChild { get; set; }
....
[ProtoMember(4, DataFormat = DataFormat.Group)]
public List<SomeOtherType> SomeChildren { get { return someChildren; } }
The deserialization in protobuf-net is very forgiving (by default there is an optional strict mode), and it will happily deserialize groups in place of length-prefix, and length-prefix in place of groups (meaning: any data you have already stored somewhere should work fine).

1.2G of memory is dangerously close to the managed memory limit for 32 bit .Net processes. My guess is the serialization triggers an OutOfMemoryException and all hell breaks loose.
You should try to use several smaller serializations rather than a gigantic one, or move to a 64bit process.
Cheers,
Florian

Mapping structs to memory in c#, is it worth it? Or is there a better way

I'm sending some packets of data across the network and they arrive in byte[]s, lets say the structure is
[int, int, byte, int]
If this was c++ I would declare a struct* and point to the byte[]. I'm doing this project in c# and I'm not sure whether it is worth it with marshalling overhead, or if there is a better way to handle it in c#, I'm all ears.
update, for clarity
Basically, what he is doing
Marshaling a Byte array to a C# structure
Except I'm wondering if it is worth it.

I think marshaling is the best option. You could parse the byte array by yourself using BitConverter, but that would require more work on your part and is not as flexible.

The only real reason to do it that way would be to squeeze every last bit of performance out of the system. In my opinion, you're better off writing it using BitConverter to make sure it's working. Then, if getting the data is a performance bottleneck, consider doing the marshaling.
For example, given a struct:
struct MyStruct
{
private int f1;
private int f2;
private byte f3;
private int f4;
public MyStruct(int i1, int i2, byte b1, int i4)
{
f1 = i1;
f2 = i2;
f3 = b1;
f4 = i4;
}
// assume there are public get accessors
}
Then you can get create a new one from the buffer with:
var s = new MyStruct(BitConverter.ToInt32(buff, 0),
BitConverter.ToInt32(buff, 4),
BitConverter.ToUInt8(buff, 8),
BitConverter.ToInt32(buff, 9));
That's a whole lot easier to write and verify than the marshaling, and probably will be fast enough for your needs.

Well, I guess everyone has their own 'favourite' way. When receiving protocol units over a byte stream in any OO language, I usually fire every received byte into a 'ProtocolUnit' class instance by calling its 'bool addByte() method. A state-machine in the class handles the bytes and error/sanity checks the assembled fields. If a ProtocolUnit has been received in its entirety, the addByte() method function returns true to indicate to the caller that a PDU has been correctly assembled. Usually, the instance is then queued off to whatever is going to handle it and a new ProtocolUnit created, (or depooled), so it can start to assemble the next PDU.
It's implicit that the start of a message can be identified so that, in case of an error, the state-machine can either reset itself, so dumping the erroneous data, or by returning 'true' to the addByte() call, setting a suitable errorMessage that the caller can check to decide what to do, (eg. if the errorMess property is "" then queue to handler else queue to error logger).
I'm sure that you consider this a massive overkill, but it works for me :)
Rgds,
Martin
PS try to avoid protocols where the length is transmitted at the start and is the only way to identify message start/end. This is very fragile and prone to explosions, especially with non-secure transports like UDP. Even with TCP, I have known a x****x router that would occasionally add a null to packets...

As an alternative to BitConverter, wrap every byte[] in a MemoryStream, and extract the fields using a BinaryReader. Similar, but the stream maintains the offsets for you.