We Keep Coding

How does a TCP packet arrive when using the Socket api in C#

I have been reading about TCP packet and how they can be split up any number of times during their voyage. I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message. BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
Then I read that the lower layers (ethernet?, IP?) will actually re-assemble packets transparently.
My question is, in C#, am I guaranteed to receive a full "message" over TCP? In other words, if I send 32 bytes, will I necessarily receive those 32 bytes in "one-go" (one call to ReceiveAsync())? Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix?
Also, could I receive more than one message in a single call to ReceiveAsync()? Say one "protobuf message" is 32 bytes. I send 2 of them. Could I potentially receive 48 bytes in "one go" and then 16 in another?
I know this question shows up easily on google, but I can never tell if it's in the correct context (talking about the actual TCP protocol, or how C# will expose network traffic to the programmer).
Thanks.

TCP is a stream protocol - it transmits a stream of bytes. That's all. Absolutely no message framing / grouping is implied. In fact, you should forget that Ethernet packets or IP datagrams even exist when writing code using a TCP socket.
You may find yourself with 1 byte available, or 10,000 bytes available to read. The beauty of the (synchronous) Berkeley sockets API is that you, as an application programmer don't need to worry about this. Since you're using a length-prefixed message format (good job!) simply recv() as many bytes as you're expecting. If there are more bytes available than the application requests, the kernel will keep the rest buffered until the next call. If there are fewer bytes available than required, the thread will either block or the call will indicate that fewer bytes were received. In this case, you can simply sleep again until data is available.
The problem with async APIs is that it requires the application to track a lot more state itself. Even this Microsoft example of Asynchronous Client Sockets is far more complicated than it needs to be. With async APIs, you still control the amount of data you're requesting from the kernel, but when your async callback is fired, you then need to know the next amount of data to request.
Note that the C# async/await in 4.5 make asynchronous processing easier, as you can do so in a synchronous way. Have a look at this answer where the author comments:
Socket.ReceiveAsync is a strange one. It has nothing to do with async/await features in .net4.5. It was designed as an alternative socket API that wouldn't thrash memory as hard as BeginReceive/EndReceive, and only needs to be used in the most hardcore of server apps.

TCP is a stream-based octet protocol. So, from the application's perspective, you can only read or write bytes to the stream.
I have been reading about TCP packet and how they can be split up any number of times during their voyage.
TCP packets are a network implementation detail. They're used for efficiency (it would be very inefficient to send one byte at a time). Packet fragmentation is done at the device driver / hardware level, and is never exposed to applications. An application never knows what a "packet" is or where its boundaries are.
I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message.
Yes. Because "message" is not a TCP concept. It's purely an application concept. Most application protocols do define a kind of "message" because it's easier to reason about.
Some application protocols, however, do not define the concept of a "message"; they treat the TCP stream as an actual stream, not a sequence of messages.
In order to support both kinds of application protocols, TCP/IP APIs have to be stream-based.
BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
That's good. Length prefixing is much easier to deal with than the alternatives, IMO.
My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No.
Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix? Also, could I receive more than one message in a single call to ReceiveAsync()?
Yes, and yes.
Even more fun:
You can get only part of your length prefix (assuming a multi-byte length prefix).
You can get any number of messages at once.
Your buffer can contain part of a message, or part of a message's length prefix.
The next read may not finish the current message, or even the current message's length prefix.
For more information on the details, see my TCP/IP .NET FAQ, particularly the sections on message framing and some example code for length-prefixed messages.
I strongly recommend using only asynchronous APIs in production; the synchronous alternative of having two threads per connection negatively impacts scalability.
Oh, and I also always recommend using SignalR if possible. Raw TCP/IP socket programming is always complex.

My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No. You will not receive a full message. A single send does not result in a single receive. You must keep reading on the receiving side until you have received everything you need.
See the example here, it keeps the read data in a buffer and keeps checking to see if there is more data to be read:
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(state.buffer, 0, bytesRead));
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
// All the data has arrived; put it in response.
if (state.sb.Length > 1)
{
response = state.sb.ToString();
}
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
See this MSDN article and this article for more details. The 2nd link goes into more details and it also has sample code.

TCP messages arrival on the same socket

I've got a 2 services that communicate using a TCP socket (the one that initiates the connection is a C++ Windows service and the receiver is a C# TCP stream) and (let's say that they might not use the same TCP connection all the time). Some of the time I'm getting half messages (where the number of bytes is miscalculated somehow) on a great network load.
I have several questions in order to resolve the issue:
Can I be sure that messages (not packets...) must follow one another, for example, if I send message 1 (that was received as half), then message 2, can I be sure that the next half of message 1 won't be after message 2?
Please have separate answer for the same TCP connection between all messages and not having the same TCP connection between them.
Is there any difference whether the sender and the receiver are on the same station?

TCP is a stream protocol that delivers a stream of bytes. It does not know (or care) about your message boundaries.
To send the data, TCP breaks the stream up into arbitrarily sized packets. [it's not really arbitrary and it is really complicated.] and send these reliably to the other end.
When you read data from a TCP socket, you get whatever data 1) has arrived, and 2) will fit in the buffer you have provided.
On the receive side you need to use code to reassemble complete messages from the TCP stream. You may have to write this yourself or you may find that it is already written for you by whatever library (if any) you are using to interact with the socket.

TCP is a streaming protocol, it doesn't have "messages" or "packets" or other boundaries. Sometimes when you receive data you might not get all that was sent, other times you could get more than one "message" in the received stream.
You have to model your protocol to handle these things, for example by including special message terminators or including a fixed-sized header including the data size.
If you don't get all of a message at once, then you have to receive again, maybe even multiple times, to get all the data that was sent.
And to answer your question, you can be sure that the stream is received in the order it was sent, or the TCP layer will give you an error. If byte A was sent before byte B then it's guaranteed that they will arrive in that order. There is also no theoretical difference if the sender and receiver is on the same system, or on different continents.

C# Packet sending using TCP streams

I'm trying to send multiple packets from a tcp server to a client using only read and write from the base "Stream" class. The problem is, I don't know what the size of the packet will be (client side). Should I send and int (4 bytes) before each packet so the client can ajust the buffer and receive only the nuber of bytes specified? I feel like there's already something in the tcp protocol that handles that but I can't seem to find it.

The process for what you are wanting to do is called Message Framing and there is no built in mecinism in TcpClient that does this for you. It is the responsibility of the higher level application layer to do it, be it your own code appending the length on the prefix of the message or some other library that handles message framing for you like WCF.
Here is a full example showing a length prefixed implementation like you suggested in your question.

Socket Data Handling -for data packing (TCP)

I'm working on a socket project. I want to send ~4kb data with TCP. I need to ask a question in here. That is: If I send ~4kb data with TCP, may my data handler function receives this data in one more time? Namely can RECEIVE(server-side) function return 2 times (If I send ~4kb data once)? If I can understand this, I'll start to make my own packet handling system.

In short: yes.
TCP transfers a stream of bytes, there is no mention of messages (and their boundaries) in a TCP connection. Even if you send two bytes they can be received in two separate chunks (theoretically of course; it's quite unlikely in practice). So you should put some marks into your data stream. I'd suggest writing a "message header" before every message where the header contains at least length of the following message's body. And take care, the header itself may also be fragmented when receiving.

TCP is a stream protocol. You receive a stream of bytes in the order you sent them. To reconstruct the message you should either prefix every message with the length of the message or use a delimiter to mark end of the message.
I don't know your business requirements but you could use a binary protocol like MQTT or a text based protocol like STOMP.

If a datagam is received by a udp client can we trust the data integrity?

Since UDP is a connection less protocol so we know there is no guarantee that data will be received by a receiver. But if the datagram is received, is it possible that data is partly /fully corrupt ?
UDP header also contains CRC, so this makes me feel that in case datagram is received the data would be reliable ? Is that correct or not ?
To more elaborate the problem I send the data from Java udp server like
// Sending in Java
InetAddress group = InetAddress.getByName("230.0.0.1");
//buf is a String.getBytes();
DatagramPacket packet = new DatagramPacket(buf, buf.length, group, 4446);
//socket is DatagramSocket in Java
socket.send(packet);
and get data in C# client like this
// client is UdpClient of C#
Byte[] data = client.Receive(ref localEp);
strData = Encoding.ASCII.GetString(data);
I am not manipulating CRC myself , at the receiver end I get the same data that was sent in buf, SO how do I check the CRC here, given I have an array of byte data.. ? or assume it would be correct (I can live with those extreme cases where CRC would match ie CRC and data are magically corrupt)

Assuming functioning hardware, OS and IP stack, single-packet integrity is assured by IP checksums.
I mention this assumptions, as I have seen many cases of IP checksumming being turned off (in the sense of allways returning "OK") for performance reasons, or as an implementation defect: Some NICs can do checksums in hardware, but bad or wrong driver parameters can ruin your day.

You can trust that the datagram is intact if its checksum (not technically a CRC) is correct, but that doesn't mean you can trust the data. There's nothing magic about the checksum formula, so a process that altered the data could merely recalculate the checksum and you'd never know.

By default OS rejects packets with invalid checksum. This mean you will receive less bad packets than proper ones. But you can configure socket to allow partial/corrupted packets to be propagated to userspace.