We Keep Coding

Check if NamedPipeClientStream write is successful

Basically the title... I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value. This is the starting code:
static void Main(string[] args){
Console.WriteLine("Client running!");
NamedPipeClientStream npc = new NamedPipeClientStream("somename");
npc.Connect();
// npc.WriteTimeout = 1000; does not work, says it is not supported for this stream
byte[] message = Encoding.UTF8.GetBytes("Message");
npc.Write(message);
int response = npc.ReadByte();
Console.WriteLine("response; "+response);
}
I've implemented a small echo message from the NamedPipeServerStream on every read. I imagine I could add some async timeout to check if npc.ReadByte(); did return a value in lets say 200ms. Similar to how TCP packets are ACKed.
Is there a better way of inspecting if namedPipeClientStream.Write() was successful?

I'd like to have same feedback on weather NamedPipeServerStream object successfully received a value
The only way to know for sure that the data you sent was received and successfully processed by the client at the remote endpoint, is for your own application protocol to include such acknowledgements.
As a general rule, you can assume that if your send operations are completing successfully, the connection remains viable and the remote endpoint is getting the data. If something happens to the connection, you'll eventually get an error while sending data.
However, this assumption only goes so far. Network I/O is buffered, usually at several levels. Any of your send operations almost certainly involve doing nothing more than placing the data in a local buffer for the network layer. The method call for the operation will return as soon as the data has been buffered, without regard for whether the remote endpoint has received it (and in fact, almost never will have by the time your call returns).
So if and when such a call throws an exception or otherwise reports an error, it's entirely possible that some of the previously sent data has also been lost in transit.
How best to address this possibility depends on what you're trying to do. But in general, you should not worry about it at all. It will typically not matter if a specific transmission has been received. As long as you can continue transmitting without error, the connection is fine, and asking for acknowledgement is just unnecessary overhead.
If you want to handle the case where an error occurs, invalidating the connection, forcing you to retry, and you want to make the broader operation resumable (e.g. you're streaming some data to the remote endpoint and want to ensure all of the data has been received, without having to resend data that has already been received), then you should build into your application protocol the ability to resume, where on reconnecting the remote endpoint reports the number of bytes it's received so far, or the most recent message ID, or whatever it is your application protocol would need to understand where it needs to start sending again.
See also this very closely-related question (arguably maybe even an actual duplicate…though it doesn't mention named pipes specifically, pretty much all network I/O will involve similar issues):
Does TcpClient write method guarantees the data are delivered to server?
There's a good answer there, as well as links to even more useful Q&A in that answer.

How does a TCP packet arrive when using the Socket api in C#

I have been reading about TCP packet and how they can be split up any number of times during their voyage. I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message. BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
Then I read that the lower layers (ethernet?, IP?) will actually re-assemble packets transparently.
My question is, in C#, am I guaranteed to receive a full "message" over TCP? In other words, if I send 32 bytes, will I necessarily receive those 32 bytes in "one-go" (one call to ReceiveAsync())? Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix?
Also, could I receive more than one message in a single call to ReceiveAsync()? Say one "protobuf message" is 32 bytes. I send 2 of them. Could I potentially receive 48 bytes in "one go" and then 16 in another?
I know this question shows up easily on google, but I can never tell if it's in the correct context (talking about the actual TCP protocol, or how C# will expose network traffic to the programmer).
Thanks.

TCP is a stream protocol - it transmits a stream of bytes. That's all. Absolutely no message framing / grouping is implied. In fact, you should forget that Ethernet packets or IP datagrams even exist when writing code using a TCP socket.
You may find yourself with 1 byte available, or 10,000 bytes available to read. The beauty of the (synchronous) Berkeley sockets API is that you, as an application programmer don't need to worry about this. Since you're using a length-prefixed message format (good job!) simply recv() as many bytes as you're expecting. If there are more bytes available than the application requests, the kernel will keep the rest buffered until the next call. If there are fewer bytes available than required, the thread will either block or the call will indicate that fewer bytes were received. In this case, you can simply sleep again until data is available.
The problem with async APIs is that it requires the application to track a lot more state itself. Even this Microsoft example of Asynchronous Client Sockets is far more complicated than it needs to be. With async APIs, you still control the amount of data you're requesting from the kernel, but when your async callback is fired, you then need to know the next amount of data to request.
Note that the C# async/await in 4.5 make asynchronous processing easier, as you can do so in a synchronous way. Have a look at this answer where the author comments:
Socket.ReceiveAsync is a strange one. It has nothing to do with async/await features in .net4.5. It was designed as an alternative socket API that wouldn't thrash memory as hard as BeginReceive/EndReceive, and only needs to be used in the most hardcore of server apps.

TCP is a stream-based octet protocol. So, from the application's perspective, you can only read or write bytes to the stream.
I have been reading about TCP packet and how they can be split up any number of times during their voyage.
TCP packets are a network implementation detail. They're used for efficiency (it would be very inefficient to send one byte at a time). Packet fragmentation is done at the device driver / hardware level, and is never exposed to applications. An application never knows what a "packet" is or where its boundaries are.
I took this to assume I would have to implement some kind of buffer on top of the buffer used for the actual network traffic in order to store each ReceiveAsync() until enough data is available to parse a message.
Yes. Because "message" is not a TCP concept. It's purely an application concept. Most application protocols do define a kind of "message" because it's easier to reason about.
Some application protocols, however, do not define the concept of a "message"; they treat the TCP stream as an actual stream, not a sequence of messages.
In order to support both kinds of application protocols, TCP/IP APIs have to be stream-based.
BTW, I am sending length-prefixed, protobuf-serialized messages over TCP.
That's good. Length prefixing is much easier to deal with than the alternatives, IMO.
My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No.
Or do I have to "store" each receive until the number of bytes received is equal to the length-prefix? Also, could I receive more than one message in a single call to ReceiveAsync()?
Yes, and yes.
Even more fun:
You can get only part of your length prefix (assuming a multi-byte length prefix).
You can get any number of messages at once.
Your buffer can contain part of a message, or part of a message's length prefix.
The next read may not finish the current message, or even the current message's length prefix.
For more information on the details, see my TCP/IP .NET FAQ, particularly the sections on message framing and some example code for length-prefixed messages.
I strongly recommend using only asynchronous APIs in production; the synchronous alternative of having two threads per connection negatively impacts scalability.
Oh, and I also always recommend using SignalR if possible. Raw TCP/IP socket programming is always complex.

My question is, in C#, am I guaranteed to receive a full "message" over TCP?
No. You will not receive a full message. A single send does not result in a single receive. You must keep reading on the receiving side until you have received everything you need.
See the example here, it keeps the read data in a buffer and keeps checking to see if there is more data to be read:
private static void ReceiveCallback(IAsyncResult ar)
{
try
{
// Retrieve the state object and the client socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket client = state.workSocket;
// Read data from the remote device.
int bytesRead = client.EndReceive(ar);
if (bytesRead > 0)
{
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(state.buffer, 0, bytesRead));
// Get the rest of the data.
client.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReceiveCallback), state);
}
else
{
// All the data has arrived; put it in response.
if (state.sb.Length > 1)
{
response = state.sb.ToString();
}
// Signal that all bytes have been received.
receiveDone.Set();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
See this MSDN article and this article for more details. The 2nd link goes into more details and it also has sample code.

TCP messages arrival on the same socket

I've got a 2 services that communicate using a TCP socket (the one that initiates the connection is a C++ Windows service and the receiver is a C# TCP stream) and (let's say that they might not use the same TCP connection all the time). Some of the time I'm getting half messages (where the number of bytes is miscalculated somehow) on a great network load.
I have several questions in order to resolve the issue:
Can I be sure that messages (not packets...) must follow one another, for example, if I send message 1 (that was received as half), then message 2, can I be sure that the next half of message 1 won't be after message 2?
Please have separate answer for the same TCP connection between all messages and not having the same TCP connection between them.
Is there any difference whether the sender and the receiver are on the same station?

TCP is a stream protocol that delivers a stream of bytes. It does not know (or care) about your message boundaries.
To send the data, TCP breaks the stream up into arbitrarily sized packets. [it's not really arbitrary and it is really complicated.] and send these reliably to the other end.
When you read data from a TCP socket, you get whatever data 1) has arrived, and 2) will fit in the buffer you have provided.
On the receive side you need to use code to reassemble complete messages from the TCP stream. You may have to write this yourself or you may find that it is already written for you by whatever library (if any) you are using to interact with the socket.

TCP is a streaming protocol, it doesn't have "messages" or "packets" or other boundaries. Sometimes when you receive data you might not get all that was sent, other times you could get more than one "message" in the received stream.
You have to model your protocol to handle these things, for example by including special message terminators or including a fixed-sized header including the data size.
If you don't get all of a message at once, then you have to receive again, maybe even multiple times, to get all the data that was sent.
And to answer your question, you can be sure that the stream is received in the order it was sent, or the TCP layer will give you an error. If byte A was sent before byte B then it's guaranteed that they will arrive in that order. There is also no theoretical difference if the sender and receiver is on the same system, or on different continents.

What mechanism is used by MSYS/Cygwin to emulate Unix domain sockets?

I'm attempting to write (in C#) a piece of software that communicates with another piece of software, built with MSYS, over (MSYS emulated) Unix domain sockets. I've learned that the "socket server" (I'm not clear on what the proper terminology is) creates a temporary file with contents such as this:
!<socket >59108 282F93E1-9E2D051A-46B57EFC-64A1852F
The 59108 corresponds to a TCP port, which the "socket server" is listening on on the loopback interface. Using a packet capture tool, I've been able to determine that the "socket client" connects to this port, and information is exchanged over the loopback interface.
I replicated this behavior in my software, and the "socket client" connects to my listening port, but no information is transmitted. I believe there's another step here, one most likely involving the GUID in the "socket" file, but I've been unable to determine what it is. What do I need to do to trigger the communication from the client?
It seems that MSYS is using Cygwin's mechanism, which involves a named event, that is (probably?) created by the "server", and signaled (apparently) by the "server", but my naive attempt at an implementation doesn't seem to be working.
I've located an email written by Conrad Scott which describes various shortcomings in the "handshaking" process, and proposes a patch which allegedly solves them. In this email, Conrad describes somewhat the process used, and he indicates that there are actually TWO events, one managed by the "server" and one managed by the "client". I've used API Monitor to look for calls to CreateEvent(), and while there are several, I cannot find one that looks like the "smoking gun" here. There are no interesting calls to CreateSemaphore() either, so it seems like Conrad's patch was never applied (or, at least, it was applied some time AFTER MSYS forked Cygwin).

It appears that both the answers from divB and Mark are correct, but they both leave out some details, so this is hopefully a bit more complete.
There are 2 different implementations here. I have not done an exhaustive investigation of who implements which implementation, but as of this writing, the current version of cygwin uses the implementation described by divB and MsysGit uses the implementation described by Mark.
Initializing the server:
Create a socket (AddressFamily = IPv4, Type = Stream, Protocol = TCP). (.NET/MFC)
Bind it to loopback (127.0.0.1). (.NET/MFC)
Tell the socket to listen. (.NET/MFC)
Generate a random 16-byte GUID.
Create a file with the following contents based on the TCP port and the GUID. Using the original example where 59108 is the TCP port and 282F93E1-9E2D051A-46B57EFC-64A1852F is the GUID.
In the cygwin implementation, the socket file contents are:
!<socket >59108 s 282F93E1-9E2D051A-46B57EFC-64A1852F
And in the msysgit implementation, the socket file contents are:
!<socket >59108 282F93E1-9E2D051A-46B57EFC-64A1852F
The difference being the extra "s" between the port and the GUID.
Set the System attribute on this file.
In msysgit implementation only, Create a named wait handle with the name cygwin.local_socket.secret.58598.282F93E1-9E2D051A-46B57EFC-64A1852F (InitalState = False, Reset = AutoReset). (.NET/MFC)
58598 is derived by using a HostToNetworkOrder function on the port (as 16-bit unsigned integer). i.e. 59108 == 0xE6E4 and 58598 == 0xE4E6.
Handling connections:
Accept the incoming socket. (.NET/MFC).
In the case of the cygwin implementation only, do handshake that consists of:
Read 16 bytes. If these do not match the GUID, then fail.
Send the same 16 bytes.
Read 12 bytes as 3 32-bit integers. They are the pid, uid and gid of the calling process.
Send 12 bytes (3 32-bit integers) back. Use the pid of the server and the uid and gid that were received.
In the case of the msysgit implementation only, synchronize with the client by:
Get the port of the incoming socket. For this example, we'll say it is 63524.
Open existing wait handle for the client. (.NET/MFC). You need to convert the port to network byte order just like we did for the server. So, for this example, the name is cygwin.local_socket.secret.9464.282F93E1-9E2D051A-46B57EFC-64A1852F
Signal the server and wait for the client (ToSignal = server, WaitOn = client, Timeout = 10000 msec, ExitContext/Alertable = False). (.NET/MFC). Not 100% sure about the ExitContext/Alertable parameter, but False seems to work.
Hand off the socket to the (hopefully already existing) code for whatever it is you are doing (which in the case of all three of us, seems to be an ssh agent).

So at least for cygwin I can answer your question now: I just implemented a cygwin compatible socket server using MFC. I did it by looking into cygwin source.
It seems that there are not even events. So the patch you mentioned does not seem to have been implemented.
All that happens is:
1.) The socket file is created, the GUID ("shared key") are just random numbers.
2.) The file MUST have "system" attribute. The cygwin code does some weird permission stuff if it's on NTFS, haven't looked into that.
3.) a network socket on localhost is created with the port indicated in the socket file.
So, then, when a client connects to the socket (via TCP/IP):
4.) It first sends the 4 random numbers to the server; the server checks if they are valid
5.) The server sends them back
6.) The client sends 3 32 bit numbers: The pid, the uid and gid
7.) The server sends back his own version of these numbers.
I don't understand what's the purpose of this handshake because from a security point of view it's completely worthless.

I've worked out something that functions correctly for the build of OpenSSH (ssh-agent.exe) that comes with Git:
Setup on the server side consists of these steps:
1. Create a "secret string" that consists of four groups of eight hex digits separated by a dash ("-")
2. Listen on a local port
3. Create an EventWaitHandle with mode EventResetMode.AutoReset named cygwin.local_socket.secret.[secret string].[listen port number here with byte order reversed]
4. Write out the "socket" file, which consists of the string ![port number here, byte order NOT reversed] [secret string]
When a connection comes in, the following steps must be undertaken:
1. Open the client's event handle with EventWaitHandle.OpenExisting(), using the event name cygwin.local_socket.secret.[remote port number with byte order reversed].[secret string]
2. Signal the server's event handle and wait for the client's wait handle to be signaled with `EventWaitHandle.SignalAndWait()
I agree that it looks like the patch discussed on the mailing list was never applied. The sequence I worked out seems closer to the sequence discussed on that list, and as well, it matches the code I dug up from Cygwin.
I don't understand the disparity between what I found to work vs what divB found to work, but I did confirm that it functioned with the software I was using (Git's OpenSSH)

UDP Client receives only 1 message

I have a server client application that I am currently working on. The server is receiving data fine over a WAN and the client seems to receive the data, but the client is only receiving one communication. Is there anything over a WAN that would make a client always only receive the first return UDP communication and none of the subsequent. Thanks for the help.
Client UDP Listening code
private void receiveUDP()
{
System.Net.IPEndPoint test = new System.Net.IPEndPoint(System.Net.IPAddress.Any,UDP_PORT_NUMBER);
System.Net.EndPoint serverIP = (System.Net.EndPoint)test;
server.Bind(serverIP);
//server.Ttl = 50;
EndPoint RemoteServ = (EndPoint)listenUDP;
do
{
byte[] content = new byte[1024];
int data = server.ReceiveFrom(content, ref RemoteServ);
string message = Encoding.ASCII.GetString(content);
ProcessCommands(message);
} while (true);
}

This is a bit of a stab in the dark (since you don't provide enough code to really say what's going on definitively), but there's one major reason why you might consistently see some UDP datagrams not be delivered over a WAN, while others always arrive successfully. This reason is MTU; the Maximum Transmission Unit which can be sent in a single UDP datagram. This can easily produce behaviour such as what you're seeing if (for example), your first datagram is a short "I accept your connection" message, and you then follow that with a datagrams containing large files; the first (small) datagram is smaller than the MTU and is delivered, while the following (large) datagrams are larger than the MTU, and are discarded en route.
For UDP over a WAN, the MTU will not be higher than about 1500 bytes, and in many situations may be as low as 1200 bytes. Any packets larger than that will be silently dropped somewhere between endpoints. To send large blocks of data via UDP, you need to chop them up into pieces smaller than the MTU for the network segment across which you're transmitting them.
On a LAN, you can usually get away with sending datagrams of any size. But as soon as they're being sent over the Internet or otherwise through heterogenous networks, they're likely to be silently discarded.
If you do need to send large files, you might choose to transmit them via TCP instead; TCP automatically manages chopping data up to fit within the MTU, and ensures that its packets are all received, and are received in order; guarantees that you will not receive from datagrams sent via UDP.
As I mentioned above, this is a complete stab in the dark and may not actually be related to your actual troubles. But it's the elephant in the room, when all we have to go on is that the first packet always arrives successfully, and later packets never do.