I have the Problem, that I want to send a large string through a SocketConnection, but I can't receive the string at once because the Network is limited to 1500 bytes, so how can I Receive all bytes over the Socket without using a TCPClient, or is that just impossible.
public string ReceiveString(Socket connection)
{
byte[] buffer = new byte[1048576];
int recBytes = connection.Receive(buffer);
return UnicodeEncoding.Unicode.GetString(buffer, 0, recBytes);
}
Thats, what i have so far.
If it isn't possible, to get it over the Socket, can I get the EndPoint out of the Socket to use TCPClient as "last Option"?
TCP is a byte stream, it has no concept of messages. The size of the individual packets on the wire is irrelevant, it is just an implementation detail of the networking hardware. TCP guarantees that what you send is what you receive (but there is no 1-to-1 relationship between the size of individual sends and the size of individual reads, like there is in UDP).
The solution requires the sender to frame the string data in such a way that allows the reader to know when to stop reading. Either:
send the string length before sending the string data. The reader can then read the length first and then read the specified number of following bytes.
a. terminate the string with a unique delimiter that cannot appear in the string itself. The reader can then keep reading until it encounters the delimiter.
b. the delimiter can be closure of the connection. The reader can keep reading until a disconnect is detected.
Which solution you need to use depends on the particular protocol you are implementing. #1 is best for binary protocols, and allows for efficient memory managent, whereas #2 is more suited to text-based protocols, or streaming protocols where the final length is not know ahead of time. Sometimes protocols have to resort to #2b (HTTP and FTP both utilize it as times) when #1 and #2a are not possible.
Related
If i send 1000 bytes in TCP, does it guarantee that the receiver will get the entire 1000 bytes "togther"? or perhaps he will first only get 500 bytes, and later he'll receive the other bytes?
EDIT: the question comes from the application's point of view. If the 1000 bytes are reassembles into a single buffer before they reach the application .. then i don't care if it was fragmented in the way..
See Transmission Control Protocol:
TCP provides reliable, ordered delivery of a stream of bytes from a program on one computer to another program on another computer.
A "stream" means that there is no message boundary from the receiver's point of view. You could get one 1000 byte message or one thousand 1 byte messages depending on what's underneath and how often you call read/select.
Edit: Let me clarify from the application's point of view. No, TCP will not guarantee that the single read would give you all of the 1000 bytes (or 1MB or 1GB) packet the sender may have sent. Thus, a protocol above the TCP usually contains fixed length header with the total content length in it. For example you could always send 1 byte that indicates the total length of the content in bytes, which would support up to 255 bytes.
As other answers indicated, TCP is a stream protocol -- every byte sent will be received (once and in the same order), but there are no intrinsic "message boundaries" -- whether all bytes are sent in a single .send call, or multiple ones, they might still be received in one or multiple .receive calls.
So, if you need "message boundaries", you need to impose them on top of the TCP stream, IOW, essentially, at application level. For example, if you know the bytes you're sending will never contain a \0, null-terminated strings work fine; various methods of "escaping" let you send strings of bytes which obey no such limitations. (There are existing protocols for this but none is really widespread or widely accepted).
Basically as far as TCP goes it only guarantees that the data sent from one end to the other end will be sent in the same order.
Now usually what you'll have to do is have an internal buffer that keeps looping until it has received your 1000 byte "packet".
Because the recv command as mentioned returns how much has actually been received.
So usually you'll have to then implement a protocol on top of TCP to make sure you send data at an appropriate speed. Because if you send() all the data in one run through it will overload the under lying networking stack, and which will cause complications.
So usually in the protocol there is a tiny acknowledgement packet sent back to confirm that the packet of 1000 bytes are sent.
You decide, in your message that how many bytes your message shall contain. For instance in your case its 1000. Following is up and running C# code to achieve the same. The method returns with 1000 bytes. The abort code is 0 bytes; you can tailor that according to your needs.
Usage:
strMsg = ReadData(thisTcpClient.Client, 1000, out bDisconnected);
Following is the method:
string ReadData(Socket sckClient, int nBytesToRead, out bool bShouldDisconnect)
{
bShouldDisconnect = false;
byte[] byteBuffer = new byte[nBytesToRead];
Array.Clear(byteBuffer, 0, byteBuffer.Length);
int nDataRead = 0;
int nStartIndex = 0;
while (nDataRead < nBytesToRead)
{
int nBytesRead = sckClient.Receive(byteBuffer, nStartIndex, nBytesToRead - nStartIndex, SocketFlags.None);
if (0 == nBytesRead)
{
bShouldDisconnect = true;
//0 bytes received; assuming disconnect signal
break;
}
nDataRead += nBytesRead;
nStartIndex += nBytesRead;
}
return Encoding.Default.GetString(byteBuffer, 0, nDataRead);
}
Let us know this didn't help you (0: Good luck.
Yes, there is a chance for receiving packets part by part. Hope this msdn article and following example (taken from the article in msdn for quick review) would be helpful to you if you are using windows sockets.
void CChatSocket::OnReceive(int nErrorCode)
{
CSocket::OnReceive(nErrorCode);
DWORD dwReceived;
if (IOCtl(FIONREAD, &dwReceived))
{
if (dwReceived >= dwExpected) // Process only if you have enough data
m_pDoc->ProcessPendingRead();
}
else
{
// Error handling here
}
}
TCP guarantees that they will recieve all 1000 bytes, but not necessarily in order (though, it will appear so to the recieving application) and not necessarily all at once (unless you craft the packet yourself and make it so.).
That said, for a packet as small as 1000 bytes, there is a good chance it'll send in one packet as long as you do it in one call to send, though for larger transmissions it may not.
The only thing that the TCP layer guarantees is that the receiver will receive:
all the bytes transmitted by the sender
in the same order
There are no guarantees at all about how the bytes might be split up into "packets". All the stuff you might read about MTU, packet fragmentation, maximum segment size, or whatever else is all below the layer of TCP sockets, and is irrelevant. TCP provides a stream service only.
With reference to your question, this means that the receiver may receive the first 500 bytes, then the next 500 bytes later. Or, the receiver might receive the data one byte at a time, if that's what it asks for. This is the reason that the recv() function takes a parameter that tells it how much data to return, instead of it telling you how big a packet is.
The transmission control protocol guarantees successful delivery of all packets by requiring acknowledgment of the successful delivery of each packet to the sender by the receiver. By this definition the receiver will always receive the payload in chunks when the size of the payload exceeds the MTU (maximum transmission unit).
For more information please read Transmission Control Protocol.
The IP packets may get fragmented during retransmission.
So the destination machine may receive multiple packets - which will be reassembled back by TCP/IP stack. Depending on the network API you are using - the data will be given to you either reassembled or in RAW packets.
It depends of the stablished MTU (Maximum transfer unit). If your stablished connection (once handshaked) refers to a MTU of 512 bytes you will need two or more TCP packets to send 1000 bytes.
So I have written the code so that I can communicate with server and client.
The first question is how does the server identify that its communicating with an actual client, not someone else who's using the port, I've heard that browsers verify with servers using SHA hashing.
Second question is about the best way to send and receive data in variables, and also identifying which is which, because the current method of splitting data doesn't seem very elegant.
Server side code to receive and send data:
NetworkStream NetStream1 = TCPSocket.GetStream();
NetStream.Read(Buffer, 0, Buffer.Length);
ReceivedData = System.Text.Encoding.ASCII.GetString(Buffer);
string[] splitter = ReceivedData.Split('-');
Variable1 = splitter[0];
Variable2 = splitter[1];
//send response
SendBuffer = Encoding.ASCII.GetBytes(ResultINT1+"-"+ResultINT2);
NetStream.Write(SendBuffer, 0, SendBuffer.Length);
NetStream.Flush();
Client code to send and receive
NetworkStream SendStream = ClientSocket.GetStream();
byte[] SendBuffer = System.Text.Encoding.ASCII.GetBytes(V1+"-"+V2);
SendStream.Write(SendBuffer, 0, SendBuffer.Length);
SendStream.Flush();
//response
SendStream.Read(RecieveBuffer, 0, RecieveBuffer.Length);
string ResultString = System.Text.Encoding.ASCII.GetString(RecieveBuffer);
string[] splitted = ResultString.Split('-');
int R1 = Convert.ToInt32(splitted[0]);
int R2 = Convert.ToInt16(splitted[1]);
Provide some authentication mechanism
Use some serializer.
Your first question concerns authentication which is a huge subject and has many possible implementations although I'm not sure exactly what you mean by "someone else who's using the port". Your server should always be on the same port - that is how the client identifies a service.
Regarding your second question there are again many possibilities but I would suggest that the simplest for a beginner would be using XmlSerializer and a simple message envelope.
Create an XmlSerializable class either just using simple public properties or perhaps decorating with XmlElementAttribute, XmlRootAttribute etc.
Serialize to a MemoryStream
Write the bytes from the memory stream wrapped in an envelope (see later)
Receive a complete envelope into a byte array.
construct a MemoryStream from the byte array
Use XmlSerializer to reconstruct a copy of your original object.
The envelope is critical. The simplest one is just the binary length of the serialized object. Most protocols will typically extend that with CRC to handle possible corruption but since Ethernet uses a strong CRC and TCP is a reliable transport (albeit with a weak CRC) that is usually overkill. The key point that beginners miss is that TCP is a streaming protocol not a message based protocol thus it is perfectly possible for a sender to make a single write of say 1000 bytes and yet the receiver receives this as a number of smaller chunks. This is why you need some way to detect the end of a message such as using a length and why the receiver needs to accumulate received chunks until a complete message (and possibly part of the next) is received and can be deserialized.
This may seem complicated but unfortunately, at the TCP level, it doesn't get any simpler than that :(
The first question is how does the server identify that its communicating with an actual client, not someone else who's using the port, I've heard that browsers verify with servers using SHA hashing.
The server can identify different client by their IP addresses. See StreamReader.ReadToEnd
Second question is about the best way to send and receive data in variables, and also identifying which is which, because the current method of splitting data doesn't seem very elegant.
It depends on your protocol architecture, but a portable way to exchange values on network is to keep them in text format (this way no problem of endianness, type size...).
Said that, be careful of your variable separator : a '-' might be difficult to use with negative numbers, ' ' or ';' are more common.
You might want to define a communication protocol of some kind - a text-based protocol would be most straightforward to begin with - you can then read and write the "commands" each on a separate line.
First, there would be a "handshake", where the client would send something like "HELLO my-awesome-protocol-v1\n" and the server would respond similarly. This way you will be sure that the other person is a client, who understands the protocol or you can close a connection, which does not implement the protocol.
Then there could be some way of sending the values of variables with commands like "VAR variableName 123.45\n". You can read https://en.wikipedia.org/wiki/Text-based_protocol and see http://www.ncftp.com/libncftp/doc/ftp_overview.html for inspiration.
Im new to sockets, and Im creating a tictactoe online, I know how to make the connections with the clients and the server, but I will make a chat too.
Then I doing this, when a user chat I send a message with a prefix "CHAT: HELLO WORLD"
and when a user make a move I send a message without the prefix... this is the best way?
THX!!!
In defining a wire protocol over a stream-based protocol like TCP, you have a few options for constructing messages:
Fixed-length
All messages are the same length; every sequence of x bytes represents a new message.
Length-prefixed (variable length)
The first byte(s) of the message represent the length of the payload to follow.
String-terminated (variable length)
Read bytes from the stream until you come to a specified byte-string that represents the end of a message, i.e. the newline character \n.
If you ever intend on changing the protocol (protip: you will, even if you don't think you will), it is crucial that you include an identifier for the protocol version in each message to prevent issues when dealing with clients using an older iteration of the protocol. Clearly, this is the first thing you must determine before deciphering the rest of the payload, so this should be the first byte(s) of the message (following any length-prefix) - how could we determine the version if we don't know where it is located in every message we receive?
Typically you would go with a format that includes a packet length, type and payload.
In your case you could go with a Byte (type), Int16 (length), Byte[] (payload).
The type can be represented in code as an enum. Length would just represent the length of the payload.
public enum Byte PacketType {
PlayerMove = 1,
PlayerChat = 2
}
You need to define a protocol. Remember to allow room for additional features :-).
Eg. using regular expressions over complete lines (end with selected line terminator):
Matching ^:[a-c][1-3]:: is a move (colon, position, colon user name).
Matching ^!.*?:: is a chat message (exclamation point, name, colon, text).
and anything else (in V1) is an error.
Remember:
Data is sent in packets, you might need multiple reads from the socket to get a complete message.
Avoid ambiguity: resolving it might be x or y is hard.
Specify a text encoding (eg. UTF-8).
I assume you're using TCP?
You need to make sure you 'frame' both messages so you can identify them and also avoid potential blocking issues (in case the client stops sending while you are still expecting to read CHAT: or whatever you define). With TCP your byte order is guaranteed but reading does not guarantee a complete 'packet' so you'll need to implement some way of building up a buffer and identifying when your 'message' is complete.
A reasonably simple way of doing this is to make sure each 'message' has a header with the type and size specified.
EG:
Enumerate your message types (move and chat currently), so say 'chat' is 0x01 and your message is 1020 bytes. You can prefix your 'message' with 0x0103FC so the server knows how many bytes to expect, and build up a buffer using async socket calls until the 1020 bytes are read (or you arbitrarily decide that the client is not sending anymore)
This code works:
TcpClient tcpClient = (TcpClient)client;
NetworkStream clientStream = tcpClient.GetStream();
byte[] message = new byte[5242880];
int bytesRead;
bytesRead = clientStream.Read(message, 0, 909699);
But this returns the wrong number of bytes:
bytesRead = clientStream.Read(message, 0, 5242880);
Why? How can I fix it?
(the real data size is 1475186; the code returns the 11043 as the number of bytes)
If this is a TCP based stream, then the answer is that the rest of the data simply didn't arrive yet.
TCP is stream oriented. That means there is no relation between the number of Send/Write calls, and the number of receive events. Multiple writes can be combined together, and single writes can be split.
If you want to work with messages on TCP, you need to implement your own packeting algorithm on top of it. Typical strategies to achieve this are:
Prefix each packed by its length, usual with binary data
Use a separation sequence such as a line-break. Usual with text data.
If you want to read all data in a blocking way you can use loop until DataAvailable is true but a subsequent call to Read returns 0. (Hope I remembered that part correctly, haven't done any network programming in a while)
From MSDN:
The Read operation reads as much data as is available, up to the
number of bytes specified by the size parameter.
I.e. you have to call the Read() method in a loop until you received all data. Have a look at the sample code in MSDN.
You need to loop reading bytes from the message until the Available property on the TCP client or the DataAvailable property of the NetworkStream are 0 (= no more bytes left)
Read the Documentation:
This method reads data into the buffer parameter and returns the
number of bytes successfully read. If no data is available for
reading, the Read method returns 0. The Read operation reads as much
data as is available, up to the number of bytes specified by the size
parameter. If the remote host shuts down the connection, and all
available data has been received, the Read method completes
immediately and return zero bytes.
So it could be because of connection failure that you get each time different number, anyway you can check the result to know if its the reason.
I think the answers already here respond to your specific question quite well, but possibly more generally: If you are trying to send data over a networkStream object for the purposes of network communication check out the open source library, networkComms.net.
If i send 1000 bytes in TCP, does it guarantee that the receiver will get the entire 1000 bytes "togther"? or perhaps he will first only get 500 bytes, and later he'll receive the other bytes?
EDIT: the question comes from the application's point of view. If the 1000 bytes are reassembles into a single buffer before they reach the application .. then i don't care if it was fragmented in the way..
See Transmission Control Protocol:
TCP provides reliable, ordered delivery of a stream of bytes from a program on one computer to another program on another computer.
A "stream" means that there is no message boundary from the receiver's point of view. You could get one 1000 byte message or one thousand 1 byte messages depending on what's underneath and how often you call read/select.
Edit: Let me clarify from the application's point of view. No, TCP will not guarantee that the single read would give you all of the 1000 bytes (or 1MB or 1GB) packet the sender may have sent. Thus, a protocol above the TCP usually contains fixed length header with the total content length in it. For example you could always send 1 byte that indicates the total length of the content in bytes, which would support up to 255 bytes.
As other answers indicated, TCP is a stream protocol -- every byte sent will be received (once and in the same order), but there are no intrinsic "message boundaries" -- whether all bytes are sent in a single .send call, or multiple ones, they might still be received in one or multiple .receive calls.
So, if you need "message boundaries", you need to impose them on top of the TCP stream, IOW, essentially, at application level. For example, if you know the bytes you're sending will never contain a \0, null-terminated strings work fine; various methods of "escaping" let you send strings of bytes which obey no such limitations. (There are existing protocols for this but none is really widespread or widely accepted).
Basically as far as TCP goes it only guarantees that the data sent from one end to the other end will be sent in the same order.
Now usually what you'll have to do is have an internal buffer that keeps looping until it has received your 1000 byte "packet".
Because the recv command as mentioned returns how much has actually been received.
So usually you'll have to then implement a protocol on top of TCP to make sure you send data at an appropriate speed. Because if you send() all the data in one run through it will overload the under lying networking stack, and which will cause complications.
So usually in the protocol there is a tiny acknowledgement packet sent back to confirm that the packet of 1000 bytes are sent.
You decide, in your message that how many bytes your message shall contain. For instance in your case its 1000. Following is up and running C# code to achieve the same. The method returns with 1000 bytes. The abort code is 0 bytes; you can tailor that according to your needs.
Usage:
strMsg = ReadData(thisTcpClient.Client, 1000, out bDisconnected);
Following is the method:
string ReadData(Socket sckClient, int nBytesToRead, out bool bShouldDisconnect)
{
bShouldDisconnect = false;
byte[] byteBuffer = new byte[nBytesToRead];
Array.Clear(byteBuffer, 0, byteBuffer.Length);
int nDataRead = 0;
int nStartIndex = 0;
while (nDataRead < nBytesToRead)
{
int nBytesRead = sckClient.Receive(byteBuffer, nStartIndex, nBytesToRead - nStartIndex, SocketFlags.None);
if (0 == nBytesRead)
{
bShouldDisconnect = true;
//0 bytes received; assuming disconnect signal
break;
}
nDataRead += nBytesRead;
nStartIndex += nBytesRead;
}
return Encoding.Default.GetString(byteBuffer, 0, nDataRead);
}
Let us know this didn't help you (0: Good luck.
Yes, there is a chance for receiving packets part by part. Hope this msdn article and following example (taken from the article in msdn for quick review) would be helpful to you if you are using windows sockets.
void CChatSocket::OnReceive(int nErrorCode)
{
CSocket::OnReceive(nErrorCode);
DWORD dwReceived;
if (IOCtl(FIONREAD, &dwReceived))
{
if (dwReceived >= dwExpected) // Process only if you have enough data
m_pDoc->ProcessPendingRead();
}
else
{
// Error handling here
}
}
TCP guarantees that they will recieve all 1000 bytes, but not necessarily in order (though, it will appear so to the recieving application) and not necessarily all at once (unless you craft the packet yourself and make it so.).
That said, for a packet as small as 1000 bytes, there is a good chance it'll send in one packet as long as you do it in one call to send, though for larger transmissions it may not.
The only thing that the TCP layer guarantees is that the receiver will receive:
all the bytes transmitted by the sender
in the same order
There are no guarantees at all about how the bytes might be split up into "packets". All the stuff you might read about MTU, packet fragmentation, maximum segment size, or whatever else is all below the layer of TCP sockets, and is irrelevant. TCP provides a stream service only.
With reference to your question, this means that the receiver may receive the first 500 bytes, then the next 500 bytes later. Or, the receiver might receive the data one byte at a time, if that's what it asks for. This is the reason that the recv() function takes a parameter that tells it how much data to return, instead of it telling you how big a packet is.
The transmission control protocol guarantees successful delivery of all packets by requiring acknowledgment of the successful delivery of each packet to the sender by the receiver. By this definition the receiver will always receive the payload in chunks when the size of the payload exceeds the MTU (maximum transmission unit).
For more information please read Transmission Control Protocol.
The IP packets may get fragmented during retransmission.
So the destination machine may receive multiple packets - which will be reassembled back by TCP/IP stack. Depending on the network API you are using - the data will be given to you either reassembled or in RAW packets.
It depends of the stablished MTU (Maximum transfer unit). If your stablished connection (once handshaked) refers to a MTU of 512 bytes you will need two or more TCP packets to send 1000 bytes.