Let's say my program sends a 1000 bytes over the network (UDP). Does it guaranteed that the receiver will receive the 1000 bytes in one "batch"? Or perhaps he will need to perform sevral "reads" until he'll receive the entire message? if the later is true, how can i ensure that the order of the packets for the same message don't get "mixed up" (in order), or perhaps the protocol guarantees it?
Edit: that is, does it possible that my message will be split to sevral packets? (what if i try to send a 10000mb message, what happens then?)
You will get it all or nothing.
But there is no particular guarantee that you will receive packets exactly once in the order they were transmitted; packet loss, reordering and (less often) duplication are all possible.
There is a maximum frame size (of 65,507 bytes), send()ing packets of larger sizes will return an error.
You must provide enough buffer to receive the entire frame in one call.
UDP packets CAN be fragmented into multiple IP fragments, but the OS will drop an incomplete packet. This is therefore transparent to the application.
The receiver will get the entire packet in one call. The packet length is limited, even in theory:
Length
A 16-bit field that specifies the length in bytes of the entire
datagram: header and data. The minimum
length is 8 bytes since that's the
length of the header. The field size
sets a theoretical limit of 65,535
bytes (8 byte header + 65527 bytes of
data) for a UDP datagram. The
practical limit for the data length
which is imposed by the underlying
IPv4 protocol is 65,507 bytes.
However the real limit is much much lower, usually is safe to assume 512 bytes. See What is the largest Safe UDP Packet Size on the Internet.
UDP, unlike TCP, is not a reliable protocol. It provides no built in mechanism to ensure that the packets arrive in the proper order, or even arrive at all. That said, you can write your send/recv routines in a lock step fashion, where every time a packet is sent, the sender must wait to receive an ACK before sending again. If an ACK is not received after some specified timeout, the packet must be resent. This way you ensure that packets are received in the proper order. (For more information, check out the RFC for the TFTP protocol, which uses this strategy.)
Finally, if possible, you may want to consider using TCP instead.
Data sent using UDP is grouped in packets, so if you send x amount of bytes then IF the receiver receives the packet he'll receive x amount of bytes.
However, your packets might not even arrive, or they may arrive out of order.
With UDP Lite you can request to receive partially corrupted packets. This can be useful for video and VoIP services.
Related
I have a server that sends telemetry data of varying size to receivers, or as I'll call them, clients, via NetworkStream. I aim for maximum speed, so I want minimal delays. The frequency is uncontrolled, so I'd like to use an infinite loop in my clients, that uses NetworkStream.Read to read the data, process it, then repeat.
My problem is that sometimes, if two packets are sent very quickly, and a client has a slow internet connection, the two packets will be received as a continous stream, resulting unprocessable data. A half-solution I found (mostly to confirm that this is indeed the error) is to have a small delay after/before each transmission, using System.Threading.Thread.Sleep(100), but not only I find Sleep a botchy solution, it's inconsistent, as it also slows down clients with a good connection, and the problem may persist with an even worse connection.
What I'd like to do is to have the server send a gap between each transmission, providing a separation regardless of internet speed, as NetworkStream.Read should finish after the current continous stream ends. I don't understand deeply the working of a NetworkStream, and have no idea what a few bytes of empty stream look like or how it could be implemented. Any ideas?
I would strong advise changing the protocol instead if you possibly can. TCP is a stream-based protocol, and any attempt to effectively ignore that is likely to be unreliable.
Instead, I'd suggest changing to make it a stream of messages where each message has a prefix indicating the length of the body of the message. That way it doesn't matter if a particular message is split across multiple packets or received in the same packet as other messages. It also makes reading easier for clients: they know exactly how much data to read, so can just do that in a simple loop.
If you're concerned that the length prefix will introduce too much overhead (if the data is often small) you could potentially make the protocol slightly more complicated with a single message containing a whole batch of information (multiple telemetry items).
But fundamentally, it's worth assuming that the data will be split into multiple packets, combined again etc. Don't assume that one write operation corresponds to one read operation.
You don't specifically mention the ProtocolType you're using on your NetworkStream but TCP is bound to fail your requirements. Intermediate routers/switches have no way to know your intent is to separate packets by time and will not respect that desire. Furthermore, TCP, being stream oriented, delivers packets in order, and it has error correction against dropped packets and corrupt packets. On any occurrence of one or the other it will hold all further packets until the error packet is retransmitted - and then you'll probably get them all in a bunch.
Use UDP and implement throttling on the receiving (i.e., client) side - throwing out data if you're falling behind.
I want to send huge buffers (from 100MB to 1GB) of data by TCP. I solved it by dividing buffer to smaller (approximately 1MB buffers) and sending by socket.send(). Each call of socket.send() method, send part of data (smaller buffer) packed in specific structure: [start byte(1B), Timestamp(4B), Command(4B), Length of data(4B), Data to send(?B), CRC(1B), End byte(1B)]. Everything works fine, when only one huge buffer is sent by specific port. But when I try to send in the same time buffer with another data (very small, e.g. 20 bytes) using the same TCP port, then TCP mixed data in buffers and it's not possible to decode buffer any more. 'Start byte' and 'end byte' in the buffer are not useful to find start and end of the buffer, because it's probable that these bytes appear in data.
EDIT: Issue does not affect the order or IDs between packages but bytes in the packages. At the beginning everything works fine and each buffer is decoded properly. After a while it is not possible to decode buffer, because it contains incorrect data. It looks like bytes in the buffer were moved or changed. Fields at the beginning of the buffer (timestamp, command, length) contain impossible values. So when I want to get length of sent data, I get e.g. value like -1534501133 instead of 1048556 (1048556 is a correct maximum size of the sent data in one package). It happens randomly but it is always connected with the moment when smaller independent buffer is sent. The additional information is, that the smaller buffers are sent repetitively using timers and the problem happens in random moments. Sometimes it is even possible to send whole data (e.g. 300 MB) without problem but it happens very rarely.
I hope, I described it clearly enough.
Do you have any suggestions how to avoid this problem?
Tag your data with a unique id so you know what data relates to what message. Also, separate the packet header from the packet payload.
So, your first request would be [ID][PACKETTYPE][TIME][COMMAND][LENGTH][CRC]
Second would be [ID][PACKETTYPE][DATA]
You can then match up the IDs with the types of packet. So packet type would be 'HEADER' or 'PAYLOAD', the header would contain the meta data for the payload allowing you to make sure that it doesnt get mixed up with other data.
I am using the NetworkStream of a TcpClient to send bytes from a .NET server to a Silverlight Client that receives them via a Socket. Sending from the client is done via Socket.SendAsync.
My question is, what is the minimum amount of bytes I can expect to be received "in one go" on both sides, without the need to send the message length too and put several byte messages together by myself.
Thanks,
Andrej
You absolutely should send the message length. Network streams are precisely that - streams of information. Basically there are three ways of recognising the end of a message:
Prefix the data with the length (don't forget that it's at least theoretically possible that you won't even get all of the length data in one packet)
Use a message delimiter/terminator
Only send one message each way per connection, and close the connection afterwards
It depends on your network settings, but the default length for network nodes is 1500 bytes and most modems take something off that. So most homes have 1460 bytes packet size.
The optimum size for your situation can be calculated
But people can always have their own settings, so there's no guarantee that you get the optimal packet size for all clients.
I want to know the amount of data arrived at my updclient including damaged/bad data, packet header etc. i want to calculate throughput using udp.
Thanks.
UdpClient is a very high-level interface which doesn't provide access to raw packet data. To get such information, you will need to use some low-level API and process the packets yourself.
However, in practice, the chances of a packet getting damaged in transit are very low - most of the time, you either get the correct packet or you don't get the packet at all. The packet headers are generally of constant size (8 bytes for the UDP header and usually 20 bytes for the IP header), so you can just add this value to the size of each datagram (which is returned by UdpClient.Receive) to get the total packet size.
I'm trying to send a large amount of data (more than 50 MB) using C# UdpClient.
So at first I split the data into 65507 byte blocks and send them in a loop.
for(int i = 0; i < packetCount; i++)
myUdpClient.Send(blocks[i], block[i].Length, remoteEndPoint);
My problem is that only the first packets can be received.
During sending the first packet the networkload increases rapidly to 100%, and then the other packets cannot be received.
I want to get as much data throughput as possible.
I'm sorry for my English!
Thanks for your help in advance.
For all those people saying to use TCP....are foolishly wrong. Although TCP is reliable and the window maintained by the kernel it's fairly "set and forget" protocol, but when it comes to the guy wanting to use 100% of his throughput, TCP will not do (it throttles too hard, and the wait for an ACK automatically puts at least 50% trashed because of the RTT).
To the original question, you are sending UDP packets nonstop in that for-loop, the window fills up and then any new data is dropped immediately and doesn't even attempt to go on the line. You are also splitting your data too large. I would recommend building your own throttle mechanism that starts off with 2k segments per second, and slowly ramps up. Each "segment" contains a SEQ (sequence identifier for acknowledgements or ACK) and OFF (offset inside the file for this data set). As the data is being tagged, let the server keep track of these tags. When the other side gets them, it stores the SEQ numbers in an ACK list, and any missing SEQ numbers are placed into a NACK timer list, when the timer runs out (if they haven't been received) it moves to a NACK list. The receiver should send 5 or so ACKs from the ACK list along with up to 5 NACKs in a single transmission every couple seconds or so. If the sender receives these messages and there are any NACKs, it should immediately throttle down and resend the missing fragment before continuing. The data that is ACKed can be freed from memory.
Good luck!
I don't know about .Net implementation specifically, it might be buffering your data, but UDP datagram is normally limited by the link MTU, which is 1500 on normal ethernet (subtract 20 bytes for IP header and 8 bytes of UDP header.)
UDP is explicitly allowed to drop and reorder the datagrams, and there's no flow control as in TCP.
Exceeding the socket send buffer on the sender side will have the network stack ignore following send attempts until the buffer space is available again (you need to check the return value of the send() for that.)
Edit:
I would strongly recommend going with TCP for large file transfers. TCP gives you sequencing (you don't have to keep track of dropped and re-ordered packets.) It has advanced flow control (so fast sender does not overwhelm a slow receiver.) It also does Path MTU discovery (i.e. finds out optimal data packetization and avoids IP fragmentation.) Otherwise you would have to re-implement most of these features yourself.
I hate to say it but you need to sleep the thread. You are overloading your throughput. UDP is not very good for lossless data transfer. UDP is for when you don't mind dropping some packets.
Reliably - no, you won't do it with UDP.
As far as I understand, this makes sense for sending to multiple computers at a time (broadcasting).
In this case,
establish a TCP connection with each of them,
split the data into blocks,
give each block an ID,
send list of IDs to each computer with TCP connection,
broadcast data with UDP,
inform clients (via TCP) that data transmission is over,
than clients should ask to resend the dropped packets