I'm building a File Sharing Program, and I would like to know if it's better, while using Sockets, to receive and send byte per byte, or a fixed amount. I'm sending messages of Login, Actual file size list, etc, of 512 bytes, and 65536, when sending and receiving files.
it is depend on your usage and goal:
for High Performance when in non-faulty environment:
choose 1500 bytes
for bad and faulty environment:
choose lower sizes but not byte per byte
It's always better to use reasonably sized blocks for efficiency reasons. Typical network packets are around 1500 bytes in size (Ethernet) and every packet carries a bunch of necessary overhead (such as protocol, destination address and port etc.).
Single bytes is the worst (in terms of efficiency) that you can do.
Handling 1500 or so bytes at a time will be much more efficient than one byte at a time. That is about the size of a typical Ethernet frame.
Keep in mind that you are using a stream of bytes: any concept of message or record is up to you to implement.
Related
I was wondering about the order of sent and received bytes by/from TCP socket.
I got implemented socket, it's up and working, so that's good.
I have also something called "a message" - it's a byte array that contains string (serialized to bytes) and two integers (converted to bytes). It has to be like that - project specifications :/
Anyway, I was wondering about how it is working on bytes:
In byte array, we have order of bytes - 0,1,2,... Length-1. They sit in memory.
How are they sent? Is last one the first to send? Or is it the first? Receiving, I think, is quite easy - first byte to appear gets on first free place in buffer.
I think a little image I made nicely shows what I mean.
They are sent in the same order they are present in memory. Doing otherwise would be more complex... How would you do if you had a continuous stream of bytes? Wait that the last one has been sent and then reverse all? Or this inversion should work "packet by packet"? So each block of 2k bytes (or whatever is the size of the TCP packets) is internally reversed but the order of the packets is "correct"?
Receiving, I think, is quite easy - first byte to appear gets on first free place in buffer.
Why on the earth the sender should reverse the bytes but the receiver shouldn't? If you build a symmetric system, both do an action or none does it!
Note that the real problem is normally the one of endianness. The memory layout of an int on your computer could be different than the layout of an int of another computer. So one of the two computers could have to reverse the 4 bytes of the int. But endianness is something that is resolved "primitive type" by "primitive type". Many internet protocols, for historical reason, are Big Endian, while Intel CPUs are Little Endians. Even internal fields of TCP are Big Endian (see Big endian or Little endian on net?), but here we are speaking of fields of TCP, not of the data moved by the TCP protocol.
I have an ASP.NET MVC app that is an Azure Web Role. It talks to another system over a socket. The files returned by the remote server are on average 100,000 bytes in length.
Currently I am using a 1024 byte buffer in a read loop when reading data from the socket (see code below). I am not sure if it matters, but both systems are in the same Cloud Service with the same data centre affinity.
Would I be better off changing the size of the buffer to 100,000 bytes? Or would that put more of a load on the server, especially if a large number of sessions were in progress? Speed is the ultimate consideration here.
byte[] myReadBuffer = new byte[1024];
int numberOfBytesRead = 0;
// Incoming message may be larger than the buffer size.
do
{
numberOfBytesRead = myNetworkStream.Read(myReadBuffer, 0, myReadBuffer.Length);
// Append the new bytes to our memory stream.
memStream.Write(myReadBuffer, 0, numberOfBytesRead);
}
while (myNetworkStream.DataAvailable);
Typically, the buffer array size offers diminishing returns. It does slightly impact the amount of memory used by your application: larger buffers held in memory take up more space. However, all that the buffer really serves to do is take bytes from the socket and hold them. As a result, you can only ever read the number of bytes equivalent to the size of your buffer at one time.
I often use a buffer of 4096 bytes. I've benchmarked it at 1024, 2048, and for fun, 100000. The performance wasn't much different between the arrays, but the memory usage did change due to the buffer size.
One best practice is to use a smaller buffer in order to read data from the socket more frequently, and depending on what you do with the data, potentially hold less data in memory at once as well. This keeps the socket from overflowing with queue'd data which has not been processed. What I mean is: if processing the bytes you receive takes 1 ms per byte, it's better to read 1024 bytes at once (~1 second between socket reads) than it would be to read 4096 bytes at once (~4 seconds between socket reads). More frequent reads of the socket data means less data queued at the socket.
EDIT: also worth noting, your loop will not always work currently. Whether or not data is available to read is happenstance and depends on many factors which you do not control such as WAN hops and delays in sending or receiving data. What you must do is use or invent a protocol that tells you how many bytes you expect to receive for a given message, and then read in the loop until either 1) you've received all expected bytes or 2) you reach a timeout at which point you feel it is taking too long. Your loop probably works currently because it's being done locally or over LAN, but it's not reliable and in fact prone to error.
Hi
I have TCP/IP client server application. i want to send large serialized object around 1MB through sockets.
Is it possible to get better performance by splitting byte array to for example 10 chunks of arrays and open a socket for each and send them Async compared to opening one socket and send all large data through it ?
Thanks
Splitting the data to less than the MTU will introduce more overhead as there will be more packets - this will actually slow things down. What you are proposing is already being done as part of the protocol i.e. splitting and re-assembling. I would experiment with sending less data e.g. compression.
No, this doesn't speed up the transfer under normal conditions, it only adds overhead. It would only help if you have a slow network segment which is quite busy otherwise and the traffic is shaped per TCP connection.
Make sure that your sockets code is efficient, because wrong buffer and therefore packet sizes, synchroneous operation and other stuff may slow the transfer down.
I'm sending data to an extremely old system via TCP. I need to send 2000 bytes in one packet, and I need it not to be split up (what happens when I write out 2000 bytes via a socket).
While, yes, I shouldn't have to care about this at an application level -- I in fact do care about this, because I have no other options on the older system, everything MUST be received in a single packet.
Is there something less terrible than calling netcat?
Unless you are on a link with jumbo frames the usual MTU on the ethernet is 1500. Subtract IP (20 bytes) and TCP headers (at least 20 bytes). So no luck with 2000 bytes in a single packet.
i am using the win32 waveform api's in a C# app to make a voip system. all is going well, however i need some way of compressing the audio data on the fly.
so basically the audio data comes into a 'record' buffer of size 150 bytes, and then this buffer is sent over udp, and at the remote end, the 150 bytes are received and put into a 'play' buffer.
so i need some way of compressing/decompressing the data just before the udp->send and just after the udp->recv. normal compression algorithms dont work with audio, including the .NET GZip class.
does anyone know of a library that i can use that will help me do this ?
thanks in advance...
150 bytes is an unbelievably small buffer for audio data--less than 5 milliseconds for e.g. 16 KHz mono. I'm no expert but I think regardless of the compression scheme you choose, your compression ratio will suffer greatly for using such a small buffer. Besides that there is significant overhead for each packet you send.
That said, if you are sending speech data, take a look at Speex for lossy compression (I have found it very effective at compressing speech, but the sound quality is terrible for music.)
I would think you'd want to batch up those 150-byte chunks to get better compression.
Although, even at small buffer sizes like that, you can still get some compression.
If the built-in GZipStream isn't working you could try the GZipStream that is included in DotNetZip. There is also a ZlibCodec class available in DotNetZip that implements the Codec pattern - this may facilitate compressing in 150-byte blocks.
The component you're looking for is more well-known as a coder/decoder, or codec, and there are many options when it comes to picking one.
As suggested above, I'd look into Speex. It's well supported, and now the defacto standard for Flash Player.
I assume that by the size you are setting your buffers that latency is an issue (the bigger the buffer, the bigger the latency), so don't go for a codec that has a high decompressed frame size, because it introduces high latency. This more or less rules out MP3... for voice at 5khz output sample rate (it wouldn't serve much purpose going higher), the minimum decompressed frame size is 576 samples, or ~100ms of data that must be encoded prior to send. This means a bothway latency of over 200ms before you've even considered the network part of the problem.