Is there any way to pass the Length of uncertain Stream to WCF Service?
Unsertain Stream means the stream of
The stream provides its length only after process and writing the data.
e.g. GZipStream
Background
I'm making a WCF Service receiving multiple Streams from client.
As WCF Streaming only allows one stream in the message, I decided to concatenate all streams into one stream and divide it in server code.
The streams client provides will contains variable kinds of stream like FileStream, MemoryStreamwith data from DataTable serialization and
using (var fileStream = new FileStream(filePath, FileMode.Open))
using (var memoryStream = new MemoryStream())
using (var concatStream = new ConcatenatedStream(fileStream, memoryStream))
{
client.UploadStreams(concatStream);
}
ConcatenatedStream is a Stream implementation suggested in c# - How do I concatenate two System.Io.Stream instances into one? - Stack Overflow.
In server side, Length of each Streams will be needed to divide single stream to multiple streams.
As I want to save memory in client side, I decided to use PullStream.
PullStream will Write buffer on demand of Read.
But this causes a big problem. I cannot get Length of PullStream before starting streaming.
Any helps will be appreciated.
Thanks
Let's make it simple:
If you have the length of a part of the stream on client before you start pushing it to server you can append a structure before the payload and read that structure on server. That is a standard data transfer template. Doing so i.e. appending a header before each payload you give your server a hint on how long the next part is going to be.
If you do not have the length of a part of the stream on client before you start pushing it to server, you are going to have to 'insert' the header inside the payload. That's not very intuitive and not that useful but it does work. I used such a thing when I had my data prepared asynchronously on client and the first buffers were ready before the length was known. In this scenario you are going to need a so called marker i.e. a set of bytes that could not be found anywhere in the stream but before the header.
This scenario is the toughest of the 3 to implement when done for the first time. Buckle up. In order to do it right you should create an artificial structure of your stream. Such a structure is used for streaming video over network and called Network Abstraction Layer or NAL, read about it. It is also called stream format AnnexB from the h264 standard. You should abstract from the field in which the standard is described, the idea is very versatile.
In short the payload is divided into parts, so called NAL Units or NALUs, each part has a byte sequence which marks it's start, then goes the type indicator and length of the current NALU, then follows the payload of the NALU. For your purposes you would need to implement NALUs of two types:
Main data payload
Metadata
After you imagine how your stream should look like, you have to grip on the idea of "stream encoding". Those are fearsome words but do not worry. You just have to ensure that the byte sequence that is used to mark the start of the NALU is never met inside the payload of the NALU. In order to achieve that you are going to implement some replacement tactic. Browse for samples.
When you are done thinking this through and before you dive into that, think twice about it. Might be the scenario 3 would fit you easier.
In the case you are sure you will never have to process a part of the streamed data you can greatly simplify the scenario i.e. totally skip the stream encoding and implement something like this:
Client Stream principal code:
private byte[] mabytPayload;
private int mintCurrentPayloadPosition;
private int? mintTotalPayloadLength;
private bool mblnTotalPayloadLengthSent;
public int Read(byte[] iBuffer, int iStart, int iLength)
{
if (mintTotalPayloadLength.HasValue && !mblnTotalPayloadLengthSent)
{
//1. Write the packet type (0)
//3. Write the total stream length (4 bytes).
...
mblnTotalPayloadLengthSent = true;
}
else
{
//1. Write the packet type (1)
//2. Write the packet length (iLength - 1 for example, 1 byte is for
//the type specification)
//3. Write the payload packet.
...
}
}
public void TotalStreamLengthSet(int iTotalStreamLength)
{
mintTotalPayloadLength = iTotalStreamLength;
}
Server stream reader:
Public void WCFUploadCallback(Stream iUploadStream)
{
while(!endOfStream)
{
//1. Read the packet type.
if (normalPayload)
{
//2.a Read the payload packet length.
//2.b Read the payload.
}
else
{
//2.c Read the total stream length.
}
}
}
In the scenario where your upload is no-stop and the metadata about the stream is ready on client long after the payload, that happens as well, you are going to need two channels i.e. one channel for payload stream and another channel with metadata where you server will answer to the client with another question like 'what did you just started sending me' or 'what have you sent me' and the client will explain itself in the next message.
If you are ready to stick to one of the scenarios, one could give you some further details and/or recommendations.
Related
I have a TcpClient class on a client and server setup on my local machine. I have been using the Network stream to facilitate communications back and forth between the 2 successfully.
Moving forward I am trying to implement compression in the communications. I've tried GZipStream and DeflateStream. I have decided to focus on DeflateStream. However, the connection is hanging without reading data now.
I have tried 4 different implementations that have all failed due to the Server side not reading the incoming data and the connection timing out. I will focus on the two implementations I have tried most recently and to my knowledge should work.
The client is broken down to this request: There are 2 separate implementations, one with streamwriter one without.
textToSend = ENQUIRY + START_OF_TEXT + textToSend + END_OF_TEXT;
// Send XML Request
byte[] request = Encoding.UTF8.GetBytes(textToSend);
using (DeflateStream streamOut = new DeflateStream(netStream, CompressionMode.Compress, true))
{
//using (StreamWriter sw = new StreamWriter(streamOut))
//{
// sw.Write(textToSend);
// sw.Flush();
streamOut.Write(request, 0, request.Length);
streamOut.Flush();
//}
}
The server receives the request and I do
1.) a quick read of the first character then if it matches what I expect
2.) I continue reading the rest.
The first read works correctly and if I want to read the whole stream it is all there. However I only want to read the first character and evaluate it then continue in my LongReadStream method.
When I try to continue reading the stream there is no data to be read. I am guessing that the data is being lost during the first read but I'm not sure how to determine that. All this code works correctly when I use the normal NetworkStream.
Here is the server side code.
private void ProcessRequests()
{
// This method reads the first byte of data correctly and if I want to
// I can read the entire request here. However, I want to leave
// all that data until I want it below in my LongReadStream method.
if (QuickReadStream(_netStream, receiveBuffer, 1) != ENQUIRY)
{
// Invalid Request, close connection
clientIsFinished = true;
_client.Client.Disconnect(true);
_client.Close();
return;
}
while (!clientIsFinished) // Keep reading text until client sends END_TRANSMISSION
{
// Inside this method there is no data and the connection times out waiting for data
receiveText = LongReadStream(_netStream, _client);
// Continue talking with Client...
}
_client.Client.Shutdown(SocketShutdown.Both);
_client.Client.Disconnect(true);
_client.Close();
}
private string LongReadStream(NetworkStream stream, TcpClient c)
{
bool foundEOT = false;
StringBuilder sbFullText = new StringBuilder();
int readLength, totalBytesRead = 0;
string currentReadText;
c.ReceiveBufferSize = DEFAULT_BUFFERSIZE * 100;
byte[] bigReadBuffer = new byte[c.ReceiveBufferSize];
while (!foundEOT)
{
using (var decompressStream = new DeflateStream(stream, CompressionMode.Decompress, true))
{
//using (StreamReader sr = new StreamReader(decompressStream))
//{
//currentReadText = sr.ReadToEnd();
//}
readLength = decompressStream.Read(bigReadBuffer, 0, c.ReceiveBufferSize);
currentReadText = Encoding.UTF8.GetString(bigReadBuffer, 0, readLength);
totalBytesRead += readLength;
}
sbFullText.Append(currentReadText);
if (currentReadText.EndsWith(END_OF_TEXT))
{
foundEOT = true;
sbFullText.Length = sbFullText.Length - 1;
}
else
{
sbFullText.Append(currentReadText);
}
// Validate data code removed for simplicity
}
c.ReceiveBufferSize = DEFAULT_BUFFERSIZE;
c.ReceiveTimeout = timeOutMilliseconds;
return sbFullText.ToString();
}
private string QuickReadStream(NetworkStream stream, byte[] receiveBuffer, int receiveBufferSize)
{
using (DeflateStream zippy = new DeflateStream(stream, CompressionMode.Decompress, true))
{
int bytesIn = zippy.Read(receiveBuffer, 0, receiveBufferSize);
var returnValue = Encoding.UTF8.GetString(receiveBuffer, 0, bytesIn);
return returnValue;
}
}
EDIT
NetworkStream has an underlying Socket property which has an Available property. MSDN says this about the available property.
Gets the amount of data that has been received from the network and is
available to be read.
Before the call below Available is 77. After reading 1 byte the value is 0.
//receiveBufferSize = 1
int bytesIn = zippy.Read(receiveBuffer, 0, receiveBufferSize);
There doesn't seem to be any documentation about DeflateStream consuming the whole underlying stream and I don't know why it would do such a thing when there are explicit calls to be made to read specific numbers of bytes.
Does anyone know why this happens or if there is a way to preserve the underlying data for a future read? Based on this 'feature' and a previous article that I read stating a DeflateStream must be closed to finish sending (flush won't work) it seems DeflateStreams may be limited in their use for networking especially if one wishes to counter DOS attacks by testing incoming data before accepting a full stream.
The basic flaw I can think of looking at your code is a possible misunderstanding of how network stream and compression works.
I think your code might work, if you kept working with one DeflateStream. However, you use one in your quick read and then you create another one.
I will try to explain my reasoning on an example. Assume you have 8 bytes of original data to be sent over the network in a compressed way. Now let's assume for sake of an argument, that each and every byte (8 bits) of original data will be compressed to 6 bits in compressed form. Now let's see what your code does to this.
From the network stream, you can't read less than 1 byte. You can't take 1 bit only. You take 1 byte, 2 bytes, or any number of bytes, but not bits.
But if you want to receive just 1 byte of the original data, you need to read first whole byte of compressed data. However, there is only 6 bits of compressed data that represent the first byte of uncompressed data. The last 2 bits of the first byte are there for the second byte of original data.
Now if you cut the stream there, what is left is 5 bytes in the network stream that do not make any sense and can't be uncompressed.
The deflate algorithm is more complex than that and thus it makes perfect sense if it does not allow you to stop reading from the NetworkStream at one point and continue with new DeflateStream from the middle. There is a context of the decompression that must be present in order to decompress the data to their original form. Once you dispose the first DeflateStream in your quick read, this context is gone, you can't continue.
So, to resolve your issue, try to create only one DeflateStream and pass it to your functions, then dispose it.
This is broken in many ways.
You are assuming that a read call will read the exact number of bytes you want. It might read everything in one byte chunks though.
DeflateStream has an internal buffer. It can't be any other way: Input bytes do not correspond 1:1 to output bytes. There must be some internal buffering. You must use one such stream.
Same issue with UTF-8: UTF-8 encoded strings cannot be split at byte boundaries. Sometimes, your Unicode data will be garbled.
Don't touch ReceiveBufferSize, it does not help in any way.
You cannot reliably flush a deflate stream, I think, because the output might be at a partial byte position. You probably should devise a message framing format in which you prepend the compressed length as an uncompressed integer. Then, send the compressed deflate stream after the length. This is decodable in a reliable way.
Fixing these issues is not easy.
Since you seem to control client and server you should discard all of this and not devise your own network protocol. Use a higher-level mechanism such as web services, HTTP, protobuf. Anything is better than what you have there.
Basically there are a few things wrong with the code I posted above. First is that when I read data I'm not doing anything to make sure the data is ALL being read in. As per microsoft documentation
The Read operation reads as much data as is available, up to the
number of bytes specified by the size parameter.
In my case I was not making sure my reads would get all the data I expected.
This can be accomplished simply with this code.
byte[] data= new byte[packageSize];
bytesRead = _netStream.Read(data, 0, packageSize);
while (bytesRead < packageSize)
bytesRead += _netStream.Read(data, bytesRead, packageSize - bytesRead);
On top of this problem I had a fundamental issue with using DeflateStream - namely I should not use DeflateStream to write to the underlying NetworkStream. The correct approach is to first use the DeflateStream to compress data into a ByteArray, then send that ByteArray using the NetworkStream directly.
Using this approach helped to correctly compress data over the network and property read the data on the other end.
You may point out that I must know the size of the data, and that is true. Every call has a 8 byte 'header' that includes the size of the compressed data and the size of the data when it is uncompressed. Although I think the second was utimately not needed.
The code for this is here. Note the variable compressedSize serves 2 purposes.
int packageSize = streamIn.Read(sizeOfDataInBytes, 0, 4);
while (packageSize!= 4)
{
packageSize+= streamIn.Read(sizeOfDataInBytes, packageSize, 4 - packageSize);
}
packageSize= BitConverter.ToInt32(sizeOfDataInBytes, 0);
With this information I can correctly use the code I showed you first to get the contents fully.
Once I have the full compressed byte array I can get the incoming data like so:
var output = new MemoryStream();
using (var stream = new MemoryStream(bufferIn))
{
using (var decompress = new DeflateStream(stream, CompressionMode.Decompress))
{
decompress.CopyTo(output);;
}
}
output.Position = 0;
var unCompressedArray = output.ToArray();
output.Close();
output.Dispose();
return Encoding.UTF8.GetString(unCompressedArray);
I have created a simple server using socket programming in C# which will receive a file from the client side. My sample code segment is given below.
I want to add some restrictions. I want to make a limit on the file size (such as 4 KB or 2 KB) and allowable file formats (such as .doc, .txt, .cpp, etc.) which will be sent to the client as soon as the client connects to the server so that the client can send files accordingly. How will I do that?
Sample code segment:
using System;
using System.Collections.Generic;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Text;
namespace FileTransfer
{
class Program
{
static void Main(string[] args)
{
// Listen on port 1234
TcpListener tcpListener = new TcpListener(IPAddress.Any, 1234);
tcpListener.Start();
Console.WriteLine("Server started");
//Infinite loop to connect to new clients
while (true)
{
// Accept a TcpClient
TcpClient tcpClient = tcpListener.AcceptTcpClient();
Console.WriteLine("Connected to client");
byte[] data = new byte[1024];
NetworkStream ns = tcpClient.GetStream();
int recv = ns.Read(data, 0, data.Length);
StreamReader reader = new StreamReader(tcpClient.GetStream());
//Will add some lines to add restrictions...
}
}
}
}
Which additional lines will I have to add to the code to send the restrictions to client?
Basically I think mainly you need two things:
define application protocol as suggested in other answer
and handle partial read/writes
For handling partial reads (not sure how much such function is needed for write) you may use function like below:
public static void ReadWholeArray (Stream stream, byte[] data)
{
int offset=0;
int remaining = data.Length;
while (remaining > 0)
{
int read = stream.Read(data, offset, remaining);
if (read <= 0)
throw new EndOfStreamException
(String.Format("End of stream reached with {0} bytes left to read", remaining));
remaining -= read;
offset += read;
}
}
Thing is traditional Stream.Read() doesn't guarantee to read as many bytes as you told it, this method on the other hand, will ensure to have read as many bytes as specified in data.Length parameter. So you can use such function to implement the desired application protocol instead.
Some relevant information about such application protocols you will find here too
Ok this is for example how the server could send file length limit and the file extension:
// Send string
string ext = ".txt";
byte [] textBytes = Encoding.ASCII.GetBytes(ext);
ns.Write(textBytes, 0, textBytes.Length);
// Now, send integer - the file length limit parameter
int limit = 333;
byte[] intBytes = BitConverter.GetBytes(limit);
ns.Write(intBytes, 0, intBytes.Length); // send integer - mind the endianness
But you will still need some kind of protocol otherwise you should let client read the "full" stream and parse these data later somehow, which isn't trivial if the data doesn't have fixed length etc - otherwise how will the client distinguish which part of the message is text, which integer?
You seem to be making the classical socket mistake. The given code and explanation seem to assume sockets handle in messages. They don't. When used this way, you're using streaming internet sockets, which provide a stream, not messages.
You don't show any code that does the actual sending, so I'm guessing that you just pump a file's data to the other side and close the connection. How else will you know you've successfully transferred an entire file?
This set of rules that client and server have to follow in order to usefully exchange data through sockets is called an application protocol. You will have to have one, otherwise you'll just be sending data to $deity knows where, and you'll have no control over it at all. This means server nor client will know what's going on, they'll just be sending and receiving data and hoping all goes well. So there's not "a few lines" you have to add to your code, you'll have to restructure it entirely.
There are many ways to define an application protocol and many options to choose from, so I'm going to show you an arbitrary one: a textual explanation of messages that are prefixed with an ID and a payload length (if applicable), both in unspecified numeric variables. You could choose little-endian four-byte unsigned integers, for example.
Messages in this format are known as "Type/Length/Value" or TLV.
So we define these messages:
ID Name Direction Description Payload
1 ServerHello Server -> Client The server sends this message None.
to every connecting client. Or maybe server or
protocol version.
2 MaxUpload Server -> Client Sent after the ServerHello. Maximum upload size
in bytes.
3 AllowedExts Server -> Client Allowed upload extensions, The allowed extensions.
comma-separated. Sent after
MaxUpload message.
10 IncomingFile Client -> Server There's a file coming. The file name.
11 FileUpload Client -> Server The file to upload. The file data.
Sent after IncomingFile.
Now all that's required is to implement this application protocol in server and client and you're done.
You also have to decide what to do if a client or server doesn't adhere to the prototol. It can for example send a message that you can't parse, an unknown message ID, a message length that you don't want to support, an out-of-order message (FileUpload before IncomingFile) or a message that isn't conform the messages sent earlier, like a client uploading a larger file than the server said it would accept or an invalid extension. You also have to think about "acknowledgement" or response messages, like the server telling the client "OK, go ahead, send the next message".
All in all, this is a very broad question and not answered easily. I tried to address that in my comment to your question, which got removed. So here you have your answer.
You can learn more about this on the web, for example Beej's Guide to Network Programming as linked to by Giorgi (be sure to read the entire guide) and Stephen Cleary's blog.
Im writing an Server/Client Application which works with SSL(over SSLStream), which has to do many things(not only file receiving/sending). Currently, It works so: Theres only one connection. I always send the data from the client/server using SSLStream.WriteLine() and receive it using SSLStream.ReadLine(), because I can send all informations over one connection and I can send from all threads without destroying the data.
Now I wanted to implement the file sending and receiving. Like other things in my client/server apps, every message has a prefix (like cl_files or sth) and a base64 encoded content part(prefix and content are seperated by |). I implemented the file sharing like that: The uploader send to the receiver a message about the total file size and after that the uploader sends the base64 encoded parts of the file over the prefix r.
My problem is that the file sharing is really slow. I got around 20KB/s from localhost to localhost. I have also another problem. If I increase the size of the base64 encoded parts of the file(which makes file sharing faster), the prefix r doesnt go out to the receiver anymore(so the datas couldnt be identified).
How can I make it faster?
Any help will be greatly appreciated.
My(propably bad) code is for the client:
//its running inside a Thread
FileInfo x = new FileInfo(ThreadInfos.Path);
long size = x.Length; //gets total size
long cursize = 0;
FileStream fs = new FileStream(ThreadInfos.Path, FileMode.Open);
Int16 readblocks = default(Int16);
while (cursize < size) {
byte[] buffer = new byte[4096];
readblocks = fs.Read(buffer, 0, 4096);
ServerConnector.send("r", getBase64FromBytes(buffer));//It sends the encoded Data with the prefix r over SSLStream.WriteLine
cursize = cursize + Convert.ToInt64(readblocks);
ThreadInfos.wait.setvalue((csize / size) * 100);//outputs value to the gui
}
fs.Close();
For the Server:
case "r"://switch case for prefixes
if (isreceiving)
{
byte[] buffer = getBytesFromBase64(splited[1]);//splited ist the received Line over ReadLine splitted by the seperator "|"
rsize = rsize + buffer.LongLength;
writer.Write(buffer, 0, buffer.Length);//it writes the decoded data into the file
if (rsize == rtotalsize)//checks if file is completed
{
writer.Close();
}
}
break;
Your problem stems from the fact that you are performing what is essentially a binary operation through a text protocol and you are exacerbating that problem by doing it over an encrypted channel. I'm not going to re-invent this for you, but here are some options...
Consider converting to an HTTPS client/server model instead of reinventing the wheel. This will give you a well-defined model for PUT/GET operations on files.
If you can not (or will not) convert to HTTPS, consider other client/server libraries that provide a secure transport and well-defined protocol for binary data. For example, I often use protobuf-csharp-port and protobuf-csharp-rpc to provide a secure protocol and transport within our datacenter or local network.
If you are stuck with your transport being a raw SslStream, try using a well-defined and proven binary serialization framework like protobuf-csharp-port or protobuf-net to define your protocol.
Lastly, if you must continue with the framework you have, try some http-like tricks. Write a name/value pair as text that defines the raw-binary content that follows.
First of all base64 over ssl will be slow anyway, ssl itself is slower then raw transport. File transfers are not done over base64 now days, http protocol is much more stable than anything else and most libraries on all platforms are very well stable. Base64 takes more size then actual data, plus the time to encode.
Also, your following line may be a problem.
ThreadInfos.wait.setvalue((csize / size) * 100);//outputs value to the gui
If your this line is blocking, then this will slow down for every 4kb. Updating for every 4kb is also not right, unless a progress value from previous value differs by significant amount, there is no need to update ui for it.
I'd give a try of gzip compress before/after the network. From my experience, it helps. I'd say some code like this could help :
using(GZipStream stream = new GZipStream(sslStream, CompressionMode.Compress))
{
stream.Write(...);
stream.Flush();
stream.Close();
}
Warning : It may interfer with SSL if the Flush is not done. and it will need some tests... and I didn't try to compile the code.
I think Akash Kava is right.
while (cursize < size) {
DateTime start = DateTime.Now;
byte[] buffer = new byte[4096];
readblocks = fs.Read(buffer, 0, 4096);
ServerConnector.send("r", getBase64FromBytes(buffer));
DateTime end = DateTime.Now;
Console.Writline((end-start).TotalSeconds);
cursize = cursize + Convert.ToInt64(readblocks);
ThreadInfos.wait.setvalue((csize / size) * 100);
end = DateTime.Now;
Console.Writline((end-start).TotalSeconds);
}
By doing this you can find out where is the bottle neck.
Also the way you sending data packets to server is not robust.
Is it possible to paste your implementation of
ThreadInfos.wait.setvalue((csize / size) * 100);
This is my Server side code:
public void ReceivingData(object sender, EventArgs e)
{
while (mysocket.Connected)
{
buffer = new byte[accepted.SendBufferSize];
int bytesRead = accepted.Receive(buffer);
MemoryStream Data = new MemoryStream(buffer);
if ( picbox.InvokeRequired)
{
picbox.Invoke(new MethodInvoker(delegate { picbox.Image = Image.FromStream(Data); }));
}
}
}
The connection gets established and the file is being received without any issue. However the image gets distorted on Transfer. I do not understand why this is happening. Here is the screenshot:
I remember i had to format the strings which i used to send over sockets using Encoding.ASCII.GetString(StringToFormat). What do i need to do in case of Images?
In your ReceivingData callback you may not receive all the data back in one pop. Some data can be partially received and the rest of it in a subsequent (or multiple) callbacks and it will be your task to re-assemble the original message.
You will need to define a protocol to ensure that you have read all necessary data.
You could for example use base64 to encode the image on the server and decode it on the client. You would need to know how many bytes you should anticipate. This can be done, either by prefixing your response with the total bytes that the client should anticipate or by having a special marker (such as byte value 0x00) to distinguish message boundaries.
Using Base64 will also have the effect of increasing file sizes by 33% since base64 basically encodes every 6bits of the incoming stream to an 8bit readable character. So for every 3 'real' bytes you would like to transfer you will need 4 encoded bytes.
How can a C# app easily communicate with an instance of itself present on another computer, which is on the same network, and transfer files and data?
Assuming the computers on the network have fixed local IP addresses, and they each know each others IPs. Would there also be a way to communicate if the IPs are unknown? based on some discovery protocol?
I heard the "Bonjour" service from Apple was a good protocol. Can we communicate via it from our Windows apps? Or do you have to use "sockets". I'm primarily looking for libraries or sample code that can fulfill my need easily, I don't want to develop my own TCP-based protocol or anything hardcore!
You can use System.Net.Sockets class to communicate and it have a method for sending file Socket.SendFile.
Update:
this is a good example for file sharing and sending file from C# help
The great thing about files and sockets in C# is that they're both exposed as streams. Copying a large file from one stream to another is pretty simple:
byte[] data = new byte[1024];
while(true) {
int bytesRead = filestream.read(data,0,data.Length);
if (bytesRead==0) break;
netstream.write(data,0,bytesRead);
}
Then just close the socket when you're done.
If you want to send metadata (filenames, sizes) or don't want to close the connection, you need some sort of protocol to handle this. FTP uses two seperate sockets(one for metadata, one for data; this is called out-of-band communication). If you're on a LAN with no firewalls, that can be perfectly acceptable. On the other hand, if you want to do internet transfer, getting a single port open is a difficult-enough task, and two is unbearable. If you don't care too much about performance, you could encode the bytes in base64 encoding, which makes sure that they're within a certain byte range. With base64, you can seperate messages with newlines or other non-alphanumeric characters. Then in the first message include the filename, size, or whatever, then send the data as a second message, then send a "that's the whole file" message so the client knows it's done.
Another tactic for messages is using an escape sequence. For instance, take your bytestream and replace each instance of '\0' with '\0\0'. Now use '\0\1' to signal the end-of-message, which is guaranteed not to be contained in your data message. Decode the '\0\0' back to '\0' on the receiving end. This works well enough in C, but I find that, in practice, looping through each byte can be slower than reading whole buffers in C#.
The best way is to adopt some sort of adaptive-length protocol. For instance, send the data in chunks of a certain size (say 512 bytes). Before each chunk, send a 32bit int representing the size of the chunk via System.BitConverter. So messages look like this (english):
Here's 512 bytes:
[data]
Here's 512 bytes:
[data]
Here's 32 bytes:
[data]
Here's 4 bytes:
That was the whole file
The advantage here is that you can make the copy/read buffers work for you (reading 512 bytes at a time), meaning your throughput is limited by your network stack instead of your C# code. The client reads the fixed-length 32-bit int that lets it know the size of the buffer it should use for the next [data] segment.
Here's some code to write messages like that:
logger.logger.debug("Sending message of length " + length);
byte[] clength = System.BitConverter.GetBytes(buffer.Length);
plaintextStream.Write(clength,0,clength.Length);
plaintextStream.Write(buffer,0,buffer.Length);
plaintextStream.Flush();
And here's some code to read them:
byte[] intbuf = new byte[int_32_size];
int offset = 0;
while (offset < int_32_size)
{
int read = 0;
read = d.plaintextStream.Read(intbuf,offset,int_32_size - offset);
offset += read;
}
int msg_size = System.BitConverter.ToInt32(intbuf,0);
//allocate a new buffer to fill the message
byte[] msg_buffer = new byte[msg_size];
offset = 0;
while (offset < msg_size)
{
int read = 0;
read = d.plaintextStream.Read(msg_buffer,offset,msg_size - offset);
offset += read;
}
return msg_buffer;
For transferring the files / data, you can use the TcpClient/TcpListener classes, which is nice abstractions over the grittier socket functionality. Or, you could simply have the application as a HTTP server using the HttpListener class, if that is easier/more appropiate for your application.
For discovery, if you are able to have a central server; then you could have each client connect to the server at startup, to register itself and retrieve a list of other online clients and their IP's. Subsequent communication can then take place directly between the clients.
A variation of this scheme, is to let the central server act as a proxy, which all traffic between the clients flow through. This would be mostly helpful to overcome firewall or routing issues if the clients is not on the same network (so it's propably not needed for your scenario).
To copy files you may also want to look at File Synchronization Provider that is part of Microsoft Sync Framework. http://msdn.microsoft.com/en-us/sync/bb887623.