We Keep Coding

TcpListener + TcpClient - wait for client to read data before closing

I am building a simple HTTP server for PDF files with TcpClient. It works well, however the TcpClient closes before the browser is downloading of the PDF is being finished. How can I force TcpClient to wait until the remote client get everything that is written before closing?
//pdf is byte[]
TcpListener server = new TcpListener(address, port);
server.Start();
TcpClient client = server.AcceptTcpClient(); //Wait for connection
var ns = client.GetStream();
string headers;
using (var writer = new StringWriter())
{
writer.WriteLine("HTTP/1.1 200 OK");
//writer.WriteLine("Accept: text/html");
writer.WriteLine("Content-type: application/pdf");
writer.WriteLine("Content-length: " + pdf.Length);
writer.WriteLine();
headers = writer.ToString();
}
var bytes = Encoding.UTF8.GetBytes(headers);
ns.Write(bytes, 0, bytes.Length);
ns.Write(pdf, 0, pdf.Length);
Thread.Sleep(TimeSpan.FromSeconds(10)); //Adding this line fixes the problem....
client.Close();
server.Stop();
Can I replace that ugly 'Thread.Sleep' hack?
EDIT: The code below works, based on the answers:
TcpListener Server = null;
public void StartServer()
{
Server = new TcpListener(IPAddress.Any, Port);
Server.Start();
Server.BeginAcceptTcpClient(AcceptClientCallback, null);
}
void AcceptClientCallback(IAsyncResult result)
{
var client = Server.EndAcceptTcpClient(result);
var ns = client.GetStream();
string headers;
byte[] pdf = //get pdf
using (var writer = new StringWriter())
{
writer.WriteLine("HTTP/1.1 200 OK");
//writer.WriteLine("Accept: text/html");
writer.WriteLine("Content-type: application/pdf");
writer.WriteLine("Content-length: " + pdf.Length);
writer.WriteLine();
headers = writer.ToString();
}
var bytes = Encoding.UTF8.GetBytes(headers);
ns.Write(bytes, 0, bytes.Length);
ns.Write(pdf, 0, pdf.Length);
client.Client.Shutdown(SocketShutdown.Send);
byte[] buffer = new byte[1024];
int byteCount;
while ((byteCount = ns.Read(buffer, 0, buffer.Length)) > 0)
{
}
client.Close();
Server.Stop();
}

The main issue in your code is that your server (the file host) neglected to read from the socket it's writing the file to, and so has no way to detect, never mind wait for, the client shutting down the connection.
The code could be way better, but at a minimum you could probably get it to work by adding something like this just before your client.Close(); statement:
// Indicate the end of the bytes being sent
ns.Socket.Shutdown(SocketShutdown.Send);
// arbitrarily-sized buffer...most likely nothing will ever be written to it
byte[] buffer = new byte[4096];
int byteCount;
while ((byteCount = ns.Read(buffer, 0, buffer.Length)) > 0)
{
// ignore any data read here
}
When an endpoint initiates a graceful closure (e.g. by calling Socket.Shutdown(SocketShutdown.Send);), that will allow the network layer to identify the end of the stream of data. Once the other endpoint has read all of the remaining bytes that the remote endpoint has sent, the next read operation will complete with a byte length of zero. That's that other endpoint's signal that the end-of-stream has been reached, and that it's time to close the connection.
Either endpoint can initiate the graceful closure with the "send" shutdown reason. The other endpoint can acknowledge it once it's finished sending whatever it wants to send by using the "both" shutdown reason, at which point both endpoints can close their sockets (or streams or listeners or whatever other higher-level abstraction they might be using the wrap the socket).
Of course, in a properly-implemented protocol, you'd know in advance whether any data would ever actually be sent by the remote endpoint. If you know none ever will be, you could get away with a zero-length buffer, and if you know some data might be sent back from the client, then you'd actually do something with that data (as opposed to the empty loop body above).
In any case, the above is strictly a kludge, to get the already-kludged code you posted to work. Please don't mistake it for something intended to be seen in production-quality code.
All that said, the code you posted is a long way from being all that good. You aren't just implementing a basic TCP connection, but apparently are trying to reimplement the HTTP protocol. There's no point in doing that, as .NET already has HTTP server functionality built in (see e.g. System.Net.HttpListener). If you do intend to reinvent the HTTP server, you need a lot more code than the code you posted. The lack of error-handling alone is a major flaw, and will cause all kinds of headaches.
If you intend to write low-level network code, you should do a lot more research and experimentation. One very good resource is the Winsock Programmer’s FAQ. It's primary focus is, of course, programmers targeting the Winsock API. But there is a wealth of general-purpose information there as well, and in any case all of the various socket APIs are very similar, as they are all based on the same low-level concepts.
You may also want to review various existing Stack Overflow Q&A. Here are a couple of ones closely related to your specific problem:
How to correctly use TPL with TcpClient?
Send a large file over tcp connection
Do be careful though. There's almost as much bad advice out there as good. There's no shortage of people going around acting like they are experts in network programming when they aren't, so take everything you read with a grain of salt (including my advice above!).

C# TcpClient and NetworkStream to send/receive data over TCP. Why not using "using" block and can I read data right after writing it to stream?

I am reading documentation on using TcpClient and NetworkStream to write and read from a Tcp connection stream. But I notice that all Microsoft's documentation is not using "using" block even thought both TcpClient and NetworkStream implement IDisposable.
So, I was wondering why is this.
I also notice in few examples that they write to stream and then immediately after, read from the stream. This does not feel right but I am new to networking but I wrote something like the code below to do exactly that.
While the code works, I am not sure how good and reliable it is though. Specifically, if data is not available to be read, once I check for it, the execution will move on. At that point, data may be available for reading but then it is to late, I already passed that point.
My questions are capitalized as code comments below
using (TcpClient client = new TcpClient()) // WHY NOT USING?
{
client.Connect(ip, port);
using (NetworkStream stream = client.GetStream()) // WHY NOT USING?
{
byte[] messageBytes = ...;
if (stream.CanWrite)
{
await stream.WriteAsync(messageBytes, 0, messageBytes.Length);
stream.Flush();
}
// IS THIS DATA GOING TO BE AVAILABLE AT THIS POINT?
// IF NOT, THEN YOU WILL HAVE NO OPPORTUNITY TO READ DATA
// SINCE NOTHING IS GOING TO TELL YOU WHEN IT IS GOING TO BE AVAILABLE.
// BY THE TIME IT IS AVAILABLE, THE EXECUTION MAY PASS THIS POINT, WHAT THEN???
if (stream.CanRead)
{
byte[] buffer = newe byte[1024];
int readSoFar = 0;
StringBuilder builder = new StringBuilder();
while(stream.DataAwailable)
{
readSoFar = await stream.ReadAsync(buffer, 0, buffer.Length);
builder.AppendFormat("{0}", Encoding.ASCII.GetString(buffer, 0, readSoFar));
}
string msg = builder.ToString();
}
}
}

TcpClient/NetworkStream/Socket have only one unmanaged resource under the hood. It is socket itself. Your code can have only 1 using block, and you will be ok. In your example, when code leaves using (NetworkStream) block, it closes underlying socket, and TcpClient.Dispose actually does nothing.
About Read/ReadAsync. The data is not available immediately. But these methods wait for one of 2 events: data appears on the socket, or other party closes tcp socket. As soon as 1 byte appears on the socket, these functions immediately return result. If another party closes connection, these functions return 0, which actually indicates end of stream.
Usually you shouldn't use DataAvailable property. In your example it can lead to unexpected behavior. You can just use Read/ReadAsync until they return zero.

issue with getting TcpClient server response as string

so long story short, i am trying to develop an application which requires having Tcp Connection to a server. having no previous experiance in this field i ended up with having none of my functions working. so i decided to make a simple console app just to check my commands and their responses. the part about setting up the connections went well, so does the part about wrting into network stream(i think) but i hit a wall when trying to display server respons:
each time my porgram reaches the line where it has to Encode server respons and WriteLine, my console application goes all black and all texts goes away. can you tell what's wrong with the code i am using?everything up until the part where console trys to display response goes well, i checked. i want to have server's respones as normal strings(if possible i guess...)
static void ServerTalk(TcpClient tcpClient,NetworkStream networkStream, StreamWriter streamWriter, StreamReader streamReader)
{
//Messaging methods:
Console.WriteLine("Client: ");
string message = Console.ReadLine();
streamWriter.WriteLine(message);
streamWriter.Flush();
//Response methode:
byte[] data = new byte[tcpClient.ReceiveBufferSize];
int ret = networkStream.Read(data, 0, data.Length);
string respond = Encoding.ASCII.GetString(data).TrimEnd();
Console.WriteLine("Server: ");
Console.WriteLine(respond);
ServerTalk(tcpClient, networkStream, streamWriter, streamReader);
}

you need to process only ret bytes, you are translating all bytes in the array.
copy to a new array that is 'ret' long and then decode.
var retarr = new byte[ret];
Array.Copy(data,retarr,ret);
string respond = Encoding.ASCII.GetString(retarr);
More importantly note that this method of tcp communication is not going to work. You are assuming that when you send x bytes in one block you will receive x bytes in one block, this is not so, you can send 1 100 byte message and receive 100 1 byte messages
This means that you task is much more complex. YOu need to devise a way of sending self describing data - so that you know its completely received. classically you send a length followed by the data. You then keep looping in the receive code till you have received that many bytes

Send / receive file over TCP [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
TcpClient send data and receive data over network
Loop until TcpClient response fully read
I am trying to send a file from a server to a client over TCP.
Server-side code, sending file:
NetworkStream netStream = client.GetStream();
FileStream fs = new FileStream("usb.exe",FileMode.Open, FileAccess.Read);
byte[] data = new byte[fs.Length];
fs.Read(data,0, data.Length);
fs.Flush();
fs.Close();
netStream.Write(data, 0, data.Length);
netStream.Flush();
Client-side code, receive file:
FileStream str = new FileStream("usb.exe", FileMode.Create, FileAccess.Write);
byte[] data = new byte[1024];
while ((dataCitit = netStream.Read(data,0, data.Length)) > 0)
{
Thread.Sleep(25);
Application.DoEvents();
str.Write(data, 0, dataCitit);
totalbytes += dataCitit;
}
str.Close();
Can someone point where I am getting it wrong ?
The file has 1036 kb, and it sends only 1032 kb and then gets stuck it won't get out the while loop on the client-side.
Also if I close the server and open it really quick it sends the last bytes and the files sends completely. (this file opens perfectly)
I think it`s a problem on the server side not sending all the bytes but why and where...

Well this is a problem in your server-side code to start with:
fs.Read(data,0, data.Length);
You're ignoring the value returned by Read. Never do that. With FileStream you're possibly okay, but I personally wouldn't trust it anyway. If you're using .NET 4, you don't need to do this anyway - just use Stream.CopyTo.
On the client-side code, your biggest initial problem is that you're doing all of this on the UI thread. That's a terrible idea - the UI will freeze if there's a network glitch, as the Read call is blocking.
Again, just use Stream.CopyTo, but do it in a background thread.
Additionally, in all of these cases, use a using statement for the streams, so that you close them cleanly whatever happens.
That's all just general hygiene. Now, as for why you're hanging...
... you're not closing the network stream on the server side. Therefore you never reach the end of the stream on the client side. If you only need to use the connection for a single file, then the answer is simple: just close the connection on the server side.
If, however, you need to use the same connection for multiple files, then you need more protocol - you need to some way of indicating the end of the data. There are three common ways of doing that:
Write the length of the data before the data itself, then on the reading side, first read the length, then read that many bytes, failing if the stream finishes before you've done so. This requires that you know how much data you're going to write before you start writing.
Use an "end of data" marker which you can detect on the reading side; this is a pain in general, as it requires escaping the marker if it appears in the text itself.
A variation on the first approach, where you write a length-prefixed chunk at a time, then a zero-length chunk to indicate "end of data". This is pretty flexible, but obviously a bit more work than the first approach if the first approach actually works for you.

how to write the stream to memory stream?

public void doprocess(TcpClient client)
{
MemoryStream ms = new MemoryStream();
Stream clStream = client.GetStream();
byte[] buffer_1 = new byte[8192];
int count = clStream.Read(buffer_1, 0, buffer_1.Length);
while (count > 0)
{
ms.Write(buffer_1, 0, count);
//the below line doesn't gives response and code hangs here
count = clStream.Read(buffer_1, 0, buffer_1.Length);
}
}
Is there any other way to write one stream to another? I want to use this Stream twice, which is why I need to write it to the MemoryStream.

In .NET 4 the copying part is really easy:
MemoryStream ms = new MemoryStream();
client.GetStream().CopyTo(ms);
If you're not using .NET 4, then code similar to what you've already got is basically the same thing.
However, note that this (and any other attempt) will only work if the network stream has been closed - otherwise the Read call will block waiting for more data. I suspect that's what's going wrong for you - and it's a fundamental problem.
If your network stream isn't closed, then it's unclear how you'd really want it to behave - should the two readers of the "split" stream basically read any buffered data, but then block until there's new data otherwise? The buffered data could be removed when it's been read from both streams, of course. If that is what you're after, I don't think there's anything in .NET to help you particularly - and it's probably pretty tricky.
If you could give us more context of what you're trying to do (and what the TcpClient is connecting to) that would really help.

The reason why your code hangs on clStream.Read is because you are tying to read 8192 bytes from the socket but on the other side no-one is writing that many bytes. So the client just sits there and waits for the other side to send the required number of bytes. Depending on the protocol you are trying to implement over TCP there must be some indincation from the server how much data it intends to send so that the client knows in advance and only tries to read that many bytes.
For example in the HTTP protocol the server sends in the headers the Content-Length header to indicate to the clients how much data is going to be sent in the body.