We Keep Coding

EndReceive in Async Socket without strings

I'm following this example about the creation of an async tcp listener in C#.
MSDN Example
I see that all data is encoded as string to check for message completeness. More precisely, every message sent is already a string, which we append the 'EOF' char to for string termination.
The server side part i'm talking about is in this snippet:
public static void ReadCallback(IAsyncResult ar) {
String content = String.Empty;
// Retrieve the state object and the handler socket
// from the asynchronous state object.
StateObject state = (StateObject) ar.AsyncState;
Socket handler = state.workSocket;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0) {
// There might be more data, so store the data received so far.
state.sb.Append(Encoding.ASCII.GetString(
state.buffer, 0, bytesRead));
// Check for end-of-file tag. If it is not there, read
// more data.
content = state.sb.ToString();
if (content.IndexOf("<EOF>") > -1) {
// All the data has been read from the
// client. Display it on the console.
Console.WriteLine("Read {0} bytes from socket. \n Data : {1}",
content.Length, content );
// Echo the data back to the client.
Send(handler, content);
} else {
// Not all data received. Get more.
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
}
Is there a way, as i usually do with TcpListener/TcpClient classes, to check if received bytes are available on the socket?
I mean something like this:
private void HandleClientConnection(TcpClient client)
{
NetworkStream clientStream = client.GetStream();
MemoryStream memoryStream = new MemoryStream();
while (true)
{
int read = clientStream.ReadByte();
if (read != -1)
{
memoryStream.WriteByte((byte)read);
}
else
{
break;
}
}
}
I'm aware that i probably misunderstood this example, or at least the Begin/End part and the "legacy" async pattern. But this is my goal, do you know some way to get it working without involving strings?

You said : "Is there a way to check if received bytes are available on the socket?"
In general 'EndReceive' will block the thread until data is available. So you don't need to do anything because 'EndReceive' is doing all the job for you.
'bytesRead' is an int that shows you how much data you have received.
a quote from docs.microsoft
The EndReceive method will block until data is available.1
But if you are using a SYNC socket (which you are not) then it's another topic.

async Telnet Server data receiving issues

I am writing a telnet server using the async Begin/End methods. The issue that I am having is determining what within my buffer is actual data and what is not. Network coding is a bit new to me, but I've tried to research this and have not been able to find a answer.
public bool Start(IGame game)
{
// Get our server address information.
IPHostEntry serverHost = Dns.GetHostEntry(Dns.GetHostName());
IPEndPoint serverEndPoint = new IPEndPoint(IPAddress.Any, this.Port);
// Instance the server socket, bind it to a port and begin listening for connections.
this._ServerSocket = new Socket(serverEndPoint.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
this._ServerSocket.Bind(serverEndPoint);
this._ServerSocket.Listen(this.MaxQueuedConnections);
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
return true;
}
private void Connect(IAsyncResult result)
{
var player = new BasePlayer();
try
{
player.Game = this.Game;
player.Connection = this._ServerSocket.EndAccept(result);
lock (this.Connections)
{
this.Connections.Add(player);
}
// Pass all of the data handling for the player to itself.
player.Connection.BeginReceive(player.Buffer, 0, player.BufferSize, SocketFlags.None, new AsyncCallback(player.ReceiveData), player);
// Fetch the next incoming connection.
this._ServerSocket.BeginAccept(new AsyncCallback(Connect), this._ServerSocket);
}
catch (Exception)
{
}
}
and then the player.ReceiveData..
public void ReceiveData(IAsyncResult result)
{
int bytesRead = this.Connection.EndReceive(result);
if (bytesRead > 0)
{
// TODO: Parse data received by the user.
//Queue the next receive data.
this.Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None, new AsyncCallback(ReceiveData), this);
var str = System.Text.Encoding.Default.GetString(Buffer);
}
else
{
this.Disconnect(result);
}
}
So when I call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
I imagine there is a better way to do this? Can someone please point me in the right direction as to how I should determine what the data is within my buffer or perhaps a way that I can do this without having to use a predetermined buffer size.

So when call BeginReceive, I need to provide a buffer of a predetermined size. In doing that, I end up with unused bytes in my buffer array. They all have the value of 0, so I am assuming that I can loop through the array and build a new one starting at index 0 and working until I hit a value of 0.
No, that's not what you should do. Instead, in your callback (ReceiveData) you're already calling EndReceive - and the result of that is the number of bytes you read. That's how much of the buffer you should use.
However, you should copy the data you've read out of the buffer before you call BeginReceive again, otherwise you may end up with the next bit of data overwriting the just-read data before you get to use it.
So something like:
string text = Encoding.ASCII.GetString(Buffer, 0, bytesRead);
Connection.BeginReceive(this.Buffer, 0, this.BufferSize, SocketFlags.None,
new AsyncCallback(ReceiveData), this);
I would not suggest that you use Encoding.Default to convert the bytes to text - instead, you should decide which encoding you're using, and stick to that. If you use an encoding which isn't always one-byte-per-character, you'll end up in a slightly trickier situation, as then you might end up receiving a buffer with part of a character. At that point you need to keep a Decoder which can maintain state about partial characters read.

How to read all requested data using NetworkStream.BeginRead?

There is a code of async server. Client sends Header - size of Data Block + Data Block.
Server reads asynchronously first Header and then Data Block.
I need, after I read Data Block run the BeginRead for Header reading part, to make threads async.
PROBLEM:
When I got DataCallBack, in line:
int bytesRead = ns.EndRead(result);
I get not all buffer i asked to read in
mc.Client.GetStream().BeginRead(mc.DataBuffer, 0, size, new AsyncCallback(DataCallBack), mc);
If client send 1MB of Data I can get different number of "bytesRead".
QUESTION:
How to force "BeginRead" to read all data from connection. It should cause the new loop of Header - Data.
MyClient - simply wrapper over TcpClient;
CODE:
public void DoAcceptTcpClientCallback(IAsyncResult ar)
{
TcpListener listener = (TcpListener)ar.AsyncState;
TcpClient client = listener.EndAcceptTcpClient(ar);
client.NoDelay = false;
// client.ReceiveBufferSize = 1024*1024;
listener.BeginAcceptTcpClient(new AsyncCallback(DoAcceptTcpClientCallback), listener);
MyClient mc = new MyClient(client);
ContinueRead(0,mc);
}
public void ContinueRead(int size, MyClient mc)
{
if (size != 0)
{
mc.DataBuffer = new byte[size];
mc.Client.GetStream().BeginRead(mc.DataBuffer, 0, size, new AsyncCallback(DataCallBack), mc);
}
mc.Client.GetStream().BeginRead(mc.HeaderBuffer, 0, 4, new AsyncCallback(HeaderCallBack), mc);
}
private void HeaderCallBack(IAsyncResult result)
{
MyClient mc = (MyClient)result.AsyncState;
NetworkStream ns = mc.Stream;
int bytesRead = ns.EndRead(result);
if (bytesRead == 0)
throw new Exception();
mc.TotalLengs = BitConverter.ToInt32(mc.HeaderBuffer, 0);
ContinueRead(mc.TotalLengs, mc);
}
private void DataCallBack(IAsyncResult result)
{
MyClient mc = (MyClient)result.AsyncState;
NetworkStream ns = mc.Stream;
int bytesRead = ns.EndRead(result);
if (bytesRead == 0)
throw new Exception();
BAD CODE - MAKES ASYNC READING - SYNC
while (bytesRead < mc.TotalLengs)
{
bytesRead += ns.Read(mc.DataBuffer, bytesRead, mc.TotalLengs - bytesRead);
}
END BAD CODE
ContinueRead(0, mc);
ProcessPacket(mc.DataBuffer, mc.IP);
}

"If client send 1MB of Data I can get different number of "bytesRead"."
Yes...this is simply how TCP works under the hood. You can't change this. TCP guarantees the order of packets, not how they are grouped. The hardware and traffic conditions along the route the packets travel determine how that data is grouped (or un-grouped).
"How to force "BeginRead" to read all data from connection."
TCP has no idea how much data is being sent. As far as it is concerned, the connection is simply an endless stream of bytes; therefore it cannot read "all data" since there is no end to the data (from its perspective). TCP also has no notion of what a "complete message" is with respect to your application. It is up to you, the programmer, to develop a protocol that allows your application to know when all data has been sent.
If you are expecting a certain number of bytes, then keep a running sum of the values returned by EndRead() and stop when that magic number is hit.

How to determine the end of a file transfer (data) from the client to the server (Socket)

Simple situation:
The client on a Socket sends a pieces(e.g., 256 byte) of the file (data) in the format byte []to the server. The server receives the data asynchronously.
How to determine when a file (data) is transmitted completely? (Server-side)
Here's the code on server-side responsible for receiving data
public static void ReadCallback(IAsyncResult ar)
{
String content = String.Empty;
// Retrieve the state object and the handler socket
// from the asynchronous state object.
StateObject state = (StateObject)ar.AsyncState;
Socket handler = state.workSocket;
// Read data from the client socket.
int bytesRead = handler.EndReceive(ar);
if (bytesRead > 0)
{
BinaryWriter writer = new BinaryWriter(File.Open(#"D:\test.png", FileMode.Append));
writer.Write(state.buffer, 0, bytesRead);
writer.Close();
// All the data has been read from the
// client. Display it on the console.
Console.WriteLine("Read {0} bytes from socket!",
bytesRead);
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
Is there a method that allows to make the some following?
if (bytesRead > 0)
{
....
if(state.buffer!=end of receive)
{
handler.BeginReceive(state.buffer, 0, StateObject.BufferSize, 0,
new AsyncCallback(ReadCallback), state);
}
}
Or, I may to try add some information to the this byte[] object (e.g., some string with <EOF> tag)
but I must to analyse this info on each step.
May I do this check more simple and how? Or use another way ...

The only way to do it is to sent a header (of specific size) in front of every message. So, the each message should consist of header and body. The data stream should look like this:
[HEADER][BODY][HEADER][SOME BIGGER BODY][HEADER][SOME EXTRA BIG BODY]
As I said, the header should be of specific size and should contain some custom service fields inlcuding the size of the message's body in bytes. In your case the header could contain only body size, i.e. int value (4 bytes). The receive process should look like this:
Receive 4 bytes (header);
Retrieve body size from header (just convert header to int);
Receive the number of bytes specified in header (i.e. receive message body);
Handle message body;
Go to 1.
I know, it may seem complicated for you, but it is the common way to do it. But you can simplify the code by using Rx library. After implementing some extensions methods for socket (WhenReadExact, the implementation could easily be found over Internet, for example here), the code will look like this:
var whenFileReceived = from header in socket.WhenReadExact(4)
let bodySize = BitConverter.ToInt32(header)
from body in socket.WhenReadExact(bodySize)
select body;
whenFileReceived.Subscribe(
file =>
{
// Handle file here
});

Good way to send a large file over a network in C#?

I am trying to build an application that can request files from a service running on another machine in the network. These files can be fairly large (500mb + at times). I was looking into sending it via TCP but I'm worried that it may require that the entire file be stored in memory.
There will probably only be one client. Copying to a shared directory isn't acceptable either. The only communication required is for the client to say "gimme xyz" and the server to send it (and whatever it takes to ensure this happens correctly).
Any suggestions?

Here is an easier way. Using BITS (Background Intelligent Transfer Service). Its already built into WinXP and Vista. Its basically what drives Windows Updates.
http://blogs.msdn.com/powershell/archive/2009/01/11/transferring-large-files-using-bits.aspx
http://blogs.msdn.com/jamesfi/archive/2006/12/23/how-to-use-bits-to-transfer-files.aspx
Here is a nice managed BITS wrapper someone wrote and how to use it.
http://www.codeproject.com/KB/cs/Managed_BITS.aspx

You can use sockets in .NET to transfer files and data.

You might want to consider WCF streaming.

This article may help you. It is about sending large files in .NET.
Check the link:
http://codetechnic.blogspot.com/2009/02/sending-large-files-over-tcpip.html

Be careful with BITS. It is a very good protocol but not a critical part of the windows update program. We found that hardly any of our corporate clients allowed the BITS update onto their machines; therefore we couldn't build an app that relied on it.

If FTP was an option then I'd go with that for the sake of simplicity. Otherwise you're into a world of TCP/IP socket programming.

Use FTP via the open source edtFTPnet library. Fast and simple.

Use TransmitFile (which is a Win32 function; perhaps it's a method of the .NET library as well).

If files exist physically on the machine why not just put them in a folder, make that folder a virtual directory in IIS, and use Content-Based Routing and/or URL Rewriting to route the requests to them.

Personally I'd go for something that balances speed, reliability and economical code, so I'd base it on a TCP network stream. The client-side of the code would look like this:
internal class Client
{
private FileStream _fs;
private long _expectedLength;
public void GetFileFromServer(string localFilename)
{
if (File.Exists(localFilename))
File.Delete(localFilename);
_fs = new FileStream(localFilename, FileMode.Append);
var ipEndpointServer = new IPEndPoint(IPAddress.Parse({serverIp}), {serverPort});
// an object that wraps tcp client
var client = new TcpClientWrapper(ipEndpointServer, "");
client.DataReceived += DataReceived;
}
private void DataReceived(object sender, DataReceivedEventArgs e)
{
var data = e.Data;
// first packet starts with 4 bytes dedicated to the length of the file
if (_expectedLength == 0)
{
var headerBytes = new byte[4];
Array.Copy(e.Data, 0, headerBytes, 0, 4);
_expectedLength = BitConverter.ToInt32(headerBytes, 0);
data = new byte[e.Data.Length - 4];
Array.Copy(e.Data, 4, data, 0, data.Length);
}
_fs.WriteAsync(e.Data, 0, e.Data.Length);
if (_fs.Length >= _expectedLength)
{
// transfer has finished
}
}
}
Then have a server class to serve the file. Note how the whole file isn't loaded into memory, instead it is read in chunks from a FileStream.
internal class Server
{
private TcpServer _tcpServer;
private NetworkStream _stream;
public void StartServer()
{
// fire up a simple Tcp server
_tcpServer = new TcpServer({serverPort}, "test");
_tcpServer.ClientConnected += ClientConnected;
}
private void ClientConnected(object sender, TcpClientConnectedEventArgs e)
{
// an incoming client has been detected ... send the file to that client!
_stream = e.Client.GetStream();
SendFileToClient({pathToFile});
}
private void SendFileToClient(string pathToFile)
{
// open the file as a stream and send in chunks
using (var fs = new FileStream(pathToFile, FileMode.Open))
{
// send header which is file length
var headerBytes = new byte[4];
Buffer.BlockCopy(BitConverter.GetBytes(fs.Length + 4), 0, headerBytes, 0, 4);
_stream.Write(headerBytes, 0, 4);
// send file in block sizes of your choosing
var buffer = new byte[100000];
int bytesRead = 0;
while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)
{
_stream.Write(buffer, 0, bytesRead);
}
_stream.Flush();
}
}
}
The TcpClientWrapper is pretty much boiler plate code with the System.Net.Sockets.TcpClient object and the underlying NetworkStream object. I don't really need to post this as well, but just to give some pointers ther construction would contain something like this:
_tcp = new Net.TcpClient();
_tcp.Connect(remoteEp);
_stream = _tcp.GetStream();
_stream.BeginRead(_receivedData, 0, _receivedData.Length, DataReceivedAsync, null);
and the DataReceivedAsync method is boilerplate socket data handling and would raise an event o share the received data back to the consumer (the client in this case):
private void DataReceivedAsync(IAsyncResult ar)
{
var receivedBytes = _stream.EndRead(ar);
if (receivedBytes > 0)
{
var data = new byte[receivedBytes];
Array.Copy(_receivedData, 0, data, 0, receivedBytes);
DataReceived?.Invoke(this, new DataReceivedEventArgs(data));
_receivedData = new byte[ReceiveBufferSize];
_stream.BeginRead(_receivedData, 0, _receivedData.Length, DataReceivedAsync, null);
}
}
The event to ship data from the wrapper back to the client:
public EventHandler<DataReceivedEventArgs> DataReceived;
public class DataReceivedEventArgs : EventArgs
{
public DataReceivedEventArgs(byte[] data) { Data = data; }
public byte[] Data { get; }
}