HTTPS proxy implementation, how to detect a completed request - c#

I'm attempting to write a simple async https proxy server in c#.
I would like to know how I should detect/handle when the request is complete, and how to exit my bActive loop, assuming a loop like this is appropriate.
Would really appreciate some pointers on if my approach is correct and what I could do to improve the logic.
The issue I seem to be running into is that the time it takes for an endpoint to respond along with the network delay means I DataAvailable doenst always have data but there may still be some sending. Requiring a sleep and another attmempt which in turn causes the long completion time in requests.
Listen for TCP connection
Extract CONNECT header and open a connection to the requested server
Copy the requestStream to proxyStream
Copy the proxyStream to the requestStream
Sleep waiting for data and repeat 3 - 4 until no data is avaiable on both streams. Then break out of the loop and close connection.
public async Task Start()
{
listener.Start();
while (listen)
{
if (listener.Pending())
{
HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100); //<--- timeout
}
}
}
private static async Task HandleClient(TcpClient clt)
{
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
int count;
const string connectText = "connect";
const string hostText = "Host: ";
bool bActive = true;
List<Task> tasks = new List<Task>();
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
proxyStream.ReadTimeout = 100;
proxyStream.WriteTimeout = 100;
while (bActive)
{
if (proxyStream.DataAvailable && hostHeaderAvailable == 0)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
requestStream.ReadTimeout = 100;
requestStream.WriteTimeout = 100;
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
// delay here seems to prevent the following proxyStream.read from failing as data is not yet avaiable
// without it the loop runs and has to timeout before running again
await Task.Delay(1);
}
}
hostHeaderAvailable++;
if (requestStream == null || !requestClient.Connected || !clt.Connected)
{
bActive = false;
break;
}
Console.WriteLine(proxyStream.DataAvailable || requestStream.DataAvailable);
if (proxyStream.DataAvailable || requestStream.DataAvailable)
{
Task task = proxyStream.CopyToAsync(requestStream);
Task task2 = requestStream.CopyToAsync(proxyStream);
tasks.Add(task);
tasks.Add(task2);
await Task.WhenAll(tasks).ConfigureAwait(false);
bActive = false;
break;
}
await Task.Delay(10);
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
clt.Close();
}
An older attempt that used ReadAsync/WriteAsync too longer to response and still had the timeout issue.
Listen for TCP connection
Extract CONNECT header and open a connection to the requested server
Read data from requestStream and copy to proxyStream
Wait checking if data is avaiable on either stream
If data avaiable read from proxyStream and write to requestStream
If data avaiable read from requestStream and write to proxyStream
Sleep waiting for data and repeat 5 - 6 until no data is avaiable on eitboth streams. Then break out of the loop and close connection.
private static TcpListener listener = new TcpListener(IPAddress.Parse("192.168.0.25"), 13000);
private static bool listen = true;
public async Task Start()
{
listener.Start();
while (listen)
{
if (listener.Pending())
{
await HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100);
}
}
}
private static async Task HandleClient(TcpClient clt)
{
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
int count;
const string connectText = "connect";
const string hostText = "Host: ";
bool bActive = true;
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
while (bActive)
{
while (proxyStream.DataAvailable)
{
// handle connect
if (hostHeaderAvailable == 0)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
// delay here seems to prevent the following proxyStream.read from failing as data is not yet avaiable
// without it the loop runs and has to timeout before running again
await Task.Delay(20);
}
}
hostHeaderAvailable++;
if (requestClient.Connected && hostHeaderAvailable > 1)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
await requestStream.WriteAsync(bytes, 0, count);
}
}
while (requestStream.DataAvailable)
{
count = await requestStream.ReadAsync(bytes, 0, bytes.Length);
await proxyStream.WriteAsync(bytes, 0, count);
}
// attempt to detect a timeout / end of data avaiable
var timeout = 0;
while (!proxyStream.DataAvailable && !requestStream.DataAvailable)
{
if (timeout > 5)
{
bActive = false;
break;
}
await Task.Delay(10);
timeout++;
}
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
UPDATE
As per AgentFire's answer I have now come to the following working code:
public static async Task HandleDisconnect(TcpClient tcp, TcpClient tcp2, CancellationToken cancellationToken)
{
while (true)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
Console.WriteLine("The requesting client has dropped its connection.");
cancellationToken = new CancellationToken(true);
break;
}
}
if (tcp2.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp2.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Server disconnected
Console.WriteLine("The destination client has dropped its connection.");
cancellationToken = new CancellationToken(true);
break;
}
}
await Task.Delay(1);
}
}
private static async Task HandleClient(TcpClient clt)
{
List<Task> tasks = new List<Task>();
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
const string connectText = "connect";
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
proxyStream.ReadTimeout = 100;
proxyStream.WriteTimeout = 100;
if (proxyStream.DataAvailable && hostHeaderAvailable == 0)
{
await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
requestStream.ReadTimeout = 100;
requestStream.WriteTimeout = 100;
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
}
}
hostHeaderAvailable++;
CancellationToken cancellationToken = new CancellationToken(false);
Task task = proxyStream.CopyToAsync(requestStream, cancellationToken);
Task task2 = requestStream.CopyToAsync(proxyStream, cancellationToken);
Task handleConnection = HandleDisconnect(clt, requestClient, cancellationToken);
tasks.Add(task);
tasks.Add(task2);
tasks.Add(handleConnection);
await Task.WhenAll(tasks).ConfigureAwait(false);
// close conenctions
clt.Close();
clt.Dispose();
requestClient.Close();
requestClient.Dispose();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
UPDATE
Attempt at using CancellationTokenSource
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken cancellationToken = source.Token;
TaskFactory factory = new TaskFactory(cancellationToken);
tasks.Add(factory.StartNew(() => {proxyStream.CopyToAsync(requestStream);}, cancellationToken));
tasks.Add(factory.StartNew(() => {requestStream.CopyToAsync(proxyStream);}, cancellationToken));
tasks.Add(factory.StartNew(async () => {
//wait for this to retur, then cancel the token
await HandleDisconnect(clt, requestClient);
source.Cancel();
}, cancellationToken));
try
{
await factory.ContinueWhenAll(tasks.ToArray(),
(results) =>
{
Console.WriteLine("Tasks complete");
}, cancellationToken);
}
catch (AggregateException ae)
{
foreach (Exception e in ae.InnerExceptions)
{
if (e is TaskCanceledException)
Console.WriteLine("Unable to compute mean: {0}",
((TaskCanceledException)e).Message);
else
Console.WriteLine("Exception: " + e.GetType().Name);
}
}
finally
{
source.Dispose();
}
UPDATE
public static class extensionTcpClient{
public static bool CheckIfDisconnected(this TcpClient tcp)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
return false;
}
}
return true;
}
}
class ProxyMaintainer
{
private static TcpListener listener = new TcpListener(IPAddress.Parse("192.168.0.25"), 13000);
public ProxyMaintainer()
{
}
public async Task Start()
{
Console.WriteLine("###############################");
Console.WriteLine("Listening on 192.168.0.25:13000");
Console.WriteLine("###############################\n");
listener.Start();
while (listen)
{
if (listener.Pending())
{
HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100); //<--- timeout
}
}
}
private static async Task Transport(NetworkStream from, NetworkStream to, Func<bool> isAlivePoller, CancellationToken token)
{
byte[] buffer = new byte[4096];
while (isAlivePoller())
{
while (from.DataAvailable)
{
int read = await from.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false);
await to.WriteAsync(buffer, 0, read, token);
}
// Relieve the CPU a bit.
await Task.Delay(10, token).ConfigureAwait(false);
}
}
private static async Task HandleClient(TcpClient clientFrom)
{
var hostHeaderAvailable = 0;
int count;
var bytes = new byte[clientFrom.ReceiveBufferSize];
const string connectText = "connect";
NetworkStream toStream = null;
using (var fromStream = clientFrom.GetStream())
using(TcpClient clientTo = new TcpClient())
using (var manualStopper = new CancellationTokenSource())
{
count = await fromStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await clientTo.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
toStream = clientTo.GetStream();
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await fromStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
}
bool Poller() => clientFrom.CheckIfDisconnected() && clientTo.CheckIfDisconnected();
Task one = Transport(fromStream, toStream, Poller, manualStopper.Token);
Task two = Transport(toStream, fromStream, Poller, manualStopper.Token);
await Task.WhenAll(one, two).ConfigureAwait(false);
//await one; await two; // To get exceptions if you want them and there are any.
// Alternatively, you can use Task.WhenAll to get exceptions aggregated for you.
}
Console.WriteLine("Closing connection");
}
}

Well, tell you what. The data availability, when it comes to HTTP, lies only in one parameter (if we omit things like WebSocket), which is called Connection and is passed as a Header as a one of two possible states: Close or Keep-Alive.
If Close is chosen by the client, the server is obliged to close the conection as soon as the request is served, whereas Keep-Alive tells the server that, if it doesn't want to, it may leave connection open for another request.
Let's consider both cases.
If client chooses Keep-Alive, the connection will persist and work as intended, indefinetely. But:
If either side drops the connection, there is an easy way to detect that. This piece of code was found on StackOverflow and it was told that it still works perfectly:
public static bool CheckIfDisconnected(this TcpClient tcp)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
return true;
}
}
return false;
}
So I believe that you, as a proxy-server, are not obliged to manage connection states at all and can leave it to the actual communication parties. All you have to do is to detect when either of your connections - proxy or request - is dropped, drop the other one and call it a day.
P.S. Now, you also asked about asynchronicity.
I must add that TCP connections are considered full-duplex. which means you are free to create two async-running tasks, both reading and writing to their own sinks. My thoughts, it would be the optimal course of action.

To answer your other question
You are still using Stream.CopyToAsync which, as I have told you, is not going to succeed as long as any communicating party decides to wait a bit before sending another chunk of data.
You are also somewhat overcomplicating your solution.
I would put it this way:
async Task Transport(NetworkStream from, NetworkStream to, Func<bool> isAlivePoller, CancellationToken token)
{
byte[] buffer = new byte[4096];
while (isAlivePoller())
{
while (from.DataAvailable)
{
int read = await from.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false);
await to.WriteAsync(buffer, 0, read, token).ConfigureAwait(false);
}
// Relieve the CPU a bit.
await Task.Delay(100, token).ConfigureAwait(false);
}
}
And then in your main code:
using TcpClient clientFrom = ...;
using TcpClient clientTo = ...;
using var fromStream = clientFrom.GetStream();
using var toStream = clientTo.GetStream();
using var manualStopper = new CancellationTokenSource();
bool Poller() => clientFrom.CheckIfDisconnected() && clientTo.CheckIfDisconnected();
Task one = Transport(fromStream, toStream, Poller, stopper.Token);
Task two = Transport(toStream, fromStream, Poller, stopper.Token);
await Task.WhenAny(one, two).ConfigureAwait(false);
//await one; await two; // To get exceptions if you want them and there are any.
// Alternatively, you can use Task.WhenAll to get exceptions aggregated for you.
And you are pretty much done here.

Related

C# No argument given that corresponds to the required formal parameter 'stream' of 'Receiver(NetworkStream) Error CS7036

I'm trying to create a C# Socket multi-user server (and eventually client).
I want this server to be able to accept multiple clients (who all join and leave at different random moments) and at the same time displays their data (that the server received).
Here is my SocketMultiServer code:
using System.Net.Sockets;
using System.Text;
using System;
using System.Threading;
// Threading
Thread MainThread = Thread.CurrentThread;
Console.Title = "Multi Server";
// Aanmaken van server
TcpListener listener = new TcpListener(System.Net.IPAddress.Any, 6969);
listener.Start(5);
Console.WriteLine("Server online...");
// 2 threads
Thread ListenerThread = new Thread(o => Listener(listener));
Thread ReceiverThread = new Thread(o => Receiver());
//Thread ReceiverThread = new Thread(o => Receiver(NetworkStream stream));
ListenerThread.Start();
ReceiverThread.Start();
static void Listener(TcpListener listener)
{
Console.WriteLine("TEST 2");
TcpClient client = listener.AcceptTcpClient();
Console.WriteLine("Client connected...");
NetworkStream stream = client.GetStream();
}
static void Receiver(NetworkStream stream)
{
while (true)
{
byte[] buffer = new byte[1024];
stream.Read(buffer, 0, buffer.Length);
int recv = 0;
foreach (byte b in buffer)
{
if (b != 0)
{
recv++;
}
}
string request = Encoding.UTF8.GetString(buffer, 0, recv);
Console.WriteLine(request);
}
}
I'm having the error: "Error CS7036 There is no argument given that corresponds to the required formal parameter 'stream' of 'Receiver(NetworkStream)' " at l17
Thread ReceiverThread = new Thread(o => Receiver());
My Client code (work in progress)
using System.Net.Sockets;
using System.Text;
using System;
TcpClient client = new TcpClient("127.0.0.1", 6969);
Console.WriteLine("Connected...");
Console.Write("Username > ");
string username = Console.ReadLine();
string username_ = username + (": ");
while (true)
{
Console.WriteLine("Message to send > ");
string msg = Console.ReadLine();
if (msg == "exit()")
{
break;
}
else
{
// msg converten
int byteCount = Encoding.ASCII.GetByteCount(msg + 1);
byte[] sendData = new byte[byteCount];
sendData = Encoding.ASCII.GetBytes(username_ + msg);
NetworkStream stream = client.GetStream();
stream.Write(sendData, 0, sendData.Length);
}
}
Does someone have a fix? I'm quite new to C#
Thanks in advance!
Your primary issue is that you need to pass NetworkStream to Receiver.
However, this is not the way to do a multi-receive server anyway. You are best off using Task and async.
CancellationTokenSource cancellation = new CancellationTokenSource();
static async Task Main()
{
Console.Title = "Multi Server";
Console.WriteLine("Press ENTER to stop server...");
var task = Task.Run(RunServer);
Console.ReadLine(); // wait for ENTER
cancellation.Cancel();
await task;
}
static async Task RunServer()
{
TcpListener listener;
try
{
listener = new TcpListener(IPAddress.Any, 6969);
listener.Start(5);
Console.WriteLine("Server online...");
while (!cancellation.IsCancellationRequested)
{
var client = await listener.AcceptTcpClientAsync();
Console.WriteLine("Client connected...");
Task.Run(async () => await Receiver(client), cancellation.Token);
}
}
catch (OperationCancelException)
{ //
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
if (listener?.Active == true)
listener.Stop();
}
}
static async Task Receiver(TcpClient client)
{
try
{
using client;
using var stream = client.GetStream();
var buffer = new byte[1024];
while (!cancellation.IsCancellationRequested)
{
var bytesReceived = await stream.ReadAsync(buffer, 0, buffer.Length);
if (bytesReceived == 0)
break;
string request = Encoding.UTF8.GetString(buffer, 0, bytesReceived);
Console.WriteLine(request);
}
}
catch (OperationCancelException)
{ //
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
}
You should have similar async code for the client. Don't forget to dispose everything with using

I have a problem getting response on websocket

I use the websocket API for speech recognition.
I send audio to the server and receive a response.
With this websocket API, it is possible to obtain voice recognition results while streaming voice data.
I can get the final recognition result, but I can not get the recognition result on the way.
The following shows the referenced code.
https://gist.github.com/arjun-g/75961830d363cc9265e4ac2ca095168b
After various trials, I found that line:85 was a problem.
This part seems to be waiting for the final result.
I may not have received the result on the way at all.
Please tell me why I can't get result on the way.
private async Task Connect(ClientWebSocket clientWebSocket)
{
CancellationTokenSource cts = new CancellationTokenSource();
clientWebSocket.Options.SetRequestHeader("Authorization", $"Bearer {this.accessToken}");
Uri connection = new Uri($"wss://hogehoge");
await clientWebSocket.ConnectAsync(connection, cts.Token);
}
The above is the part of the connect method.
Here is my whole code.
private async Task Connect(ClientWebSocket clientWebSocket)
{
CancellationTokenSource cts = new CancellationTokenSource();
clientWebSocket.Options.SetRequestHeader("Authorization", $"Bearer {this.accessToken}");
Uri connection = new Uri($"wss://hogehoge");
await clientWebSocket.ConnectAsync(connection, cts.Token);
}
static async Task SendAudio(ClientWebSocket ws)
{
ArraySegment<byte> closingMessage = new ArraySegment<byte>(Encoding.UTF8.GetBytes(
"{\"command\": \"recog-break\"}"
));
using (FileStream fs = File.OpenRead("audio.raw"))
{
byte[] b = new byte[3200];
while (fs.Read(b, 0, b.Length) > 0)
{
await ws.SendAsync(new ArraySegment<byte>(b), WebSocketMessageType.Binary, true, CancellationToken.None);
}
await ws.SendAsync(closingMessage, WebSocketMessageType.Text, true, CancellationToken.None);
}
}
// prints results until the connection closes or a delimeterMessage is recieved
private async Task HandleResults(ClientWebSocket ws)
{
var buffer = new byte[60000];
while (true)
{
var segment = new ArraySegment<byte>(buffer);
var result = await ws.ReceiveAsync(segment, CancellationToken.None);
if (result.MessageType == WebSocketMessageType.Close)
{
return;
}
int count = result.Count;
while (!result.EndOfMessage)
{
if (count >= buffer.Length)
{
await ws.CloseOutputAsync(WebSocketCloseStatus.InvalidPayloadData, "That's too long", CancellationToken.None);
return;
}
segment = new ArraySegment<byte>(buffer, count, buffer.Length - count);
result = await ws.ReceiveAsync(segment, CancellationToken.None);
count += result.Count;
}
var message = Encoding.UTF8.GetString(buffer, 0, count);
this.textBox2.AppendText(message);
}
}
private async void button1_Click_1(object sender, EventArgs e)
{
ClientWebSocket cws = new ClientWebSocket();
await this.Connect(cws);
await Task.WhenAll(SendAudio(cws), HandleResults(cws));
}

Web API service hangs on reading the stream

Description: I am modifying the ASP.NET Core Web API service (hosted in Windows Service) that supports resumable file uploads. This works fine and resumes file uploads in many failure conditions except one described below.
Problem: When the service is on ther other computer and the client is on mine and I unplug the cable on my computer, the client detects the absence of network while the service hangs on fileSection.FileStream.Read(). Sometimes the service detects the failure in 8 min, sometimes in 20, sometimes never.
I also noticed that after I unplug cable and stop the client, the service becomes stuck at Read() function and the file size is x KB, but when the service finally detects the exception some time later, it writes additional 4 KB to the file. This is weird because I turned off buffering and the buffer size is 2 KB.
Question: How to properly detect the absence of network on the service, or timeout properly, or cancel the request
The service code:
public static async Task<List<(Guid, string)>> StreamFileAsync(
this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = fileSection.FileStream.Read(buffer, 0, buffer.Length)) > 0) // HANGS HERE
{
outfile.Write(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
}
}
catch (Exception e)
{
...
}
}
}
The client code:
var request = CreateRequest(fileTransferId, boundary, header, footer, filePath, offset, headers, null);
using (Stream formDataStream = request.GetRequestStream())
{
formDataStream.ReadTimeout = 60000;
formDataStream.Write(Encoding.UTF8.GetBytes(header), 0, header.Length);
byte[] buffer = new byte[2048];
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
fs.Seek(offset, SeekOrigin.Begin);
for (int i = 0; i < fs.Length - offset;)
{
int k = await fs.ReadAsync(buffer, 0, buffer.Length);
if (k > 0)
{
await Task.Delay(100);
await formDataStream.WriteAsync(buffer, 0, k);
}
i = i + k;
}
}
formDataStream.Write(footer, 0, footer.Length);
}
var uploadingResult = request.GetResponse() as HttpWebResponse;
private static HttpWebRequest CreateRequest(
Guid fileTransferId,
string boundary,
string header,
byte[] footer,
string filePath,
long offset,
NameValueCollection headers,
Dictionary<string, string> postParameters)
{
var url = $"{_BaseAddress}v1/ResumableUpload?fileTransferId={fileTransferId}";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "multipart/form-data; boundary=\"" + boundary + "\"";
request.UserAgent = "Agent 1.0";
request.Headers.Add(headers); // custom headers
request.Timeout = 120000;
request.KeepAlive = true;
request.AllowReadStreamBuffering = false;
request.ReadWriteTimeout = 120000;
request.AllowWriteStreamBuffering = false;
request.ContentLength = CalculateContentLength(filePath, offset, header, footer, postParameters, boundary);
return request;
}
What I tried:
I added these in into config files:
Tried to set timeout on the server
var host = new WebHostBuilder().UseKestrel(o => { o.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(2);})
Used async and non-async Read()
Tried with keep alive and without
Tried to abort the request when network was restored: request?.Abort();
Tried to set formDataStream.ReadTimeout = 60000;
Since I did not find a better way, I decided to add a timeout to the reading stream and saving it to the file. The good example was posted here: https://blogs.msdn.microsoft.com/pfxteam/2012/10/05/how-do-i-cancel-non-cancelable-async-operations/
public static async Task<List<(Guid, string)>> StreamFileAsync(this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
await SaveMyFile(...);
}
catch (OperationCanceledException){...}
catch (Exception){...}
}
}
private static async Task SaveMyFile(...)
{
var cts = CancellationTokenSource.CreateLinkedTokenSource(myOtherCancellationToken);
cts.CancelAfter(streamReadTimeoutInMs);
var myReadTask = StreamFile(transferInfo, fileSection, cts.Token);
await ExecuteMyTaskWithCancellation(myReadTask, cts.Token);
}
private static async Task<T> ExecuteMyTaskWithCancellation<T>(Task<T> task, CancellationToken cancellationToken)
{
var tcs = new TaskCompletionSource<bool>();
using (cancellationToken.Register(s => ((TaskCompletionSource<bool>) s).TrySetResult(true), tcs))
{
if (task != await Task.WhenAny(task, tcs.Task))
{
throw new OperationCanceledException(cancellationToken);
}
}
return await task;
}
private static async Task<bool> StreamFile(...)
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = await fileSection.FileStream.ReadAsync(buffer, 0, buffer.Length, token)) > 0)
{
if (token.IsCancellationRequested)
{
break;
}
await outfile.WriteAsync(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
return true;
}
}

StreamSocket C# Client is only able to write one time to server

I have been trying to setup a client-server application using StreamSockets in c#. I am able to initially connect to the server (ConnectAsync) and following to write and read the stream. If the client sends another stream to the server using the method WriteToServer the event on the server side is not being triggered (SocketListener_ConnectionReceived). I'm sending a xmlSerialized object "Message" to the server.
While debugging I don't receive any errors. On the client side though the after having used "WriteToServer" and moving forward to "ReadFromServer" the code gets obviously stuck in the line below since the server doesn't reply
int bufferLength = inStream.ReadByte();
I hope that someone call help. I am honestly not sure what the issues is because the "Write"-method is being used the same way both times the client attempts to write to the server.
Client is a Windows 10 computer and Server a Raspberry pi running Windows 10 IoT.
The class inside the client application which handles the connection looks like this.
StreamSocket socket;
HostName remoteHostName, localHostName;
string serviceAddress = "1337";
EndpointPair endPointPair;
Boolean isConnected;
Socket test;
public Connection()
{
remoteHostName = new HostName("172.16.20.202"); // might have to change based on pi's ip address
//remoteHostName = new HostName("172.16.20.4");
localHostName = new HostName(getLocalIpAddress());
endPointPair = new EndpointPair(localHostName, serviceAddress, remoteHostName, serviceAddress);
socket = new StreamSocket();
socket.Control.NoDelay = true;
socket.Control.QualityOfService = SocketQualityOfService.LowLatency;
}
private string getLocalIpAddress()
{
var icp = NetworkInformation.GetInternetConnectionProfile();
if (icp?.NetworkAdapter == null) return null;
var hostname =
NetworkInformation.GetHostNames()
.SingleOrDefault(
hn =>
hn.IPInformation?.NetworkAdapter != null && hn.IPInformation.NetworkAdapter.NetworkAdapterId
== icp.NetworkAdapter.NetworkAdapterId);
// the ip address
return hostname?.CanonicalName;
}
public async Task StartConnection()
{
try
{
if (isConnected)
{
await socket.CancelIOAsync();
socket.Dispose();
socket = null;
isConnected = false;
}
await socket.ConnectAsync(endPointPair);
isConnected = true;
}
catch (Exception exc)
{
if (Windows.Networking.Sockets.SocketError.GetStatus(exc.HResult) == SocketErrorStatus.Unknown)
{
throw;
}
Debug.WriteLine("Connect failed with error: " + exc.Message);
socket.Dispose();
isConnected = false;
socket = null;
//return null;
}
}
public async Task WriteToServer(Message msg)
{
try
{
using (DataWriter writer = new DataWriter(socket.OutputStream))
{
writer.WriteBytes(serialize(msg));
await writer.StoreAsync();
writer.DetachStream();
writer.Dispose();
}
}
catch (Exception exc)
{
Debug.WriteLine("Write failed with error: " + exc.Message);
}
}
public async Task<Library.Message> ReadFromServer()
{
try
{
Stream inStream = socket.InputStream.AsStreamForRead();
int bufferLength = inStream.ReadByte();
byte[] serializedMessage = new byte[bufferLength];
await inStream.ReadAsync(serializedMessage, 0, bufferLength);
await inStream.FlushAsync();
Library.Message incomingMessage;
using (var stream = new MemoryStream(serializedMessage))
{
var serializer = new XmlSerializer(typeof(Library.Message));
incomingMessage = (Library.Message)serializer.Deserialize(stream);
}
return incomingMessage;
}
catch (Exception exc)
{
Debug.WriteLine("Read failed with error: " + exc.Message);
return null;
}
}
private byte[] serialize(Message msg)
{
byte[] serializedMessage, returnArray;
var serializer = new XmlSerializer(typeof(Library.Message));
using (var stream = new MemoryStream())
{
serializer.Serialize(stream, msg);
serializedMessage = stream.ToArray();
stream.Dispose();
}
int bufferLength = serializedMessage.Length;
returnArray = new byte[serializedMessage.Length + 1];
serializedMessage.CopyTo(returnArray, 1);
returnArray[0] = (byte)bufferLength;
Debug.WriteLine("ClientReturnArrayLength: " + returnArray.Length);
Debug.WriteLine("ClientFirstByte: " + returnArray[0]);
return returnArray;
}
The server side looks like this
public Task DispatcherPriority { get; private set; }
public MainPage()
{
this.InitializeComponent();
dispatcher = CoreWindow.GetForCurrentThread().Dispatcher;
hostName = new HostName(getLocalIpAddress());
clients = new List<Client>();
}
/// <summary>
/// Gets the ip address of the host
/// </summary>
/// <returns></returns>
private string getLocalIpAddress()
{
var icp = NetworkInformation.GetInternetConnectionProfile();
if (icp?.NetworkAdapter == null) return null;
var hostname =
NetworkInformation.GetHostNames()
.SingleOrDefault(
hn =>
hn.IPInformation?.NetworkAdapter != null && hn.IPInformation.NetworkAdapter.NetworkAdapterId
== icp.NetworkAdapter.NetworkAdapterId);
// the ip address
return hostname?.CanonicalName;
}
async void setupSocketListener()
{
if (socketListener != null)
{
await socketListener.CancelIOAsync();
socketListener.Dispose();
socketListener = null;
}
socketListener = new StreamSocketListener();
socketListener.Control.QualityOfService = SocketQualityOfService.LowLatency;
socketListener.ConnectionReceived += SocketListener_ConnectionReceived;
await socketListener.BindServiceNameAsync("1337");
listBox.Items.Add("server started.");
clients.Clear();
}
private async void SocketListener_ConnectionReceived(StreamSocketListener sender, StreamSocketListenerConnectionReceivedEventArgs args)
{
HostName ip = args.Socket.Information.RemoteAddress;
string port = args.Socket.Information.RemotePort;
try
{
Stream inStream = args.Socket.InputStream.AsStreamForRead();
int bufferLength = inStream.ReadByte();
byte[] serializedMessage = new byte[bufferLength];
await inStream.ReadAsync(serializedMessage, 0, bufferLength);
await inStream.FlushAsync();
Message incomingMessage;
using (var stream = new MemoryStream(serializedMessage))
{
var serializer = new XmlSerializer(typeof(Message));
incomingMessage = (Message)serializer.Deserialize(stream);
}
/// <summary>
/// 1 = Connected
/// 2 = SentNote
/// 3 = Login
/// </summary>
switch (incomingMessage.Mode)
{
case 1:
onClientConnect(ip, port, incomingMessage.Username, args.Socket);
break;
case 2:
onNoteReceived(incomingMessage);
break;
case 3:
//handle login
break;
}
}
catch (Exception msg)
{
Debug.WriteLine(msg);
}
}
private async void onNoteReceived(Message msg)
{
foreach (var client in clients)
{
//if (client.Username != msg.Username)
//{
using (DataWriter writer = new DataWriter(client.Socket.OutputStream))
{
writer.WriteBytes(serialize(msg));
await writer.StoreAsync();
writer.DetachStream();
writer.Dispose();
}
//}
}
}
private void buttonStartServer_Click(object sender, RoutedEventArgs e)
{
setupSocketListener();
}
private async void notifyClients(string username)
{
Message msg = new Message();
msg.Username = username;
foreach (var client in clients)
{
//if (client.Username != msg.Username)
//{
using (DataWriter writer = new DataWriter(client.Socket.OutputStream))
{
writer.WriteBytes(serialize(msg));
await writer.StoreAsync();
writer.DetachStream();
writer.Dispose();
}
//}
}
}
private async void onClientConnect(HostName ip, string port, string username, StreamSocket socket)
{
clients.Add(new Client(ip, port, username, socket));
await dispatcher.RunAsync(CoreDispatcherPriority.Normal, () =>
{
listBox.Items.Add("User: " + username + " on IP " + ip + " is connected.");
});
notifyClients(username);
}
private byte[] serialize(Message msg)
{
byte[] serializedMessage, returnArray;
var serializer = new XmlSerializer(typeof(Message));
using (var stream = new MemoryStream())
{
serializer.Serialize(stream, msg);
serializedMessage = stream.ToArray();
stream.Dispose();
}
int bufferLength = serializedMessage.Length;
returnArray = new byte[serializedMessage.Length + 1];
serializedMessage.CopyTo(returnArray, 1);
returnArray[0] = (byte)bufferLength;
Debug.WriteLine("ClientReturnArrayLength: " + returnArray.Length);
Debug.WriteLine("ClientFirstByte: " + returnArray[0]);
return returnArray;
}
}
This is the message class which I am using to send over the network.
public string Username { get; set; }
/// <summary>
/// 1 = startConnection
/// 2 = SendNote
/// 3 = Login
/// </summary>
public int Mode { get; set; }
public Piano PianoNote { get; set; }
public string Instrument { get; set; }
public Message()
{
}
}
public enum Piano { a1, a1s, b1, c1 };
**Edit: **
Message framing:
byte[] prefix = BitConverter.GetBytes(serializedMessage.Length);
returnArray = new byte[serializedMessage.Length + prefix.Length];
prefix.CopyTo(returnArray, 0);
serializedMessage.CopyTo(returnArray, prefix.Length);
Reading the message:
byte[] prefix = new byte[4];
await inStream.ReadAsync(prefix, 0, 4);
int bufferLength = BitConverter.ToInt32(prefix, 0);
Half-Open:
Instead of reading synchronous I switched to async reading the first 4 bytes as seen above.
I am able to initially connect to the server (ConnectAsync) and following to write and read the stream. If the client sends another stream to the server using the method WriteToServer the event on the server side is not being triggered (SocketListener_ConnectionReceived).
Take a good look at those names. You're calling ConnectAsync once and then WriteToServer twice, and only seeing SocketListener_ConnectionReceived once. There's only once connection, so yes, ConnectionReceived would only trigger once.
That's just scratching the surface, though. There's a few other very subtle issues wrong with this code.
One that sticks out is a lack of proper message framing, as I describe on my blog. You're using a single-byte length prefix, so on the wire it's OK (though limited to 256 bytes, which doesn't go far with XML). But the reading of the messages is incorrect; in particular, Stream.ReadAsync may read between 1 and bufferLength bytes, or it may return 0.
Another problem is that it's subject to the half-open problem, as I describe on my blog. In particular, int bufferLength = inStream.ReadByte(); will block indefinitely in a half-open situation. You should only use asynchronous methods for all network streams, and periodically write while waiting for data to arrive.
In summary, I strongly recommend that you use self-hosted SignalR instead of raw sockets. Raw socket programming is extremely difficult to do correctly, especially because incorrect code often happens to work correctly in a local environment.

Downloading files not affecting UI

We are currently downloading files using HttpClient because our backend requires certificate.
I have a control - FileRowwhich is an UI element with some code-behind methods for file downloading, like this one:
if (FileIsDownloaded == false)
{
await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Low, () =>
{
DataManager.Instance.DownloadFile(this);
});
}
if (ThumbnailIsDownloaded == false)
{
await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Low, () =>
{
DataManager.Instance.DownloadThumbnail(this);
});
}
Downloading single item is fine but when i click download all ( about 50 items ) the whole UI starts to freeze.
As you can see, i have tried to give requests low priority - but still same result.
Answers to common questions:
1) Yes the files should be downloadable all at one time, not one after another.
2) DataManager.Instance.DownloadThumbnail(this) - i do this to give refference to current control so that i could report a progress in a progress bar.
Any suggestions?
EDIT:
Downloading looks like this:
public async void DownloadFile(FileRow fileRow)
{
//Lot of checking for if file exist, if version is the same
string LocalFilename = await DownloadManager.DownloadFile(fileRow.MyFile.file.id, fileRow.MyFile.file.version, fileRow.MyFile.file.filename,fileRow);
// next is just using the filename string
}
And finally my download:
public static async Task<string> DownloadFileOfCustomerAssetRow(int? id, int? version, string filename, FileRow fileRow)
{
try
{
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
customerAssetRow.CurrentFileDownload = aClient;
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(WebServices.BackendStartUrl + "getFileData?id=" + id + "&version=" + version, HttpCompletionOption.ResponseHeadersRead);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync(filename, Windows.Storage.CreationCollisionOption.GenerateUniqueName);
fileRow.FileName = file.Name;
using (var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite))
{
Stream stream = await response.Content.ReadAsStreamAsync();
IInputStream inputStream = stream.AsInputStream();
ulong totalBytesRead = 0;
while (true)
{
IBuffer buffer = new Windows.Storage.Streams.Buffer(1024);
buffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.None);
if (buffer.Length == 0)
{
break;
}
totalBytesRead += buffer.Length;
fileRow.Progress.Value = fileRow.Progress.Value + 1024;
await fs.WriteAsync(buffer);
}
inputStream.Dispose();
}
fileRow.Progress.Visibility = Visibility.Collapsed;
return file.Name;
}
catch (Exception e)
{
ErrorReporter.ReportError("Error in DownloadManager.cs in function DownloadFile.", e);
return "";
}
}
It's possible that your async method continuations are interrupting the UI thread too much. Try creating a stronger separation between your background logic (DownloadFileOfCustomerAssetRow) and your UI (FileRow) by introducing an IProgress<T> reporter. Then ensure that every await in your background logic has a ConfigureAwait(false) on it.
public static async Task<string> DownloadFileOfCustomerAssetRow(int? id, int? version, string filename, IProgress<int> progress)
{
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
customerAssetRow.CurrentFileDownload = aClient;
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(WebServices.BackendStartUrl + "getFileData?id=" + id + "&version=" + version, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync(filename, Windows.Storage.CreationCollisionOption.GenerateUniqueName).ConfigureAwait(false);
fileRow.FileName = file.Name;
using (var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite).ConfigureAwait(false))
{
Stream stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
IInputStream inputStream = stream.AsInputStream();
ulong totalBytesRead = 0;
while (true)
{
IBuffer buffer = new Windows.Storage.Streams.Buffer(1024);
buffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.None).ConfigureAwait(false);
if (buffer.Length == 0)
{
break;
}
totalBytesRead += buffer.Length;
if (progress != null)
progress.Report(totalBytesRead);
await fs.WriteAsync(buffer).ConfigureAwait(false);
}
inputStream.Dispose();
}
return file.Name;
}

Categories