Web API service hangs on reading the stream - c#

Description: I am modifying the ASP.NET Core Web API service (hosted in Windows Service) that supports resumable file uploads. This works fine and resumes file uploads in many failure conditions except one described below.
Problem: When the service is on ther other computer and the client is on mine and I unplug the cable on my computer, the client detects the absence of network while the service hangs on fileSection.FileStream.Read(). Sometimes the service detects the failure in 8 min, sometimes in 20, sometimes never.
I also noticed that after I unplug cable and stop the client, the service becomes stuck at Read() function and the file size is x KB, but when the service finally detects the exception some time later, it writes additional 4 KB to the file. This is weird because I turned off buffering and the buffer size is 2 KB.
Question: How to properly detect the absence of network on the service, or timeout properly, or cancel the request
The service code:
public static async Task<List<(Guid, string)>> StreamFileAsync(
this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = fileSection.FileStream.Read(buffer, 0, buffer.Length)) > 0) // HANGS HERE
{
outfile.Write(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
}
}
catch (Exception e)
{
...
}
}
}
The client code:
var request = CreateRequest(fileTransferId, boundary, header, footer, filePath, offset, headers, null);
using (Stream formDataStream = request.GetRequestStream())
{
formDataStream.ReadTimeout = 60000;
formDataStream.Write(Encoding.UTF8.GetBytes(header), 0, header.Length);
byte[] buffer = new byte[2048];
using (FileStream fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read))
{
fs.Seek(offset, SeekOrigin.Begin);
for (int i = 0; i < fs.Length - offset;)
{
int k = await fs.ReadAsync(buffer, 0, buffer.Length);
if (k > 0)
{
await Task.Delay(100);
await formDataStream.WriteAsync(buffer, 0, k);
}
i = i + k;
}
}
formDataStream.Write(footer, 0, footer.Length);
}
var uploadingResult = request.GetResponse() as HttpWebResponse;
private static HttpWebRequest CreateRequest(
Guid fileTransferId,
string boundary,
string header,
byte[] footer,
string filePath,
long offset,
NameValueCollection headers,
Dictionary<string, string> postParameters)
{
var url = $"{_BaseAddress}v1/ResumableUpload?fileTransferId={fileTransferId}";
HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url);
request.Method = "POST";
request.ContentType = "multipart/form-data; boundary=\"" + boundary + "\"";
request.UserAgent = "Agent 1.0";
request.Headers.Add(headers); // custom headers
request.Timeout = 120000;
request.KeepAlive = true;
request.AllowReadStreamBuffering = false;
request.ReadWriteTimeout = 120000;
request.AllowWriteStreamBuffering = false;
request.ContentLength = CalculateContentLength(filePath, offset, header, footer, postParameters, boundary);
return request;
}
What I tried:
I added these in into config files:
Tried to set timeout on the server
var host = new WebHostBuilder().UseKestrel(o => { o.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(2);})
Used async and non-async Read()
Tried with keep alive and without
Tried to abort the request when network was restored: request?.Abort();
Tried to set formDataStream.ReadTimeout = 60000;


Since I did not find a better way, I decided to add a timeout to the reading stream and saving it to the file. The good example was posted here: https://blogs.msdn.microsoft.com/pfxteam/2012/10/05/how-do-i-cancel-non-cancelable-async-operations/
public static async Task<List<(Guid, string)>> StreamFileAsync(this HttpRequest request, DeviceId deviceId, FileTransferInfo transferInfo)
{
var boundary = GetBoundary(MediaTypeHeaderValue.Parse(request.ContentType), DefaultFormOptions.MultipartBoundaryLengthLimit);
var reader = new MultipartReader(boundary, request.Body);
var section = await reader.ReadNextSectionAsync(_cancellationToken);
if (section != null)
{
var fileSection = section.AsFileSection();
var targetPath = transferInfo.FileTempPath;
try
{
await SaveMyFile(...);
}
catch (OperationCanceledException){...}
catch (Exception){...}
}
}
private static async Task SaveMyFile(...)
{
var cts = CancellationTokenSource.CreateLinkedTokenSource(myOtherCancellationToken);
cts.CancelAfter(streamReadTimeoutInMs);
var myReadTask = StreamFile(transferInfo, fileSection, cts.Token);
await ExecuteMyTaskWithCancellation(myReadTask, cts.Token);
}
private static async Task<T> ExecuteMyTaskWithCancellation<T>(Task<T> task, CancellationToken cancellationToken)
{
var tcs = new TaskCompletionSource<bool>();
using (cancellationToken.Register(s => ((TaskCompletionSource<bool>) s).TrySetResult(true), tcs))
{
if (task != await Task.WhenAny(task, tcs.Task))
{
throw new OperationCanceledException(cancellationToken);
}
}
return await task;
}
private static async Task<bool> StreamFile(...)
{
using (var outfile = new FileStream(transferInfo.FileTempPath, FileMode.Append, FileAccess.Write, FileShare.None))
{
var buffer = new byte[DefaultCopyBufferSize];
int read;
while ((read = await fileSection.FileStream.ReadAsync(buffer, 0, buffer.Length, token)) > 0)
{
if (token.IsCancellationRequested)
{
break;
}
await outfile.WriteAsync(buffer, 0, read);
transferInfo.BytesSaved = read + transferInfo.BytesSaved;
}
return true;
}
}

Related

HTTPS proxy implementation, how to detect a completed request

I'm attempting to write a simple async https proxy server in c#.
I would like to know how I should detect/handle when the request is complete, and how to exit my bActive loop, assuming a loop like this is appropriate.
Would really appreciate some pointers on if my approach is correct and what I could do to improve the logic.
The issue I seem to be running into is that the time it takes for an endpoint to respond along with the network delay means I DataAvailable doenst always have data but there may still be some sending. Requiring a sleep and another attmempt which in turn causes the long completion time in requests.
Listen for TCP connection
Extract CONNECT header and open a connection to the requested server
Copy the requestStream to proxyStream
Copy the proxyStream to the requestStream
Sleep waiting for data and repeat 3 - 4 until no data is avaiable on both streams. Then break out of the loop and close connection.
public async Task Start()
{
listener.Start();
while (listen)
{
if (listener.Pending())
{
HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100); //<--- timeout
}
}
}
private static async Task HandleClient(TcpClient clt)
{
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
int count;
const string connectText = "connect";
const string hostText = "Host: ";
bool bActive = true;
List<Task> tasks = new List<Task>();
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
proxyStream.ReadTimeout = 100;
proxyStream.WriteTimeout = 100;
while (bActive)
{
if (proxyStream.DataAvailable && hostHeaderAvailable == 0)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
requestStream.ReadTimeout = 100;
requestStream.WriteTimeout = 100;
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
// delay here seems to prevent the following proxyStream.read from failing as data is not yet avaiable
// without it the loop runs and has to timeout before running again
await Task.Delay(1);
}
}
hostHeaderAvailable++;
if (requestStream == null || !requestClient.Connected || !clt.Connected)
{
bActive = false;
break;
}
Console.WriteLine(proxyStream.DataAvailable || requestStream.DataAvailable);
if (proxyStream.DataAvailable || requestStream.DataAvailable)
{
Task task = proxyStream.CopyToAsync(requestStream);
Task task2 = requestStream.CopyToAsync(proxyStream);
tasks.Add(task);
tasks.Add(task2);
await Task.WhenAll(tasks).ConfigureAwait(false);
bActive = false;
break;
}
await Task.Delay(10);
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
clt.Close();
}
An older attempt that used ReadAsync/WriteAsync too longer to response and still had the timeout issue.
Listen for TCP connection
Extract CONNECT header and open a connection to the requested server
Read data from requestStream and copy to proxyStream
Wait checking if data is avaiable on either stream
If data avaiable read from proxyStream and write to requestStream
If data avaiable read from requestStream and write to proxyStream
Sleep waiting for data and repeat 5 - 6 until no data is avaiable on eitboth streams. Then break out of the loop and close connection.
private static TcpListener listener = new TcpListener(IPAddress.Parse("192.168.0.25"), 13000);
private static bool listen = true;
public async Task Start()
{
listener.Start();
while (listen)
{
if (listener.Pending())
{
await HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100);
}
}
}
private static async Task HandleClient(TcpClient clt)
{
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
int count;
const string connectText = "connect";
const string hostText = "Host: ";
bool bActive = true;
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
while (bActive)
{
while (proxyStream.DataAvailable)
{
// handle connect
if (hostHeaderAvailable == 0)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
// delay here seems to prevent the following proxyStream.read from failing as data is not yet avaiable
// without it the loop runs and has to timeout before running again
await Task.Delay(20);
}
}
hostHeaderAvailable++;
if (requestClient.Connected && hostHeaderAvailable > 1)
{
count = await proxyStream.ReadAsync(bytes, 0, bytes.Length);
await requestStream.WriteAsync(bytes, 0, count);
}
}
while (requestStream.DataAvailable)
{
count = await requestStream.ReadAsync(bytes, 0, bytes.Length);
await proxyStream.WriteAsync(bytes, 0, count);
}
// attempt to detect a timeout / end of data avaiable
var timeout = 0;
while (!proxyStream.DataAvailable && !requestStream.DataAvailable)
{
if (timeout > 5)
{
bActive = false;
break;
}
await Task.Delay(10);
timeout++;
}
}
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
UPDATE
As per AgentFire's answer I have now come to the following working code:
public static async Task HandleDisconnect(TcpClient tcp, TcpClient tcp2, CancellationToken cancellationToken)
{
while (true)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
Console.WriteLine("The requesting client has dropped its connection.");
cancellationToken = new CancellationToken(true);
break;
}
}
if (tcp2.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp2.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Server disconnected
Console.WriteLine("The destination client has dropped its connection.");
cancellationToken = new CancellationToken(true);
break;
}
}
await Task.Delay(1);
}
}
private static async Task HandleClient(TcpClient clt)
{
List<Task> tasks = new List<Task>();
var bytes = new byte[clt.ReceiveBufferSize];
var hostHeaderAvailable = 0;
NetworkStream requestStream = null;
const string connectText = "connect";
try
{
using (NetworkStream proxyStream = clt.GetStream())
using (TcpClient requestClient = new TcpClient())
{
proxyStream.ReadTimeout = 100;
proxyStream.WriteTimeout = 100;
if (proxyStream.DataAvailable && hostHeaderAvailable == 0)
{
await proxyStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await requestClient.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
requestStream = requestClient.GetStream();
requestStream.ReadTimeout = 100;
requestStream.WriteTimeout = 100;
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await proxyStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
}
}
hostHeaderAvailable++;
CancellationToken cancellationToken = new CancellationToken(false);
Task task = proxyStream.CopyToAsync(requestStream, cancellationToken);
Task task2 = requestStream.CopyToAsync(proxyStream, cancellationToken);
Task handleConnection = HandleDisconnect(clt, requestClient, cancellationToken);
tasks.Add(task);
tasks.Add(task2);
tasks.Add(handleConnection);
await Task.WhenAll(tasks).ConfigureAwait(false);
// close conenctions
clt.Close();
clt.Dispose();
requestClient.Close();
requestClient.Dispose();
}
}
catch (Exception e)
{
Console.WriteLine(e.ToString());
}
}
UPDATE
Attempt at using CancellationTokenSource
CancellationTokenSource source = new CancellationTokenSource();
CancellationToken cancellationToken = source.Token;
TaskFactory factory = new TaskFactory(cancellationToken);
tasks.Add(factory.StartNew(() => {proxyStream.CopyToAsync(requestStream);}, cancellationToken));
tasks.Add(factory.StartNew(() => {requestStream.CopyToAsync(proxyStream);}, cancellationToken));
tasks.Add(factory.StartNew(async () => {
//wait for this to retur, then cancel the token
await HandleDisconnect(clt, requestClient);
source.Cancel();
}, cancellationToken));
try
{
await factory.ContinueWhenAll(tasks.ToArray(),
(results) =>
{
Console.WriteLine("Tasks complete");
}, cancellationToken);
}
catch (AggregateException ae)
{
foreach (Exception e in ae.InnerExceptions)
{
if (e is TaskCanceledException)
Console.WriteLine("Unable to compute mean: {0}",
((TaskCanceledException)e).Message);
else
Console.WriteLine("Exception: " + e.GetType().Name);
}
}
finally
{
source.Dispose();
}
UPDATE
public static class extensionTcpClient{
public static bool CheckIfDisconnected(this TcpClient tcp)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
return false;
}
}
return true;
}
}
class ProxyMaintainer
{
private static TcpListener listener = new TcpListener(IPAddress.Parse("192.168.0.25"), 13000);
public ProxyMaintainer()
{
}
public async Task Start()
{
Console.WriteLine("###############################");
Console.WriteLine("Listening on 192.168.0.25:13000");
Console.WriteLine("###############################\n");
listener.Start();
while (listen)
{
if (listener.Pending())
{
HandleClient(await listener.AcceptTcpClientAsync());
}
else
{
await Task.Delay(100); //<--- timeout
}
}
}
private static async Task Transport(NetworkStream from, NetworkStream to, Func<bool> isAlivePoller, CancellationToken token)
{
byte[] buffer = new byte[4096];
while (isAlivePoller())
{
while (from.DataAvailable)
{
int read = await from.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false);
await to.WriteAsync(buffer, 0, read, token);
}
// Relieve the CPU a bit.
await Task.Delay(10, token).ConfigureAwait(false);
}
}
private static async Task HandleClient(TcpClient clientFrom)
{
var hostHeaderAvailable = 0;
int count;
var bytes = new byte[clientFrom.ReceiveBufferSize];
const string connectText = "connect";
NetworkStream toStream = null;
using (var fromStream = clientFrom.GetStream())
using(TcpClient clientTo = new TcpClient())
using (var manualStopper = new CancellationTokenSource())
{
count = await fromStream.ReadAsync(bytes, 0, bytes.Length);
var text = Encoding.UTF8.GetString(bytes);
Console.WriteLine(text);
if (text.ToLower().StartsWith(connectText))
{
// extract the url and port
var host = text.Remove(0, connectText.Length + 1);
var hostIndex = host.IndexOf(" ", StringComparison.Ordinal);
var hostEntry = host.Remove(hostIndex).Split(new[] { ":" }, StringSplitOptions.None);
// connect to the url and prot supplied
await clientTo.ConnectAsync(hostEntry[0], Convert.ToInt32(hostEntry[1]));
toStream = clientTo.GetStream();
// send 200 response to proxyStream
const string sslResponse = "HTTP/1.0 200 Connection established\r\n\r\n";
var sslResponseBytes = Encoding.UTF8.GetBytes(sslResponse);
await fromStream.WriteAsync(sslResponseBytes, 0, sslResponseBytes.Length);
}
bool Poller() => clientFrom.CheckIfDisconnected() && clientTo.CheckIfDisconnected();
Task one = Transport(fromStream, toStream, Poller, manualStopper.Token);
Task two = Transport(toStream, fromStream, Poller, manualStopper.Token);
await Task.WhenAll(one, two).ConfigureAwait(false);
//await one; await two; // To get exceptions if you want them and there are any.
// Alternatively, you can use Task.WhenAll to get exceptions aggregated for you.
}
Console.WriteLine("Closing connection");
}
}

Well, tell you what. The data availability, when it comes to HTTP, lies only in one parameter (if we omit things like WebSocket), which is called Connection and is passed as a Header as a one of two possible states: Close or Keep-Alive.
If Close is chosen by the client, the server is obliged to close the conection as soon as the request is served, whereas Keep-Alive tells the server that, if it doesn't want to, it may leave connection open for another request.
Let's consider both cases.
If client chooses Keep-Alive, the connection will persist and work as intended, indefinetely. But:
If either side drops the connection, there is an easy way to detect that. This piece of code was found on StackOverflow and it was told that it still works perfectly:
public static bool CheckIfDisconnected(this TcpClient tcp)
{
if (tcp.Client.Poll(0, SelectMode.SelectRead))
{
byte[] buff = new byte[1];
if (tcp.Client.Receive(buff, SocketFlags.Peek) == 0)
{
// Client disconnected
return true;
}
}
return false;
}
So I believe that you, as a proxy-server, are not obliged to manage connection states at all and can leave it to the actual communication parties. All you have to do is to detect when either of your connections - proxy or request - is dropped, drop the other one and call it a day.
P.S. Now, you also asked about asynchronicity.
I must add that TCP connections are considered full-duplex. which means you are free to create two async-running tasks, both reading and writing to their own sinks. My thoughts, it would be the optimal course of action.

To answer your other question
You are still using Stream.CopyToAsync which, as I have told you, is not going to succeed as long as any communicating party decides to wait a bit before sending another chunk of data.
You are also somewhat overcomplicating your solution.
I would put it this way:
async Task Transport(NetworkStream from, NetworkStream to, Func<bool> isAlivePoller, CancellationToken token)
{
byte[] buffer = new byte[4096];
while (isAlivePoller())
{
while (from.DataAvailable)
{
int read = await from.ReadAsync(buffer, 0, buffer.Length, token).ConfigureAwait(false);
await to.WriteAsync(buffer, 0, read, token).ConfigureAwait(false);
}
// Relieve the CPU a bit.
await Task.Delay(100, token).ConfigureAwait(false);
}
}
And then in your main code:
using TcpClient clientFrom = ...;
using TcpClient clientTo = ...;
using var fromStream = clientFrom.GetStream();
using var toStream = clientTo.GetStream();
using var manualStopper = new CancellationTokenSource();
bool Poller() => clientFrom.CheckIfDisconnected() && clientTo.CheckIfDisconnected();
Task one = Transport(fromStream, toStream, Poller, stopper.Token);
Task two = Transport(toStream, fromStream, Poller, stopper.Token);
await Task.WhenAny(one, two).ConfigureAwait(false);
//await one; await two; // To get exceptions if you want them and there are any.
// Alternatively, you can use Task.WhenAll to get exceptions aggregated for you.
And you are pretty much done here.

Trouble writing zip content to device. Writing as hexa data not xml

I'm trying to download a zip file that contains an xml file and save the xml file to the device.
I'm able to save the file inside the zip but when I open the file it contains hexa data and not xml.
What I'm doing:
I'm using http client to make a get call. I get stream content from the response and using dependency injection, I pass the stream to a method that writes to the device.
In share code:
public static async Task<bool> Download()
{
var url = "some url";
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
var httpClient = new HttpClient(handler);
try
{
var response = await httpClient.GetAsync(url);
var total = response.Content.Headers.ContentLength.HasValue ? response.Content.Headers.ContentLength.Value : -1L;
var stream = await response.Content.ReadAsStreamAsync();
DependencyService.Get<IFileHelper>().WriteToFilePath("download", stream, total);
}
catch (Exception e)
{
//Handle exception
}
return true;
}
in iOS:
public async void WriteToFilePath(string filename, Stream responseStream, long contentSize)
{
var downloadFilePath = GetLocalFilePath(filename);
int receivedBytes = 0;
using (FileStream fileOut = File.Create(downloadFilePath))
{
using (Stream stream = responseStream)
{
byte[] buffer = new byte[4096];
for (;;)
{
int bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
await Task.Yield();
break;
}
else
{
System.Diagnostics.Debug.WriteLine(bytesRead);
fileOut.Write(buffer, 0, bytesRead);
}
receivedBytes += bytesRead;
System.Diagnostics.Debug.WriteLine("receivedBytes: " + receivedBytes + " contentSize: " + contentSize);
}
}
}

how to return a xlsx file using memorystream web api

I am successfully making a web request, creating xlsx file and saving it to a directory. I would like to stream it and return it to the client without actually saving it on the server.
this is what i am currently using, it works fine
private string generateStudyTemplate(string requestId)
{
var serviceUrl = ConfigurationManager.AppSettings["serviceUrl"];
// create webRequest
HttpWebRequest webRequest = createWebRequest(serviceUrl + "/" + requestId);
// begin async call to web request
IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null);
// suspend this thread until call is complete. You might want to
// do something usefull here like update your UI
asyncResult.AsyncWaitHandle.WaitOne();
// get the response from the completed web request
var filename = string.Format("{0}.xlsx", "NewWorkbook");
string physicalPath = HttpContext.Current.Server.MapPath("/FilesForExport");
string relativePath = Path.Combine(physicalPath, filename).Replace("\\", "/");
var filePath = relativePath;
// var filePath = directory + "\\NewWorkbook.xlsx";
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
var str = webResponse.GetResponseStream();
var inBuf = new byte[webResponse.ContentLength];
var bytesToRead = Convert.ToInt32(inBuf.Length);
var bytesRead = 0;
while (bytesToRead > 0)
{
var n = str.Read(inBuf, bytesRead, bytesToRead);
if (n == 0)
break;
bytesRead += n;
bytesToRead -= n;
}
var fstr = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Write);
fstr.Write(inBuf, 0, bytesRead);
fstr.Close();
}
return filePath;
}
private static HttpWebRequest createWebRequest(string url)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
webRequest.ContentType = "text/xml;charset=\"utf-8\"";
webRequest.Accept = "text/xml";
webRequest.Method = "GET";
return webRequest;
}
Here is what I have put together from some other examples.
public HttpResponseMessage GenerateMarketStudyResult([FromBody]Result id)
{
if (id.requestId == null)
{
throw new HttpResponseException(new HttpResponseMessage(HttpStatusCode.BadRequest));
}
var serviceUrl = ConfigurationManager.AppSettings["serviceUrl"];
var streamContent = new PushStreamContent((outputStream, httpContext, transportContent) =>
{
try
{
HttpWebRequest webRequest = createWebRequest(serviceUrl + "/" + id.requestId);
IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null);
asyncResult.AsyncWaitHandle.WaitOne();
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
using (MemoryStream memoryStream = new MemoryStream())
{
var str = webResponse.GetResponseStream();
var inBuf = new byte[webResponse.ContentLength];
var bytesToRead = Convert.ToInt32(inBuf.Length);
var bytesRead = 0;
while (bytesToRead > 0)
{
var n = str.Read(inBuf, bytesRead, bytesToRead);
if (n == 0)
break;
bytesRead += n;
bytesToRead -= n;
}
memoryStream.Write(inBuf, 0, bytesRead);
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
outputStream.Close();
}
});
streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
streamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
streamContent.Headers.ContentDisposition.FileName = "reports.xlsx";
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = streamContent
};
return result;
}
I am not getting any exceptions, but the xlsx file is returning with 0bytes.
The breakpoint here
memoryStream.Write(inBuf, 0, bytesRead);
here is the javascript serving the returned file
$http.post('/api/GenerateMarketStudyResult/', Result, { responseType: 'arraybuffer' })
.success(function (response, status, headers, config) {
saveAs(new Blob([response], { type: "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" }), 'reports.xlsx');
})
shows that the
bytesRead = 112336

I assume that you write a web service which acts as a proxy between your JavaScript and some third party web service.
First of all, if you use at least .NET 4.0, you can use the Stream.CopyTo method to copy a stream to another.
So instead of this:
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
var str = webResponse.GetResponseStream();
var inBuf = new byte[webResponse.ContentLength];
var bytesToRead = Convert.ToInt32(inBuf.Length);
var bytesRead = 0;
while (bytesToRead > 0)
{
var n = str.Read(inBuf, bytesRead, bytesToRead);
if (n == 0) break;
bytesRead += n;
bytesToRead -= n;
}
var fstr = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Write);
fstr.Write(inBuf, 0, bytesRead);
fstr.Close();
}
You could write:
using (var webResponse = webRequest.EndGetResponse(asyncResult))
using (var fstr = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Write))
{
webResponse.GetResponseStream().CopyTo(fstr);
}
Second, assuming you use WCF to build a web service, you could pipe the response to a memory stream, and return it. (dont forget to reset the stream's position after you finished writing)
Put together:
[WebGet(UriTemplate = "GenerateMarketStudyResult/{id}")]
public Stream GenerateMarketStudyResult(string id)
{
var serviceUrl = ConfigurationManager.AppSettings["serviceUrl"];
// create webRequest
HttpWebRequest webRequest = createWebRequest(serviceUrl + "/" + id);
// begin async call to web request
IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null);
// suspend this thread until call is complete. You might want to
// do something usefull here like update your UI
asyncResult.AsyncWaitHandle.WaitOne();
var memStream = new MemoryStream();
// var filePath = directory + "\\NewWorkbook.xlsx";
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
webResponse.GetResponseStream().CopyTo(memStream);
}
memStream.Position = 0;
var response = WebOperationContext.Current.OutgoingResponse;
response.ContentType = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
response.ContentLength = (int)memStream.Length;
return memStream;
}
EDIT:
You didn't copy the contents of the memoryStream to the outputStream. You can omit the memoryStream. Try this:
public HttpResponseMessage GenerateMarketStudyResult([FromBody]Result id)
{
if (id.requestId == null)
{
throw new HttpResponseException(new HttpResponseMessage(HttpStatusCode.BadRequest));
}
var serviceUrl = ConfigurationManager.AppSettings["serviceUrl"];
var streamContent = new PushStreamContent((outputStream, httpContext, transportContent) =>
{
try
{
HttpWebRequest webRequest = createWebRequest(serviceUrl + "/" + id.requestId);
IAsyncResult asyncResult = webRequest.BeginGetResponse(null, null);
asyncResult.AsyncWaitHandle.WaitOne();
using (WebResponse webResponse = webRequest.EndGetResponse(asyncResult))
{
webResponse.GetResponseStream().CopyTo(outputStream);
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
}
finally
{
outputStream.Close();
}
});
streamContent.Headers.ContentType = new MediaTypeHeaderValue("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet");
streamContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("attachment");
streamContent.Headers.ContentDisposition.FileName = "reports.xlsx";
var result = new HttpResponseMessage(HttpStatusCode.OK)
{
Content = streamContent
};
return result;
}

Async FTP Upload List of Files

I'm trying to loop through a list of images and ftp them to a server. It's partially working except for it is still blocking my UI thread. Even though my ftp function is async, I'm guessing because my calling method is not, I'm not getting the results I'm looking for. Here's what I got. What am I doing wrong?
public void UploadPictures()
{
//loop through each picture and upload
for (int i = 0; i < this.items.Count; i++) {
byte[] bytes;
if (System.IO.Path.GetExtension (this.items [i].FileName.ToUpper()) == ".JPG") {
using (var imageData = this.items[i].Image.AsJPEG())
{
bytes = new byte[imageData.Length];
Marshal.Copy(imageData.Bytes, bytes, 0, Convert.ToInt32(imageData.Length));
}
var test=UploadPhoto(bytes, this.items[i].FileName);
}
if (System.IO.Path.GetExtension (this.items [i].FileName.ToUpper()) == ".PNG") {
using (var imageData = this.items[i].Image.AsPNG())
{
bytes = new byte[imageData.Length];
Marshal.Copy(imageData.Bytes, bytes, 0, Convert.ToInt32(imageData.Length));
}
var test=UploadPhoto(bytes, this.items[i].FileName);
}
}
}
public static async Task<string> UploadPhoto(byte[] photoBytes, string filename)
{
FtpWebRequest request = (FtpWebRequest)FtpWebRequest.Create("ftp://XXXXXXXX/" + filename);
request.Method = WebRequestMethods.Ftp.UploadFile;
request.Credentials = new NetworkCredential ("user", "pass");
request.UseBinary = true;
request.ContentLength = photoBytes.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(photoBytes, 0,photoBytes.Length);
}
WebResponse ftpResp = await (Task<WebResponse>)request.GetResponseAsync ();
return ftpResp.ToString();
}

Mark UploadPictures as async and await on UploadPhoto.

So I called my UploadPictures function using Task run and that worked. Not sure if thats the most correct way to do it.
Task.Run( () =>
{
UploadPictures();
});

Downloading files not affecting UI

We are currently downloading files using HttpClient because our backend requires certificate.
I have a control - FileRowwhich is an UI element with some code-behind methods for file downloading, like this one:
if (FileIsDownloaded == false)
{
await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Low, () =>
{
DataManager.Instance.DownloadFile(this);
});
}
if (ThumbnailIsDownloaded == false)
{
await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Low, () =>
{
DataManager.Instance.DownloadThumbnail(this);
});
}
Downloading single item is fine but when i click download all ( about 50 items ) the whole UI starts to freeze.
As you can see, i have tried to give requests low priority - but still same result.
Answers to common questions:
1) Yes the files should be downloadable all at one time, not one after another.
2) DataManager.Instance.DownloadThumbnail(this) - i do this to give refference to current control so that i could report a progress in a progress bar.
Any suggestions?
EDIT:
Downloading looks like this:
public async void DownloadFile(FileRow fileRow)
{
//Lot of checking for if file exist, if version is the same
string LocalFilename = await DownloadManager.DownloadFile(fileRow.MyFile.file.id, fileRow.MyFile.file.version, fileRow.MyFile.file.filename,fileRow);
// next is just using the filename string
}
And finally my download:
public static async Task<string> DownloadFileOfCustomerAssetRow(int? id, int? version, string filename, FileRow fileRow)
{
try
{
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
customerAssetRow.CurrentFileDownload = aClient;
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(WebServices.BackendStartUrl + "getFileData?id=" + id + "&version=" + version, HttpCompletionOption.ResponseHeadersRead);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync(filename, Windows.Storage.CreationCollisionOption.GenerateUniqueName);
fileRow.FileName = file.Name;
using (var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite))
{
Stream stream = await response.Content.ReadAsStreamAsync();
IInputStream inputStream = stream.AsInputStream();
ulong totalBytesRead = 0;
while (true)
{
IBuffer buffer = new Windows.Storage.Streams.Buffer(1024);
buffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.None);
if (buffer.Length == 0)
{
break;
}
totalBytesRead += buffer.Length;
fileRow.Progress.Value = fileRow.Progress.Value + 1024;
await fs.WriteAsync(buffer);
}
inputStream.Dispose();
}
fileRow.Progress.Visibility = Visibility.Collapsed;
return file.Name;
}
catch (Exception e)
{
ErrorReporter.ReportError("Error in DownloadManager.cs in function DownloadFile.", e);
return "";
}
}

It's possible that your async method continuations are interrupting the UI thread too much. Try creating a stronger separation between your background logic (DownloadFileOfCustomerAssetRow) and your UI (FileRow) by introducing an IProgress<T> reporter. Then ensure that every await in your background logic has a ConfigureAwait(false) on it.
public static async Task<string> DownloadFileOfCustomerAssetRow(int? id, int? version, string filename, IProgress<int> progress)
{
HttpClientHandler aHandler = new HttpClientHandler();
aHandler.ClientCertificateOptions = ClientCertificateOption.Automatic;
HttpClient aClient = new HttpClient(aHandler);
customerAssetRow.CurrentFileDownload = aClient;
aClient.DefaultRequestHeaders.ExpectContinue = false;
HttpResponseMessage response = await aClient.GetAsync(WebServices.BackendStartUrl + "getFileData?id=" + id + "&version=" + version, HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);
var file = await ApplicationData.Current.LocalFolder.CreateFileAsync(filename, Windows.Storage.CreationCollisionOption.GenerateUniqueName).ConfigureAwait(false);
fileRow.FileName = file.Name;
using (var fs = await file.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite).ConfigureAwait(false))
{
Stream stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);
IInputStream inputStream = stream.AsInputStream();
ulong totalBytesRead = 0;
while (true)
{
IBuffer buffer = new Windows.Storage.Streams.Buffer(1024);
buffer = await inputStream.ReadAsync(
buffer,
buffer.Capacity,
InputStreamOptions.None).ConfigureAwait(false);
if (buffer.Length == 0)
{
break;
}
totalBytesRead += buffer.Length;
if (progress != null)
progress.Report(totalBytesRead);
await fs.WriteAsync(buffer).ConfigureAwait(false);
}
inputStream.Dispose();
}
return file.Name;
}

Categories