I have HTTP server written using HttpListener and want zero-copy technology for sending files to clients.
Is there are any option to use TransmitFile to respond?
I assume you're referring to HttpResponse.TransmitFile? HttpListener doesn't buffer the response content, so you just need to write directly to the output stream.
You can use an extension method like this to mimic the ASP.NET behavior:
public static void TransmitFile(this HttpListenerResponse response, string fileName)
{
using (var fileStream = File.OpenRead(filename))
{
response.ContentLength64 = fileStream.Length;
fileStream.CopyTo(response.OutputStream);
}
}
Related
I want to send file as byte[] to another PC via HTTP POST method. What is the most efficient way to assemble the file from byte[] on the other side? I am using File.ReadAllBytes method to get byte[] from file.
If you are using tcp the network protocol will make sure that your stream is coming in the right order and without dropped parts. Therefore, the simplest readstream will be the most efficient. If you want to use parallel routes and play with the datagrams.
If the file is large you will have to transmit and receive it in chunks. But the IP streams can hide that from you.
For example: https://learn.microsoft.com/en-us/dotnet/api/system.net.sockets.tcpclient.getstream?view=netframework-4.7.2
This is what worked for me. I used this method to call api method and send file as byte[]. I tried sending whole byte[] but api method wasn't able to recive it.
private static async void SendFiles(string path)
{
var bytes = File.ReadAllBytes(path);
var length = bytes.Length;
foreach (var b in bytes)
{
length--;
string sendFilesUrl = $"http://api/communication/sendF/{b}/{length}";
StringContent queryString = new StringContent(bytes.ToString(), Encoding.UTF8, "application/x-www-form-urlencoded");
HttpResponseMessage response = await client.PostAsync(sendFilesUrl, queryString);
string responseBody = await response.Content.ReadAsStringAsync();
}
}
My api method is
[HttpPost]
[Route("sendF/{b}/{length}")]
public HttpResponseMessage SendF([FromUri]byte[] b, [FromUri]int length)
{
if(length != 0)
{
bytes.AddRange(b);
}
else
{
File.WriteAllBytes(#"G:\test\test.exe", bytes.ToArray<byte>());
}
return CreateResponse(client);
}
This code works for me but it takes very long to pass all bytes if the file is large. Currently I'm searching for more efficient way of sending bytes. One solution that came to my mind is to send byte[] in chunks
My program uses HttpClient to send a GET request to a Web API, and this returns a file.
I now use this code (simplified) to store the file to disc:
public async Task<bool> DownloadFile()
{
var client = new HttpClient();
var uri = new Uri("http://somedomain.com/path");
var response = await client.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
var fileName = response.Content.Headers.ContentDisposition.FileName;
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.Content.CopyToAsync(fs);
return true;
}
}
return false;
}
Now, when this code runs, the process loads all of the file into memory. I actually would rather expect the stream gets streamed from the HttpResponseMessage.Content to the FileStream, so that only a small portion of it is held in memory.
We are planning to use that on large files (> 1GB), so is there a way to achieve that without having all of the file in memory?
Ideally without manually looping through reading a portion to a byte[] and writing that portion to the file stream until all of the content is written?
It looks like this is by-design - if you check the documentation for HttpClient.GetAsync() you'll see it says:
The returned task object will complete after the whole response
(including content) is read
You can instead use HttpClient.GetStreamAsync() which specifically states:
This method does not buffer the stream.
However you don't then get access to the headers in the response as far as I can see. Since that's presumably a requirement (as you're getting the file name from the headers), then you may want to use HttpWebRequest instead which allows you you to get the response details (headers etc.) without reading the whole response into memory. Something like:
public async Task<bool> DownloadFile()
{
var uri = new Uri("http://somedomain.com/path");
var request = WebRequest.CreateHttp(uri);
var response = await request.GetResponseAsync();
ContentDispositionHeaderValue contentDisposition;
var fileName = ContentDispositionHeaderValue.TryParse(response.Headers["Content-Disposition"], out contentDisposition)
? contentDisposition.FileName
: "noname.dat";
using (var fs = new FileStream(#"C:\test\" + fileName, FileMode.Create, FileAccess.Write, FileShare.None))
{
await response.GetResponseStream().CopyToAsync(fs);
}
return true
}
Note that if the request returns an unsuccessful response code an exception will be thrown, so you may wish to wrap in a try..catch and return false in this case as in your original example.
Instead of GetAsync(Uri) use the the GetAsync(Uri, HttpCompletionOption) overload with the HttpCompletionOption.ResponseHeadersRead value.
The same applies to SendAsync and other methods of HttpClient
Sources:
docs (see remarks)
https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.getasync?view=netcore-1.1#System_Net_Http_HttpClient_GetAsync_System_Uri_System_Net_Http_HttpCompletionOption_
The returned Task object will complete based on the completionOption parameter after the part or all of the response (including content) is read.
.NET Core implementation of GetStreamAsync that uses HttpCompletionOption.ResponseHeadersRead https://github.com/dotnet/corefx/blob/release/1.1.0/src/System.Net.Http/src/System/Net/Http/HttpClient.cs#L163-L168
HttpClient spike in memory usage with large response
HttpClient.GetStreamAsync() with custom request? (don't mind the comment on response, the ResponseHeadersRead is what does the trick)
Another simple and quick way to do it is:
public async Task<bool> DownloadFile(string url)
{
using (MemoryStream ms = new MemoryStream()) {
new HttpClient().GetStreamAsync(webPath).Result.CopyTo(ms);
... // use ms in what you want
}
}
now you have the file downloaded as stream in ms.
Im trying to create a web service which gets to a URL e.g. www.domain.co.uk/prices.csv and then reads the csv file. Is this possible and how? Ideally without downloading the csv file?
You could use:
public string GetCSV(string url)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
StreamReader sr = new StreamReader(resp.GetResponseStream());
string results = sr.ReadToEnd();
sr.Close();
return results;
}
And then to split it:
public static void SplitCSV()
{
List<string> splitted = new List<string>();
string fileList = getCSV("http://www.google.com");
string[] tempStr;
tempStr = fileList.Split(',');
foreach (string item in tempStr)
{
if (!string.IsNullOrWhiteSpace(item))
{
splitted.Add(item);
}
}
}
Though there are plenty of CSV parsers out there and i would advise against rolling your own. FileHelpers is a good one.
// Download the file to a specified path. Using the WebClient class we can download
// files directly from a provided url, like in this case.
System.Net.WebClient client = new WebClient();
client.DownloadFile(url, csvPath);
Where the url is your site with the csv file and the csvPath is where you want the actual file to go.
In your Web Service you could use the WebClient class to download the file, something like this ( I have not put any exception handling, not any using or Close/Dispose calls, just wanted to give the idea you can use and refine/improve... )
using System.Net;
WebClient webClient = new WebClient();
webClient.DownloadFile("http://www.domain.co.uk/prices.csv");
then you can do anything you like with it once the file content is available in the execution flow of your service.
if you have to return it to the client as return value of the web service call you can either return a DataSet or any other data structure you prefer.
Sebastien Lorion's CSV Reader has a constructor that takes a Stream.
If you decided to use this, your example would become:
void GetCSVFromRemoteUrl(string url)
{
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
using (CsvReader csvReader = new CsvReader(response.GetResponseStream(), true))
{
int fieldCount = csvReader.FieldCount;
string[] headers = csvReader.GetFieldHeaders();
while (csvReader.ReadNextRecord())
{
//Do work with CSV file data here
}
}
}
The ever popular FileHelpers also allows you to read directly from a stream.
The documentation for WebRequest has an example that uses streams. Using a stream allows you to parse the document without storing it all in memory
I want to send two different lines of XML to a socket as follows:
<GetServerTime UpTimeAtRequest="1919"/>
<Subscribe FrontId="priceFeed" URI="ffo:/price/productCode=VP-4-6-"/>
I'm using Hercules to test this, but it won't let me send it in that format. How should I delimit or format the XML above so I can send it directly to the socket with Hercules after connecting to the appropriate ip address and port?
I would be happy to send this using a WebClient or something in C# also.
Thanks.
I've no idea what Hercules is, but sending arbitrary data over a client is easy:
using (var client = new TcpClient())
{
client.Connect(host, porg);
using (var stream = client.GetStream())
{
// Or some other encoding, of course...
byte[] data = Encoding.UTF8.GetBytes(xmlString);
stream.Write(data, 0, data.Length);
// Whatever else you want to do...
}
}
WebClient is pretty simple to use (assuming your socket is using the HTTP protocol), so using UploadString would look something like this:
Uri uri = new Uri(#"http://www.mywebsite.com/someurl");
string myXml = "<insert valid xml here> /"
using (WebClient wc = new WebClient()
{
wc.UploadString(uri, myXml);
}
I'd only worry that your xml isn't valid as it has two root nodes and no xml header.
How to download first 200 bytes of a file via HTTP protocol using C#?
I believed it could be done like this:
WebClient wc = new WebClient();
byte[] buffer = new byte[200];
using (var stream = wc.OpenRead(fileName))
{
stream.Read(buffer, 0, 200);
}
but when wc.OpenRead it called it downloads the whole file.
You need to set a Range Header on your WebClient before you invoke the OpenRead method.
See: http://msdn.microsoft.com/en-us/library/system.net.webclient.headers.aspx