We Keep Coding

Send and receive large file over streams in ASP.NET Web Api C#

I'm working on a project where I need to send large audio files via streams from a client to a server. I'm using the ASP.NET Web Api to communicate between client and server. My client has a "SendFile" method which I believe works fine, but I don't know how to make my server receive the data I'm sending via a stream. My client code looks like this so far:
private const int MAX_CHUNK_SIZE = (1024 * 5000);
private HttpWebRequest webRequest = null;
private FileStream fileReader = null;
private Stream requestStream = null;
public bool SendAudio(string uri, string file)
{
byte[] fileData;
fileReader = new FileStream(file, FileMode.Open, FileAccess.Read);
webRequest = (HttpWebRequest)WebRequest.Create(uri);
webRequest.Method = "POST";
webRequest.ContentLength = fileReader.Length;
webRequest.Timeout = 600000;
webRequest.Credentials = CredentialCache.DefaultCredentials;
webRequest.AllowWriteStreamBuffering = false;
requestStream = webRequest.GetRequestStream();
long fileSize = fileReader.Length;
long remainingBytes = fileSize;
int numberOfBytesRead = 0, done = 0;
while (numberOfBytesRead < fileSize)
{
SetByteArray(out fileData, remainingBytes);
done = WriteFileToStream(fileData, requestStream);
numberOfBytesRead += done;
remainingBytes -= done;
}
fileReader.Close();
return true;
}
public int WriteFileToStream(byte[] fileData, Stream requestStream)
{
int done = fileReader.Read(fileData, 0, fileData.Length);
requestStream.Write(fileData, 0, fileData.Length);
return done;
}
private void SetByteArray(out byte[] fileData, long bytesLeft)
{
fileData = bytesLeft < MAX_CHUNK_SIZE ? new byte[bytesLeft] : new byte[MAX_CHUNK_SIZE];
}
My server looks like this:
[HttpPost]
[ActionName("AddAudio")]
public async Task<IHttpActionResult> AddAudio([FromUri]string name)
{
try
{
isReceivingFile = true;
byte[] receivedBytes = await Request.Content.ReadAsByteArrayAsync();
if (WriteAudio(receivedBytes, name) == true)
{
isReceivingFile = false;
return Ok();
}
else
{
isReceivingFile = false;
return BadRequest("ERROR: Audio could not be saved on server.");
}
}
catch (Exception ex)
{
isReceivingFile = false;
return BadRequest("ERROR: Audio could not be saved on server.");
}
}
public bool WriteAudio(byte[] receivedBytes, string fileName)
{
string file = Path.Combine(#"C:\Users\username\Desktop\UploadedFiles", fileName);
using (FileStream fs = File.Create(file))
{
fs.Write(receivedBytes, 0, receivedBytes.Length);
}
return true;
}
The server code has the original code I wrote for it, before deciding to try and make it work with streams. The server code still works if I send a small file (under 30 MB), but if I send a large file my server gets a "outofmemoryexception". I can't figure out how to make the server take in the data via a stream. In my search for solutions I've come across a lot of examples with sockets and TCPClient, but that's not how we want to do it on this project. Can anybody help, please?

if I send a large file my server gets a "outofmemoryexception"
Well, it's reading the entire stream into memory right here:
byte[] receivedBytes = await Request.Content.ReadAsByteArrayAsync();
What you want to do is copy the stream from one location to another, without loading it all into memory at once. Something like this should work:
[HttpPost]
[ActionName("AddAudio")]
public async Task<IHttpActionResult> AddAudio([FromUri]string name)
{
try
{
string file = Path.Combine(#"C:\Users\username\Desktop\UploadedFiles", fileName);
using (FileStream fs = new FileStream(file, FileMode.Create, FileAccess.Write,
FileShare.None, 4096, useAsync: true))
{
await Request.Content.CopyToAsync(fs);
}
return Ok();
}
catch (Exception ex)
{
return BadRequest("ERROR: Audio could not be saved on server.");
}
}

Can't consume byte[] or stream from WCF to create an image

UPDATE: Got it! Answer posted below... I will clean up the question so it may be more useful to someone else.
I have a service that uses a library to generate thumbnails from html pages using a WebBrowser control. That works fine but I need to create a WCF service that allows you to pass in a uri, then the service converts that to a thumbnail and returns it. I've tried it as a Stream and a byte[]. It creates the image in the service, I save it as a file just to prove it, but when I consume it I get data (more than I send) and I am unable to save it as a viewable image. Any suggestions would be appreciated. I am not a WCF expert so I'm hoping I've just missed something that will be easy to spot.
Here is the service, it's hosted in a console app and the Main() procedure is included.
namespace RawImageService
{
[ServiceContract]
public interface IImageServer
{
[OperationContract]
[WebInvoke(Method = "GET", UriTemplate = "/image/?uri={uri}")]
Stream HtmlToImage(string uri);
}
public class Service : IImageServer
{
public Stream HtmlToImage(string uri)
{
string path = #"D:\Templates\HtmlServiceImage.bmp";
if( File.Exists(path) )
File.Delete(path);
if (string.IsNullOrEmpty(uri))
{
return null;
}
else
{
if ((uri.IndexOf("file:", System.StringComparison.Ordinal) < 0) &&
(uri.IndexOf("http", System.StringComparison.Ordinal) < 0))
uri = "http://" + uri;
Thumbnail.Uri = uri;
try
{
Bitmap bitmap =
HtmlToThumbnail.WebsiteThumbnail.GetThumbnail(Thumbnail.Uri, Thumbnail.Width,
Thumbnail.Hight, Thumbnail.ThumbWidth,
Thumbnail.ThumbHight);
using (MemoryStream ms = new MemoryStream())
{
bitmap.Save(ms, ImageFormat.Jpeg);
bitmap.Save(path, ImageFormat.Jpeg);
ms.Position = 0;
WebOperationContext.Current.OutgoingResponse.ContentType = "image/jpeg";
return ms;
}
}
catch (Exception)
{
throw;
}
return null;
}
}
static void Main(string[] args)
{
string baseAddress = "http://" + Environment.MachineName + ":8000/";
ServiceHost host = new ServiceHost(typeof(Service), new Uri(baseAddress));
host.AddServiceEndpoint(typeof(IImageServer), new WebHttpBinding(), "").Behaviors.Add(new WebHttpBehavior());
host.Open();
Console.WriteLine("Service is running");
Console.Write("Press ENTER to close the host");
Console.ReadLine();
host.Close();
}
}
}
Here is my attempt to consume the service:
private static void Main(string[] args)
{
string uri = string.Concat("http://localhost:8000",
string.Format("/image/?uri={0}", "file:///D:/Templates/Test4.htm"));
tryThis(uri);
}
public static void tryThis(string uri)
{
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;
request.Method = "GET";
// Get response
using (WebResponse response = request.GetResponse() as WebResponse)
{
using (Stream stream = response.GetResponseStream())
{
byte[] buffer = new byte[response.ContentLength];
MemoryStream ms = new MemoryStream();
int bytesRead, totalBytesRead = 0;
do
{
bytesRead = stream.Read(buffer, 0, buffer.Length);
totalBytesRead += bytesRead;
ms.Write(buffer, 0, bytesRead);
} while (bytesRead > 0);
string path = #"D:/templates/fs.jpg";
if (File.Exists(path))
File.Delete(path);
var fs = new FileStream(path, FileMode.Create);
fs.Write(ms.ToArray(), 0, totalBytesRead);
fs.Flush();
fs.Close();
}
}
}
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
There is nothing additional in the .config files. I'm going to start looking into that but my understanding is I can configure everything in the app itself.
I don't care which way I get it working, I just need to get it working. I appreciate any suggestions. Thanks.

I got the Stream method working. Here is the code. I can access it like this:
http://localhost:8000/image/?uri=file:///D:/Templates/Test4.htm
Here is the new method. If you replace the code above with this it works.
public Stream HtmlToImage(string uri)
{
string path = #"D:\Templates\HtmlServiceImage.bmp";
if( File.Exists(path) )
File.Delete(path);
if (string.IsNullOrEmpty(uri))
{
return null;
}
else
{
if ((uri.IndexOf("file:", System.StringComparison.Ordinal) < 0) &&
(uri.IndexOf("http", System.StringComparison.Ordinal) < 0))
uri = "http://" + uri;
Thumbnail.Uri = uri;
try
{
Bitmap bitmap =
HtmlToThumbnail.WebsiteThumbnail.GetThumbnail(Thumbnail.Uri, Thumbnail.Width,
Thumbnail.Hight, Thumbnail.ThumbWidth,
Thumbnail.ThumbHight);
MemoryStream ms = new MemoryStream();
bitmap.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
ms.Position = 0;
WebOperationContext.Current.OutgoingResponse.ContentType = "image/jpeg";
return ms;
}
catch (Exception)
{
throw;
}
return null;
}
}

To download a zipped file from FTP

I am trying to do this using C#(Winforms).
The code I am using is giving me a string as an output, but I need to have a zipped file.
I am using the following code
try
{
FtpWebRequest request = (FtpWebRequest)WebRequest.Create(
"ftp:SITENAME/FILENAME.zip");
request.Method = WebRequestMethods.Ftp.DownloadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential("", "");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
//StreamWriter writer = new StreamWriter(responseStream);
MessageBox.Show(reader.ReadtoEnd);
MessageBox.Show("Download Complete, status {0}" + response.StatusCode);
reader.Close();
response.Close();
}
catch (NotSupportedException ne)
{
MessageBox.Show(ne.Message);
}
I have an idea that I have to use Open source library from SharpZipLib to convert this string to a zipped file. But,I cannot find any sample code to show me how to do that.
I would really appreciate if someone can guide me through the process.
Thanks,
Sidhanshu

This might do just that.
using ICSharpCode.SharpZipLib.BZip2;
public static string Unzip(byte[] compressedbytes)
{
string result;
MemoryStream m_msBZip2 = null;
BZip2InputStream m_isBZip2 = null;
m_msBZip2 = new MemoryStream(compressedbytes);
// read final uncompressed string size stored in first 4 bytes
using (BinaryReader reader = new BinaryReader(m_msBZip2, System.Text.Encoding.ASCII))
{
Int32 size = reader.ReadInt32();
m_isBZip2 = new BZip2InputStream(m_msBZip2);
byte[] bytesUncompressed = new byte[size];
m_isBZip2.Read(bytesUncompressed, 0, bytesUncompressed.Length);
m_isBZip2.Close();
m_msBZip2.Close();
result = Encoding.ASCII.GetString(bytesUncompressed, 0, bytesUncompressed.Length);
reader.Close();
}
return result;
}
public static byte[] Zip(string sBuffer)
{
byte[] result;
using (MemoryStream m_msBZip2 = new MemoryStream())
{
Int32 size = sBuffer.Length;
// Prepend the compressed data with the length of the uncompressed data (firs 4 bytes)
using (BinaryWriter writer = new BinaryWriter(m_msBZip2, System.Text.Encoding.ASCII))
{
writer.Write(size);
using (BZip2OutputStream m_osBZip2 = new BZip2OutputStream(m_msBZip2))
{
m_osBZip2.Write(Encoding.ASCII.GetBytes(sBuffer), 0, sBuffer.Length);
m_osBZip2.Close();
}
writer.Close();
result = m_msBZip2.ToArray();
m_msBZip2.Close();
}
}
return result;
}

How to use httpwebrequest to pull image from website to local file

I'm trying to use a local c# app to pull some images off a website to files on my local machine. I'm using the code listed below. I've tried both ASCII encoding and UTF8 encoding but the final file is not an correct. Does anyone see what I'm doing wrong? The url is active and correct and show the image just fine when I put the address in my browser.
private void button1_Click(object sender, EventArgs e)
{
HttpWebRequest lxRequest = (HttpWebRequest)WebRequest.Create("http://www.productimageswebsite.com/images/stock_jpgs/34891.jpg");
// returned values are returned as a stream, then read into a string
String lsResponse = string.Empty;
HttpWebResponse lxResponse = (HttpWebResponse)lxRequest.GetResponse();
using (StreamReader lxResponseStream = new StreamReader(lxResponse.GetResponseStream()))
{
lsResponse = lxResponseStream.ReadToEnd();
lxResponseStream.Close();
}
byte[] lnByte = System.Text.UTF8Encoding.UTF8.GetBytes(lsResponse);
System.IO.FileStream lxFS = new FileStream("34891.jpg", FileMode.Create);
lxFS.Write(lnByte, 0, lnByte.Length);
lxFS.Close();
MessageBox.Show("done");
}

nice image :D
try using the following code:
you needed to use a BinaryReader, 'cause an image file is binary data and thus not encoded in UTF or ASCII
edit: using'ified
HttpWebRequest lxRequest = (HttpWebRequest)WebRequest.Create(
"http://www.productimageswebsite.com/images/stock_jpgs/34891.jpg");
// returned values are returned as a stream, then read into a string
String lsResponse = string.Empty;
using (HttpWebResponse lxResponse = (HttpWebResponse)lxRequest.GetResponse()){
using (BinaryReader reader = new BinaryReader(lxResponse.GetResponseStream())) {
Byte[] lnByte = reader.ReadBytes(1 * 1024 * 1024 * 10);
using (FileStream lxFS = new FileStream("34891.jpg", FileMode.Create)) {
lxFS.Write(lnByte, 0, lnByte.Length);
}
}
}
MessageBox.Show("done");

Okay, here's the final answer. It uses a memorystream as a way to buffer the data from the reaponsestream.
private void button1_Click(object sender, EventArgs e)
{
byte[] lnBuffer;
byte[] lnFile;
HttpWebRequest lxRequest = (HttpWebRequest)WebRequest.Create("http://www.productimageswebsite.com/images/stock_jpgs/34891.jpg");
using (HttpWebResponse lxResponse = (HttpWebResponse)lxRequest.GetResponse())
{
using (BinaryReader lxBR = new BinaryReader(lxResponse.GetResponseStream()))
{
using (MemoryStream lxMS = new MemoryStream())
{
lnBuffer = lxBR.ReadBytes(1024);
while (lnBuffer.Length > 0)
{
lxMS.Write(lnBuffer, 0, lnBuffer.Length);
lnBuffer = lxBR.ReadBytes(1024);
}
lnFile = new byte[(int)lxMS.Length];
lxMS.Position = 0;
lxMS.Read(lnFile, 0, lnFile.Length);
}
}
}
using (System.IO.FileStream lxFS = new FileStream("34891.jpg", FileMode.Create))
{
lxFS.Write(lnFile, 0, lnFile.Length);
}
MessageBox.Show("done");
}

A variation of the answer, using async await for async file I/O. See Async File I/O on why this is important.
Download png and write to disk using BinaryReader/Writer
string outFile = System.IO.Path.Combine(outDir, fileName);
// Download file
var request = (HttpWebRequest) WebRequest.Create(imageUrl);
using (var response = await request.GetResponseAsync()){
using (var reader = new BinaryReader(response.GetResponseStream())) {
// Read file
Byte[] bytes = async reader.ReadAllBytes();
// Write to local folder
using (var fs = new FileStream(outFile, FileMode.Create)) {
await fs.WriteAsync(bytes, 0, bytes.Length);
}
}
}
Read all bytes extension method
public static class Extensions {
public static async Task<byte[]> ReadAllBytes(this BinaryReader reader)
{
const int bufferSize = 4096;
using (var ms = new MemoryStream())
{
byte[] buffer = new byte[bufferSize];
int count;
while ((count = reader.Read(buffer, 0, buffer.Length)) != 0) {
await ms.WriteAsync(buffer, 0, count);
}
return ms.ToArray();
}
}
}

You can use the following method to download an image from a web site and save it, using the Image class:
WebRequest req = WebRequest.Create(imageUrl);
WebResponse resp = req.GetResponse();
Image img = Image.FromStream(resp.GetResponseStream());
img.Save(filePath + fileName + ".jpg");

Stream a PDF to a web page failing

I have a URL to a PDF and I want to serve up the PDF to my page viewer.
I can successfully (I think) retrieve the PDF file. Then when I do the Response.BinaryWrite() I get a "The file is damaged and could not be repaired" error from the adobe reader.
Here's the code I have:
protected void Page_Load(object sender, EventArgs e)
{
if (IsPostBack)
{
byte[] output = DoWork("Http://localhost/test.pdf");
Response.Clear();
Response.ContentType = "application/pdf";
Response.AddHeader("content-disposition", "inline; filename=MyPDF.PDF");
Response.AddHeader("content-length", output.Length.ToString());
Response.BinaryWrite(output);
Response.End();
}
}
public byte[] DoWork(string requestUrl)
{
byte[] responseData;
HttpWebRequest req = null;
HttpWebResponse resp = null;
StreamReader strmReader = null;
try
{
req = (HttpWebRequest)WebRequest.Create(requestUrl);
using (resp = (HttpWebResponse)req.GetResponse())
{
byte[] buffer = new byte[resp.ContentLength];
BinaryReader reader = new BinaryReader(resp.GetResponseStream());
reader.Read(buffer, 0, buffer.Length);
responseData = buffer;
}
}
finally
{
if (req != null)
{
req = null;
}
if (resp != null)
{
resp.Close();
resp = null;
}
}
return responseData;
}

Apparently, I need to use ReadBytes() For some reason, when reading a PDF from a URL, You don't get all of the bytes that you requested.
protected void Page_Load(object sender, EventArgs e)
{
if (IsPostBack)
{
byte[] output = DoWork("Http://localhost/test.pdf");
Response.Clear();
Response.ContentType = "application/pdf";
Response.AddHeader("content-disposition", "attachment");
Response.AddHeader("content-length", output.Length.ToString());
Response.BinaryWrite(output);
Response.End();
}
}
public byte[] DoWork(string requestUrl)
{
byte[] responseData;
HttpWebRequest req = null;
HttpWebResponse resp = null;
StreamReader strmReader = null;
try
{
req = (HttpWebRequest)WebRequest.Create(requestUrl);
using (resp = (HttpWebResponse)req.GetResponse())
{
byte[] buffer = new byte[resp.ContentLength];
using (BinaryReader reader = new BinaryReader(resp.GetResponseStream()))
{
buffer = reader.ReadBytes(buffer.Length);
reader.Close();
}
responseData = buffer;
}
}
finally
{
if (req != null)
{
req = null;
}
if (resp != null)
{
resp.Close();
resp = null;
}
}
return responseData;
}

Try saving the resulting file to your disk. Then open the file with a text editor. Maybe there are some errors in your script/source file.

You can also try to use FileStream the read the file
string strPath = Request.PhysicalApplicationPath
+ "\\document\\Test.pdf";
FileStream fStream = new FileStream
(strPath, FileMode.Open, FileAccess.Read);
StreamReader sReader = new StreamReader(fStream);

Try to flush the response after the binary write...
Response.BinaryWrite(output);
Response.Flush();
Response.End();
Alternatively, instead of inline, try to output the PDF as an attachment:
Response.AddHeader("Content-Disposition", "attachment;filename=MyPDF.PDF");

When I did this in Perl the other day (as a quick hack for our intranet), the crux of the script was:
binmode(STDOUT);
print "Content-type: application/pdf\n\n";
binmode(FILE);
print <FILE>;
close(FILE);
The key points being to make sure that the input and output streams are in binary mode, i.e. as you've found, the PDF needs to be interpreted as binary data throughout the chain.

You might be able to simplify your code quite a bit by using the WebClient class
Here's the MSDN documentation. Its not as cumbersome as the lower level HttpWebRequest class.