I'm using Response.Filter in order to implement stream compression in accordance with HTTP Request Header Accept-Encoding
Here's the important stuff:
if (AcceptEncoding.Contains("deflate") || AcceptEncoding == "*")
{
HttpApp.Response.Filter = new DeflateStream(PreviousOutputStream, CompressionMode.Compress);
HttpApp.Response.AppendHeader("Content-Encoding", "deflate");
}
By and large this works as intended. However, I'm in a situation where I'm using an ActionResult on an MVC Controller to serve up files to the user agent:
Response.Clear();
Response.Headers["Content-Type"] = contentType;
Response.Headers["Content-Length"] = contentLength;
if (Request.QueryString["dl"] == "1")
{
Response.Headers["Content-Disposition"] = "attachment; filename=" + fileInfo.Name;
}
Response.Flush();
Response.TransmitFile(fileInfo.FullName);
To be more exact, the action method returns new EmptyResult() after the Response.TransmitFile() call. This works exactly as intended without the Response.Filter modification.
In this situation, the response entity reaches the user agent garbled and unintelligible. FireFox's Poster addon shows empty entities or jumbled entities coming back.
If you can help it, definitely look for alternatives, because manually doing compression in ASP.NET is NOT fun. But, if you are as hard-headed as I am, I submit to you the following.
First of all: do NOT use .NET's built-in compression stream classes. They are buggy and can truncate bytes off the end of streams at random. I've been using DotNetZip with good results: http://dotnetzip.codeplex.com/
Now, some additional notes:
Response.TransmitFile() doesn't work with response filtering.
Response.BinaryWrite() doesn't work with response filtering, so you can't loop over the contents of the file and write it out that way.
Response.OutputStream doesn't work with response filtering, so you can't loop over the contents of the file and write it out THAT way, either.
Response.WriteFile() DOES work with response filtering, but it loads the entire file into memory and keeps it there until the client closes the connection, which doesn't work well for large files.
And to make things just that little bit more fun: response filtering stops working if you set Response.BufferOutput to false. (I just spent literally hours figuring that out)
Obviously, there's a LOT of different issues surrounding response filtering and writing to the output stream. Using Reflector and lots of experimentation, this is the best ("best" being in terms of working correctly in a variety of scenarios) solution I've found thus far:
Write a class that extends Encoding and call it BinaryEncoding. Implement all the methods so that they copy characters and bytes correctly, but of course doing the necessary type casts.
Set Response.ContentEncoding to an instance of BinaryEncoding (you can use the singleton pattern quite successfully for this).
Open your file with FileStream.
Create a new StreamReader(fileStream, new BinaryEncoding(), false). That "false" parameter is very important, it stops the StreamReader from eating byte order marks and overriding your BinaryEncoding.
Allocate a buffer of char[] (I've found that 32KB is a good size).
Then, in a loop:
int n = StreamReader.Read(buffer, 0, buffer.Length);
Response.Write(buffer, 0, n);
Response.Flush();
Until n is 0.
Be warned: this method results in fairly high CPU usage. On a 100 megabit LAN, CPU usage on one core goes to around 40-50% for a single client downloading at 10MB/sec. I wish I could find a better way... If I had Reflector Pro, I might be able to find one.
If you are using IIS7 or IIS7.5 I would suggest using the HTTP Compression module rather than rolling your own. It might help solve the problem.
http://technet.microsoft.com/en-us/library/cc771003(WS.10).aspx
Related
Currently, I have a feature on an ASP.NET website where the user can play back MP3 Files. The code looks something like this:
Response.Clear();
Response.ContentType = "audio/mpeg";
foreach (DataChunk leChunk in db.Mp3Files.First(mp3 => mp3.Mp3ResourceId.Equals(id)).Data.Chunks.OrderBy(chunk => chunk.ChunkOrder))
{
Response.BinaryWrite(leChunk.Data);
}
Unfortunately, if a larger MP3 file is selected, the audio does not begin to play until the entire file is downloaded, which can cause a noticeable delay. Is there any way to get the MP3 to start playing immediately, even though the entire file may not yet be transferred?
You should be able to do what you want by writing to the outpstream of the response, i.e.:
Response.OutputStream.Write
It is also probably a good idea to check previously if Response.IsClientConnected and give up if not.
I found a demo that allows playback of mp3 files from an asp.net web application:
http://aspsnippets.com/Articles/Save-MP3-Audio-Files-to-database-and-display-in-ASPNet-GridView-with-Play-and-Download-option.aspx
try this:
Response.BufferOutput = false; //sets chunked encoding
Response.ContentType = "audio/mpeg";
using (var bw = new BinaryWriter(Response.OutputStream))
{
foreach (DataChunk leChunk in db.Mp3Files.First(mp3 => mp3.Mp3ResourceId.Equals(id)).Data.Chunks.OrderBy(chunk => chunk.ChunkOrder))
{
if (Response.IsClientConnected) //avoids the host closed the connection exception
{
bw.Write(leChunk.Data);
}
}
}
Also, go yo your web.config file and do this if you still have problems with chunked encoding:
<system.webServer>
<asp enableChunkedEncoding="true" />
</system.webServer>
The error you reported above about the host being closing the connection is happening probably because you are opening the page using the browser and when the browser reads the content type, it opens the media player and closes itself who had the opened connection which was then closed, causing that error, so to avoid this, you need to check periodically whether your client is still connected or not.
Finally, I would use a Generic Handler (.ashx) or a custom handler and set a .mp3 extension for this if you are using a aspx page to avoid the unnecessary overhead of the web page.
I hope this helps.
Try setting Response.BufferOutput = false before streaming the response.
If the location of the MP3 files are publicly available to your user then an alternative approach could be to just return the MP3's URL and use the HTML 5 audio tags in your mark up to stream the music. I am pretty sure that the default behaviour of the audio tag would be to stream the file rather than wait until the whole file has downloaded.
One method to support this would be implementing HTTP byte range requests.
By default I don't believe that ASP.NET does this, and definitely won't if using any of the code in the questions or the answer.
You can implement this manually with a little work though. Another option, which would be much less dev work, would be to let IIS serve a static file. I assume that isn't an option though.
Here's an example implementation:
http://www.codeproject.com/Articles/820146/HTTP-Partial-Content-In-ASP-NET-Web-API-Video
I'm compressing a log file as data is written to it, something like:
using (var fs = new FileStream("Test.gz", FileMode.Create, FileAccess.Write, FileShare.None))
{
using (var compress = new GZipStream(fs, CompressionMode.Compress))
{
for (int i = 0; i < 1000000; i++)
{
// Clearly this isn't what is happening in production, just
// a simply example
byte[] message = RandomBytes();
compress.Write(message, 0, message.Length);
// Flush to disk (in production we will do this every x lines,
// or x milliseconds, whichever comes first)
if (i % 20 == 0)
{
compress.Flush();
}
}
}
}
What I want to ensure is that if the process crashes or is killed, the archive is still valid and readable. I had hoped that anything since the last flush would be safe, but instead I am just ending up with a corrupt archive.
Is there any way to ensure I end up with a readable archive after each flush?
Note: it isn't essential that we use GZipStream, if something else will give us the desired result.
An option is to let Windows handle the compression. Just enable compression on the folder where you're storing your log files. There are some performance considerations you should be aware of when copying the compressed files, and I don't know how well NT compression performs in comparision to GZipStream or other compression options. You'll probably want to compare compression ratios and CPU load.
There's also the option of opening a compressed file, if you don't want to enable compression on the entire folder. I haven't tried this, but you might want to look into it: http://social.msdn.microsoft.com/forums/en-US/netfxbcl/thread/1b63b4a4-b197-4286-8f3f-af2498e3afe5
Good news: GZip is a streaming format. Therefore corruption at the end of the stream cannot affect the beginning which was already written.
So even if your streaming writes are interrupted at an arbitrary point, most of the stream is still good. You can write yourself a little tool that reads from it and just stops at the first exception it sees.
If you want an error-free solution I'd recommend splitting the log into one file every x seconds (maybe x = 1 or 10?). Write into a file with extensions ".gz.tmp" and rename to ".gz" after the file was completely written and closed.
Yes, but it's more involved than just flushing. Take a look at gzlog.h and gzlog.c in the zlib distribution. It does exactly what you want, efficiently adding short log entries to a gzip file, and always leaving a valid gzip file behind. It also has protection against crashes or shutdowns during the process, still leaving a valid gzip file behind and not losing any log entries.
I recommend not using GZIPStream. It is buggy and does not provide the necessary functionality. Use DotNetZip instead as your interface to zlib.
I am trying to implement file download feature in asp.net application. The application would be used by say around 200 users concurrently to download various files.
It would be hosted on IIS 7. I do not want the application server to crash because of multiple requests coming concurrently.
I am assuming that by calling Context.Response.Flush() in a loop, I am flushing out all the file data that I would have read till then, so application memory usage would be kept uniform. What other optimizations can I make to the current code or what other approach should be used in a scenario like this?
The requests would be for various files and the file sizes can be anywhere between 100 KB to 10 MB.
My current code is like this:
FileStream inStr = null;
byte[] buffer = new byte[1024];
String fileName = #"C:\DwnldTest\test.doc";
long byteCount; inStr = File.OpenRead(fileName);
Response.AddHeader("content-disposition", "attachment;filename=test.doc");
while ((byteCount = inStr.Read(buffer, 0, buffer.Length)) > 0)
{
if (Context.Response.IsClientConnected)
{
Context.Response.ContentType = "application/msword";
//Context.Response.BufferOutput = true;
Context.Response.OutputStream.Write(buffer, 0, buffer.Length);
Context.Response.Flush();
}
}
You can use Response.TransmitFile to save server memory when sending files.
Response.ContentType = "application/pdf";
Response.AddHeader("content-disposition", "attachment; filename=testdoc.pdf");
Response.TransmitFile(#"e:\inet\www\docs\testdoc.pdf");
Response.End();
In your code example, you're not closing / disposing inStr. That could affect performance.
Another more simple way to do this would be to use the built in method:
WriteFile
It should already be optimized and will take care of opening / closing files for you.
Maybe you want to use FileSystemWatcher class to check if the file was modified, and read it into memory only while such change was detected. For rest of the time just return the byte array that is already stored in memory. I don't know if HttpResponse.WriteFile method is sensitive for such file modification changes, or if always reads a file from given path, but this also seems to be a good option to use, as it is served by framework out of the box.
Since you are sending an existing file to the client, consider using HttpResponse.TransmitFile (http://msdn.microsoft.com/en-us/library/12s31dhy.aspx).
Looking at the .NET code it seems that this will forward the file writing to IIS instead of reading/writing it in ASP.NET process. HttpResponse.WriteFile(string, false) and HttpResponse.Write(string) seems to do the same thing.
In order to verify that the file sending is relayed to IIS, at HttpResponse.Output property - it should be of type HttpWriter. The HttpWriter._buffers array should now contain a new element HttpFileResponseElement).
Of course, you should always investigate if caching is appropriate in your scenario and test if it is being used.
I'm making tests with ASP.NET HttpHandler for download a file writting directly on the response stream, and I'm not pretty sure about the way I'm doing it. This is a example method, in the future the file could be stored in a BLOB in the database:
public void GetFile(HttpResponse response)
{
String fileName = "example.iso";
response.ClearHeaders();
response.ClearContent();
response.ContentType = "application/octet-stream";
response.AppendHeader("Content-Disposition", "attachment; filename=" + fileName);
using (FileStream fs = new FileStream(Path.Combine(HttpContext.Current.Server.MapPath("~/App_Data"), fileName), FileMode.Open))
{
Byte[] buffer = new Byte[4096];
Int32 readed = 0;
while ((readed = fs.Read(buffer, 0, buffer.Length)) > 0)
{
response.OutputStream.Write(buffer, 0, readed);
response.Flush();
}
}
}
But, I'm not sure if this is correct or there is a better way to do it.
My questions are:
When I open the url with the browser, appears the "Save File" dialog... but it seems like the server has started already to push data into the stream before I click "Save", is that normal?
If I remove the line"response.Flush()", when I open the url with the browser, ... I see how the web server is pushing data but the "Save File" dialog doesn't come up, (or at least not in a reasonable time fashion) why?
When I open the url with a WebRequest object, I see that the HttpResponse.ContentLength is "-1", although I can read the stream and get the file. What is the meaning of -1? When is HttpResponse.ContentLength going to show the length of the response? For example, I have a method that retrieves a big xml compresed with deflate as a binary stream, but in that case... when I access it with a WebRequest, in the HttpResponse I can actually see the ContentLength with the length of the stream, why?
What is the optimal length for the Byte[] array that I use as buffer for optimal performance in a web server? I've read that is between 4K and 8K... but which factors should I consider to make the correct decision.
Does this method bloat the IIS or client memory usage? or is it actually buffering the transference correctly?
Sorry for so many questions, I'm pretty new in web development :P
Cheers.
Yes; this is normal.
If you never flush, the browser doesn't get any response until the server finishes (Not even the Content-Disposition header). Therefore, it doesn't know to show a file dialog.
The Content-Length header only gets set if the entire response is buffered (If you never flush) or if you set it yourself. In this case, you can and should set it yourself; write
response.AppendHeader("Content-Length", new FileInfo(path).Length.ToString());
I recommend 4K; I don't have any hard basis for the recommendation.
This method is the best way to do it. By calling Flush inside the loop, you are sending the response down the wire immediately, without any buffering. However, for added performance, you can use GZIP compression.
Yes, it is buffering.
Flush pushes the cached content to the browser. If it is never pushed, you won't get a save dialog box.
Hard to tell without seeing the exact files/URLs/Streams you are using.
I think the factors depends on how sluggish your page is, really. You will have better performance toward 4k. And perhaps, the lower value will be better to accommodate slower connections.
See #1 & 2.
For #3 you need to set the content-length header in your http-response. Many of those values come from http headers.
I believe you can change the bufferring by changing a buffering property on the response object to false. Haven't done it in a while so I don't remember what it might be.
I have a custom HttpHandler that invokes a webservice to get a file. In test, I invoke the production webservice and the HttpHandler returns the file correctly. When I test it in the production environment on the server, it works as well. However, if I invoke the HttpHandler from a remote client (not on the server) the filename and size are set correctly, but the file bytes that are downloaded are zero. Any ideas?
So here's the deal. I created a multipart range handler (you need to implement the RFC in order to stream content to, say, an iPhone or Adobe Reader). The spec is suppose to enable handling a file when the client requests a range of bytes instead of the whole array. The issue with my handler came when the client wanted the whole BLOB:
if (context.Request.Headers[HEADER_RANGE] != null)
{
...
}
else
{
context.Response.ContentType = contentItem.MimeType;
addHeader(context.Response, HEADER_CONTENT_DISPOSITION, "attachment; filename=\"" + contentItem.Filename + "\"");
addHeader(context.Response, HEADER_CONTENT_LENGTH, contentItem.FileBytes.Length.ToString());
context.Response.OutputStream.Write(contentItem.FileBytes, 0, contentItem.FileBytes.Length);
}
Notice anything missing???
I forgot to include:
context.Response.Flush();
After adding that line of code, it started working in the production environment. I find it very odd, however, that this was working on the server and not on any clients. Anyone able to shed any light on why that would be?