I have been working on getting gzip/deflate compression working on Web API responses. I have been using the code from Github - MessageHandlers.Compression. However it didn't appear to work. There was no Content-Encoding header appearing in the Google Developer console or in Firebug in Firefox and the Content-Length was consistently set to the uncompressed size of the data. So I kept stripping out the code until I ended up with the following:
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
// Send the request to the web api controller
var response = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
// Compress uncompressed responses from the server
if (response.Content != null && request.Headers.AcceptEncoding.IsNotNullOrEmpty())
{
var content = response.Content;
var bytes = content.ReadAsByteArrayAsync().Result;
if (bytes != null && bytes.Length > 1024)
{
// The data has already been serialised to JSon by this point
var compressedBytes = Compress(bytes);
response.Content = new ByteArrayContent(compressedBytes);
var headers = response.Content.Headers;
headers.Remove("Content-Type");
headers.ContentLength = compressedBytes.Length;
headers.ContentEncoding.Clear();
headers.ContentEncoding.Add("gzip");
headers.Add("Content-Type", "application/json");
}
}
return response;
}
private static byte[] Compress(byte[] input)
{
using (var compressStream = new MemoryStream())
{
using (var compressor = new GZipStream(compressStream, CompressionMode.Compress))
{
compressor.Write(input, 0, input.Length);
compressor.Close();
return compressStream.ToArray();
}
}
}
When I initially did this I made a mistake, and set the content encoding in the header to 'gzip' when I used a DeflateStream in the Compress method. As you would expect I got an error in the browser, however the response headers were correct(!). That is, the Content-Encoding header was set and the Content-Length was correct. As well, looking at the raw data I could clearly see if was compressed. As soon as I corrected my error though the problem returned.
What I am wondering is do the latest versions of the browser decompress the content behind the scenes, or is there actually something wrong with my code? Responses are sent in Json format
Any help much appreciated.
EDIT
I tried dumping the headers to a log file in the following methods in Global.asax (listed in the order they appeared in the log):
Application_PreSendRequestHeaders
Application_EndRequest
Application_PreSendRequestContent
In each case the required headers were there, even though they didn't appear in the Google developer console. I then took a look at the solution at Code Project. When run from the command line everything worked as anticipated. However when I called the web server from Google Chrome I got the exact same result. That is, no Content-Encoding header, and no indication as to whether the content had been compressed or not. However with the developer console open it's easy to see this header at other sites (stack overflow for instance). I have to assume this is therefore something to do with compressed responses from web api services. It's so hard to know though if this is actually working client side.
In case anyone doesn't want to read all the comments the answer came from Jerry Hewett (jerhewet). Namely that anti-virus software intercepts the response before it gets to the browser. The anti-virus software decompresses the data, no doubt as part of the scanning process. Huge thanks to Jerry for his help here.
Related
I have a ASP.NET MVC 5 project in which i am trying to download a pdf. I thought this would be pretty straightforward, but apparently not so, as the pdf somehow gets formatted wrongly when sent to the browser. Chrome will read the pdf, but all special characters and images looks corrupted, and Edge simply refuses to open it.
The pdf file is written in Danish, so it contains letters like 'æ', 'ø' and 'å', and also contains various images.
I've simplified my code current code, but the issue is still the same:
[HttpGet]
public HttpResponseMessage DownloadDocument()
{
try
{
var localFilePath = "C:\\*somepath*\\Testpdf.pdf";
var fileStream = new FileStream(localFilePath, FileMode.Open, FileAccess.Read);
fileStream.Seek(0, SeekOrigin.Begin);
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
response.Content = new StreamContent(fileStream);
response.Content.Headers.Add("Content-Disposition", "inline;filename=\"Testpdf.pdf\"");
response.Content.Headers.Add("Content-Length", fileStream.Length.ToString());
response.Content.Headers.Add("Content-Name", "Testpdf.PDF");
response.Content.Headers.Add("Content-Type", "application/pdf;charset=UTF-8");
return response;
}
catch (Exception e)
{
throw e;
}
}
It reads a pdf from the disk and assigns the filestream to the content of the HttpResponseMessage. In the example I'm using a StreamContent, but the result is still the same if switch to ByteArrayContent.
I've also tried writing the file back onto the disk, with no issues at all. I've litterally just done the same behaviour in a dotnetcore 2.0 web project, with no issues what-so-ever, which leads me to believe that the formatters in of the Web API is parsing the content wrong.
I'm at my wits end here, so any input would be much appreciated.
I fixed this issue by removing the jwt "Authorization" token from the Request Headers. I changed it to get the token from the url instead, by making a alternate provider, following this article:
https://leastprivilege.com/2013/10/31/retrieving-bearer-tokens-from-alternative-locations-in-katanaowin/
I have started to get some issues where I am over the limit with my headers.
I have a MVC controller, calling a WebApi Controller/Service.
I know the trigger, it is a saml-token (xml) that I've converted to base64.
No, I don't have control of the SecurityToken service...so JWT is not an option at this time. Trust me, I've raised my concerns several times.
We use the saml to secure the WebApi Controller(s) using a custom delegating handler that reads the custom-header and transforms it to a ClaimPrincipal...
I have seen gzip code examples for dealing with the Response, but after hours of googling, I haven't found if there is a way to compress my custom header (or all of them if that's the only way)...for the ~Request.
Ideally I would be able to compress the
"X-My-Custom-Header"
and deal with uncompressing it on the webapi side....
So I'm at a loss to know if this is even possible. This is the first time I've ever had to deal with a way too big header issue.
Sample MVC code below. As an FYI, the windows-credentials are sent over, but that contains the Identity that runs the AppPool that runs the MVC.
My custom header is the saml that is associated with the specific logged in User. Thus why I need to send it over and consider it separately from the windows-identity.
using (var client = new HttpClient(HttpClientHandlerFactory.GetWindowsAuthenticationHttpClientHandler()))
{
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
string base64SerializedToken = "SuperDuperLongBase64String_IMeanSuperDuper";
client.DefaultRequestHeaders.Add("X-My-Custom-Header", base64SerializedToken);
Uri baseUri = new Uri("http:www.mywebapiservice.com");
Uri destinationUri = new Uri(baseUri, "doSomething");
HttpResponseMessage response = client.PostAsJsonAsync(new Uri(new Uri(this._baseUri), destinationUri.ToString()).ToString(), accountName).Result;
if (response.IsSuccessStatusCode)
{
returnItem = response.Content.ReadAsAsync<MyCustomReturnObject>().Result;
}
else
{
string errorMessage = response.Content.ReadAsStringAsync().Result;
throw new InvalidOperationException(errorMessage);
}
}
public static class HttpClientHandlerFactory
{
public static HttpClientHandler GetWindowsAuthenticationHttpClientHandler()
{
HttpClientHandler returnHandler = new HttpClientHandler()
{
UseDefaultCredentials = true,
PreAuthenticate = true
};
return returnHandler;
}
}
Considering that a SAML token can be several kilobytes in size, depending on the number of claims, sending it as a header is probably a bad idea. Even if you can get it to work now there's no guarantee that it will continue to work if the claim count grows.
Since there is no standard for header compression you will have to modify both ends of the conversation to do something about it. That being the case, why not simply add the SAML token as part of the request body in your API?
If that's really not going to fly (I get that project managers are often painful when it comes to things like this) then you'll have to look into using something like GZipStream to pack the XML, but at some point you're still going to run into problems. This is a bandaid, not a solution.
No, there's no standard for header compression in HTTP. This might have something to do with the fact that you'd need to read the headers to know if (and how) the headers are compressed.
If you don't have a way to decompress whatever manual compression you figure out on the other side, you're out of luck.
I recently came across a Chrome issue which I think is worth sharing it with you.
I worked on a self written API using an HttpHandler which primary should return json data. But when an error occures I wanted to display an html file. That worked pretty well in IE and FF, but not in Chrome.
Looking to the developer tools revealed this error: net::ERR_INCOMPLETE_CHUNKED_ENCODING
Google said not very much about this issue while it was seen very much. All I got to know was, that it was magically disappearing after some time.
I found out it lays on this lines of code:
result.StoreResult(context);
context.Response.Flush();
context.Response.Close(); //<-- this causes the error
After removing the last line it worked well. I don´t know why only Chrome had/has an issue with that, but it seemed as if I closed the response stream before chrome finished reading it.
I hope it helps those of you coming across the same or a similar issue.
Now my question:
How is the best pratice in closing/flushing the response stream? Are there any rules?
According to ASP.NET sets the transfer encoding as chunked on premature flushing the Response:
ASP.NET transfers the data to the client in chunked encoding (Transfer-Encoding: chunked), if you prematurely flush the Response stream for the Http request and the Content-Length header for the Response is not explicitly set by you.
Solution: You need to explicitly set the Content-Length header for the Response to prevent ASP.NET from chunking the response on flushing.
Here's the C# code that I used for preventing ASP.NET from chunking the response by setting the required header:
protected void writeJsonData (string s) {
HttpContext context=this.Context;
HttpResponse response=context.Response;
context.Response.ContentType = "text/json";
byte[] b = response.ContentEncoding.GetBytes(s);
response.AddHeader("Content-Length", b.Length.ToString());
response.BinaryWrite(b);
try
{
this.Context.Response.Flush();
this.Context.Response.Close();
}
catch (Exception) { }
}
I was running into this error when generating a file and pushing it to the user for download, but only occasionally. When it didn't fail, the file was consistently 2 bytes short. Close() forcibly closes the connection, whether it's finished or not, and in my case it was not. Leaving it out, as suggested in the question, meant the resulting file contained both the generated content as well as the HTML for the entire page.
The solution here was replacing
context.Response.Flush();
context.Response.Close();
with
context.Response.End();
which does the same, but without cutting the transaction short.
In my case, the problem was cache-related and was happening when doing a CORS request.
Forcing the response header Cache-Control to no-cache resolved my issue:
[ using Symfony HttpFoundation component ]
<?php
$response->headers->add(array(
'Cache-Control' => 'no-cache'
));
I was also getting same error. This issue was with web server user permission on cache folder.
On the offchance that someone is landing here as a result of issues with their ASP.net Core project, I was able to resolve by adding the IIS middleware.
This is done by adding UseIISIntegration when instantiating your webhost instance.
Once I had the same problem and the main reason was lying in my controller return type.
If you try to return a C# object just as-is, you will only get net::ERR_INCOMPLETE_CHUNKED_ENCODING so don't forget to serialize your complex objects before sending them out for java script client (or View).
i.e. my controller return type was :
public async Task<List<ComplexModel>> GetComplexModelList(){
return new List<ComplexModel>()
}
Which caused INCOMPLETE_CHUNKED_ENCODING error, so I tried to fix my mistake with something like:
using Newtonsoft.Json;
...
public async Task<string> GetComplexModelList(){
return JsonConvert.SerializeObject(new List<ComplexModel>())
}
I am trying to upload some avi file to server. It works fine with HttpRequest but i need to continue uploading even if i suspend app so thats why i am trying to use BackgroundUploader. I am following this guideline on msdn http://msdn.microsoft.com/en-us/library/windows/apps/jj152727.aspx. So my code looks something like this.
StorageFile storageFile = KnownFolders.VideosLibrary.GetFileAsync("fileName");
BackgroundUploader uploader = new BackgroundUploader();
uploader.Method = "POST";
uploader.SetRequestHeader("Content-Type", "multipart/form-data; boundary=" + boundary);
var fs = await storageFile.OpenAsync(Windows.Storage.FileAccessMode.ReadWrite);
IInputStream aaaa = fs.GetInputStreamAt(0);
UploadOperation upload = uploader.CreateUploadFromStreamAsync(new Uri("uploadUri"), aaaa);
await HandleUploadAsync(upload, true);
the rest is same as on MSDN. And i am getting exception Unsupported media type (415) in method HandleUploadAsync on line
await upload.StartAsync().AsTask(cts.Token, progressCallback);
What am i doing wrong? Or what can cause this kind of exception?
EDIT : I solved my problem as i commented down here and in my answer. I think at the end i am basically just sending some data to server that are recognized and interpreted as i want to. So if i use BackgroundUploader i am not only uploading some file i am also sending information about how am i doing that(as i mentioned in my answer). So by the same idea i can also upload folder to server and by that i am not sending any actual content only some description about how to do that. And if i compare request that i am making by HttpRequest and BackgroundUploader they are equal and thats what i wanted.
So the problem part was the header of request. I have some header in my request that is recognized by server and i was trying to put it to BackgroundUploader through SetRequestheader method but it did not work. As Kieqic suggested i used Fiddler and by that i compare request made by HttpRequest and BackgroundUploader. I found out they are completely different. So through SetRequestheader i add some parts like expected Content-Type and for the rest parts of header to make them equal i add it before content of my file as array of bytes. And this works so conclusion is in my case using Fiddler that helped my how to construct request header.
I have a WCF Service hosted in IIS/ASP.NET that accepts HTTP Post (not form post) of serialized objects.
If the client sends malformed requests (eg they're not serializing the object correctly) I'd like to log the message sent up.
We're already using ELMAH to capture unhandled exceptions, so simply attaching the post data would be the easiest option.
I can get the current HttpContext during an exception, however this does only contains the HTTP Header information.
My question is this: Is there some way of capturing the original HTTP POST request body? Or, failing that - a better way (without a reverse proxy) of capturing the input that caused the error?
Edit: Just to clarify, running packet-level capturing at all times isn't really suitable. I'm after a solution that I can deploy to Production servers, and which will have clients outside our control or ability to monitor.
Edit #2: A suggestion was made to access the Request.InputStream - this doesn't work if you're trying to read after WCF has read the request off the stream.
A sample piece of code to see how I've tried using this is here.
StringBuilder log = new StringBuilder();
var request = HttpContext.Current.Request;
if (request.InputStream != null)
{
log.AppendLine(string.Format("request.InputStream.Position = \"{0}\"", request.InputStream.Position));
if (request.InputStream.Position != 0)
{
request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
using (StreamReader sr = new StreamReader(request.InputStream))
{
log.AppendLine(string.Format("Original Input: \"{0}\"", sr.ReadToEnd()));
}
}
else
{
log.AppendLine("request.Inputstream = null");
}
log.ToString();
The ouput of log.ToString() is:
request.InputStream.Position = "0"
Original Input: ""
By the time it gets to your service the request is processed and not available to you.
However ... you could attach a message inspector. Message Inspectors allow you to fiddle with the message before it reaches your operation implementations. You could create a buffered copy of the message, and copy it into the OperationContext.Current.
Ugly hack of course, and it will mean memory overhead as now two copies of the message are floating about for every request.
Did you look at the System.Web.Request.InputStream Property? It should have exactly what you want.
How to "rewind" the InputStream Property.
if (Request.InputStream.Position != 0)
{
Request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
Another option you should look into is capturing this information with an HTTPModule on the BeginRequest event. The data should be there at BeginRequest event because I do not believe WCF picks up the request until after PostAuthenticateEvent.
from under ASP.NET (ASP web service under IIS) the following code helps:
if (request.InputStream.Position != 0)
{
request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
WCF maybe different (that is it Disposes InputStream after reading it)
Use fiddler. Free from MS. Works great.