I have a WCF Service hosted in IIS/ASP.NET that accepts HTTP Post (not form post) of serialized objects.
If the client sends malformed requests (eg they're not serializing the object correctly) I'd like to log the message sent up.
We're already using ELMAH to capture unhandled exceptions, so simply attaching the post data would be the easiest option.
I can get the current HttpContext during an exception, however this does only contains the HTTP Header information.
My question is this: Is there some way of capturing the original HTTP POST request body? Or, failing that - a better way (without a reverse proxy) of capturing the input that caused the error?
Edit: Just to clarify, running packet-level capturing at all times isn't really suitable. I'm after a solution that I can deploy to Production servers, and which will have clients outside our control or ability to monitor.
Edit #2: A suggestion was made to access the Request.InputStream - this doesn't work if you're trying to read after WCF has read the request off the stream.
A sample piece of code to see how I've tried using this is here.
StringBuilder log = new StringBuilder();
var request = HttpContext.Current.Request;
if (request.InputStream != null)
{
log.AppendLine(string.Format("request.InputStream.Position = \"{0}\"", request.InputStream.Position));
if (request.InputStream.Position != 0)
{
request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
using (StreamReader sr = new StreamReader(request.InputStream))
{
log.AppendLine(string.Format("Original Input: \"{0}\"", sr.ReadToEnd()));
}
}
else
{
log.AppendLine("request.Inputstream = null");
}
log.ToString();
The ouput of log.ToString() is:
request.InputStream.Position = "0"
Original Input: ""
By the time it gets to your service the request is processed and not available to you.
However ... you could attach a message inspector. Message Inspectors allow you to fiddle with the message before it reaches your operation implementations. You could create a buffered copy of the message, and copy it into the OperationContext.Current.
Ugly hack of course, and it will mean memory overhead as now two copies of the message are floating about for every request.
Did you look at the System.Web.Request.InputStream Property? It should have exactly what you want.
How to "rewind" the InputStream Property.
if (Request.InputStream.Position != 0)
{
Request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
Another option you should look into is capturing this information with an HTTPModule on the BeginRequest event. The data should be there at BeginRequest event because I do not believe WCF picks up the request until after PostAuthenticateEvent.
from under ASP.NET (ASP web service under IIS) the following code helps:
if (request.InputStream.Position != 0)
{
request.InputStream.Seek(0, System.IO.SeekOrigin.Begin);
}
WCF maybe different (that is it Disposes InputStream after reading it)
Use fiddler. Free from MS. Works great.
Related
I'm reading a file from user upload and it was working synchronously. I needed to change it in order to immediately send a "received" alert to the user, then read the file asynchronously while the user would periodically poll back to see if the read was finished.
Here is what my code looks like right now:
public FileUpload SaveFile(Stream stream)
{
FileUpload uploadObj = //instantiate the return obj
var task = Task.Run(async () => await ProcessFileAsync(stream));
return upload;
}
public async Task ProcessFileAsync(Stream stream)
{
StreamReader file = new StreamReader(stream);
CsvReader csv = new CsvReader(file, CultureInfo.InvariantCulture);
while (await csv.ReadAsync())
{
//read the file
}
}
the issue I'm having is that by the time I call the csv.ReadAsync() method, the Stream object has been disposed. How do I access the Stream when I want the SaveFile() method to return a value to the user, but the act of returning disposes the Stream object?
The point here is that you're working within the constraints of ASP.NET, which abstracts away a lot of the underlying HTTP stuff.
When you say you want to process a user-uploaded file asynchronously, you want to step out of the normal order of doing things with HTTP and ASP.NET. You see, when a client sends a request with a body (the file), the server receives the request headers and kicks off ASP.NET to tell your application code that there's a new request incoming.
It hasn't even (fully) read the request body at this point. This is why you get a Stream to deal with the request, and not a string or a filename - the data doesn't have to be arrived at the server yet! Just the request headers, informing the web server about the request.
If you return a response at that point, for all HTTP and ASP.NET care, you're done with the request, and you cannot continue reading its body.
Now what you want to do, is to read the request body (the file), and process that after sending a response to the client. You can do that, but then you'll still have to read the request body - because if you return something from your action method before reading the request, the framework will think you're done with it and dispose the request stream. That's what's causing your exception.
If you'd use a string, or model binding, or anything that involves the framework reading the request body, then yes, your code will only execute once the body has been read.
The short-term solution that would appear to get you going, is to read the request stream into a stream that you own, not the framework:
var myStream = new MemoryStream();
await stream.CopyTo(myStream);
Task.Run(async () => await ProcessFileAsync(myStream));
Now you'll have read the entire request body and saved it in memory, so ASP.NET can safely dispose the request stream and send a response to the client.
But don't do this. Starting fire-and-forget tasks from a controller is a bad idea. Keeping uploaded files in memory is a bad idea.
What you actually should do, if you still want to do this out-of-band:
Save the incoming file as an actual, temporary file on your server
Send a response to the client with an identifier (the temporarily generated filename, for example a GUID)
Expose an endpoint that clients can use to request the status using said GUID
Have a background process continuously scan the directory for newly uploaded files and process them
For the latter you could hosted services or third-party tools like Hangfire.
You'll need to either do this if the environment warrants:
var result = task.Result;
//do stuff
...or
public Task<FileUpload> SaveFile(Stream stream)
{
var uploadObj = //instantiate the return obj
await ProcessFileAsync(stream);
return uploadObj;
}
See here for a thorough discussion on fire-and-forget if you go that route:
Web Api - Fire and Forget
I am at a loss and would welcome any and all advice. This code is deceptively simple, but Async might be complicating things and causing issues.
POST method works when data bytes is less than 3802. Data is truncated at 3800 or 3801 when the POST data is greater than 3802. This took a while to catch for us because most of the time data is < 3800. I finally added the debug logging and saw the disconnect.
I have been doing extra reading on async and don't currently believe this code has an issue. I know async void will cause fire and forget, but we have a try/catch, so I expect errors to show up in the logs without taking the whole program out due to an exception. I would welcome suggestions for improvement however.
[HttpPost("{instance}/{objectType}")]
public async void Post( [FromRoute] string instance, [FromRoute] string objectType )
{
Log.Debug("Incoming post(instance/objectype)");
string content = "";
try
{
using (var reader = new StreamReader(Request.Body, Encoding.UTF8))
{
// have a scenario where conlen below is always cut off at 3800 or 3801 bytes
// wild guess is that it is a async/threadpool issue?
content = await reader.ReadToEndAsync();
}
var conlen = (content != null) ? content.Length : 0;
var contentbegin = (content != null) ? content.Substring(0, Math.Min(conlen, 80)) : "";
Log.Debug("POST {objecttype} {contentlen} {contentbegin}", objectType, conlen, contentbegin);
await ProcessPost(objectType, content, instance);
}
catch (Exception ex)
{
Log.Error("Exception in Post {msg} {trace}", ex.Message, ex.StackTrace);
Log.Error("Exception in Post offending message {msg} ", content);
}
}
Here is an example of the logs. You can see the size of the data is > 3800, but alas, we are getting cut off when we go to read it.
[19:27:57 INF] Request starting HTTP/1.0 POST http://backend.example.com/orderinput application/json 5459
[19:27:57 INF] Executing action method Controllers.ValuesController.Post (Test) with arguments (["Production", "orderinput"]) - ModelState is Valid
[19:27:57 DBG] Incoming post(instance/objectype)
[19:27:57 INF] Executed action Controllers.ValuesController.Post (Test) in 0.4701ms
[19:27:57 INF] Request finished in 0.789ms 200
[19:27:57 DBG] POST orderinput 3801 {"SAMPLE":"1234567890.........
[19:27:57 DBG] Enter ProcessPost
[19:27:57 DBG] Persisting post
[19:27:57 ERR] Exception in ProcessPost DB Persist Unexpected end of content while loading JObject. Path 'customer_id', line 1, position 3801.
EDITS:
WHY DO I THINK THIS CODE IS AN ISSUE?
At the end of line 1 in the logs, ASP.NET has helpfully put the content length, 5459. Line 6 has my calculation and it shows 3801. These two numbers are always the same unless the length exceeds 3802.
HOW IS THE DATA SUBMITTED? HAVE I TRIED DOING SO?
Data is coming from another web service not under my control through an NGINX proxy. Thanks for the suggestion I will exercise this both with and without the proxy, above and below 3802 and report the results back here. Given the disconnect between the ASP.NET logs and my calculation, it seems like a long shot, but all roads are worth going down.
I threw away the codebase that was written by another developer and started over from scratch to fix this issue. I think the fact that sync/async was being mixed indiscriminately was the issue, and this weird symptom was the result. I used nito asyncex in my final solution to isolate the rabbit library from Async calls, and for the Asp.Net web API, I had all incoming data post to a queue, and the queue ran in a separate thread that exclusively talked to rabbit. No issues at all since the rewrite.
I recently came across a Chrome issue which I think is worth sharing it with you.
I worked on a self written API using an HttpHandler which primary should return json data. But when an error occures I wanted to display an html file. That worked pretty well in IE and FF, but not in Chrome.
Looking to the developer tools revealed this error: net::ERR_INCOMPLETE_CHUNKED_ENCODING
Google said not very much about this issue while it was seen very much. All I got to know was, that it was magically disappearing after some time.
I found out it lays on this lines of code:
result.StoreResult(context);
context.Response.Flush();
context.Response.Close(); //<-- this causes the error
After removing the last line it worked well. I don´t know why only Chrome had/has an issue with that, but it seemed as if I closed the response stream before chrome finished reading it.
I hope it helps those of you coming across the same or a similar issue.
Now my question:
How is the best pratice in closing/flushing the response stream? Are there any rules?
According to ASP.NET sets the transfer encoding as chunked on premature flushing the Response:
ASP.NET transfers the data to the client in chunked encoding (Transfer-Encoding: chunked), if you prematurely flush the Response stream for the Http request and the Content-Length header for the Response is not explicitly set by you.
Solution: You need to explicitly set the Content-Length header for the Response to prevent ASP.NET from chunking the response on flushing.
Here's the C# code that I used for preventing ASP.NET from chunking the response by setting the required header:
protected void writeJsonData (string s) {
HttpContext context=this.Context;
HttpResponse response=context.Response;
context.Response.ContentType = "text/json";
byte[] b = response.ContentEncoding.GetBytes(s);
response.AddHeader("Content-Length", b.Length.ToString());
response.BinaryWrite(b);
try
{
this.Context.Response.Flush();
this.Context.Response.Close();
}
catch (Exception) { }
}
I was running into this error when generating a file and pushing it to the user for download, but only occasionally. When it didn't fail, the file was consistently 2 bytes short. Close() forcibly closes the connection, whether it's finished or not, and in my case it was not. Leaving it out, as suggested in the question, meant the resulting file contained both the generated content as well as the HTML for the entire page.
The solution here was replacing
context.Response.Flush();
context.Response.Close();
with
context.Response.End();
which does the same, but without cutting the transaction short.
In my case, the problem was cache-related and was happening when doing a CORS request.
Forcing the response header Cache-Control to no-cache resolved my issue:
[ using Symfony HttpFoundation component ]
<?php
$response->headers->add(array(
'Cache-Control' => 'no-cache'
));
I was also getting same error. This issue was with web server user permission on cache folder.
On the offchance that someone is landing here as a result of issues with their ASP.net Core project, I was able to resolve by adding the IIS middleware.
This is done by adding UseIISIntegration when instantiating your webhost instance.
Once I had the same problem and the main reason was lying in my controller return type.
If you try to return a C# object just as-is, you will only get net::ERR_INCOMPLETE_CHUNKED_ENCODING so don't forget to serialize your complex objects before sending them out for java script client (or View).
i.e. my controller return type was :
public async Task<List<ComplexModel>> GetComplexModelList(){
return new List<ComplexModel>()
}
Which caused INCOMPLETE_CHUNKED_ENCODING error, so I tried to fix my mistake with something like:
using Newtonsoft.Json;
...
public async Task<string> GetComplexModelList(){
return JsonConvert.SerializeObject(new List<ComplexModel>())
}
What I want to be able to do: Edit HTTP Requests before they are sent off to the server
User navigates to a webpage of their choice in their browser > They encounter a request they wish to edit > they edit the request and then that gets sent to the server instead of the original one.
What I have done so far: I have captured the request, now I need help finding the code to edit it. Here is my code for capturing the request so far:
Fiddler.FiddlerApplication.BeforeRequest += sess =>
{
//Code to detect user specified URL here
}
Is it possible for me to edit the request before it is actually sent? If it can be done using the FiddlerCore API only then I'd be grateful, although I am willing to download more binaries if required.
Additional notes: I have tried streamwriters, binary writers, copy the respose into a memory stream edit it then copy it back, none of those methods work for me. Also when I try some methods my app just hangs and doesn't respond to things like pressing the X.
Maybe I'm just bad at explaining what I'm trying to achieve seems the only good answer I have has been about reponses :/
If the request reads the string "hello world" then I'd like the user to be able to change the REQUEST to say "hello there"
Such a noobish mistake I made, I thought that RequestBody was read only! Turns out I could have simply edited the response like this:
session.RequestBody = myBytes;
Really annoyed at myself for this!
In the demo app, adding the delegate is shown as:
Fiddler.FiddlerApplication.BeforeResponse += delegate(Fiddler.Session oS) {
// Console.WriteLine("{0}:HTTP {1} for {2}", oS.id, oS.responseCode, oS.fullUrl);
// Uncomment the following two statements to decompress/unchunk the
// HTTP response and subsequently modify any HTTP responses to replace
// instances of the word "Microsoft" with "Bayden". You MUST also
// set bBufferResponse = true inside the beforeREQUEST method above.
//
//oS.utilDecodeResponse(); oS.utilReplaceInResponse("Microsoft", "Bayden");
};
I am reading from the history database, and for every URL read, I am downloading it and storing the data into a string. I want to be able to determine if the link is a download link, i.e. .exe or .zip for e.g. I am assuming I need to read the headers to determine this, but I don't know how to do it with WebClient. Any suggestions?
while (sqlite_datareader.Read())
{
noIndex = false;
string url = (string)sqlite_datareader["url"];
try
{
if (url.Contains("http") && (!url.Contains(".pdf")) && (!url.Contains(".jpg")) && (!url.Contains("https")) && !isInBlackList(url))
{
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Only a test!");
String htmlCode = client.DownloadString(url);
}
}
}
Instead of loading the complete content behind the link, I would issue a HEAD request.
The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response. The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request. This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself. This method is often used for testing hypertext links for validity, accessibility, and recent modification.
Quote of http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
See these questions for C# examples
How to check if a file exists on a server using c# and the WebClient class
How to check if System.Net.WebClient.DownloadData is downloading a binary file?
You're on the right track; you'll need to examine the ResponseHeaders after a successful request:
var someType = "application/zip";
if (client.ResponseHeaders["Content-Type"].Contains(someType)) {
// this was a "download link"
}
The tricky part will be in determining what constitutes a download link since there are so many content types possible. For example, how would you decide whether XML data is a download link or not?
Try to check WebClient's ResponseHeaders collections to validate response file type.
In case, anyone has the same problem, I have used an attribute in the history places.sqlite database which came in very handy!
Places.sqlite contains a table called moz_historyvisits which contains a column visit_type. According to [1], a visit_type of 7 is a download link. Therefore, reading this value will determine if it is a download link without reading the response header or even sending out a head method.
[1] http://www.firefoxforensics.com/research/moz_historyvisits.shtml