We are recieving proportionately low but consistent 404 from server events from a channel subscription. This seems to only be via our react interface which uses the typescript adapter here:
https://docs.servicestack.net/typescript-server-events-client
If I understand correctly, the 404 returns when the client has failed to respond with a heartbeat before the time out setting, therefore the id passed is invalid.
(HeartbeatInterval =60,IdleTimeout = 180)
Looking at application insights gives up 6 out of 330 fails in an hour:
404s logged
Digging into the request, nothing seems untoward:
Request Properties
Problem is, I cannot see anything on the server throwing errors or missing ids in the reports even though DebugMode and ReturnInnerException are both true. I have run this against a local copy and server with a c# client connected to server events feed and cannot seem to get a 404 to return while observing with fiddler.
Is there a way I can catch these server side and see why they are returning a 404 ? I can only assume something is failing and returning a 404 but it is hidden behind service stacks log handling layer. Is this reported to the logging layer as is it likely to happen regually therefore discarded ?
The 404 Response in a heartbeat is due to the subscription no longer existing, you can use the OnHeartbeatInit callback to inspect each heartbeat, e.g:
Plugins.Add(new ServerEventsFeature
{
OnHeartbeatInit = req =>
{
var subscriptionId = req.QueryString["id"];
var subscription = req.TryResolve<IServerEvents>().GetSubscriptionInfo(subscriptionId);
if (subscription == null)
{
//... subscription no longer exists
}
}
});
Which will let you inspect the subscription of 404 heartbeat responses but it wont tell you why the subscription was removed.
You can handle the OnUnsubscribe to get a callback when a subscription is unsubscribed which should help identify why the subscription was removed.
To help with debugging you can upgrade to the latest v5.4.1 on MyGet which now has embedded pdbs and source-link enabled which will let you easily debug into ServiceStack source code.
Related
We got the following problem:
I am currently developing a web server implementing a specific API. The association behind that API provided specific test cases I'm using to test my implementation.
One of the test cases is:
5.3.2.12 Robustness, large resource ID
This test confirms correct error handling when sending a HTTP request with a very long location ID as URL parameter.
The url its calling looks something like this:
https://localhost:443/api/v2/functions/be13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005ebe13789-1f1e-47d0-8f8a-000000000005
Basically the tests checks, if my server responds with the correct error code if the URL is too long. (At the time of writing it is testing for Errorcode 405, but I already asked them if it shouldn't be 414)
I'm developing the server in Asp.Net 6 and it always returns Bad Request 400 in the testcase.
I don't seem to find a place to change the handling for this behaviour and I am not even sure, if I can, or if the IIS is blocking the request even before it reaches my server. I activated logging in IIS, but the request does not show in the logfile in inetpub/logs/LogFiles.
My question would be, if it is possible to tell IIS to return a different error code in this case, or if it is even possible to handle the error in my application.
What I tried:
Activating IIS Logs to see if the request is even passed to my site. (It did not)
Tried adding Filters to my Controller to see if I can catch an Exception
Checked, if Development Error Sites are called.
Breakpoints in existing middlewares are not reached.
EDIT:
I am now pretty sure now, that the request never reaches my application.
It is possible to reproduce the error by using the default site the IIS generates on windows. Just copy the whole path from above into a browser with the host http://localhost will also just produce the error 400
EDIT 2:
As #YurongDai pointed out, I tried activating failed request tracing for my IIS Site. I used the default path \logs\FailedReqLogFiles.
The folder was created, but no file is written, when I'm opening the URL above in my browser.
IIS Error 400 occurs when the server is unable to process a request sent to a web server. The most common cause of Bad Request error 400 is an invalid URL, but it can happen for other reasons as well. To resolve IIS Error 400, first make sure that you have entered the URL correctly, typos or disallowed characters in the URL are the most common causes of Bad Request errors. If the error persists after verifying the URL, please clear your browser's cache, DNS cache, and cookies and try again.
Clear your browser's cookies.
Clear your browser's cache.
Clear your DNS cache.(Execute the following command in the command prompt window: ipconfig /flushdns)
I am trying to access a page that is returning a 400 status code and then to subsequently redirect the user to homepage via middleware. Here is an example url that will throw a 400: https://testsite.com/%C3%85%C2%81%C3%83%C2%B3d%C3%85%C2%BA_Voivodeship
Note that tacking on "/%C3%85%C2%81%C3%83%C2%B3d%C3%85%C2%BA_Voivodeship" to a request on your blazor server side app/.net mvc application will throw a 400 error.
The url mentioned is present in a sitemap along with many other urls returning a 400 status code. Merely removing the urls from the sitemap is not sufficient in a practical, useful context because search engines have already read them and will continue requesting them. On top of this SEO concern, there is the obvious, but unfortunately not mentioned enough, extremely poor user experience of serving out a "Bad Request - Invalid URL
HTTP Error 400. The request URL is invalid." error message for any user-facing application. This just leaves the user stranded.
So, this issue does need to be dealt with when requested in order to course correct the user to a location where they could do something and also to help search engines that crawl websites to strongly indicate that a specific url should not be crawled any longer through a redirect(yes, for those uninformed of technical SEO, this is the correct, best-practice way to handle this scenario).
I tried to accomplish this by placing a app.use in the Configure method of Startup.cs but it never actually hits the code if you use a url like mentioned above. I am able to break; however, for any ordinary request like a 200 status code one. So, it would seem, this is not the right approach to take. Any ideas?
app.Use(async (context, next) =>
{
var statusCode = context.Response.StatusCode;
// Redirect to homepage of site when 400 status code
if (statusCode == 400)
{
context.Response.Redirect("/");
return; // short circuit
}
await next();
});
The HTTP status code 400 and corresponding message are generated by HTTP.SYS. HTTP.SYS performs URL validation and can reject the request. In such case the returned web page looks like
Bad Request - Invalid URL
HTTP Error 400. The request URL is invalid.
As the specified URL is blocked by HTTP.SYS, request never comes to application. Please see the similar question %09 in a URL causes IIS (HTTP.SYS?) to return HTTP 400 immediately for additional details. In the current case, HTTP.SYS does not like %81, %83, %85 characters.
And the corresponding messages can be found in the HTTP.SYS log located at %SystemRoot%\System32\LogFiles\HTTPERR.
If by some reason URL cannot be changed, the HTTP.SYS behavior can be configured via AllowRestrictedChars DWORD registry key located under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\HTTP\Parameters as described in Http.sys registry settings for Windows article.
If nonzero, Http.sys accepts hex-escaped chars in request URLs that decode to U+0000 - U+001F and U+007F - U+009F ranges.
The HTTP service must be restarted for the changes to take effect.
Additionally to HTTP.SYS configuration change, the middleware code needs a little improvement. The execution of the next delegate needs to be moved to the very top of the method.
app.Use(
async (context, next) =>
{
await next();
if (context.Response.StatusCode == 400)
{
context.Response.Redirect("/");
}
});
Also, as mentioned in the comments, the position of the middleware is important. The HttpResponse.Redirect method simply sets the HTTP status code to 301 or 302 and specifies the Location header. If a response has already been sent to the client by other middleware or application code, the HttpResponse.Redirect method will throw an exception.
I have a strange issue I'm trying to triage having to do with the new HttpClient on .NET Core 2.1. From this article here (https://blogs.msdn.microsoft.com/dotnet/2018/04/11/announcing-net-core-2-1-preview-2/) I know that the HttpClient has been completely re-written to use a different low level library for handling HTTP requests. I'm wondering if anyone has seen any issues with the new implementation.
What I'm seeing is a strange case where my application (.NET Core 2.1) which sends a POST request to some API periodically (every 10 seconds) a few of times every 15 min it will throw an exception with the error: An error occurred while sending the request.; The server returned an invalid or unrecognized response.
No other details are available, it's just an exception when I make a call like this:
using (var res = await _httpClient.PostAsync(uriBuilder.Uri, new StringContent(serializedRequestBody, Encoding.UTF8, "application/json")))
{
//Do something here
}
The exception caught is a System.Net.Http.HttpRequestException and it has some inner exception with the above error message.
So as I mentioned this does NOT happen all time, it happens seemingly at random, or at least I can not discern any particular pattern. All I can say is these POST requests are made once every 10 seconds 24/7 and anywhere between 5% and 10% of the POST requests fail with the above exception.
So used tcdump and piped it into wireshark to examine the requests to see what's actually happening when the requests fail and what i see is the following:
On a good POST I see: my app sends the request to server, server sends response back, my app sends ACK to server and server responds with FIN,ACK. Done. Good Stuff.
On POST which gets the above exception I see the following: my app sends the request to server, and almost immediately after (like a few milliseconds after) my application sends FIN, ACK to server.
This seems consistent with what I see in my application logs, which show that the request duration is 0 before the exception is thrown.
So what it looks like to me is, my application sends the request and then immediately after closes the connection for some reason. However, I don't understand why this happens. I tried comparing the raw HTTP requests (good POST vs bad POST) to see any differences and I can not see any difference.
One last thing to mention, is that I ONLY see this in applications running on .NET Core 2.1. When I run my application on .NET 2.0 I do not see this problem. Also when I use the same library (where the HTTP call is being made) in the .NET 4.5.1 application (I use multi-targeting to compile the library targeting .net standard and net451) I also do NOT see this problem. So it seems to affect only .NET Core 2.1
Any ideas of where I can go from here? Is there something else I should look for ? How would someone go about trying to triage this type of issue ?
[EDIT] I added a screenshot of the wireshark output which shows the last POST request the server never does not respond before the client sends FIN,ACK
[EDIT]
#Svek Pointed out something in the comments about the sequence of ACKs. I think there maybe something here, because (in the screenshot) after the very last POST there is a FIN, ACK and it shows Ack=7187, so I look back I see the previous FIN,ACK had sequence=7186. Now, I'm by far not an expect in TCP or networking so I maybe saying something completely dumb, but does that mean that the last FIN,ACK (which comes from my host to server) is essentially my host FIN,ACK'ing the previous FIN,ACK (from server to my host) and essentially closing the connection.
So since the next POST is made to the same host:port, using the same connection and yet the connection is closed (via that last FIN,ACK) that's why I never get a response back?
In my ASP.NET MVC web application, I have an external single-sign-on which is used to authenticate the user centrally in the enterprise. The SSO is supposed to return a "packet" of stuff that identifies the user uniquely, which would then be used to pass to local ASP.NET Owin Cookie Authentication (or alternatively Forms Authentication, or something like that). If for some reason the SSO-provided "packet" of stuff doesn't contained minimum required info, I'd like to handle it in the local ASP.NET level appropriately. It's a pretty severe error and not necessarily something that would commonly occur. So I thought I would do something like this:
public class AuthController() : Controller
{
public ActionResult Login(string returnUrl)
{
// Process external single-sign-on authentication
bool isSuccess = ProcessExternalAuth();
if (!success)
// return appropriate HTTP status code
else
// continue with login
}
}
But I'm not sure which HTTP status code to return. Would something like this be appropriate?
return new HttpStatusCodeResult(
HttpStatusCode.BadGateway, "SSO authentication failed.");
According to List of HTTP Status Codes, one of the 500 errors would probably be most appropriate:
Response status codes beginning with the digit "5" indicate cases in which the server is aware that it has encountered an error or is otherwise incapable of performing the request...
Error code 401 and error code 403 are typically used for authentication related errors. These however have some standards that go along with them that specify when and how they should be used. In your instance neither of these would quite fit. For example, a 403 is described as (http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html):
403 Forbidden
The server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity. If the server does not wish to make this information available to the client, the status code 404 (Not Found) can be used instead.
A good alternative would be to return code 503 (service unavailable).
503 Service Unavailable
The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay.
Errors authenticating clients should be under 401:
401 Unauthorized (RFC 7235)
Similar to 403 Forbidden, but specifically for use when authentication is required and has failed or has not yet been provided. The response must include a WWW-Authenticate header field containing a challenge applicable to the requested resource. See Basic access authentication and Digest access authentication.[36] 401 semantically means "unauthenticated", i.e. the user does not have the necessary credentials.
Note: Some sites issue HTTP 401 when an IP address is banned from the website (usually the website domain) and that specific address is refused permission to access a website.
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_Error
I want to catch a third-party made application's http/https messages with fiddlercore in C#.
I'm not sure what kind of the connection type is but what I did is
FiddlerApplication.OnReadRequestBuffer += FiddlerApplication_OnReadRequestBuffer;
FiddlerApplication.OnReadResponseBuffer += FiddlerApplication_OnReadResponseBuffer;
FiddlerApplication.AfterSessionComplete += FiddlerApplication_AfterSessionComplete;
FiddlerApplication.Startup(8888, false, true);
Then I start the application and setup proxy to fiddler's.
The AfterSessionComplete is called only once. The response code is 200 but responseBody is empty.
However, after that I keep receiving OnReadRequestBuffer and OnReadResponseBuffer. The application does keep connecting with the internet(through fiddler proxy)
In my other project I found any messages goes in and out through fiddler should contain one BeforeRequest, then some OnReadRequestBuffer, then following some OnReadResponseBuffer and finally one AfterSessionComplete.
In my mind this is the model of any single post/get process.
But here what I encountered here is that the session won't complete and request -> response -> another requests -> another response...
So how can I catch every full request and the full response to it?
Is there any callback that gives the sign of a full request or a full response is done?
How can I decrypt https base on the partial buffers that runs in and out that caught by OnReadRequestBuffer/OnReadResponseBuffer?
Edit: OnWebSocketMessage is never triggered. Someone says that this kind of protocol could be COMET. But I find nothing about how to catch COMET messages with fiddler. (And I know nothing about COMET)