I'm using HttpClient 0.6.0 from NuGet.
I have the following C# code:
var client = new HttpClient(new WebRequestHandler() {
CachePolicy =
new HttpRequestCachePolicy(HttpRequestCacheLevel.CacheIfAvailable)
});
client.GetAsync("http://myservice/asdf");
The service (this time CouchDB) returns an ETag value and status code 200 OK. There is returned a Cache-Control header with value must-revalidate
Update, here are the response headers from couchdb (taken from the visual studio debugger):
Server: CouchDB/1.1.1 (Erlang OTP/R14B04)
Etag: "1-27964df653cea4316d0acbab10fd9c04"
Date: Fri, 09 Dec 2011 11:56:07 GMT
Cache-Control: must-revalidate
Next time I do the exact same request, HttpClient does a conditional request and gets back 304 Not Modified. Which is right.
However, if I am using low-level HttpWebRequest class with the same CachePolicy, the request isn't even made the second time. This is the way I would want HttpClient also behave.
Is it the must-revalidate header value or why is HttpClient behaving differently? I would like to do only one request and then have the rest from cache without the conditional request..
(Also, as a side-note, when debugging, the Response status code is shown as 200 OK, even though the service returns 304 Not Modified)
Both clients behave correctly.
must-revalidate only applies to stale responses.
When the must-revalidate directive is present in a response received by a cache, that cache MUST NOT use the entry after it becomes stale to respond to a
subsequent request without first revalidating it with the origin server. (I.e., the cache MUST do an end-to-end revalidation every time, if, based solely on the origin server's Expires or max-age value, the cached response is stale.)
Since you do not provide explicit expiration, caches are allowed to use heuristics to determine freshness.
Since you do not provide Last-Modified caches do not need to warn the client that heuristics was used.
If none of Expires, Cache-Control: max-age, or Cache-Control: s- maxage (see section 14.9.3) appears in the response, and the response does not include other restrictions on caching, the cache MAY compute a freshness lifetime using a heuristic. The cache MUST attach Warning 113 to any response whose age is more than 24 hours if such warning has not already been added.
The response age is calculated based on Date header since Age is not present.
If the response is still fresh according to heuristic expiration, caches may use the stored response.
One explanation is that HttpWebRequest uses heuristics and that there was a stored response with status code 200 that was still fresh.
Answering my own question..
According to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.4 I would say that
a "Cache-Control: must-revalidate" without expiration states that the resource should be validated on every request.
In this case it means a conditional GET should be done every time the resource is made. So in this case System.Net.Http.HttpClient is behaving correctly and the legacy (Http)WebRequest is doing invalid behavior.
Related
I'm interested in interfacing with Discord using the Discord API. I would describe their documentation as "sparse," but maybe I'm just not looking in the right places. Most of my information comes from this page:
https://discordapp.com/developers/docs/topics/oauth2
I've already set up my Discord guild and application (and even a bot, which may be unnecessary). My specific plan to is to allow users to give my site permission to add them to a private Discord guild/server. I have a hyperlink on one of my site's pages that references this URL:
https://discordapp.com/api/oauth2/authorize?client_id=[ClientID]&scope=guilds.join&response_type=code&redirect_uri=[RedirectURI]
This part seems to work well. The user approves the request. The user is then sent back to my site with a "code" key-value pair in the query string. I think this code is what is called an "authorization code." So how can I use this authorization code to add the user to my guild? I found this page on the Discord site:
https://discordapp.com/developers/docs/resources/guild#add-guild-member
From that page I can see I need to initiate a PUT to this URL:
https://discordapp.com/api/guilds/{guild.id}/members/{user.id}
But I don't know the {user.id}. I only have an authorization code.
It also says, "... provided you have a valid oauth2 access token for the user with the guilds.join scope." I don't have an access token. Again, I only have an authorization code.
So it seems to me I need to somehow exchange this authorization code for an access token and a user ID. Can someone tell me how to do that? I've been experimenting with the following URL, but I don't know what method (GET, POST, etc.) or what parameters to send it:
https://discordapp.com/api/oauth2/token
Because I'd like to understand the intracacies of how this works, I'd prefer to know how to do this with ordinary Web requests (such as HttpWebRequest and WebClient, as opposed to using some OAuth library).
Update
I decided to read (selectively) this RFC:
https://www.rfc-editor.org/rfc/rfc6749#section-4.1.3
I have linked the section that I think is the most appropriate. It seems the correct procedure is to send a POST request to the following URL and parameters:
https://discordapp.com/api/oauth2/token
grant_type=authorization_code&code=[AuthorizationCode]&redirect_uri=[RedirectURI]&client_id=[ClientID]
This also seems to be in accordance with Peter G's answer. Unfortunately, this request fails with a 401 (Unauthorized) error. So I figured this was a dead end. I've tried it several times, but hopefully there's a solution. I received this response body:
{"error": "invalid_client"}
And I received these headers:
Connection: close
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Alt-Svc: clear
CF-RAY: [RedactedJustInCase]
Content-Length: 27
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 07 Apr 2017 01:12:19 GMT
Set-Cookie: __cfduid=[RedactedJustInCase]; expires=Sat, 07-Apr-18
01:12:19 GMT; path=/; domain=.discordapp.com; HttpOnly
Server: cloudflare-nginx
Via: 1.1 google
You're almost there as far as getting the OAuth token is concerned. You just need to use the other URL listed on the documentation you linked, https://discordapp.com/api/oauth2/token. POST to it with the following parameters: https://discordapp.com/api/oauth2/token?client_id=[ClientID]&grant_type=authorization_code&code=[AuthorizationCode]&redirect_uri=[RedirectURI]&client_secret=[Secret] where the AuthorizationCode is the return from the first URL and the Secret is the client secret you got when first registering your app.
That should get you the client token back (as well as how long it will take for the token to expire) in the response body. As for getting the User object, you need to add scope identify to the first request so you can use the token to call https://discordapp.com/developers/docs/resources/user#get-current-user (in case the link breaks, it's GET users/#me). That API will return the User object in JSON form.
Finally, you can add the user by PUT-ing to https://discordapp.com/api/guilds/[guild.id]/members/[user.id] using the user object you just got.
When using the APIs after getting the client token (the ones to get the user object and put the user in the guild), you need to put the token in the HTTP request under the authorization header using the Bearer auth scheme. Basically, that means the header should be set to "Bearer TOKEN_HERE". You should also use content-type "application/x-www-form-urlencoded" if you weren't already using it.
If there's anything you don't understand here, I strongly suggest reading about oauth from the source RFC (don't worry, these two sections are short): getting an auth code, getting a token, authenticating with Bearer scheme. Them's the breaks when you don't use a library.
We have stumble unto a problem while using WebRequestHandler and HttpRequestCachePolicy where the cached entry seems to get corrupted after a precise sequence. The scenario goes as follow:
request returns a 200 response with Cache-Control: public, must-revalidate, max-age=0 and Last-Modified.
a second request is made with Last-Modified-Since that returns a 500 response with Cache-Control: private. In our case, it is caused by the IIS server when an error occurs during the request.
a third request is sent. It contains a Last-Modified-Since header which causes the server to respond with 304.
The HttpClient result resolves to the 500 response.
My assumption is that the Last-Modified-Since header in the 3rd request should not have been present and is a bug.
The test was made in a sample console application in .NET 4.5 while using the HttpRequestCacheLevel.Default cache policy. Trying the same test in a browser doesn't seem to reproduce the problem when navigating to the URL (I verified that max-age=0 was not present in the request).
What I am trying to understand is if it's a bug in the .NET framework, if we didn't use the API correctly or if we didn't use the response headers correctly.
My application uses ASP.NET MVC 5 with OutputCache (in detail, we use MVCDonutCaching) to cache high traffic sites and expensive routes.
Some of the Actions have a Custom ActionFilter which adds a Content-Range header depending on the view model. Without caching it works like charm. When the cache is enabled the first hit is ok (Content-Range header is present in the response) - but the second one only contains Content-Type and the HTML/JSON Response and our custom Content-Range header is missing (which breaks the client functionality).
Is there any way to enable proper header caching without writing an own OutputCache implementation?
Thank you very much.
The cached response is a "304 - Not Modified" HTTP Response, and that kind of response is not expected to return entity headers (except some exceptions like "Last-Modified").
The "Content-Range" header you are trying to return is an entity header:
http://www.freesoft.org/CIE/RFC/2068/178.htm
Here is a full list of Entity headers:
https://www.rfc-editor.org/rfc/rfc2616#section-7.1
The reason why 304 is not returning entity headers is that the 304 response is not supposed to return a full representation of the target resource, since nothing changed.
The 304 (Not Modified) status code indicates that a conditional GET
or HEAD request has been received and would have resulted in a 200
(OK) response if it were not for the fact that the condition has
evaluated to false. In other words, there is no need for the server
to transfer a representation of the target resource because the
request indicates that the client, which made the request
conditional, already has a valid representation;
That means that entity headers should not be transferred again. This ensures consistency, and also has some performance benefits.
If the conditional GET used a strong cache validator (see section 13.3.3), the response SHOULD NOT include other entity-headers. Otherwise (i.e., the conditional GET used a weak validator), the response MUST NOT include other entity-headers; this prevents inconsistencies between cached entity-bodies and updated headers.
https://datatracker.ietf.org/doc/html/draft-ietf-httpbis-p4-conditional-23#section-4.1
My conclusion is that ASP.NET and IIS are interpreting this specification correctly, and what you are trying to do is NOT supported. A prove for that is that Apache, and other popular web servers do the same as explained above.
If you still need that header in your 304 you will have to identify and replace (if possible) the components responsible for filtering the 304 responses.
We had a weird issue on our site last week that seemed to be a caching issue. A version of our page was cached with Content-Type: text/vnd.wap.wml; charset=utf-8 set in the header.
After some research, I found out that asp .net uses .browser files in the %SystemRoot%\Microsoft.NET\Framework\versionNumber\CONFIG\Browsers path to determine preferred mime types for certain user agents. based on the content-type above, it looks like a Nokia phone was the first application to hit our page after a cache clear based on the content-type above, and asp stored a cached version of the page with that content-type rather than text/html. the problem with that content-type is that browsers do not recognize it, and will just display the page as plain text.
I could verify that the above scenario was the cause. I took one of our servers out of our pool, recycled the app pools for the site and reset iis, then hit the page with fiddler and passed the follow headers as a GET to our homepage.
Accept: text/html
User-Agent: NokiaN90-1/3.0545.5.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1
this returned the following content-type in the response as expected:
Content-Type: text/vnd.wap.wml; charset=utf-8
Now to fix this going forward, it would make sense for asp to cache various flavors of the page based on the content-type it will be serving, right? is there a way to configure asp to do this, or is there a better way to handle this scenario?
I believe that customarily you'd add Vary: User-Agent header if you plan to serve different content types to different clients. E.g. http://msdn.microsoft.com/en-us/library/system.web.httpcachevarybyheaders.useragent(v=vs.100).aspx
My C# app hits a web server that uses NTLM authentication.
I find that each request made to the server (using a new HttpWebRequest) is individually authenticated. In other words, every request results in a 401 response, after which an NTLM handshaking conversation occurs before I then get the actual response.
e.g.:
First GET request:
-> GET xyz
<- 401 error (WWW-Authenticate:NTLM)
-> GET xyz (Authorization:NTLM base64stuff)
<- 401 error (WWW-Authenticate:NTLM base64stuff)
-> GET xyz (Authorization: base64stuff)
<- 200
Subsequent requests:
-> GET xyz (Authorization:NTLM base64stuff)
<- 401 error (WWW-Authenticate:NTLM) //can this request be avoided?
-> GET xyz (Authorization: base64stuff)
<- 200
(initially, with PreAuthenticate set to false, the subsequent requests looked like the first request - i.e. three underlying requests per 'request')
Is there a way to 'share' the authentication performed on the first request to the server with subsequent HttpWebRequests?
I thought perhaps the UnsafeAuthenticatedConnectionSharing property would allow me to do this, but setting it to true for all HttpWebRequest objects used in the app has no effect.
However if I set PreAuthenticate to true, one less 401 response happens for each request after the first one.
Last request sent after NTLM is performed (the one that results in a 200 response) contains an auth header that tells the server that you have the correct credentials.
I'm not sure if the client class has the feature to keep this by its own, but if you find some way to retain this header and add it to your subsequent requests it should work fine.
Update: NTLM authenticates a connection, so you need to keep your connection open using Keep-Alive header. The client class should provide some settings for this.
Maybe it is a bit too late for it, but you have to set the UnsafeAuthenticatedConnectionSharing property to true in the WebRequestHandler (it extends HttpClientHandler).
This way connections are kept alive by allowing the HttpClient to "share" the authentication among other requests, allowing at the same time the connections to be kept alive (you can't do this manually even by setting the header yourself). Bear in mind that you should also have the appropriate persistent authorization in the server, either with authPersistNonNTLM for Kerberos or with authPersistSingleRequest for NTLM.