We Keep Coding

In .NET, failure to retrieve HTTP resource from W3C web site

Retrieving the resource at http://www.w3.org/TR/xmlschema11-1/XMLSchema.xsd takes around 10 seconds using the following mechanisms:
web browser
curl
Java URL.openConnection()
It's possible that the W3C site is applying some "throttling" - deliberately slowing the response to discourage bulk requests.
Trying to retrieve the same resource from a C# application on .NET, I get a timeout after about 60-70 seconds. I've tried a couple of different approaches, both with the same result:
System.Xml.XmlUrlResolver.GetEntity()
new WebClient().OpenRead(uri)
Anyone have any idea what's going on? Would another API, or some configuration options, solve the problem?

The problem is they are (probably) checking for a User-Agent string. If it's not present, they send you to purgatory. .NET's http clients do not set this by default.
So, give this a shot:
private static readonly HttpClient _client = new HttpClient();
public static async Task TestMe()
{
using (var req = new HttpRequestMessage(HttpMethod.Get,
"http://www.w3.org/TR/xmlschema11-1/XMLSchema.xsd"))
{
req.Headers.Add("user-agent",
"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X)");
using (var resp = await _client.SendAsync(req))
{
resp.EnsureSuccessStatusCode();
var data = await resp.Content.ReadAsStringAsync();
}
}
}
No idea why they do this; Maybe it's a bug in their back-end? (I sure wouldn't want to leave a socket open longer than it needs to be for no good reason). The request still takes 10-15 seconds, but it's better than the 120+ second timeout.

Is setting the Authorization header in HttpClient safe?

I'm working in a MVC5 ASP.NET project, and learned that to send authenticated requests to a WEB API from the controller I could do the following to add a token to the header(using an example code):
public static class APICaller
{
// Use a single instance for HttpClient to reduce overhead
private static readonly HttpClient client = new HttpClient();
//Set the Authorization Header
public static string SetHeader( string token )
{
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
return("Success");
}
}
Is setting the header this way on the HttpClient thread-safe? Will other users have a way to access this same token, given that there is only one instance of this HttpClient?
EDIT:
I'd like to ask one more question to get a better understanding of how it works. Would I need to add the header each time I'm making a request with the same HttpClient object?

With the approach you have, once you've set the default request header on your static instance, it will remain set without you having to keep setting it. This means that if you have multiple requests coming into your server, you could end up in a situation where the header is set for one user and then changed by another request before that first request makes it out the door.
One option to avoid this would be to use SendAsync when using user-specific authorisation headers. This allows you to tie the header to a specific message, rather than setting it as a default for the HttpClient itself.
The code is a bit more verbose, but would look something like this:
using (var httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, "http://path/to/wherever"))
{
httpRequestMessage.Headers.Authorization = new AuthenticationHeaderValue("Bearer", "TheToken");
using (var httpResponseMessage = httpClient.SendAsync(httpRequestMessage))
{
// ...
}
}
As you can see, the header is set specially on each request and therefore the issue of mixing up the headers goes away. The obvious downside is that this syntax is more verbose.

Will other users have a way to access this same token, given that there is only one instance of this HttpClient?
Yes, that is why you need to be careful when setting the default headers.
Would I need to add the header each time I'm making a request with the same HttpClient object?
No, because you set the default header all requests created with that object will have the header.
For things like a Bearer token it is better to not put in the default headers and instead put it in the request header by creating a new HttpRequestMessage object, setting the headers you need there, then using HttpClient.SendAsync( passing in the request message to send the headers along with your request.

Modify request headers per request C# HttpClient PCL

I'm currently using the System.Net.Http.HttpClient for cross platform support.
I read that it is not a good practice to instantiate a HttpClient object for each request and that you should reuse it whenever possible.
Now I have a problem while writing a client library for a service. Some API calls need to have a specific header, some MUST not include this specific header.
It seems that I can only manipulate the "DefaultRequestHeaders" which will be send with each request.
Is there an option when actually making the request with e.g. "client.PostAsync()" to modify the headers only for the specific request?
(Info: Requests can be multi threaded).
Thanks in advance!

Yes, you can create a new HttpRequestMessage, set all the properties you need to, and then pass it to SendAsync.
var request = new HttpRequestMessage() {
RequestUri = new Uri("http://example.org"),
Method = HttpMethod.Post,
Content = new StringContent("Here is my content")
}
request.Headers.Accept.Add(...); // Set whatever headers you need to
var response = await client.SendAsync(request);

Use HttpContent.Headers. Simply create HttpContent instance with required headers and pass it to PostAsync method.

How to determine if an HttpResponseMessage was fulfilled from cache using HttpClient

What is the equivalent to WebResponse.IsFromCache when using HttpClient and HttpResponseMessage?
Is there some HTTP header in the response that I can look at?

FYI: The Windows.Web.Http HttpClient (a similar API targetted at Windows 8.1 app development) does include an HttpResponseMessage.Source field that specifies where the result came from (common values are "cache" and "network").
The Windows.Web.Http classes are usable from C# and other .NET languages, from C++, and from JavaScript (when running as a WwaHost app like from the Windows app store).

Can I ask what you're trying to achieve? Are trying to avoid caching?
The reason for asking is I've looked at the source code for HttpClient (specifically HttpClientHandler) and the source for HttpWebResponse and I dont believe you can get this information from the headers.
HttpClient/HttpClientHandler does use HttpWebResponse internally however it does not expose all properties from HttpWebResponse :
private HttpResponseMessage CreateResponseMessage(HttpWebResponse webResponse, HttpRequestMessage request)
{
HttpResponseMessage httpResponseMessage = new HttpResponseMessage(webResponse.StatusCode);
httpResponseMessage.ReasonPhrase = webResponse.StatusDescription;
httpResponseMessage.Version = webResponse.ProtocolVersion;
httpResponseMessage.RequestMessage = request;
httpResponseMessage.Content = (HttpContent) new StreamContent((Stream) new HttpClientHandler.WebExceptionWrapperStream(webResponse.GetResponseStream()));
//this line doesnt exist, would be nice
httpResponseMessage.IsFromCache = webResponse.IsFromCache;// <-- MISSING!
...
}
So your options the way I see it are:
a) Look at the source code for HttpWebRequest to determine the logic for IsFromCache and retrofit this somehow into HttpClient (this may not even be possible, depends on what the logic actually does/needs)
b)ask the ASP.NET team for this property to be included with HttpResponseMessage. either directly as a property or perhaps they could 'keep' the HttpWebResponse
Neither of these options are that great sorry, hence my original question, what are you trying to acheive?

I've been struggling with this scenario recently as well.
What I needed was an integration test to verify that:
Responses for a newly created resource had the correct headers set by the server.
Subsequent requests for that resource were fulfilled from the client-cache.
Responses for an existing resource had the correct headers set by the server as well.
What I ended up doing was a twofold check:
A non-caching HttpClient to check the initial response:
new WebRequestHandler
{
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = new CookieContainer(),
CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Refresh)
};
var client = new HttpClient(handler)
and a second HTTP client to check the client-side cache:
new WebRequestHandler
{
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = new CookieContainer(),
CachePolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Default)
};
var client = new HttpClient(handler)
To verify the source of response messages I compare the HttpResponseMessage.Headers.Date values from steps 1 and 2 (which will be the same if the response came from the client cache). For my third step I can just re-use the client from the first step and append an arbitrary string to the URL.
Disclaimer: this applies to .NET Framework 4.7 and ignores best practices concerning HttpClient usage but is seems to do the trick for me in my test suite. An explicit property like the one mentioned above would be preferable but does not seem to be available. Since the last reply here is already a few years old there might be better ways to handle this, but I couldn't think of one.

System.Net.WebClient unreasonably slow

When using the System.Net.WebClient.DownloadData() method I'm getting an unreasonably slow response time.
When fetching an url using the WebClient class in .NET it takes around 10 sec before I get a response, while the same page is fetched by my browser in under 1 sec.
And this is with data that's 0.5kB or smaller in size.
The request involves POST/GET parameters and a user agent header if perhaps that could cause problems.
I haven't (yet) tried if other ways to download data in .NET gives me the same problems, but I'm suspecting I might get similar results. (I've always had a feeling web requests in .NET are unusually slow...)
What could be the cause of this?
Edit:
I tried doing the exact thing using System.Net.HttpWebRequest instead, using the following method, and all requests finish in under 1 sec.
public static string DownloadText(string url)
var request = (HttpWebRequest)WebRequest.Create(url);
var response = (HttpWebResponse)request.GetResponse();
using (var reader = new StreamReader(response.GetResponseStream()))
{
return reader.ReadToEnd();
}
}
While this (old) method using System.Net.WebClient takes 15-30s for each request to finish:
public static string DownloadText(string url)
{
var client = new WebClient();
byte[] data = client.DownloadData(url);
return client.Encoding.GetString(data);
}

I had that problem with WebRequest. Try setting Proxy = null;
WebClient wc = new WebClient();
wc.Proxy = null;
By default WebClient, WebRequest try to determine what proxy to use from IE settings, sometimes it results in like 5 sec delay before the actual request is sent.
This applies to all classes that use WebRequest, including WCF services with HTTP binding.
In general you can use this static code at application startup:
WebRequest.DefaultWebProxy = null;

Download Wireshark here http://www.wireshark.org/
Capture the network packets and filter the "http" packets.
It should give you the answer right away.

There is nothing inherently slow about .NET web requests; that code should be fine. I regularly use WebClient and it works very quickly.
How big is the payload in each direction? Silly question maybe, but is it simply bandwidth limitations?
IMO the most likely thing is that your web-site has spun down, and when you hit the URL the web-site is slow to respond. This is then not the fault of the client. It is also possible that DNS is slow for some reason (in which case you could hard-code the IP into your "hosts" file), or that some proxy server in the middle is slow.
If the web-site isn't yours, it is also possible that they are detecting atypical usage and deliberately injecting a delay to annoy scrapers.
I would grab Fiddler (a free, simple web inspector) and look at the timings.

WebClient may be slow on some workstations when Automatic Proxy Settings in checked in the IE settings (Connections tab - LAN Settings).

Setting WebRequest.DefaultWebProxy = null; or client.Proxy = null didn't do anything for me, using Xamarin on iOS.
I did two things to fix this:
I wrote a downloadString function which does not use WebRequest and System.Net:
public static async Task<string> FnDownloadStringWithoutWebRequest(string url)
{
using (var client = new HttpClient())
{
//Define Headers
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.GetAsync(url);
if (response.IsSuccessStatusCode)
{
string responseContent = await response.Content.ReadAsStringAsync();
//dynamic json = Newtonsoft.Json.JsonConvert.DeserializeObject(responseContent);
return responseContent;
}
Logger.DefaultLogger.LogError(LogLevel.NORMAL, "GoogleLoginManager.FnDownloadString", "error fetching string, code: " + response.StatusCode);
return "";
}
}
This is however still slow with Managed HttpClient.
So secondly, in Visual Studio Community for Mac, right click on your Project in the Solution -> Options -> set HttpClient implementation to NSUrlSession, instead of Managed.
Screenshot: Set HttpClient implementation to NSUrlSession instead of Managed
Managed is not fully integrated into iOS, doesn't support TLS 1.2, and thus does not support the ATS standards set as default in iOS9+, see here:
https://learn.microsoft.com/en-us/xamarin/ios/app-fundamentals/ats
With both these changes, string downloads are always very fast (<<1s).
Without both of these changes, on every second or third try, downloadString took over a minute.
Just FYI, there's one more thing you could try, though it shouldn't be necessary anymore:
//var authgoogle = new OAuth2Authenticator(...);
//authgoogle.Completed...
if (authgoogle.IsUsingNativeUI)
{
// Step 2.1 Creating Login UI
// In order to access SFSafariViewController API the cast is neccessary
SafariServices.SFSafariViewController c = null;
c = (SafariServices.SFSafariViewController)ui_object;
PresentViewController(c, true, null);
}
else
{
PresentViewController(ui_object, true, null);
}
Though in my experience, you probably don't need the SafariController.

Another alternative (also free) to Wireshark is Microsoft Network Monitor.

What browser are you using to test?
Try using the default IE install. System.Net.WebClient uses the local IE settings, proxy etc. Maybe that has been mangled?

Another cause for extremely slow WebClient downloads is the destination media to which you are downloading. If it is a slow device like a USB key, this can massively impact download speed. To my HDD I could download at 6MB/s, to my USB key, only 700kb/s, even though I can copy files to this USB at 5MB/s from another drive. wget shows the same behavior. This is also reported here:
https://superuser.com/questions/413750/why-is-downloading-over-usb-so-slow
So if this is your scenario, an alternative solution is to download to HDD first and then copy files to the slow medium after download completes.