We Keep Coding

Why is TLS relevant in non-secure http?

I am using WebClient to get data from a web resource thus:
var wc = new System.Net.WebClient();
var stream = wc.OpenRead("http://...");
// etc..
It used to work until recently, when the server would forcibly close the connection.
Based on this StackOverflow answer, I added the ServicePointManager setting thus:
var wc = new System.Net.WebClient();
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var stream = wc.OpenRead("http://...");
// etc..
… and it once again works as it did before.
I get that TLS is relevant to HTTPS and the effect of the setting is to include TLS1.2 in the handshake, and that the host site must have been recently updated to reject the older vulnerable protocol, but why is it necessary for ordinary, non-secure HTTP?

AllowAutoRedirect is set to true by default in WebClient instances.
Therefore, the request automatically follows redirection responses from your server.
Simply set AllowAutoRedirect to false and you will not follow any redirection, so you will not have to deal with SSL/TLS handshakes.
Of course, if your server does not want to serve your request with HTTP, you will not get the content you are looking for. Anyway, setting AllowAutoRedirect to false will help you confirm that the behaviour you encounter is due to a redirect.

Inspect headers set by WebClient proxy

I'm executing request through some free proxy servers, and I would like to know what headers each proxy server sets. Right now I'm visiting a page that prints out the result in the html body.
using(WebClient client = new WebClient())
{
WebProxy wp = new WebProxy("proxy url");
client.Proxy = wp;
string str = client
.DownloadString("http://www.pagethatprintsrequestheaders.com");
}
The WebClient doesn't show the modified headers, but the page prints the correct ones. Is there any way to find out what headers that are being set by the proxy without visiting a page that prints them like in my example? Do I have to create my own http listener?

When the proxy server sets its own headers, it is essentially performing its own web request. It can even hide or override some of the headers that you set using your WebProxy.
Consequently, only the target page (pagethatprintsrequestheaders.com) can reliably see the headers being set by the proxy. There is no guarantee that the proxy server will send back the headers that it had sent to the target, back to you.
To put it another way, it really depends on the proxy server implementation. if the proxy server you are using is based on Apache's ProxyPass, you'd probably see the headers being set! If it's a custom implementation, then you may not see it.
You can first try inspecting the client.ResponseHeaders property of the WebClient after your response comes back. If this does not contain headers matching what (pagethatprintsrequestheaders.com) reports, then it's indeed a custom or modified implementation.
You could then create your own proxy servers, but this is more involved. You would probably spin up an EC2 instance, install Squid/TinyProxy/YourCustomProxy on it and use that in your WebProxy call.
You may also want to modify your question and explain why you want to read the headers. There may be solutions to your overall goal that don't require reading headers at all but could be done in some other way.

It looks like your sending a request from your WebClient, through the proxy and its received by the host at www.pagethatprintsrequestheaders.com.
If the proxy is adding headers to the request, your webclient will never see them on it's request.
webclient proxys request
request with headers added
client -----------> proxy ----------------------> destination host
The webclient can only see the state of the request between it and the proxy. The proxy will create a new request to send to the destination host, and its that request to which the headers are added. It also that request that is received by the destination host (which is why when it echoes back the headers it can see those added by the proxy)
When the response comes back, the headers are set by the host. It's possible that the proxy will add some headers to the response, but even if it did, they are not likely to be the same headers it adds to a request.
response response
(forwarded by proxy) (headers set by host)
client <------------------- proxy <------------------------- destination host
Using a host that echo the headers back as part of the response payload is one option.
Another would be to use something between the proxy and the destination host to inspect the request there (e.g a packet sniffer or another proxy like Fiddler that lets you see the request headers).
If the proxy is outside of you network, getting between the proxy and the destination host will be difficult (unless the host is under your control).

HTTP 407 (Proxy Authentication Required)

I have some basic questions on HTTP authentication
1) How does client know about Server's authentication type (Basic/Digest/NTLM) of HTTP ? is this configurable at HTTP Server side?
My Answer: Server will be set with which authentication type it has to perform with the client.
So our client API(in C# HttpWebRequest API) will automatically take care of it.
Best use of Wireshrk with applying HTTP filter; you will get source and Destination IP at Internet Protocol layer. And src and dest port at Transmission control protocol and athentication type at http layer.
2) If i place squid linux proxy in between client and Server; is there any need from my client code should know about authentication type of proxy also? or authentication type is only related to the end HTTP server?
My Answer: If squid proxy is placed in between Client and Server; it wont use HTTP authentication. It may use a) DB: Uses a SQL database b) LDAP: Uses the Lightweight Directory Access Protocol. c) RADIUS: Uses a RADIUS server for login validation. and etc..
So we have to mention proxy authentication credentials in HTTP Headers.
3) Using WireShark found that there are three request from Browser to Server to fulfill single request.
a) Browser sends a request without any authentication credentials; So server responded with 401 along with relam and nonce.
WWW-Authenticate: Digest realm="realm", qop="auth", nonce="MTM1OTYyMzkyNDU4MzpiOWM0OWY0NmMzMzZlMThkMDJhMzRhYmU5NjgwNjkxYQ=="\r\n <BR>
b) The second time Browser sends request with credentials, relam, nonce, cnonce; but still server responded with 401;
WWW-Authenticate: Digest realm="realm", qop="auth", nonce="MTM1OTYyMzk0OTAyMTo3Njk3MDNhZTllZDQyYzQ5MGUxYzI5MWY2MGU5ZDg0Yw==", stale="true"\r\n
c) The third time Browser send the same request with same credentials, relam, nonce, cnonce. This time Server sends 200 ok.
My Question is in the second and third time Browser send the same request; why Server failed at second time and success at the third time. Is this because of my server implementation ? (I am having REST Java server with SPRING Security filter).
I have C# HTTP client;
where in the first time HttpWebRequest is sent without credentials though the System.Net.NetworkCredentials is set; so clinet got 407 with relam, and nonce. The second time HttpWebRequest is scuccess. There is no third request from this client like browser.
Why this difference between the browser and C# client?
My Answer: I am still don't know what is happening here: Q3.
4) Now the real issue i am facing is When SQUID LINUX PROXY came in between our Client and HTTP Server, Browser did the same three request authentication and succeeded. However C# HttpWebRequest is failed (401) at the second request and reached the cache(exception){} block and not tried for the third time.
Could you please anyone clarify me how to resolve this issue in C# client when PROXY SERVER is in between?
Below code is doing the GET Request.
HttpWebRequest request = WebRequest.Create("url") as HttpWebRequest;
request.Credentials = new NetworkCredential(loginUserName, password);
WebResponse response = request.GetResponse();
Note that our request to proxy is send via TCP protocol not with HTTP protocol. Then from PROXY to SERVER is communicated with HTTP protocol.
But HTTP request from Proxy has the info about our client ip in the HTTP header X-Forwarded-For.
Below are the possible solutions
These solutions only required if your proxy requires any authentication else ignore it.
Solution 1: working for me
request.Proxy.Credentials = System.Net.CredentialCache.DefaultNetworkCredentials;
Solution 2:
IWebProxy proxy = WebRequest.GetSystemWebProxy();
proxy.Credentials = new NetworkCredential(UserName, UserPassword, UserDomain);
request.Proxy = proxy;
Solution 3 given by Martin:
var proxy = new WebProxy ("http://localhost:3128/");
proxy.Credentials = new NetworkCredential (UserName, UserPassword, UserDomain);
request.Proxy = proxy;
Learn more about proxy authentication at
http://wiki.squid-cache.org/Features/Authentication

It's a challenge/response protocol. Usually, the client makes an initial request without any authentication headers (you may set request.PreAuthenticate = true to send the credentials with the first request).
Then, the server responds with a list of authentication methods that it supports.
If the user did not explicitly specify an authentication method using CredentialsCache, the runtime will try them all, from strongest to weakest. Some protocols (NTLM, for instance) require multiple requests from client to server. In theory, Digest should work with a single one, no idea why it's sending the request twice.
Regarding your proxy question, there are two different kinds of authentication:
The target web server's authentication, a proxy simply passes this though and you don't need any special code in your client.
In addition to that, the proxy itself may also require authentication - which may be different from the one of target web server.
You specify these using
var proxy = new WebProxy ("http://localhost:3128/");
proxy.Credentials = new NetworkCredential ("username", "password");
and then
WebRequest.DefaultWebProxy = proxy;
or
request.Proxy = proxy;
Don't set any credentials on the WebProxy if your proxy server doesn't use any authentication.
If you can't get authentication working while using a proxy server, look at the actual requests that are being sent between the three parties (web server, proxy, client) with Wireshark.

How to use proxy like browser OR CredentialCache.DefaultCredentials different between XP and 7

I am able to fix a problem with a client where they cannot authenticate through a proxy doing the following:
var proxy = WebRequest.GetSystemWebProxy();
proxy.Credentials = CredentialCache.DefaultNetworkCredentials;
service.Proxy = proxy;
This works fine for Windows XP, however on Windows 7 I get a 407 (proxy not authenticated exception). Does anybody know what the difference is, and more importantly, what I need to do to get this to work on both OS?
UPDATE
I am having the users check the following:
In the registry editor, can you go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon and let me know what the value is for CachedLogonsCount. f
In the Start box, type in Group Policy and an option to Edit Group Policy should pop up, click on it. Then go to Computer Configuration\Administrative Templates\System\User Profiles\Delete cached copies of roaming profiles and let me know if it is configured, and if so, to what is it set?
UPDATE FOR BOUNTY
So, I added the bounty. I can take a solution from here, or just an alternate means to getting through a proxy on Windows 7...
Another Update
I am not sure if this is useful or not, but we are also doing the following:
service.PreAuthenticate = true;
service.Url = "myurl";
service.Credentials = new NetworkCredential(txt_UserName.Text, txt_Password.Text);
My temporary solution
This is not really a solution, but works for now. I am using the app.config and setting the proxy to be default, with a ByPassList so that the proxy is not even used. This is only doable since the proxy does not have a strong firewall currently. For other clients, I need to get the above to work

This piece of code works for me on XP, Win7 and 2008
var webProxy = new WebProxy(WebRequest.DefaultWebProxy.GetProxy(new Uri({TheURLoftheService})));
webProxy.Credentials = CredentialCache.DefaultCredentials;
webProxy.UseDefaultCredentials = true;
service.Proxy = webProxy;

actually looks like they "fixed" it in Win7 :) Can you confirm that both client and server are specifying http 1.1
Now let's discuss as to why the browser works in this scenario. IE
uses WinINet under the hood rather than WinHTTP. If we look at the
network traces we see that IE sends HTTP/1.1, but the proxy replies
with HTTP/1.0. IE still accepts this behavior, because in the internet
scenario there are countless number of clients and servers which still
use HTTP/1.0.
WinHTTP strictly requires HTTP/1.1 compliance for keeping the
connection alive and HTTP Keep-Alives are not supported in HTTP/1.0
protocol. HTTP Keep-Alive feature was introduced in the HTTP/1.1
protocol as per RFC 2616. The server or the proxy which expects the
keep-alive should also implement the protocol correctly. WinHTTP on
Windows 7, Windows 2008 R2 are strict in terms of security wrto
protocol compliance. The ideal solution is to change the server/proxy
to use the right protocol and be RFC compliant.
http://blogs.msdn.com/b/httpcontext/archive/2012/02/21/changes-in-winhttp-on-windows-7-and-onwards-wrto-http-1-0.aspx

Will this work?
I am using this to set proxy, so far we did not encounter an error on all windows platform
Uri address = new Uri("http://your-webservice-address");
//Get User current network credential
ICredentials credentials = CredentialCache.DefaultCredentials;
NetworkCredential credential = credentials.GetCredential(address, "Basic");
//Get HttpWebRequest
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
//Network Credential should be included on the request to avoid network issues when requesting to the web servic
request.Proxy = WebRequest.DefaultWebProxy;
request.Credentials = new NetworkCredential(credential.UserName, credential.Password, credential.Domain);

It's hard to say based on the code you've given. I'd suspect that it's either your IE settings or your proxy variables.
Check http://social.msdn.microsoft.com/Forums/en/netfxnetcom/thread/61b71194-1758-4c7b-89fe-91be7363db13 it may help.

WCF REST Consuming error

I keep getting the following error when trying to consume a webservice:
The HTTP request is unauthorized with client authentication scheme 'Basic'. The authentication header received from the server was 'Basic Realm'.
The webservice is REST written with WCF. The authentication is basic over https.
Any help fixing the error would be apreciated.
Here is the code I tried:
WebHttpBinding webBinding = new WebHttpBinding();
webBinding.Security.Mode = WebHttpSecurityMode.Transport;
webBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Basic;
ChannelFactory<ServiceReferences.BTService.FLDT_WholesaleService> factory = new ChannelFactory<ServiceReferences.BTService.FLDT_WholesaleService>(webBinding,
new EndpointAddress(
"https://wholesale.fluidata.co.uk/FLDT_BT_wholesale/Service.svc"));
factory.Endpoint.Behaviors.Add(new WebHttpBehavior());
factory.Credentials.UserName.UserName = "username";
factory.Credentials.UserName.Password = "password";
ServiceReferences.BTService.FLDT_WholesaleService proxy = factory.CreateChannel();
proxy.AvailabilityCheck("123");

As long as you expose RESTful service you may attempt to use Fiddler - http://www.fiddler2.com/fiddler2/ and/or normal HttpRequest/HttpResponse. Did you try anything like that?

Mr. Franek's answer is useful - you WILL be using Fiddler in working with WCF, period. I can add a little bit...what is happening here is that you've specified "Basic" as your authentication scheme as a client. The server is saying "I only allow 'Basic Realm'" as the authentication scheme. What is 'realm'? Basically a credential namespace:
Realm for HTTP basic authentication
Here's another helpful link: Authentication in WinHTTP
I can't find a property or method overload that carries Realm...I would probably try to construct the Authenticate-WWW header manually.
That would go something like this:
request.Headers.Add("WWW-Authenticate", string.Format("basic realm=\"{0}\", realm));
"realm" would be the value of whatever the server is expecting, e.g., "www.targetsite.com".