We Keep Coding

Hide Http(s) traffic made by my application from Fiddler

I have application which uses http to obtain data from my server like this:
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(requestString);
req.Timeout = 200 * 1000;
req.Headers.Add(String.Format("deleteme: {0}", content));
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Stream resStream = resp.GetResponseStream();
StreamReader read = new StreamReader(resStream);
html = read.ReadToEnd();
Everything works fine, but how can I hide my requests from Fiddler (something similar to Wireshark)? I want to prevent users to see it.

Fiddler works by registering itself as the http proxy in Windows.
You can disable your application from using the default proxy by setting a specific proxy (like in the code below "no proxy") anywhere in your application before making web requests:
HttpWebRequest.DefaultWebProxy = new WebProxy();
Note that this will also prevent your application from using a configured proxy when one is set-up for legitimate reasons.
This will hide the requests from Fiddler or any other tool that traces web requests by registering itself as http proxy. This will not prevent tracing the request with other tools that operate on a different level in the stack (like Wireshark)
Security by obscurity does not really work. If you want to make it impossible to read the data transferred, use actual encryption.

C# System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send

I'm getting this error on just one server running Windows Server 2003:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
Here's my code... Any ideas?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https:// URL HERE ");
//request.Headers.Add("Accept", "application/xml");
byte[] bytes;
bytes = System.Text.Encoding.ASCII.GetBytes(xml);
request.KeepAlive = false;
request.Accept = "application/xml";
request.ContentType = "application/xml; charset='UTF-8'";
request.ContentLength = bytes.Length;
request.Method = "POST";
request.Timeout = 10000;
request.ServicePoint.Expect100Continue = false;

Setting the HttpWebRequest.KeepAlive to false didn't work for me.
Since I was accessing a HTTPS page I had to set the Service Point Security Protocol to Tls12.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Notice that there are other SecurityProtocolTypes:
SecurityProtocolType.Ssl3
SecurityProtocolType.Tls
SecurityProtocolType.Tls11
So if the Tls12 doesn't work for you, try the three remaining options.
Also notice you can set multiple protocols. This is preferable on most cases.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12| SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
Edit: Since this is a choice of security standards it's obviously best to go with the latest (TLS 1.2 as of writing this), and not just doing what works. In fact, SSL3 has been officially prohibited from use since 2015 and TLS 1.0 and TLS 1.1 will likely be prohibited soon as well. source: #aske-b

I was getting the same error, using RestSharp with .NET 4.5. I tested the same URL with cURL and it worked fine. After a long time debugging I found that setting the SecurityProtocol fixed the issue.
See: "The underlying connection was closed: An unexpected error occurred on a send." With SSL Certificate

In my case, I forgot to remove the "s" from "https" when I was swapping URLs between environments. I was hitting Localhost with https on accident. Same thing would occur if you are hitting an http site with no https certificate, or an expired certificate.

This problem may occur in this case, you will want to download a link that is a filter link and you do not have permission to download that link.

I have faced with this error while I was deploying a nuget package to nexus server manually from command line with API-KEY.
I have checked the nexus server configuration and I have realized Nexus NuGet API-Key Realm is not activated. I have activated it and tried again, everythings worked fine.
So, You should check server side to confirm you have activated related realms.

I was getting this error trying to download an rss file with HttpWebRequest. When I tested in a browser, and checked the response codes, the url was fine.
After trying everything here, it occurred to me the site might be blocking based on User Agent.
Changing the User Agent string in the request worked :
let request = WebRequest.Create(url) :?> HttpWebRequest
request.UserAgent <- #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
let response = request.GetResponse()
This User Agent String came from typing "what's my user agent" in Google Chrome

This problem occurs when the client computer cannot send an HTTP request. The client computer cannot send the HTTP request because the connection has been closed or is unavailable. This problem may occur when the client computer is sending lots of data. To resolve this problem, see resolutions A, D, E, F, and O.
https://support.microsoft.com/en-us/kb/915599

Windows Authentication for OData service returns "401 - Unauthorized" in C# app, but works in browser

I am trying to add functionality to my C# app, to test a connection to an OData service which is secured with only Windows Authentication. The following block of code is what I am using to perform this test:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri(SERVICE_NAME));
CredentialCache myCache = new CredentialCache();
myCache.Add(new Uri(SERVICE_NAME), "Negotiate", new NetworkCredential(user, password));
resolver.Credentials = myCache;
// Do a simple request
request.Credentials = myCache;
request.PreAuthenticate = true;
request.KeepAlive = true;
request.AllowAutoRedirect = true;
request.CookieContainer = new CookieContainer();
object response = request.GetResponse(); // This is where the exception is thrown
When I run the above code, I receive the 401 - Unauthorized error as previously stated. However, when I have Fiddler2 running, the code works fine. So I am using Wireshark instead. In addition, the service works perfect within my browser (Chrome), and if I use Wireshark to compare the HTTP requests/responses for the Authentication, I see that they are nearly identical, except that in Chrome I have: Accept, User-Agent, Accept-Encoding, and Accept-Language headers, while my C# app does not have these. The only other difference is that my C# app sets the "Negotiate Seal" flag in the NTLM header, while Chrome does not set this flag.
Despite these differences, the authentication phase seems to work fine in the C# app, up until the service returns a 302 - Redirection, at which point the app tries a GET on the newly redirected URI, which returns a 401 again (when Chrome does the analogous GET, it receives HTTP 200 - OK, and proceeds on its merry way).
So, any ideas what could cause this? Problem with the service? or my code?
Thanks a lot!
-Erik

Ok, two whole days of research and I found the answer. Line 3 in the code above was using the full URI of the service (".../Northwind/Northwind.svc"), when the request was redirected, the credentials no longer applied to the new URI. The solution was to pass in only the beginning part ("...") of the URI. Stupid mistake on my part.

How to use proxy like browser OR CredentialCache.DefaultCredentials different between XP and 7

I am able to fix a problem with a client where they cannot authenticate through a proxy doing the following:
var proxy = WebRequest.GetSystemWebProxy();
proxy.Credentials = CredentialCache.DefaultNetworkCredentials;
service.Proxy = proxy;
This works fine for Windows XP, however on Windows 7 I get a 407 (proxy not authenticated exception). Does anybody know what the difference is, and more importantly, what I need to do to get this to work on both OS?
UPDATE
I am having the users check the following:
In the registry editor, can you go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon and let me know what the value is for CachedLogonsCount. f
In the Start box, type in Group Policy and an option to Edit Group Policy should pop up, click on it. Then go to Computer Configuration\Administrative Templates\System\User Profiles\Delete cached copies of roaming profiles and let me know if it is configured, and if so, to what is it set?
UPDATE FOR BOUNTY
So, I added the bounty. I can take a solution from here, or just an alternate means to getting through a proxy on Windows 7...
Another Update
I am not sure if this is useful or not, but we are also doing the following:
service.PreAuthenticate = true;
service.Url = "myurl";
service.Credentials = new NetworkCredential(txt_UserName.Text, txt_Password.Text);
My temporary solution
This is not really a solution, but works for now. I am using the app.config and setting the proxy to be default, with a ByPassList so that the proxy is not even used. This is only doable since the proxy does not have a strong firewall currently. For other clients, I need to get the above to work

This piece of code works for me on XP, Win7 and 2008
var webProxy = new WebProxy(WebRequest.DefaultWebProxy.GetProxy(new Uri({TheURLoftheService})));
webProxy.Credentials = CredentialCache.DefaultCredentials;
webProxy.UseDefaultCredentials = true;
service.Proxy = webProxy;

actually looks like they "fixed" it in Win7 :) Can you confirm that both client and server are specifying http 1.1
Now let's discuss as to why the browser works in this scenario. IE
uses WinINet under the hood rather than WinHTTP. If we look at the
network traces we see that IE sends HTTP/1.1, but the proxy replies
with HTTP/1.0. IE still accepts this behavior, because in the internet
scenario there are countless number of clients and servers which still
use HTTP/1.0.
WinHTTP strictly requires HTTP/1.1 compliance for keeping the
connection alive and HTTP Keep-Alives are not supported in HTTP/1.0
protocol. HTTP Keep-Alive feature was introduced in the HTTP/1.1
protocol as per RFC 2616. The server or the proxy which expects the
keep-alive should also implement the protocol correctly. WinHTTP on
Windows 7, Windows 2008 R2 are strict in terms of security wrto
protocol compliance. The ideal solution is to change the server/proxy
to use the right protocol and be RFC compliant.
http://blogs.msdn.com/b/httpcontext/archive/2012/02/21/changes-in-winhttp-on-windows-7-and-onwards-wrto-http-1-0.aspx

Will this work?
I am using this to set proxy, so far we did not encounter an error on all windows platform
Uri address = new Uri("http://your-webservice-address");
//Get User current network credential
ICredentials credentials = CredentialCache.DefaultCredentials;
NetworkCredential credential = credentials.GetCredential(address, "Basic");
//Get HttpWebRequest
HttpWebRequest request = WebRequest.Create(address) as HttpWebRequest;
//Network Credential should be included on the request to avoid network issues when requesting to the web servic
request.Proxy = WebRequest.DefaultWebProxy;
request.Credentials = new NetworkCredential(credential.UserName, credential.Password, credential.Domain);

It's hard to say based on the code you've given. I'd suspect that it's either your IE settings or your proxy variables.
Check http://social.msdn.microsoft.com/Forums/en/netfxnetcom/thread/61b71194-1758-4c7b-89fe-91be7363db13 it may help.

HttpWebRequest doesn't work except when fiddler is running

This is probably the weirdest problem I have run into. I have a piece of code to submit POST to a url. The code doesn't work neither throws any exceptions when fiddler isn't running, However, when fiddler is running, the code posts the data successfuly. I have access to the post page so I know if the data has been POSTED or not. This is probably very non-sense, But it's a situation I am running into and I am very confused.
byte[] postBytes = new ASCIIEncoding().GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://myURL);
req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10";
req.Accept = "application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
req.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3");
req.Headers.Add("Accept-Language", "en-US,en;q=0.8");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
req.ContentLength = postBytes.Length;
req.CookieContainer = cc;
Stream s = req.GetRequestStream();
s.Write(postBytes, 0, postBytes.Length);
s.Close();

If you don't call GetResponseStream() then you can't close the response. If you don't close the response, then you end up with a socket in a bad state in .NET. You MUST close the response to prevent interference with your later request.

Close the HttpWebResponse after getting it.
I had the same problem, then I started closing the responses after each request, and Boom, no need to have fiddler running.
This is a pseudo of a synchronous code:
request.create(url);
///codes
httpwebresponse response = (httpwebresponse)request.getresponse();
/// codes again like reading it to a stream
response.close();

I had a similar problem recently. Wireshark would show the HTTPWebRequest not leave the client machine unless Fiddler was running. I tried removing proxy settings, but that didn't fix the problem for me. I tried everything from setting the request to HttpVersion.Version10, enabling/disabling SendChuck, KeepAlive, and a host of other settings. None of which worked.
Ultimately, I just checked if .Net detected a proxy and had the request attempt to ignore it. That fixed my issue with request.GetResponse() throwing an immediate exception.
IWebProxy proxy = request.Proxy;
if (request.Proxy != null)
{
Console.WriteLine("Removing proxy: {0}", proxy.GetProxy(request.RequestUri));
request.Proxy = null;
}

In my case when I had the same situation (POST only works when Fiddler is running) the code was sending the POST from an application running on IISExpress in a development environment behind a proxy to an external server. Apparently even if you have proxy settings configured in Internet Options the environment IIS is running in may not have access to them. In my work environment I simply had to update web.config with the path to our proxy's configuration script. You may need to tweak other proxy settings. In that case your friend is this MSDN page that explains what they are: http://msdn.microsoft.com/en-us/library/sa91de1e.aspx.
Ultimately I included the following in the application's web.config and then the POST went through.
<configuration>
<system.net>
<defaultProxy>
<proxy scriptLocation="http://example.com:81/proxy.script" />
</defaultProxy>
</system.net>
</configuration>

Well i have faced similar problem few weeks back and the reason was that when fiddler is running it changes the proxy settings to pass the request through Fiddler but when its closed the proxy somehow still remains and thus doesn't allow your request to go ahead on internet.
I tried by setting the IE's and Firefox's network settings to not to take any proxy and it worked.
Try this, may it be the same problem...

I ran into the same problem with Python - requests to a local server were failing with a 404, but then when I ran them with Fiddler running they were working correctly.
The real clue to the problem here is that Fiddler works by acting as a proxy for HTTP traffic so that all requests from the local machine go through Fiddler rather than straight out into the network.
In the exact situation I was in, I was making requests to a local server, regular traffic passes through a proxy and in Local Area Network (LAN) Settings for the network connection the Proxy server pane the Bypass proxy server for local addresses option was checked.
My suspicion is that the "Bypass proxy server for local addresses" is not necessarily picked up by the programming language, but the proxy server details are. Fiddler is aware of that policy, so requests through Fiddler work but requests direct from the programming language don't.
By setting the proxy for the request for the local server to nothing, it worked correctly from code. Obviously, that could be a gotcha if you find yourself moving from an internal to external server during deployment.

I faced the same scenario : I was POSTing to an endpoint behind Windows Authentication.
Fiddler keeps a pool of open connections, but your C# test or powershell script does not when it runs without fiddler.
So you can make the test/script to also maintain a pool of open authenticated connections, by setting the property UnsafeAuthenticatedConnectionSharing to true on your HttpWebRequest. Read more about it here, microsoft KB. In both cases in that article, you can see that they are making two requests. The first one is a simple GET or HEAD to get the authentication header (to complete the handshake), and the second one is the POST, that will use the header obtained before.
Apparently you cannot (sadness) directly do the handshake with POST http requests.

Always use using construct. it make sure all resource release after call
using (HttpWebResponse responseClaimLines = (HttpWebResponse)requestClaimLines.GetResponse())
{
using (StreamReader reader = new StreamReader(responseClaimLines.GetResponseStream()))
{
responseEnvelop = reader.ReadToEnd();
}
}
add following entries to webconfig file
<system.net>
<connectionManagement>
<add address="*" maxconnection="30"/>

I found the solution in increasing the default number of connections
ServicePointManager.DefaultConnectionLimit = 10000;