We Keep Coding

C# HttpClient HTTPs TLS Data Included

I have a few ipv4 address which I am attempting to screen scrape on. I have a basic program written in C# .Net 5.0 which uses HttpClient to make the GET request with a web server. My client works perfectly for a while until the rate limit happens. (Predictable). Which, I then presume I can change my IPv4 address (through Outbound NAT) and it should work in theory. However it doesn't. I get the same error.
I have confirmed that the outbound IPV4 switches correctly. I wrote a small script in C# with https://www.ipify.org/ to get my outbound Ipv4.
I have tried to make this connection in Google Chrome after the rate limit, it works fine. I tried to make the query in Postman on the same machine, it works fine.
Its almost like C# has some type of certificate it uses, and the server is banning the certificate. However I think I nulled this by trying to run my agent on multiple machines.
What type of data is seen in the TLS handshake that could yield this type of detection?
I notice that C# normally uses TLS 1.1 or 1.2, however Google Chrome uses 1.3. But I think I tested this theory too by disabling TLS 1.3 in Postman, and it works. I also tried to run in very old Internet Explorer, and it works.
Also, I did copy all the params/body and headers into the C# application to mimic fully. I also tried to curl the request, it works too.
Please understand this question is less about the code and more about the principal. To my understand of a curl request the only thing that is noticeable on the first request is the TLS process followed by the actual requests/headers/body. Even if an answer is not provided, pointing to the direct and me learning is completely acceptable.
Code Subset :
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
if (proxy != null)
handler.Proxy = proxy;
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("authority", "awebsite.itsawebsite.com");
client.DefaultRequestHeaders.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
client.DefaultRequestHeaders.Add("accept-language", "en-US,en;q=0.9");
//More Headers
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36");
var response = client.GetAsync("https://secure.website.com").Result;
var stringResponse = response.Content.ReadAsStringAsync().Result;

Windows Store Application Http Requests Only Work After Manifest Change

I am currently building a windows store application and I am running into what I now think is a bug.
All my http requests fail except when I make a change to the application manifest then they will work on the first run but straight after that the next web request will fail.
The strange thing is that in order for me to get it to work again I will have to remove a capability from the manifest, even if it is an important one such as the internet capability then the application will work!
Here are the headers I am passing in my HttpClient request:
requestMessage.Headers.TryAddWithoutValidation("Connection", "keep-alive");
requestMessage.Headers.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate, sdch");
requestMessage.Headers.TryAddWithoutValidation("Accept-Language", "en-US,en;q=0.8");
requestMessage.Headers.TryAddWithoutValidation("Host", "xx.xx.x.xxx");
requestMessage.Headers.TryAddWithoutValidation("Cache-Control", "no-cache");
requestMessage.Headers.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.107 Safari/537.36");
requestMessage.Headers.TryAddWithoutValidation("Upgrade-Insecure-Requests", "1");
Even when I get a web request to hit the server it will always fail on the next call.
Here is the exception I get when it fails:
InnerException = {"An attempt was made to access a socket in a way forbidden by its access permissions 10.98.0.181:80"}
We have deployed the Web API services on IIS and the application is running on a tablet that connects to the server using a VPN.
The tablet is running Symantic Endpoint Protection which is managed externally.
Is there a caching option I can turn of on the device that could be causing this or a setting I have overlooked?

C# make POST request via internet explorer and retrieve response back in to C#?

I am writing a C# application whereby I formulate the POST strings in C# but the website I am POSTing recognizes that I am not using IE, Chrome or Firefox. Is there a way that I can "use" Internet Explorer (or either of the other two browsers) to make the POST request and then retrieve the response back in the C# (to parse the HTML)?
I have this currently:
using (var wb = new WebClient())
{
var data = new NameValueCollection();
//Any key-value arguments for the POST are stored in data
var response = wb.UploadValues(url, "POST", data);
}

Yes. Forge the User-Agent HTTP header. The User-Agent header basically tells the receiving server what program is sending the packets to it.
See this StackOverflow answer on how to do just that.

client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
place after response
also
Other Response

How can I emulate a web browser http request from code?

I use C# in my WPF Project. I want to send a GET http request to a website, but I want to send it in a way, so that it will look like a request from a browser.
Now I have a program which sends a GET request and gets a response. I use WebRequest class for sending GET requests.
I know that browsers add some information to their requests like browser name, OS name and the computer name.
My question is how can I add this information to my WebRequest? To what properties all that information (browser name, OS name) should be assigned?

You should use Fiddler to capture the request that you want to simulate.
You need to look at the inspectors > raw.
This is an example of a request to the fiddler site from chrome
GET http://fiddler2.com/ HTTP/1.1
Host: fiddler2.com
Connection: keep-alive
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36
Referer: https://www.google.be/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,nl;q=0.6
You can then set each one of these headers in your webrequest (see http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.aspx).
WebRequest request = (HttpWebRequest)WebRequest.Create("http://www.test.com");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36";

Generally the information you are interested in (browser, os, etc.) is sent in the "User Agent" header along with the request. You can control the user agent with its property, here:
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.useragent.aspx
There may be other differences, I recommend using Fiddler to capture your browser traffic and then compare it to the traffic from your .NET-based web request.
http://fiddler2.com/
Enjoy.

All such information is sent via header in a web request. You can also add such information in header as key/value pair. However, you have only limited attributes which you can set using WebRequest's header property; many of them are restricted. You can also check the list of restricted header attributes in the following thread: Cannot set some HTTP headers when using System.Net.WebRequest.

How to call a web page/service from different domain?

I need to call a web page from different domain. When I call this page from browser, it responds normally. But when i call it from a server side code or from jquery ajax script, it responds empty xml.
I am trying to call a page or service like this:
http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801
this responds normally from browser. But when I write a c# code like this:
WebClient wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
wc.Headers[HttpRequestHeader.Accept] = "*/*";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8";
wc.Headers[HttpRequestHeader.Host] = "otherdomain.com";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
i get empty xml as response:
<xml></xml>
How can I get same response from server side code or client side which I got from browser?
I tried solution here: Calling Cross Domain WCF service using Jquery
So that I didnt understand what to do, I couldnt apply solution described.

How can I get same response from server side code or client side which I got from browser?
Due to the same origin policy restriction you cannot send cross domain AJAX requests from browsers.
From .NET on the other hand you could perfectly fine send this request. But probably the web server that you are trying to send the request to expects some HTTP headers such as the User-Agent header for example. So make sure that you have provided all the headers in your request that the server needs. For example to add the User-Agent header:
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
}
You could use FireBug or Chrome developer toolbar to inspect all the HTTP request headers that your browser sends along the request that works and simply add those headers.