We Keep Coding

C# HttpClient HTTPs TLS Data Included

I have a few ipv4 address which I am attempting to screen scrape on. I have a basic program written in C# .Net 5.0 which uses HttpClient to make the GET request with a web server. My client works perfectly for a while until the rate limit happens. (Predictable). Which, I then presume I can change my IPv4 address (through Outbound NAT) and it should work in theory. However it doesn't. I get the same error.
I have confirmed that the outbound IPV4 switches correctly. I wrote a small script in C# with https://www.ipify.org/ to get my outbound Ipv4.
I have tried to make this connection in Google Chrome after the rate limit, it works fine. I tried to make the query in Postman on the same machine, it works fine.
Its almost like C# has some type of certificate it uses, and the server is banning the certificate. However I think I nulled this by trying to run my agent on multiple machines.
What type of data is seen in the TLS handshake that could yield this type of detection?
I notice that C# normally uses TLS 1.1 or 1.2, however Google Chrome uses 1.3. But I think I tested this theory too by disabling TLS 1.3 in Postman, and it works. I also tried to run in very old Internet Explorer, and it works.
Also, I did copy all the params/body and headers into the C# application to mimic fully. I also tried to curl the request, it works too.
Please understand this question is less about the code and more about the principal. To my understand of a curl request the only thing that is noticeable on the first request is the TLS process followed by the actual requests/headers/body. Even if an answer is not provided, pointing to the direct and me learning is completely acceptable.
Code Subset :
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
if (proxy != null)
handler.Proxy = proxy;
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("authority", "awebsite.itsawebsite.com");
client.DefaultRequestHeaders.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
client.DefaultRequestHeaders.Add("accept-language", "en-US,en;q=0.9");
//More Headers
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36");
var response = client.GetAsync("https://secure.website.com").Result;
var stringResponse = response.Content.ReadAsStringAsync().Result;

Calling a Restful Web Service that has HttpRequestMessage as a parameter?

I am in the process of migrating one of my company's web services to a new server, and unfortunately the previous developers left us no way to test the migration of this service prior to migrating the production version. This leaves us in a harsh situation where I have to formulate a backup plan in case things go wrong when we migrate to the new server.
To understand my plan, you must first understand that the flow of execution for this web service is currently:
Customer calls platform.
Platform calls web service.
Web service responds to platform.
Platform responds to customer.
Simple enough, but the platform's changes are already in place for deployment at the flip of a switch and the developer will not be in house for the migration. Thus, they will flip the switch and leave me hoping the migration works.
I have a simple rollback plan in which the platform's developer won't be required for me to rollback. I simply inject a middle-man to the chain above which acts as a conduit to the web service for the platform:
Customer calls platform.
Platform calls conduit service.
Conduit service calls web service.
Web service responds to conduit.
Conduit responds to platform.
Platform responds to customer.
This way, if for some reason, the migrated version of the web service fails, I can fallback to the original version hosted on the old server until we can investigate what's missing and why it all went wrong (currently we have no way to do this).
Now that you have an understanding of the issue, I have a simple issue with writing the conduit to the underlying web service. I encountered a method in the web service that returns HttpResponseMessage and expects HttpRequestMessage as a request. This is rather confusing since the platform calls this method via the following URI:
test.domain.com:port/api/route/methodname
I have no access to the code under this URI assignment (which is in RPG code), so I have no idea how they are passing the data over. Currently my code is simple:
[Route("MethodName")]
[HttpPost]
public HttpResponseMessage MethodName(HttpRequestMessage request) {
try {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create($"{ServiceRoute}/api/route/methodname");
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return response; // There is a type mismatch, I know.
} catch (Exception e) {
// Log exception.
return null;
}
}
How can I call a restful web service and pass on the request message to the service?
NOTE: I understand, the snippet I've supplied will not work and has an error. I DO NOT expect anyone to just hand out code. References and explanations as to what needs to be done and why are what I'm looking for.

I'm not sure I totally understand the question, so apologies if this isn't helpful, but if your conduit truly just forwards each request as-is, you should be able to reuse the incoming HttpRequestMessage by changing the RequestUri property to the web service URI and forwarding it to the web service with an instance of HttpClient. Something like this:
[Route("MethodName")]
[HttpPost]
public async HttpResponseMessage MethodName(HttpRequestMessage request) {
request.RequestUri = $"{ServiceRoute}/api/route/methodname";
request.Method = HttpMethod.Get;
request.Headers.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
//add any additional headers, etc...
try
{
//best practice is to reuse the same HttpClient instance instead of reinstantiating per request, but this will do for an example
var httpClient = new HttpClient();
var response = await httpClient.SendAsync(request);
//perform any validation or modification of the response here, or fall back to the old web service on a failure
return response;
}
catch (Exception e)
{
// Log exception.
return null;
}
}

C# - User Agent for Web in WP8.1 Apps

One question that's been confusing me and could really do with some insight.
I need to retreive Json objects from a http service. When I tested this in a Console Window, I kept receiving a "Internal Server Error : 500" until I set the UserAgent property for the WebClient object.
Example:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36");
content = client.DownloadString(url);
Now, if I need to do the same for a WP8.1 app, how would I detect (if I need to in the first place?) the UserAgent (and set it) and be able to retrieve the data?
Thank you all.

Windows Phone 8.1 App will use HttpClient. By default there will not be a user agent set. The default user-agent for the phones web browser is:
"Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 520) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537"
You can manually set the user-agent on the HttpRequestMessage.Headers.UserAgent property.
References:
HttpClient
https://msdn.microsoft.com/en-us/library/windows/apps/xaml/windows.web.http.headers.httprequestheadercollection.aspx
User-Agent
https://msdn.microsoft.com/en-us/library/ie/hh869301(v=vs.85).aspx#ie11\

The class libraries for using http do not add any User Agents by default. See these lines from the msdn page:
By default, no user-agent header is sent with the HTTP request to the web service by the HttpClient object. Some HTTP servers, including some Microsoft web servers, require that a user-agent header be included with the HTTP request sent from the client. The user-agent header is used by the HTTP server to determine how to format some HTTP pages so they render better on the client for different web browsers and form factors (mobile phones, for example). Some HTTP servers return an error if no user-agent header is present on the client request. We need to add a user-agent header to avoid these errors using classes in the Windows.Web.Http.Headers namespace. We add this header to the HttpClient.DefaultRequestHeaders property.
For more details, refer the link below:
How to connect to an HTTP server using Windows.Web.Http.HttpClient (XAML)
Also look at the answer below (by Bret Bentzinger) for the exact user agent string.

HttpWebRequest to pretend like a browser request?

I have some code (in a Winform app) that reads this URL using HttpWebRequest.GetResponse().
For some reason, it recently starts returning 500 Internal Error when requested from my app.
(The response contains some HTML for the navigations, but doesn't have the main content I need)
On Firefox/Chrome/IE, it is still returning 200 OK.
The problem is I don't have control over their code, I don't know what it does on the backend that causes it to break when requested from my app.
Is there a way I can "pretend" to make the request from, say, Google Chrome? (just to avoid the error)

Set the HttpWebRequest.UserAgent property to the value of a real browser's user agent.
HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create("http://example.com");
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36";

How to call a web page/service from different domain?

I need to call a web page from different domain. When I call this page from browser, it responds normally. But when i call it from a server side code or from jquery ajax script, it responds empty xml.
I am trying to call a page or service like this:
http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801
this responds normally from browser. But when I write a c# code like this:
WebClient wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
wc.Headers[HttpRequestHeader.Accept] = "*/*";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8";
wc.Headers[HttpRequestHeader.Host] = "otherdomain.com";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
i get empty xml as response:
<xml></xml>
How can I get same response from server side code or client side which I got from browser?
I tried solution here: Calling Cross Domain WCF service using Jquery
So that I didnt understand what to do, I couldnt apply solution described.

How can I get same response from server side code or client side which I got from browser?
Due to the same origin policy restriction you cannot send cross domain AJAX requests from browsers.
From .NET on the other hand you could perfectly fine send this request. But probably the web server that you are trying to send the request to expects some HTTP headers such as the User-Agent header for example. So make sure that you have provided all the headers in your request that the server needs. For example to add the User-Agent header:
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
}
You could use FireBug or Chrome developer toolbar to inspect all the HTTP request headers that your browser sends along the request that works and simply add those headers.