I'm trying to obtain a JSON via a rest API using, targeting .Net 4.5
I've tried various methods in code, but the all end up in me getting:
"Authentication failed because the remote party has closed the
transportstream" .
the exact same URL works via browser and Postman.
So far, I've tried using .Net's WebClient, HttpClient and HttpWebRequest with identical results. I've tried comparing requests between Postman and my code (via RequestBin), but even when they were identical, I still kept getting back:
Authentication failed because the remote party has closed the
transport
My current code is using HttpWebRequest, but every solution will do.
I've played around with all of the security protocols, some of them will cause the API to return 404 and some will cause the server to return
"Authentication failed because the remote party has closed the
transport stream".
Here's my current code:
public string GetCityStreets()
{
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var request = (HttpWebRequest)WebRequest.Create(url);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls12 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls;
var response = (HttpWebResponse)request.GetResponse();
string jsonResponse;
using (var reader = new StreamReader(response.GetResponseStream()))
{
jsonResponse = reader.ReadToEnd();
}
return jsonResponse;
}
In my current code, the exception is thrown when the request is actually made: request.GetResponse().
What I need, essentially, is to get the JSON from the API.
Set SecurityProtocolType.Tls12 before you initalize the request:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12`
var request = WebRequest.CreateHttp(url);
If you're on Windows 7. On Windows 10, you should only need SecurityProtocolType.SystemDefault.
Note: To enable TLS1.3 (it's available in both Windows 7 and Windows 10), If you don't use .Net 4.8 or .Net Core 3.0, since there's no enumerator for it, you can set it with:
var secProtoTls13 = (SecurityProtocolType)12288;
Remove all the other SecurityProtocolType you have set there.
Setting the User-Agent header is also mandatory, otherwise you will receive a 404 (not found). Here's the FireFox header:
request.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0";
A note on the User-Agent header: this specific site doesn't activate HTTP Strict Transport Security (HSTS). But some sites do, when they see that the WebBrowser supports it. HttpWebRequest doesn't understand it, so it will simply wait for a response that never comes, since the Site is waiting for interaction.
You may want to use the IE11 header instead.
Also add this other header:
request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
The server side appears to be checking the user agent (presumably to stop bots and other code (like yours!) from hitting the endpoint). To bypass this, you will need to set the user agent to a value such that it thinks you are a web browser.
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";
works, as an example.
You may wish to consider setting the ServicePointManager.SecurityProtocol just once (at app startup) rather than on each request.
This works using HttpClient (.net 4.5 and up)
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", "C# App");
Task<string> response = client.GetStringAsync(url);
Console.WriteLine(response.Result);
Think the server requires a user agent.
Related
When attempting a web request to a site which requires TLS 1.3 https://shop.claytonengineering.com/. I'm receiving the following exception on "request.GetResponse();".
Exception: The SSL connection could not be established, see inner exception.
Inner Exception: The client and server cannot communicate, because they do not possess a common algorithm.
From Google Chrome Developer tools Security tab - "The connection to this site is encrypted and authenticated using TLS 1.3, X25519, and AES_128_GCM."
Any ideas on how to get this request to work?
HttpWebResponse response = null;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://shop.claytonengineering.com/");
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", #"1");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36";
request.Headers.Add("Sec-Fetch-Mode", #"navigate");
request.Headers.Add("Sec-Fetch-User", #"?1");
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
request.Headers.Add("Sec-Fetch-Site", #"same-origin");
request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls13;
response = (HttpWebResponse)request.GetResponse();
string html = null;
using (StreamReader stream = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
{
html = stream.ReadToEnd();
}
There are at least two parts to a successful TLS connection:
Protocol version
Ciphersuite match
In both cases both the client and server must match up to use the same protocol and cipher. You already understand that there are multiple TLS versions (1.0, 1.1, 1.2, 1.3) but there are also multiple ciphers that are used to encrypt/decrypt information.
After the 3-way handshake, the client sends its accepted versions in the "Client Hello" and a list of accepted ciphers to the server. The server finds a match and sends that information back to the client ("Server Hello"), then communication begins.
In this case, although you're both using TLS 1.3, you aren't matching up with a cipher. The server has a list of ciphers that can be found by using the test found at ssllabs.com.
The list of ciphers accepted by the server you listed above is in the image below. You can enable one of the above on your client and then the connection should succeed. I assume you're using Windows, so please see this:
https://www.howtogeek.com/221080/how-to-update-your-windows-server-cipher-suite-for-better-security/#:~:text=On%20the%20left%20hand%20side%2C%20expand%20Computer%20Configuration%2C,%E2%80%9CEnabled%E2%80%9D%20button%20to%20edit%20your%20server%E2%80%99s%20Cipher%20Suites.
I had this very same issue and found out that while .Net framework 4.8 and .Net5.0 and 6.0 all have implementations of the TLS1.3 stack, none of them seem to use them when you do not tell Windows to do so.
In my case I had to add the following registry key:
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001
For some reason these are not there by default.
I have a few ipv4 address which I am attempting to screen scrape on. I have a basic program written in C# .Net 5.0 which uses HttpClient to make the GET request with a web server. My client works perfectly for a while until the rate limit happens. (Predictable). Which, I then presume I can change my IPv4 address (through Outbound NAT) and it should work in theory. However it doesn't. I get the same error.
I have confirmed that the outbound IPV4 switches correctly. I wrote a small script in C# with https://www.ipify.org/ to get my outbound Ipv4.
I have tried to make this connection in Google Chrome after the rate limit, it works fine. I tried to make the query in Postman on the same machine, it works fine.
Its almost like C# has some type of certificate it uses, and the server is banning the certificate. However I think I nulled this by trying to run my agent on multiple machines.
What type of data is seen in the TLS handshake that could yield this type of detection?
I notice that C# normally uses TLS 1.1 or 1.2, however Google Chrome uses 1.3. But I think I tested this theory too by disabling TLS 1.3 in Postman, and it works. I also tried to run in very old Internet Explorer, and it works.
Also, I did copy all the params/body and headers into the C# application to mimic fully. I also tried to curl the request, it works too.
Please understand this question is less about the code and more about the principal. To my understand of a curl request the only thing that is noticeable on the first request is the TLS process followed by the actual requests/headers/body. Even if an answer is not provided, pointing to the direct and me learning is completely acceptable.
Code Subset :
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
if (proxy != null)
handler.Proxy = proxy;
var client = new HttpClient(handler);
client.DefaultRequestHeaders.Add("authority", "awebsite.itsawebsite.com");
client.DefaultRequestHeaders.Add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9");
client.DefaultRequestHeaders.Add("accept-language", "en-US,en;q=0.9");
//More Headers
client.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36");
var response = client.GetAsync("https://secure.website.com").Result;
var stringResponse = response.Content.ReadAsStringAsync().Result;
I am in the process of migrating one of my company's web services to a new server, and unfortunately the previous developers left us no way to test the migration of this service prior to migrating the production version. This leaves us in a harsh situation where I have to formulate a backup plan in case things go wrong when we migrate to the new server.
To understand my plan, you must first understand that the flow of execution for this web service is currently:
Customer calls platform.
Platform calls web service.
Web service responds to platform.
Platform responds to customer.
Simple enough, but the platform's changes are already in place for deployment at the flip of a switch and the developer will not be in house for the migration. Thus, they will flip the switch and leave me hoping the migration works.
I have a simple rollback plan in which the platform's developer won't be required for me to rollback. I simply inject a middle-man to the chain above which acts as a conduit to the web service for the platform:
Customer calls platform.
Platform calls conduit service.
Conduit service calls web service.
Web service responds to conduit.
Conduit responds to platform.
Platform responds to customer.
This way, if for some reason, the migrated version of the web service fails, I can fallback to the original version hosted on the old server until we can investigate what's missing and why it all went wrong (currently we have no way to do this).
Now that you have an understanding of the issue, I have a simple issue with writing the conduit to the underlying web service. I encountered a method in the web service that returns HttpResponseMessage and expects HttpRequestMessage as a request. This is rather confusing since the platform calls this method via the following URI:
test.domain.com:port/api/route/methodname
I have no access to the code under this URI assignment (which is in RPG code), so I have no idea how they are passing the data over. Currently my code is simple:
[Route("MethodName")]
[HttpPost]
public HttpResponseMessage MethodName(HttpRequestMessage request) {
try {
HttpWebRequest request = (HttpWebRequest)WebRequest.Create($"{ServiceRoute}/api/route/methodname");
request.Method = "GET";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
return response; // There is a type mismatch, I know.
} catch (Exception e) {
// Log exception.
return null;
}
}
How can I call a restful web service and pass on the request message to the service?
NOTE: I understand, the snippet I've supplied will not work and has an error. I DO NOT expect anyone to just hand out code. References and explanations as to what needs to be done and why are what I'm looking for.
I'm not sure I totally understand the question, so apologies if this isn't helpful, but if your conduit truly just forwards each request as-is, you should be able to reuse the incoming HttpRequestMessage by changing the RequestUri property to the web service URI and forwarding it to the web service with an instance of HttpClient. Something like this:
[Route("MethodName")]
[HttpPost]
public async HttpResponseMessage MethodName(HttpRequestMessage request) {
request.RequestUri = $"{ServiceRoute}/api/route/methodname";
request.Method = HttpMethod.Get;
request.Headers.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36";
//add any additional headers, etc...
try
{
//best practice is to reuse the same HttpClient instance instead of reinstantiating per request, but this will do for an example
var httpClient = new HttpClient();
var response = await httpClient.SendAsync(request);
//perform any validation or modification of the response here, or fall back to the old web service on a failure
return response;
}
catch (Exception e)
{
// Log exception.
return null;
}
}
I'm getting this error on just one server running Windows Server 2003:
System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.
Here's my code... Any ideas?
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https:// URL HERE ");
//request.Headers.Add("Accept", "application/xml");
byte[] bytes;
bytes = System.Text.Encoding.ASCII.GetBytes(xml);
request.KeepAlive = false;
request.Accept = "application/xml";
request.ContentType = "application/xml; charset='UTF-8'";
request.ContentLength = bytes.Length;
request.Method = "POST";
request.Timeout = 10000;
request.ServicePoint.Expect100Continue = false;
Setting the HttpWebRequest.KeepAlive to false didn't work for me.
Since I was accessing a HTTPS page I had to set the Service Point Security Protocol to Tls12.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
Notice that there are other SecurityProtocolTypes:
SecurityProtocolType.Ssl3
SecurityProtocolType.Tls
SecurityProtocolType.Tls11
So if the Tls12 doesn't work for you, try the three remaining options.
Also notice you can set multiple protocols. This is preferable on most cases.
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12| SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
Edit: Since this is a choice of security standards it's obviously best to go with the latest (TLS 1.2 as of writing this), and not just doing what works. In fact, SSL3 has been officially prohibited from use since 2015 and TLS 1.0 and TLS 1.1 will likely be prohibited soon as well. source: #aske-b
I was getting the same error, using RestSharp with .NET 4.5. I tested the same URL with cURL and it worked fine. After a long time debugging I found that setting the SecurityProtocol fixed the issue.
See: "The underlying connection was closed: An unexpected error occurred on a send." With SSL Certificate
In my case, I forgot to remove the "s" from "https" when I was swapping URLs between environments. I was hitting Localhost with https on accident. Same thing would occur if you are hitting an http site with no https certificate, or an expired certificate.
This problem may occur in this case, you will want to download a link that is a filter link and you do not have permission to download that link.
I have faced with this error while I was deploying a nuget package to nexus server manually from command line with API-KEY.
I have checked the nexus server configuration and I have realized Nexus NuGet API-Key Realm is not activated. I have activated it and tried again, everythings worked fine.
So, You should check server side to confirm you have activated related realms.
I was getting this error trying to download an rss file with HttpWebRequest. When I tested in a browser, and checked the response codes, the url was fine.
After trying everything here, it occurred to me the site might be blocking based on User Agent.
Changing the User Agent string in the request worked :
let request = WebRequest.Create(url) :?> HttpWebRequest
request.UserAgent <- #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
let response = request.GetResponse()
This User Agent String came from typing "what's my user agent" in Google Chrome
This problem occurs when the client computer cannot send an HTTP request. The client computer cannot send the HTTP request because the connection has been closed or is unavailable. This problem may occur when the client computer is sending lots of data. To resolve this problem, see resolutions A, D, E, F, and O.
https://support.microsoft.com/en-us/kb/915599
I need to call a web page from different domain. When I call this page from browser, it responds normally. But when i call it from a server side code or from jquery ajax script, it responds empty xml.
I am trying to call a page or service like this:
http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801
this responds normally from browser. But when I write a c# code like this:
WebClient wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
wc.Headers[HttpRequestHeader.Accept] = "*/*";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8";
wc.Headers[HttpRequestHeader.Host] = "otherdomain.com";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
i get empty xml as response:
<xml></xml>
How can I get same response from server side code or client side which I got from browser?
I tried solution here: Calling Cross Domain WCF service using Jquery
So that I didnt understand what to do, I couldnt apply solution described.
How can I get same response from server side code or client side which I got from browser?
Due to the same origin policy restriction you cannot send cross domain AJAX requests from browsers.
From .NET on the other hand you could perfectly fine send this request. But probably the web server that you are trying to send the request to expects some HTTP headers such as the User-Agent header for example. So make sure that you have provided all the headers in your request that the server needs. For example to add the User-Agent header:
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
}
You could use FireBug or Chrome developer toolbar to inspect all the HTTP request headers that your browser sends along the request that works and simply add those headers.