I am trying to access the Racing Post website from an app - example racing post results
Accessing this from the browsers is fine, but from my app I keep getting issues. This is my final code:
private string download(string url)
{
WebResponse response = null;
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
//{"The request was aborted: Could not create SSL/TLS secure channel."}
// Setup our Web request
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
//request.CookieContainer = new CookieContainer();
//request.Timeout = timeoutSeconds * 1000;
// Retrieve data from request
response = request.GetResponse();
System.IO.Stream streamReceive = response.GetResponseStream();
System.Text.Encoding encoding = System.Text.Encoding.UTF8;
System.IO.StreamReader streamRead = new System.IO.StreamReader(streamReceive, encoding);
// return the retrieved HTML
string s = streamRead.ReadToEnd();
return s;
}
catch (Exception ex)
{
throw new Exception("Problem retrieving the webpage", ex);
}
finally
{
// Check if exists, then close the response.
if (response != null)
{
response.Close();
}
}
}
This is failing with: "The remote server returned an error: (403) Forbidden."
#Crowcoder is correct. If you turn on Fiddler and do requests both through code and browser you'll see that you simply need to add User Agent to the header in code prior to the request.GetResponse() line:
// Set User Agent
request.UserAgent =
#"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36";
Related
I'm developing an app for a school project which uses data from CoinGecko's free public API. The way I'm getting the data is with the HttpWebRequest class. At first, I was able to use the API fast, without any problems, but recently the requests have been getting stuck at the GetResponse() function for a long time. Even pinging the API takes about 30 seconds to complete. I've read a lot of posts regarding this problem, and none of the suggestions worked.
try
{
string uri = "https://api.coingecko.com/api/v3/ping";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.Method = "GET";
request.Accept =
"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng," +
"*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
request.ContentType = "application/json";
request.Headers.Add("accept-encoding", "gzip, deflate, br");
request.Headers.Add("accept-language", "en-US,en;q=0.9");
request.Headers.Add("cache-control", "max-age=0");
request.Headers.Add("if-none-match", "W/\"c1f074e1bdf979ac5dc291f2c0acf9e4\"");
request.Headers.Add("sec-ch-ua", "\" Not A;Brand\"; v = \"99\", \"Chromium\"; v = \"96\", \"Google Chrome\"; v = \"96\"");
request.Headers.Add("sec-ch-ua-mobile", "?0");
request.Headers.Add("sec-ch-ua-platform", "\"Windows\"");
request.Headers.Add("sec-fetch-dest", "document");
request.Headers.Add("sec-fetch-mode", "navigate");
request.Headers.Add("sec-fetch-site", "none");
request.Headers.Add("sec-fetch-user", "?1");
request.Headers.Add("upgrade-insecure-requests", "1");
request.UserAgent =
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)" +
" Chrome/97.0.4692.71 Safari/537.36 Edg/97.0.1072.55";
request.Proxy = null;
using HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode != HttpStatusCode.OK && response.StatusCode != HttpStatusCode.Created)
{
throw new ApplicationException("REST client error code: "
+ response.StatusCode.ToString());
}
using Stream stream = response.GetResponseStream();
if (stream == null)
{
throw new ApplicationException("Error: Null response");
}
using StreamReader reader = new StreamReader(stream);
return reader.ReadToEnd();
}
catch (Exception ex)
{
return ex.ToString();
}
It works with the browser so I copied all the browser request headers and added them to the code but they didn't help. I have also tried: Increasing the number of connections, playing around with all the security protocol types, disabling my firewall. Could it be that the API is detecting and slowing down my requests since they're not "human"?
Set Windows to prioritise IPv4 by typing the following in the command line:
netsh interface ipv6 set prefixpolicy ::ffff:0:0/96 46 4
Source
My method to get tokenKey is :
HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create(url);
httpRequest.Method = "POST";
httpRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 OPR/52.0.2871.99";
string tokenResponse = null;
HttpClient client = new HttpClient();
HttpResponseMessage response = null;
try
{
client.DefaultRequestHeaders.Add("x-api-key", "key");
if (method.Equals("POST"))
{
httpRequest.Accept = "application/json";
httpRequest.ContentType = "application/json";
var data = #"{""username"":#"""+ login + #""",""password"" :#"""+ password + #"""}";
using (var streamWriter = new StreamWriter(httpRequest.GetRequestStream()))
{
streamWriter.Write(data);
}
var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
tokenResponse = streamReader.ReadToEnd();
}
}
}
catch (Exception ex)
{
throw new Exception(ex.Message);
}
return tokenResponse;
after var httpResponse = (HttpWebResponse)httpRequest.GetResponse();
this error msg: The underlying connection was closed: Unexpected error on a send.
x-api-key is okay but i don't put the original as the username and password
i do it easy with postman
postman request done
Can anyone help me understand where I'm going wrong?
Could this be a case of mismatch TLS versions? See this answer here:
C# HttpWebRequest The underlying connection was closed: An unexpected error occurred on a send
Also it looks like your HttpWebRequest won't have the correct headers, because you are setting them on the HttpClient object instead. The HttpClient also doesn't seem to be being used for this call.
See documentation, how to use HttpWebRequest and how to write body properly to request. I think the problem is with a request stream.
I am integrating with Magento 2, using RESTful APIs. When I use postman, it works like charm, while in C# code, it returns "Unauthorized 401" exception.
However, It was working in C# code earlier, but suddenly it stopped working.
I have tried every way, I tried (WebRequest, HTTPClient & RESTsharp) the same exception returned.
Also, I am using Fiddler 4 to catch & match the requests, I used Fiddler to C# plugins to extract C# code, also I used the RESTsharp Code of Postman same exception returned.
The remote server returned an error: (401) Unauthorized.
//Calls request functions sequentially.
private string MakeRequests()
{
HttpWebResponse response;
if (Request_hatolna_co(out response))
{
//Success, possibly uses response.
string responseText = ReadResponse(response);
response.Close();
return responseText;
}
else
{
//Failure, cannot use response.
return "";
}
}
private static string ReadResponse(HttpWebResponse response)
{
using (Stream responseStream = response.GetResponseStream())
{
Stream streamToRead = responseStream;
if (response.ContentEncoding.ToLower().Contains("gzip"))
{
streamToRead = new GZipStream(streamToRead, CompressionMode.Decompress);
}
else if (response.ContentEncoding.ToLower().Contains("deflate"))
{
streamToRead = new DeflateStream(streamToRead, CompressionMode.Decompress);
}
using (StreamReader streamReader = new StreamReader(streamToRead, Encoding.UTF8))
{
return streamReader.ReadToEnd();
}
}
}
private bool Request_hatolna_co(out HttpWebResponse response)
{
response = null;
try
{
//Create a request to URL.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://MAGENTO.co/index.php/rest//V1/orders/items?searchCriteria[filter_groups][0][filters][0][field]=item_id&searchCriteria[filter_groups][0][filters][0][value]=1");
//Set request headers.
request.KeepAlive = true;
request.Headers.Set(HttpRequestHeader.Authorization, "Bearer xxxxxxxxxxxxxxxxxxxxxx");
request.Headers.Add("Postman-Token", #"1181fa03-4dda-ae84-fd31-9d6fbd035614");
request.Headers.Set(HttpRequestHeader.CacheControl, "no-cache");
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36";
request.ContentType = "application/json";
request.Accept = "*/*";
request.Headers.Set(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9,ar;q=0.8,la;q=0.7");
request.Headers.Set(HttpRequestHeader.Cookie, #"store=default; private_content_version=f16533d4f181d42a1b3f386fa6d2cdf1");
//Get response to request.
response = (HttpWebResponse)request.GetResponse();
}
catch (WebException e)
{
//ProtocolError indicates a valid HTTP response, but with a non-200 status code (e.g. 304 Not Modified, 404 Not Found)
if (e.Status == WebExceptionStatus.ProtocolError) response = (HttpWebResponse)e.Response;
else return false;
}
catch (Exception)
{
if (response != null) response.Close();
return false;
}
return true;
}
Why Postman-Token has been set in C# code? Remove it and then try.
The problem was in the URL, where Magento's (Server's) Admin Changed it to [HTTPS] instead of [HTTP].
That concludes the difference between [Postman, Insomnia, or any other API app] & the C# code, that C# doesn't handle the [HTTP vs HTTPs], while the API app can handle it.
I am doing an http request to the unknown server. When i make request from browser then it works like a charm but whenever i make the same request from my application then i get 503 Server unavailable.
I searched for this error and found that this is caused due to server temporary unavailable and another answered as the server believe that this call is made from a bot application.
I made same kind of request to the same server and there is only a small change in the uri. All previous calls are working except this one.
private static Root_BoardingStationList Request_BoardingStation(string uri, string auth,string greq, string sp_csr)
{
Root_BoardingStationList rootBoaringStationEnquiry = null;
string captchaImageURL = string.Empty;
string html = string.Empty;
try
{
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Referer = "https://www.irctc.co.in/nget/train-list";
request.Headers.Set("Authorization", "Bearer " + login_authorization_bearer);
request.Headers.Set("greq", greq);
request.Headers.Set("spa-csrf-token", sp_csr);
request.Proxy = null;
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.None;
request.CookieContainer = new CookieContainer();
request.Method = method;
request.ContentType = contentType;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
request.Connection = "keepalive";
using (var response_ = (HttpWebResponse)request.GetResponse())
{
var responseString = new StreamReader(response_.GetResponseStream()).ReadToEnd();
}
}
catch (WebException ex)
{
ExceptionLog.HandleException(ex);
using (var sr = new StreamReader(ex.Response.GetResponseStream()))
html = sr.ReadToEnd();
}
return rootBoaringStationEnquiry;
}
Please check and help.
I have found the solution for above as
csrf-token was mismatched.
I have corrected csrf-token and this works like a charm.
I was using
request.Headers.Set("spa-csrf-token", sp_csr);
and sent wrong sp_csr value to server.
I have this setup in my WebSecurityConfigurerAdapter to allow my client application to send POST request to the "/commands/" path on server:
#Override
protected void configure(HttpSecurity http) throws Exception {
http.authorizeRequests()
.antMatchers("/").permitAll()
.antMatchers("/commands/**").permitAll()
.antMatchers("/files/**").authenticated()
.and().
formLogin();
}
GET requests are fine,however the csrf seems be required for POST requests after this setup. I get following result if I don't login:
{
"timestamp": 1497904660159,
"status": 403,
"error": "Forbidden",
"message": "Could not verify the provided CSRF token because your session was not found.",
"path": "/commands/add"
}
If I login and attach the cookies from login request with C# client code, I will get following error:
{
"timestamp":1497897646380,
"status":403,
"error":"Forbidden",
"message":"Could not verify the provided CSRF token because your session was not found.",
"path":"/commands/add"
}
My C# code client for post looks like this:
public String SendJsonCommandByPost(String url, string data)
{
try
{
WebRequest req = HttpWebRequest.Create(url);
req.Proxy = null;
req.Method = "POST";
req.Timeout = TIMEOUT;
((HttpWebRequest)req).CookieContainer = myCookieContainer;
PrintCookies(myCookieContainer);
req.Headers.Add("X-CSRF-TOKEN", _csrftoken);
req.ContentType = "application/json";
((HttpWebRequest)req).UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
byte[] postdata = Encoding.UTF8.GetBytes(data);
req.ContentLength = postdata.Length;
Stream stream = req.GetRequestStream();
stream.Write(postdata, 0, postdata.Length);
stream.Flush();
stream.Close();
string source;
Console.WriteLine(req.Headers);
using (HttpWebResponse response = (HttpWebResponse)req.GetResponse())
{
using (StreamReader reader = new StreamReader(req.GetResponse().GetResponseStream()))
{
source = reader.ReadToEnd();
}
req.GetResponse().Close();
return source;
}
}
catch (Exception exp)
{
Console.WriteLine(exp);
if (exp is WebException)
{
var webexp = (WebException)exp;
Console.WriteLine(webexp.Response.Headers);
TextReader reader = new StreamReader(webexp.Response.GetResponseStream());
Console.WriteLine(reader.ReadToEnd());
}
return null;
}
}
May I know what could cause this kind of issue? Thank you!
add this line.
http.csrf().disable();
By default csrf is enabled so your post requests are getting blocked. Try this. It works for me