I need to call a web page from different domain. When I call this page from browser, it responds normally. But when i call it from a server side code or from jquery ajax script, it responds empty xml.
I am trying to call a page or service like this:
http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801
this responds normally from browser. But when I write a c# code like this:
WebClient wc = new WebClient();
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
wc.Headers[HttpRequestHeader.Accept] = "*/*";
wc.Headers[HttpRequestHeader.AcceptCharset] = "ISO-8859-1,utf-8;q=0.7,*;q=0.3";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip,deflate,sdch";
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8";
wc.Headers[HttpRequestHeader.Host] = "otherdomain.com";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
i get empty xml as response:
<xml></xml>
How can I get same response from server side code or client side which I got from browser?
I tried solution here: Calling Cross Domain WCF service using Jquery
So that I didnt understand what to do, I couldnt apply solution described.
How can I get same response from server side code or client side which I got from browser?
Due to the same origin policy restriction you cannot send cross domain AJAX requests from browsers.
From .NET on the other hand you could perfectly fine send this request. But probably the web server that you are trying to send the request to expects some HTTP headers such as the User-Agent header for example. So make sure that you have provided all the headers in your request that the server needs. For example to add the User-Agent header:
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5";
var response = wc.DownloadString("http://www.otherdomain.com/oddsData.jsp?odds_flash_id=11&odds_s_type=1&odds_league=all&odds_period=all&me_select_string=&q=93801");
Response.Write(response);
}
You could use FireBug or Chrome developer toolbar to inspect all the HTTP request headers that your browser sends along the request that works and simply add those headers.
Related
I'm trying to obtain a JSON via a rest API using, targeting .Net 4.5
I've tried various methods in code, but the all end up in me getting:
"Authentication failed because the remote party has closed the
transportstream" .
the exact same URL works via browser and Postman.
So far, I've tried using .Net's WebClient, HttpClient and HttpWebRequest with identical results. I've tried comparing requests between Postman and my code (via RequestBin), but even when they were identical, I still kept getting back:
Authentication failed because the remote party has closed the
transport
My current code is using HttpWebRequest, but every solution will do.
I've played around with all of the security protocols, some of them will cause the API to return 404 and some will cause the server to return
"Authentication failed because the remote party has closed the
transport stream".
Here's my current code:
public string GetCityStreets()
{
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var request = (HttpWebRequest)WebRequest.Create(url);
ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 |
SecurityProtocolType.Tls12 |
SecurityProtocolType.Tls11 |
SecurityProtocolType.Tls;
var response = (HttpWebResponse)request.GetResponse();
string jsonResponse;
using (var reader = new StreamReader(response.GetResponseStream()))
{
jsonResponse = reader.ReadToEnd();
}
return jsonResponse;
}
In my current code, the exception is thrown when the request is actually made: request.GetResponse().
What I need, essentially, is to get the JSON from the API.
Set SecurityProtocolType.Tls12 before you initalize the request:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12`
var request = WebRequest.CreateHttp(url);
If you're on Windows 7. On Windows 10, you should only need SecurityProtocolType.SystemDefault.
Note: To enable TLS1.3 (it's available in both Windows 7 and Windows 10), If you don't use .Net 4.8 or .Net Core 3.0, since there's no enumerator for it, you can set it with:
var secProtoTls13 = (SecurityProtocolType)12288;
Remove all the other SecurityProtocolType you have set there.
Setting the User-Agent header is also mandatory, otherwise you will receive a 404 (not found). Here's the FireFox header:
request.UserAgent = "Mozilla/5.0 (Windows NT 10; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0";
A note on the User-Agent header: this specific site doesn't activate HTTP Strict Transport Security (HSTS). But some sites do, when they see that the WebBrowser supports it. HttpWebRequest doesn't understand it, so it will simply wait for a response that never comes, since the Site is waiting for interaction.
You may want to use the IE11 header instead.
Also add this other header:
request.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
The server side appears to be checking the user agent (presumably to stop bots and other code (like yours!) from hitting the endpoint). To bypass this, you will need to set the user agent to a value such that it thinks you are a web browser.
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36";
works, as an example.
You may wish to consider setting the ServicePointManager.SecurityProtocol just once (at app startup) rather than on each request.
This works using HttpClient (.net 4.5 and up)
var url = "https://data.gov.il/api/action/datastore_search?resource_id=a7296d1a-f8c9-4b70-96c2-6ebb4352f8e3&q=26";
var client = new HttpClient();
client.DefaultRequestHeaders.Add("User-Agent", "C# App");
Task<string> response = client.GetStringAsync(url);
Console.WriteLine(response.Result);
Think the server requires a user agent.
One question that's been confusing me and could really do with some insight.
I need to retreive Json objects from a http service. When I tested this in a Console Window, I kept receiving a "Internal Server Error : 500" until I set the UserAgent property for the WebClient object.
Example:
WebClient client = new WebClient();
client.Headers.Add("user-agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36");
content = client.DownloadString(url);
Now, if I need to do the same for a WP8.1 app, how would I detect (if I need to in the first place?) the UserAgent (and set it) and be able to retrieve the data?
Thank you all.
Windows Phone 8.1 App will use HttpClient. By default there will not be a user agent set. The default user-agent for the phones web browser is:
"Mozilla/5.0 (Mobile; Windows Phone 8.1; Android 4.0; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 520) like iPhone OS 7_0_3 Mac OS X AppleWebKit/537 (KHTML, like Gecko) Mobile Safari/537"
You can manually set the user-agent on the HttpRequestMessage.Headers.UserAgent property.
References:
HttpClient
https://msdn.microsoft.com/en-us/library/windows/apps/xaml/windows.web.http.headers.httprequestheadercollection.aspx
User-Agent
https://msdn.microsoft.com/en-us/library/ie/hh869301(v=vs.85).aspx#ie11\
The class libraries for using http do not add any User Agents by default. See these lines from the msdn page:
By default, no user-agent header is sent with the HTTP request to the web service by the HttpClient object. Some HTTP servers, including some Microsoft web servers, require that a user-agent header be included with the HTTP request sent from the client. The user-agent header is used by the HTTP server to determine how to format some HTTP pages so they render better on the client for different web browsers and form factors (mobile phones, for example). Some HTTP servers return an error if no user-agent header is present on the client request. We need to add a user-agent header to avoid these errors using classes in the Windows.Web.Http.Headers namespace. We add this header to the HttpClient.DefaultRequestHeaders property.
For more details, refer the link below:
How to connect to an HTTP server using Windows.Web.Http.HttpClient (XAML)
Also look at the answer below (by Bret Bentzinger) for the exact user agent string.
I am writing a C# application whereby I formulate the POST strings in C# but the website I am POSTing recognizes that I am not using IE, Chrome or Firefox. Is there a way that I can "use" Internet Explorer (or either of the other two browsers) to make the POST request and then retrieve the response back in the C# (to parse the HTML)?
I have this currently:
using (var wb = new WebClient())
{
var data = new NameValueCollection();
//Any key-value arguments for the POST are stored in data
var response = wb.UploadValues(url, "POST", data);
}
Yes. Forge the User-Agent HTTP header. The User-Agent header basically tells the receiving server what program is sending the packets to it.
See this StackOverflow answer on how to do just that.
client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
place after response
also
Other Response
I have some code (in a Winform app) that reads this URL using HttpWebRequest.GetResponse().
For some reason, it recently starts returning 500 Internal Error when requested from my app.
(The response contains some HTML for the navigations, but doesn't have the main content I need)
On Firefox/Chrome/IE, it is still returning 200 OK.
The problem is I don't have control over their code, I don't know what it does on the backend that causes it to break when requested from my app.
Is there a way I can "pretend" to make the request from, say, Google Chrome? (just to avoid the error)
Set the HttpWebRequest.UserAgent property to the value of a real browser's user agent.
HttpWebRequest webRequest = (HttpWebRequest) WebRequest.Create("http://example.com");
webRequest.UserAgent = #"Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36";
how to set the http headers when sending http request to a server so that the server thinks the requests are coming from firefox
One needs to alter the HTTP Header's User-Agent value, to something that looks like the following:
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
In C#/ASP.NET (assuming this is follow-up on previous question)
using (WebClient myBrowser= new WebClient())
{
myBrowser.Headers["User-Agent"] = some_string_like_the_one_above;
// Download data.
byte[] httpResp = myBrowser.DownloadData(some_url);
// Here to exploit the data returned from the server
}