I'm querying Wikipedia using the following code, but I always get an error (403 forbidden). When I type the exact same url in my browser, however, it works. I've been using the same code before to query other web apis, so I am not sure what's causing the trouble.
private static string query(string text)
{
text = text.Replace(" ", "%20");
string url = "http://en.wikipedia.org/w/api.php?action=opensearch&search=" + text + "&format=json&callback=spellcheck";
WebClient client = new WebClient();
client.Headers.Add("User-Agent", "whatever"); // <-- this line was missing
try
{
string response = client.DownloadString(url);
return response;
}
catch(Exception e)
{
Console.WriteLine(e.Message);
return null;
}
}
Try setting the user agent header to something that matches your browser. If this doesn't work, fire up Fiddler, take a peek at your browser headers and copy them to your web request.
http://msdn.microsoft.com/en-us/library/system.net.webclient.headers.aspx
EDIT
The advice I gave was generic. Please observe the policies of the website you are downloading from, as spoofing a browser user-agent may contravene policy or be considered malicious by default:
http://meta.wikimedia.org/wiki/User-Agent_policy :
Do not copy a browser's user agent for your bot, as bot-like behavior with a browser's user agent will be assumed malicious.
Related
Hey guys,
I have a problem with my code. Since about a week my code is not working anymore without any changes. I am pretty sure, that my could should work. All I get is Error 404: forbidden.
Below is a snippet of my Code. I also read about adding a header of the webclient, which did not help. Any other suggestions? I am sorry if my syntax is not that good, it is my first post on stackoverflow.
Thanks in advance!
string epicId = "ManuelNotManni";
WebClient webClient = new WebClient();
Uri uri = new Uri("https://api.tracker.gg/api/v2/rocket-league/standard/profile/epic/");
string result = String.Empty;
try
{
string website = $"{uri.ToString()}{epicId}?";
result = webClient.DownloadString(website);
}
catch (Exception ex)
{
Console.WriteLine($"Error:\n{ex}");
Console.ReadLine();
}
finally
{
webClient.Dispose();
}
This is the exact error:
System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponse()
at System.Net.WebClient.GetWebResponse(WebRequest request)
at System.Net.WebClient.DownloadBits(WebRequest request, Stream writeStream)
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadString(Uri address)
at System.Net.WebClient.DownloadString(String address)
at TestProject.Program.Main(String[] args) in > C:\Users\Manue\source\repos\TestProject\Program.cs:line 17
You're right. Your code should work fine.
Issue is that URL you're requesting which is actually:
https://api.tracker.gg/api/v2/rocket-league/standard/profile/epic/ManuelNotManni?
This returns a 403 status code in any case - no matter if you use a browser, your code or for example postman.
I suggest to have a look at the response body while using postman.
It shows this
<html class="no-js" lang="en-US">
<!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta name="captcha-bypass" id="captcha-bypass" />
Tracker.gg wants API users to register their apps with them before they're given access to the API.
What you need to do is to first head to their Getting Started page. Here you will have to create an app, which should give you an authentication key.
When you have done this, you want to change your code slightly to add the Authentication Header. Like so for example:
var webClient = new WebClient();
webclient.Headers.Add("TRN-Api-Key", "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX")
As a sidenote, WebClient has been deprecated and it's recommended to use HttpClient from now on. Here's your code with HttpClient instead:
var epicId = "ManuelNotManni";
var httpClient = new HttpClient();
httpClient.DefaultRequestHeaders.Add("TRN-Api-Key", "YOUR API KEY GOES HERE");
// Simplifying Uri creation:
var uri = new Uri($"https://api.tracker.gg/api/v2/rocket-league/standard/profile/epic/{epicId}");
var result = string.Empty; // C# prefers lowercase string
try
{
var response = await httpClient.GetAsync(uri);
if (response.IsSuccessStatusCode)
{
result = await response.Content.ReadAsStringAsync();
}
else
{
Console.WriteLine($"Unable to retrieve data for {epicId}.");
Console.WriteLine($"Statuscode: {response.StatusCode}");
Console.WriteLine($"Reason: {response.ReasonPhrase}");
}
}
catch (Exception ex)
{
Console.WriteLine($"Error:\n{ex}");
Console.ReadLine();
}
finally
{
httpClient.Dispose();
}
This happens when we violate the Firewall rule set by Cloudflare, you can visit this blog for more details.
https://community.cloudflare.com/t/community-tip-fixing-error-1020-access-denied/66439
I use an javascript picker to get file from my google drive, it's work well and i get then download url and acces_token from my drive
I would like to download bytes array from this file from my server, then i path the url and acces_token to it (with ajax), no problem
on server i would get data with this code (it's worked !!!)
public void DownloadFile(string url, string AccessToken)
{
try
{
byte[] BB;
using (WebClient wc = new WebClient())
{
wc.Headers.Add("Authorization", "Bearer " + AccessToken);
BB = wc.DownloadData(url);
}
}
catch (Exception ex)
{
throw ex;
}
}
now i get an 403 error ??
url are like "https://content.googleapis.com/drive/v2/files.....?key=mykey....."
what's changed on google server ?
thanks
If you want to download a file using the Drive API and setting the access token in the header, you have to do it by calling this url:
https://www.googleapis.com/drive/v2/files/[file-Id]?alt=media
As it's told in the Response section in the Files: get endpoint. It's really important the url parameter alt=media in order to make it work. Don't use an API Key.
I don't want to display the 404 message to the user, I want the message to be displayed in the normal page instead.
public ActionResult Download(string fileName, string filePath){
try{
filePath = (filePath != null && filePath != "") ? Cryptography.DecryptAESFromBase64(filePath) : filePath;
if (!filePath.Contains(fileName))
{
var fs = System.IO.File.OpenRead(Server.MapPath(filePath + "_" + fileName));
return File(fs, "application/octet-stream", fileName);
}
else
{
byte[] data;
this._documentService.DownloadAmazonObject(filePath, out data);
return File(data, "application/octet-stream", fileName);
}
}
catch{
throw new HttpException("Couldn't find ", ex);
}
}
This is the code that I have, I keep having the throw new HttpException display in the yellow page.
This answer depends a lot on your whole setup. Whats happening is basically the following: The browser sends a request to your HTTP server. It might be part of your program itself or an application server which in turn hands the request over to your application after transforming the HTTP request into objects and a function call. This is called binding or deserialization.
The application works with the objects (strings in your case) and throws an Exception which the application server in turn transforms into a Response with the HttpStatus 404.
Depending on your setup you have to tell your web server resp. application server how to handle erroneous status codes by explicitly defining an error page.
The dirty alternative would be to catch the exception and return the message as part of a valid response. But that basically bypasses the way errors are handled in HTTP so I would very much discourage it.
Hope this helps a bit :-)
I'm using below script to retrieve HTML from an URL.
string webURL = #"https://nl.wiktionary.org/wiki/" + word.ToLower();
using (WebClient client = new WebClient())
{
string htmlCode = client.DownloadString(webURL);
}
The variable word can be any word. In case there is no WIKI page for the "word" be retrieved the code is ending in error with code 404, while retrievng the URL with a browser opens a WIKI page, saying there is no page for this item yet.
What I want is that the code always gets the HTML, also when the WIKI page says there is no info yet. I do not want to avoid the error 404 with a try and catch.
Does anyone has an idea why this is not working with a Webclient?
try this. You can catch the 404 error content in a try catch block.
var word = Console.ReadLine();
string webURL = #"https://nl.wiktionary.org/wiki/" + word.ToLower();
using (WebClient client = new WebClient() { })
{
try
{
string htmlCode = client.DownloadString(webURL);
}
catch (WebException exception)
{
string responseText=string.Empty;
var responseStream = exception.Response?.GetResponseStream();
if (responseStream != null)
{
using (var reader = new StreamReader(responseStream))
{
responseText = reader.ReadToEnd();
}
}
Console.WriteLine(responseText);
}
}
Console.ReadLine();
Since this WIKI-server use case-sensitive url mapping, just don't modify case of URL to harvest (remove ".ToLower()" from you code).
Ex.:
Lower case:
https://nl.wiktionary.org/wiki/categorie:onderwerpen_in_het_nynorsk
Result: HTTP 404(Not Found)
Normal (unmodified) case:
https://nl.wiktionary.org/wiki/Categorie:Onderwerpen_in_het_Nynorsk
Result: HTTP 200(OK)
Also, keep in mind what most (if not all) WiKi servers (including this one) generates custom 404 pages, so in browser they looks like "normal" pages, but, despite this, they are serving with 404 http code.
Bounty Question
I am using c# 3.5 Window Forms Application. I am using the code mentioned in the accepted answer. and I am getting below error
The remote server returned an error: (401) Unauthorized.
Sample code to verify the UserName and Password will be really appreciated
Bounty Question Ends
I have an application with the following use-case: when the user first starts using the application, he inputs his username and password. Then, at a much later stage, the application may update his status.
Currently I'm using Twitterizer, but I believe the question is beyond the scope of the specific library I'm using. Following are the two relevant lines of code:
Twitter twitter = new Twitter("username", "password", "source");
twitter.Status.Update("update");
The construction of the Twitter object does not throw an exception if the username/password are incorrect. This is probably because nothing is sent at this point. On the other hand, the status update does throw an exception if the username/password are invalid.
My problem is that I want to validate the username/password at the point of user input, not when trying to post the update.
How can I validate the username/password without posting anything (in Twitterizer or otherwise)?
Taking a quick look at the verify_credentials API as mentioned by peSHIr, I wrote a little routine which seems to do the trick. It's late, but I was able to test it a couple of times and seems to work.
In my function, I am just returning true if I I get an HttpResponseCode.OK, and false if I get anything else or an exception is thrown. If twitter does not like the uid/password an exception will be thrown with a 401 error (not authorized.)
public bool CheckTwitterCredentials(string UserName, string Password)
{
// Assume failure
bool Result = false;
// A try except block to handle any exceptions
try {
// Encode the user name with password
string UserPass = Convert.ToBase64String(
System.Text.Encoding.UTF8.GetBytes(UserName + ":" + Password));
// Create our HTTP web request object
HttpWebRequest Request =
(HttpWebRequest)WebRequest.Create("http://twitter.com/account/verify_credentials.xml");
// Set up our request flags and submit type
Request.Method = "GET";
Request.ContentType = "application/x-www-form-urlencoded";
// Add the authorization header with the encoded user name and password
Request.Headers.Add("Authorization", "Basic " + UserPass);
// Use an HttpWebResponse object to handle the response from Twitter
HttpWebResponse WebResponse = (HttpWebResponse)Request.GetResponse();
// Success if we get an OK response
Result = WebResponse.StatusCode == HttpStatusCode.OK;
} catch (Exception Ex) {
System.Diagnostics.Debug.WriteLine("Error: " + Ex.Message);
}
// Return success/failure
return Result;
}
You could try to use the API call account/verify_credentials. Hopefully the API library you use already supports this.. Twitter is notorious now for really hating third party programmers, so unless you have good reason to do something with Twitter, just stay away...
I have used other Twitter Libraries but none of them support checking the username and password for validitity. This might be because Twitter API does not have the facility to validate the username and password unless we try to do something which requires authentication.
One thing you can do is try to get friend list or any other methods that requires authentication.
Twitter API hasn't supported username/password in years. Instead, you have OAuth, which lets the user authorize your application to act on their behalf. Twitter has an account/verify_credentials endpoint you can use to verify whether the user who's tokens you have still authorizes your app. Here's an example of how you could call this endpoint with LINQ to Twitter:
var accounts =
from acct in twitterCtx.Account
where acct.Type == AccountType.VerifyCredentials
select acct;
You can visit Account/VerifyCredentials documentation for more details:
as #joe-mayo informed, you have to switch to OAuth. twitter expired v1 of their API and they documented that in following url https://dev.twitter.com/docs/faq#17750.
Here's a function i wrote that will verify twitter username and password in C# :
public bool isTwitterValid(string username, string password)
{
try
{
string user = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(username + ":" + password));
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://twitter.com/statuses/verify.xml");
request.Method = "POST";
request.ServicePoint.Expect100Continue = false;
request.Headers.Add("Authorization", "Basic " + user);
request.ContentType = "application/x-www-form-urlencoded";
WebResponse response = request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());
string responseString = reader.ReadToEnd();
reader.Close();
}
catch (Exception ex)
{
if (ex.Message.Contains("404")) { return true; }
}
return false;
}