Requesting to github-api using .net HttpClient says Forbidden - c#

I am trying to getting data from this github api link https://api.github.com/users/arif2009 . It works fine using posman get request.
But if i try to get data using .net HttpClient then it says Forbidden.
C# code:
using (var client = new HttpClient())
{
client.BaseAddress = new Uri("https://api.github.com/");
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await client.GetAsync("users/arif2009");
if (response.IsSuccessStatusCode)
{
var data = response.Content.ReadAsAsync<GithubUser>();
}
}
Response:
Can anybody tell me where i made the mistake?

I took a look at your requests using Telerik's Fiddler and found the following.
Request with your code:
GET https://api.github.com/users/arif2009 HTTP/1.1
Accept: application/json
Host: api.github.com
Request from Postman:
GET https://api.github.com/users/arif2009 HTTP/1.1
Host: api.github.com
Connection: keep-alive
Accept: application/json
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
Postman-Token: d560ee28-b1e8-ece5-2612-87371ddcb295
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB,en;q=0.9,ja-JP;q=0.8,ja;q=0.7,en-US;q=0.6
The obvious missing header seemed to be "User-Agent", so I added this:
client.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue("product", "1")); // set your own values here
Which produced the following request:
GET https://api.github.com/users/arif2009 HTTP/1.1
Accept: application/json
User-Agent: product/1
Host: api.github.com
And returned the following response:
HTTP/1.1 200 OK
Date: Wed, 17 Jul 2019 15:19:35 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 1249
Server: GitHub.com
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 52
X-RateLimit-Reset: 1563380375
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: "1df3e0be6e824ca684f27963806533da"
Last-Modified: Tue, 16 Jul 2019 05:58:59 GMT
X-GitHub-Media-Type: github.v3
Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
Vary: Accept-Encoding
X-GitHub-Request-Id: D5E0:8D63:8BE92C:A799AE:5D2F3C86
{"login":"arif2009","id":6396346,"node_id":"MDQ6VXNlcjYzOTYzNDY=","avatar_url":"https://avatars0.githubusercontent.com/u/6396346?v=4","gravatar_id":"","url":"https://api.github.com/users/arif2009","html_url":"https://github.com/arif2009","followers_url":"https://api.github.com/users/arif2009/followers","following_url":"https://api.github.com/users/arif2009/following{/other_user}","gists_url":"https://api.github.com/users/arif2009/gists{/gist_id}","starred_url":"https://api.github.com/users/arif2009/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/arif2009/subscriptions","organizations_url":"https://api.github.com/users/arif2009/orgs","repos_url":"https://api.github.com/users/arif2009/repos","events_url":"https://api.github.com/users/arif2009/events{/privacy}","received_events_url":"https://api.github.com/users/arif2009/received_events","type":"User","site_admin":false,"name":"Arif","company":"#BrainStation-23 ","blog":"https://arif2009.github.io/","location":"Bangladesh","email":null,"hireable":true,"bio":"Software Engineer | Full Stack | Web Developer | Technical Writer","public_repos":15,"public_gists":2,"followers":9,"following":7,"created_at":"2014-01-14T05:03:47Z","updated_at":"2019-07-16T05:58:59Z"}

this API reference from github says that 'User-Agent' is a required header:
All API requests MUST include a valid User-Agent header. Requests with no User-Agent header will be rejected.
Postman automatically adds its own User-Agent to the call when this is not provided by the user. (This is nicely demonstrated by #John's answer).
simply adding this header will resolve your issue:
client.DefaultRequestHeaders.UserAgent.Add(new ProductInfoHeaderValue("yourAppName", "yourVersionNumber"));

You may try to put the complete URI into the GetAsync instead using the BaseAddress, as you do not give the GetAsync an URI, but only a string.
When sending a HttpRequestMessage with a relative Uri, the message Uri will be added to the BaseAddress property to create an absolute Uri.
https://learn.microsoft.com/de-de/dotnet/api/system.net.http.httpclient.baseaddress?view=netframework-4.8

Related

Getting Microsoft.Identity (former AzureAD) to work with Swagger

I have an ASP.Net 5 Web API which is secured using The Microsoft.identity.Web package, so it is backed by Azure Active Directory.
The authentication in the API itself is working fine and without any problems.
I'm struggling when I want to get the authorization to work inside Swagger UI.
I'm using the Authorization code flow and everything seems fine at first (I get to the Microsoft login screen, can enter my credentials and receive an authorization code).
However after Swagger UI got the authorization code, it calls the token endpoint at https://login.microsoftonline.com/organizations/oauth2/v2.0/token.
The response from that call is 99% percent fine, except that it is missing the Allow-Origin-Header so the response is blocked by the browser itself and cannot reach the Swagger UI JavaScript which would then set the token it received from that response.
What am I missing here to get that header in the response?
This is the code in my Startup.cs
services.AddSwaggerGen(c =>
{
c.AddSecurityDefinition("msid", new Microsoft.OpenApi.Models.OpenApiSecurityScheme
{
Type = Microsoft.OpenApi.Models.SecuritySchemeType.OAuth2,
Flows = new Microsoft.OpenApi.Models.OpenApiOAuthFlows
{
AuthorizationCode = new Microsoft.OpenApi.Models.OpenApiOAuthFlow
{
AuthorizationUrl = new System.Uri("https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize"),
TokenUrl = new System.Uri("https://login.microsoftonline.com/organizations/oauth2/v2.0/token"),
Scopes = new Dictionary<string, string>
{
{ "api://myClientId/access", "access" }
}
}
}
});
c.AddSecurityRequirement(new Microsoft.OpenApi.Models.OpenApiSecurityRequirement
{
{
new Microsoft.OpenApi.Models.OpenApiSecurityScheme
{
Reference = new Microsoft.OpenApi.Models.OpenApiReference {Type = Microsoft.OpenApi.Models.ReferenceType.SecurityScheme, Id = "msid" }
},
new [] { "api://myClientId/access" }
}
});
});
This is the request which is sent from Swagger UI to https://login.microsoftonline.com/organizations/oauth2/v2.0/token
POST https://login.microsoftonline.com/organizations/oauth2/v2.0/token HTTP/1.1
Host: login.microsoftonline.com
Connection: keep-alive
Content-Length: 1086
Pragma: no-cache
Cache-Control: no-cache
sec-ch-ua: "Chromium";v="94", "Microsoft Edge";v="94", ";Not A Brand";v="99"
Accept: application/json, text/plain, */*
Content-Type: application/x-www-form-urlencoded
X-Requested-With: XMLHttpRequest
sec-ch-ua-mobile: ?0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4585.0 Safari/537.36 Edg/94.0.972.0
sec-ch-ua-platform: "Windows"
Origin: https://localhost:5003
Sec-Fetch-Site: cross-site
Sec-Fetch-Mode: cors
Sec-Fetch-Dest: empty
Referer: https://localhost:5003/
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.9,en;q=0.8,en-US;q=0.7
grant_type=authorization_code&code=hereIsMyLongAuthorizationCode&redirect_uri=https%3A%2F%2Flocalhost%3A5003%2Fswagger%2Foauth2-redirect.html
This is the response
HTTP/1.1 200 OK
Cache-Control: no-store, no-cache
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
x-ms-request-id: 683dc687-7211-400b-ab02-bccdc6e9ba00
x-ms-ests-server: 2.1.11898.12 - WEULR1 ProdSlices
report-to: {"group":"network-errors","max_age":86400,"endpoints":[{"url":"https://identity.nel.measure.office.net/api/report?catId=GW+estsfd+dub2"}]}
nel: {"report_to":"network-errors","max_age":86400,"success_fraction":0.001,"failure_fraction":1.0}
Set-Cookie: fpc=...; expires=Fri, 03-Sep-2021 13:57:11 GMT; path=/; secure; HttpOnly; SameSite=None
Set-Cookie: x-ms-gateway-slice=estsfd; path=/; secure; samesite=none; httponly
Set-Cookie: stsservicecookie=estsfd; path=/; secure; samesite=none; httponly
Referrer-Policy: strict-origin-when-cross-origin
Date: Wed, 04 Aug 2021 13:57:10 GMT
Content-Length: 1763
{"token_type":"Bearer","scope":"api://myClientId/access","expires_in":3599,"ext_expires_in":3599,"access_token":"theToken"}
The problem was that I was using the AuthorizationCode-Flow which is only suitable for backend applications, because the client secret needs to be transmitted there.
The correct way was to use the Implicit-Flow while keeping everything else the same. That flow is intended for JS applications where it is impossible to securely send a client secret without the user being able to see it.

Angular 405 Method not allowed and "Provisional Headers are shown" error

This is driving me insane. I am trying to send a simple request to my api server and appending the header. It is first giving me this Provisional Headers are shown message then it will fail on the next request
Provisional headers are shown
Accept: application/json, text/plain, */*
Access-Control-Allow-Credentials: true
Cache-Control: no-cache
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Pragma: no-cache
Referer: https://mywebsite.com/
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.106 Safari/537.36
If I add the below data in my intercept method then I get the error.
intercept(request: HttpRequest<any>, next: HttpHandler): Observable<HttpEvent<any>> {
request = request.clone({ headers: request.headers.set('Access-Control-Allow-Credentials', 'true') });
request = request.clone({ headers: request.headers.set('Cache-Control', 'no-cache') });
request = request.clone({ headers: request.headers.set('Pragma', 'no-cache') });
request = request.clone({ headers: request.headers.set('Expires', 'Sat, 01 Jan 2000 00:00:00 GMT') });
return next.handle(request);
}
If I remove it and do not add the headers then I don't get an error but eventually I will need to send other data in the header like an authtoken and user id.
Why am I getting this 405 error when appending to the headers?
Also why is it telling me method not allowed when I clearly am allowing Options and Post methods
I had this same problem about a month ago.
You need to create a basic http ok response for the Options verb. Put this in your base api controller. The options request is sent first and it needs an ok response callback before it can proceed with passing Options to the API
public HttpResponseMessage Options()
{
return new HttpResponseMessage { StatusCode = HttpStatusCode.OK };
}
Try add
API Key in header (e.g.: x-my-api-key)
API Value in header (e.g.: 7e807890-c777-466a-897)
Hope this could help...

Google Drive: Error in retrieving access and refresh tokens

I am trying to access google drive from my app for WP7. But when i try to get access token in exchange for Authorization code, I get BAD REQUEST from server.
My POST request as seen in Fidler:
POST https://accounts.google.com/o/oauth2/token HTTP/1.1
Accept: */*
Referer: file:///Applications/Install/7128457C-3AF4-41C4-A606-742068B1463F/Install/
Content-Length: 240
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
User-Agent: NativeHost
Host: accounts.google.com
Connection: Keep-Alive
Cache-Control: no-cache
code=<*Authorization_Code*>&
client_id=<*My_Client_Id*>&
client_secret=<*My_Client_Secret*>&
redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&
grant_type=authorization_code
Response from server:
HTTP/1.1 400 Bad Request
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Sat, 07 Sep 2013 14:05:35 GMT
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Alternate-Protocol: 443:quic
Transfer-Encoding: chunked
21
{
"error" : "invalid_request"
}
0
My Code:
StringBuilder postData = new StringBuilder();
postData.AppendFormat("{0}={1}", "code", HttpUtility.UrlEncode(AuthorizationCode));
postData.AppendFormat("&\n{0}={1}", "client_id", HttpUtility.UrlEncode(ClientId));
postData.AppendFormat("&\n{0}={1}", "client_secret", HttpUtility.UrlEncode(ClientSecret));
postData.AppendFormat("&\n{0}={1}", "redirect_uri", HttpUtility.UrlEncode("urn:ietf:wg:oauth:2.0:oob"));
postData.AppendFormat("&\n{0}={1}", "grant_type", HttpUtility.UrlEncode("authorization_code"));
WebClient client = new WebClient();
client.UploadStringCompleted += TokenResponse;
client.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
client.UploadStringAsync(new Uri("https://accounts.google.com/o/oauth2/token",UriKind.Absolute), "POST", postData.ToString());
I get this result both on emulator as well as Lumia 820. I also tried without using HttpUtility in POST request but didn't work. Any help?
Its likely due to the fact that you are adding a new line via \n between all the param/value pairs.
I do it without that without it and it works - https://github.com/entaq/GoogleAppsScript/blob/master/IO2013/YouTubeAnalytics/oauth2.gs#L25

Does it matter that a servicestack.net OPTIONS request returns a 404?

I'm using the method described at https://github.com/ServiceStack/ServiceStack/wiki/New-Api to enable CORS. This seems to work, i.e. I get the correct Access-Control headers back in the response, but the status code is a 404. Does this matter?
For completeness I'd prefer to return a non-error code. Reading http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html it seems as though I ought to be returning a 200. What's the best way to do this?
The service method definition is:-
[EnableCors]
public void Options(ApiKeyRequest apiKeyRequest)
{
}
The HTTP request is:-
OPTIONS http://myreallycoolwebapi/apikeyrequests HTTP/1.1
Host: myreallycoolwebapi
Connection: keep-alive
Access-Control-Request-Method: POST
Origin: http://localhost:8000
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11
Access-Control-Request-Headers: origin, content-type, accept
Accept: */*
Referer: http://localhost:8000/index.html
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
and the response is:-
HTTP/1.1 404 Not Found
Server: nginx
Date: Sat, 17 Nov 2012 18:06:01 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 3
Connection: keep-alive
Cache-Control: private
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Headers: Content-Type
404
UPDATE 1
This is my resource type.
[Route("/apikeyrequests", "POST")]
public class ApiKeyRequest : IReturn<ApiKeyRequest>
{
public string EmailAddress { get; set; }
}
The POST requests work just fine.
I still haven't seen anything matching the expected route /apikeyrequests.
Do you have a custom route definition, e.g:
[Route("/apikeyrequests")]
public class ApiKeyRequest {}
Added for the request?
The CORS spec itself explicitly indicates that a non-200 preflight response is an error. See the "If the response has an HTTP status code that is not 200" section here:
http://www.w3.org/TR/cors/#cross-origin-request-with-preflight-0
I don't know what the motivation is for this requirement. Perhaps its because a 404 could introduce confusion as to whether the endpoint actually exists.

Parsing from web - script changes source code content?

I have been trying to parse a php generated web page(not site) for some time. I tried parsing using xpath through HTMLAgility in C# as well as PHP. At first I thought I was not parsing correctly due to incorrect values.
Later, I found that actually I am parsing it correctly. But there is a script in that page which is changing the value when loading. how, I don't know.
I am new to parsing, so here is what happening according to me:
I am downloading the source code of the content. The part I want to parse is somewhat like this:
<b id="solved_b">0</b>
When the page loads, the script in the source code changes the value to something other than 0.
When I parse using xpath, the original value, i.e. 0 is parsed, instead of the script changed value.
So, how can I parse the changed value instead of the original one?
the page I am trying to parse is
http://felix-halim.net/uva/hunting.php?id=59756
here is the snippet in HTMLAgility:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using HtmlAgilityPack;
namespace ParseFelix {
class WebParser {
string webUrl;
public WebParser(string url) {
webUrl = "http://felix-halim.net/uva/hunting.php?id=59756";
HtmlWeb htmlWeb = new HtmlWeb();
HtmlDocument htmldoc = htmlWeb.Load(webUrl);
var username = htmldoc.DocumentNode.SelectSingleNode("/html/body/div/h2/i");
var submittedStats = htmldoc.DocumentNode.SelectSingleNode(".//*[#id=\"submissions_b\"]");
string content = htmldoc.DocumentNode.InnerHtml;
//System.IO.File.WriteAllText("D:\\exp\\felix\\parsed.txt", content);
var acceptedStats = htmldoc.DocumentNode.SelectSingleNode(".//*[#id=\"solved_b\"]");
Console.WriteLine("Username is {0}, you submitted {1} solutions, and {2} were accepted", username.InnerText, submittedStats.InnerText, acceptedStats.InnerText);
}
}
}
Well what you are trying to do is parse javascript right?
And that is AFAIK not possible (aside from writing your own parser or using an existing one).
What you want is to read the manipulated DOM, and that is not trivial at all
Use Fiddler - you will see that site makes ajax queries and some values gets from json:
POST http://felix-halim.net/uva/service2.php HTTP/1.1
Host: felix-halim.net
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:2.0) Gecko/20100101 Firefox/4.0
Accept: */*
Accept-Language: lt
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Proxy-Connection: keep-alive
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://felix-halim.net/uva/hunting.php?id=59756
Content-Length: 73
Cookie: PHPSESSID=o6if4t4vqadv7ia6vbqcfcvi75
Pragma: no-cache
Cache-Control: no-cache
{"method":"uva2.chat_update","params":[12150,59756,"guest",3127,8744317]}
And response:
HTTP/1.0 200 OK
Date: Fri, 15 Apr 2011 05:32:08 GMT
Server: Apache
X-Powered-By: PHP/5.2.15
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html
Content-Length: 2069
[null,[["0","guest","1302842169"],["0","guest","1302793161"]<SKIPPED>

Categories