Trying to read web page with HttpClient - c#

I try to read some web pages with HttpClient (C#) and with some pages I get Response status code does not indicate success: 404 (No such file)? Other work just fine.
Here is the sample of code:
static readonly HttpClient client = new HttpClient();
private static async void GetHtmlAsync()
{
var url = "https://www.yourwebpage.com";
try
{
HttpResponseMessage response = await client.GetAsync(url);
response.EnsureSuccessStatusCode();
string responseBody = await response.Content.ReadAsStringAsync();
Console.WriteLine(responseBody);
}
catch (HttpRequestException e)
{
Console.WriteLine("\nException Caught!");
Console.WriteLine("Message :{0} ", e.Message);
}
}
also try to add Header to client, with no success:
client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0");
client.DefaultRequestHeaders.Add("Accept", text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
client.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Connection", "keep-alive");
client.DefaultRequestHeaders.Add("Host", "www.betexplorer.com");
client.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");
Solution:
I fix this problem with using RestSharp library.


Here is the solution.
I used RestSharp library and it work perfectly.
private static async void GetHtmlAsync()
{
var url = "https://www.yourwebpage.com";
try
{
var restClient = new RestClient(url);
var request = new RestRequest("", Method.GET);
restClient.ExecuteAsync(request, response =>
{
Console.WriteLine(response.Content);
});
}
catch (HttpRequestException e)
{
Console.WriteLine("\nException Caught!");
Console.WriteLine("Message :{0} ", e.Message);
}
}


You have to pretend to be a browser.
For example, add the following header
client.DefaultRequestHeaders.Add("User-Agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0");

Related

API error 401 Unauthorized laravel C Sharp

I am developing an application using C # that calls a RESTApi de laravel. Through Postman it works correctly, but instead of C# no, it returns error 401.
If I remove the following if in the Laravel controller it works fine:
if ($ request-> isJson ()) {
The header Content-Type is set to application/json
Laravel Code
function getResult(Request $request, $id)
{
if ($request->isJson()) {
// Eloquent
$times = Result::selectRaw('THE SELECT')
->where('ID', $Id)
->get();
$result = [];
foreach($times as $key => $time)
{
...........
}
sort($result);
return response()->json(['results'=>$result], 200);
}
return response()->json(['error' => 'Unauthorized'], 401, []);
}
C#
public static async Task<dynamic> GETTimes(int eventID, int stageID)
{
using (var client = new HttpClient())
{
client.BaseAddress = new Uri(baseUrl);
client.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json");
client.DefaultRequestHeaders.TryAddWithoutValidation("cache-control", "no-cache");
client.DefaultRequestHeaders.Add("User-Agent", #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.106 Safari/537.36");
// Add the Authorization header with the AccessToken.
//client.DefaultRequestHeaders.Add("Authorization", "Bearer " + accessToken);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
// create the URL string.
string url = string.Format("api/v1/events/{0}/results/{1}", eventID, stageID);
// make the request
HttpResponseMessage response = await client.GetAsync(url);
// parse the response and return the data.
string jsonString = await response.Content.ReadAsStringAsync();
object responseData = JsonConvert.DeserializeObject(jsonString);
return (dynamic)responseData;
}
}

C# HttpClient Authentication

I cant force to work HttpClient Post request...
or rather, I can't understand what I'm doing wrong..
My code below and this's sketch
#region Method 2
HttpClient h = new HttpClient();
var values = new Dictionary<string, string>
{
//{ "csrf_token" , "aacc347b22df2a7b3a20c9674ba59cf4279c8d16:1552967983844" },
//{ "track_id", "501c86f19b745703e9ca54991e90dc3a87" },
//{ "password", "Oguhokuu13" }
};
CookieContainer container = new CookieContainer();
h.DefaultRequestHeaders.Add("Accept", "application/json, text/javascript, */*; q=0.01");
h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br");
h.DefaultRequestHeaders.Add("Accept-Language", "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7");
h.DefaultRequestHeaders.Add("Connection", "keep-alive");
h.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded")); // Content-Type
h.DefaultRequestHeaders.AcceptCharset.Add(new StringWithQualityHeaderValue("UTF-8"));
h.DefaultRequestHeaders.Add("Origin", "https://passport.yandex.ru");
h.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36");
// h.DefaultRequestHeaders.Add("Referer", "https://passport.yandex.ru/auth/welcome?retpath=https%3A%2F%2Ftaxi-cabinet.mobile.yandex.ru%2Fnews%2F");
h.DefaultRequestHeaders.Add("Host", "passport.yandex.ru");
h.DefaultRequestHeaders.Add("X-Requested-With", "XMLHttpRequest");
var content = new FormUrlEncodedContent(values);
var response = h.PostAsync("https://passport.yandex.ru/auth/add", content);
response.Wait();
var responseString = response.Result.Content.ReadAsStringAsync().Result;
textBox1.Text = responseString.ToString();
#endregion
Then I get response:
\u001f?\b\0\0\0\0\0\0\u0003sN???/Q\b?\u000f\u000eQ?O,-??OLI?\u0002\0d(,h\u0016\0\0\0"
Idk what I'm doing wrong

Why does Httpclient always return 401 Unauthorized error?

I try to get some data from a web, but always get 401 Unauthorized error. I guess this is caused by my request header setting is wrong.
Could we can set Accept, ContentType by using TryAddWithoutValidation?
How can I set the correct request header in HttpClient?
And I try to set the header like as following:
public static async Task<ResultInfo> Get(string url, string cookie = null)
{
url =$"https://www.zhihu.com/api/v4/questions/46508954/answers?sort_by=default&include=data%5B%2A%5D.is_normal%2Cis_sticky%2Ccollapsed_by%2Csuggest_edit%2Ccomment_count%2Ccan_comment%2Ccontent%2Ceditable_content%2Cvoteup_count%2Creshipment_settings%2Ccomment_permission%2Cmark_infos%2Ccreated_time%2Cupdated_time%2Crelationship.is_authorized%2Cis_author%2Cvoting%2Cis_thanked%2Cis_nothelp%2Cupvoted_followees%3Bdata%5B%2A%5D.author.badge%5B%3F%28type%3Dbest_answerer%29%5D.topics&limit=20&offset=20";
; var result = new ResultInfo()
{
IsSuccessStatusCode = false
};
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
using (var client = new HttpClient(handler))
{
try
{
cookie = "_zap=bb597634-86d0-4636-a516-89e76b4b5335; q_c1=b2e3af8dde034209a6948fcd9fd0e70f|1515131469000|1509512040000; d_c0=\"AJBrsuC5AA2PTqu9qOLgytIZlpXzmzgRgcE=|1516167026\"; _xsrf=ec351d2e-f0b2-4b2e-9eff-a3262a9faf10; capsion_ticket=\"2|1:0|10:1516261057|14:capsion_ticket|44:NzU1ZGRjYTA3Y2U3NGVjZWE4MWIzMzllNjA1NjMxZGQ=|b73bb13fe6a623db85478f5eb46ed51906674b89f29e4ef1a067fce6dcd90f54\"; z_c0=\"2|1:0|10:1516261068|4:z_c0|92:Mi4xMzMwVEFBQUFBQUFBa0d1eTRMa0FEU1lBQUFCZ0FsVk56SnhOV3dETFF5VGlla1ZDUllwcUVMRVhMWk4wV1duOVJn|f2fcc1750b966a19dd202ddc2eb42dca9dc9f573421bc3a8675a072e603ca897\"; __utma=51854390.1320208938.1517560249.1517560249.1517560249.1; __utmb=51854390.0.10.1517560249; __utmc=51854390; __utmz=51854390.1517560249.1.1.utmcsr=zhihu.com|utmccn=(referral)|utmcmd=referral|utmcct=/question/38824940; __utmv=51854390.100-1|2=registration_date=20130816=1^3=entry_date=20130816=1";
var replace = cookie.Replace("\"", "");
client.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "*/*");
client.DefaultRequestHeaders.TryAddWithoutValidation("ContentType", "application/x-www-form-urlencoded");
client.DefaultRequestHeaders.TryAddWithoutValidation("UserAgent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36");
client.DefaultRequestHeaders.TryAddWithoutValidation("Cookie", replace);
client.DefaultRequestHeaders.TryAddWithoutValidation("accept-encoding", "gzip/deflate");
//client.DefaultRequestHeaders.TryAddWithoutValidation("accept-language", "en-US,en;q=0.9");
client.DefaultRequestHeaders.TryAddWithoutValidation("AllowAutoRedirect","false");
if (!string.IsNullOrEmpty(url))
{
var data = await client.GetAsync(url);
if (data.IsSuccessStatusCode)
{
var content = await data.Content.ReadAsStringAsync();
result.IsSuccessStatusCode = true;
result.Data = await data.Content.ReadAsStringAsync();
}
else
{
result.Message = data.StatusCode.ToString();
}
}
}
catch (Exception e)
{
result.Message = e.Message;
throw;
}
}
return result;
}

"Navigating" a CSRF-protected website using HTTP requests and cookies keeps redirecting to the login page, only working in Postman

I am trying to get info from a website that has no public API. I can't give the website's name because you literally can only sign in with a work account created on their end, so there's no way for other people to test this without my credentials (which I am not giving out).
class Client
{
public string PHPSESSID { get; set; }
public Client()
{
this.Login();
}
public void Login()
{
string csrf;
RestClient client;
RestRequest csrfRequest = new RestRequest(Method.GET);
csrfRequest.AddHeader("accept-language", "en-US,en;q=0.8");
csrfRequest.AddHeader("accept-encoding", "gzip, deflate, br");
csrfRequest.AddHeader("dnt", "1");
csrfRequest.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
csrfRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
csrfRequest.AddHeader("upgrade-insecure-csrfRequests", "1");
RestRequest request = new RestRequest(Method.POST);
request.AddHeader("x-requested-with", "XMLHttpRequest");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");
request.AddHeader("referer", "{WEBSITE}/login");
request.AddHeader("origin", "{WEBSITE}");
request.AddHeader("host", "{WEBSITE}");
request.AddHeader("dnt", "1");
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddHeader("content-length", "118");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
do
{
client = new RestClient("{WEBSITE}/login");
response = client.Execute(csrfRequest);
if (String.IsNullOrEmpty(this.PHPSESSID))
this.setPHPID(response);
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/login_check");
RestRequest bak = request.Clone();
request.AddCookie("PHPSESSID", this.PHPSESSID);
request.AddParameter("application/x-www-form-urlencoded", "form%5Busername%5D={USERNAME}&form%5Bpassword%5D={PASSWORD}&_csrf_token=" + csrf, ParameterType.RequestBody);
client.FollowRedirects = true;
response = client.Execute(request);
request = bak.Clone();
} while (response.Content.Contains("error"));
}
public string FindCSRF(string input)
{
int find = input.IndexOf("value=", input.IndexOf("_token")) + 8;
int find2 = input.IndexOf('"', find);
--find;
return input.Substring(find, find2-find);
}
public void setPHPID(IRestResponse response)
{
try
{
string cookie = response.Headers.ToList().Find(p => p.Name.Contains("Set-Cookie")).Value.ToString();
if(String.IsNullOrEmpty(cookie))
this.PHPSESSID = response.Cookies[0].Value;
else
{
int cookieStart = cookie.IndexOf("PHPSESSID=") + 10;
this.PHPSESSID = cookie.Substring(cookieStart, cookie.IndexOf(';', cookieStart) - cookieStart);
}
}
catch
{
this.PHPSESSID = response.Cookies[0].Value;
}
}
public void switchBranch(string branchID)
{
RestRequest request = new RestRequest(Method.GET);
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("referer", "{WEBSITE}/message-board");
request.AddHeader("dnt", "1");
request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
request.AddHeader("upgrade-insecure-requests", "1");
RestRequest branchRequest = new RestRequest(Method.POST);
branchRequest.AddHeader("accept-language", "en-US,en;q=0.8");
branchRequest.AddHeader("accept-encoding", "gzip, deflate, br");
branchRequest.AddHeader("referer", "{WEBSITE}/message-board");
branchRequest.AddHeader("dnt", "1");
branchRequest.AddHeader("content-type", "application/x-www-form-urlencoded");
branchRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
branchRequest.AddHeader("x-branchRequested-with", "XMLHttpbranchRequest");
branchRequest.AddHeader("origin", "{WEBSITE}");
branchRequest.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
RestClient client;
string csrf;
do
{
client = new RestClient("{WEBSITE}/");
client.FollowRedirects = true;
RestRequest acc = request.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
response = client.Execute(acc);
if (response.Content.Contains("login"))
{
this.Login();
Console.WriteLine("Login Required");
continue;
}
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/branch-switch");
acc = branchRequest.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
acc.AddParameter("application/x-www-form-urlencoded", "office_select%5Boffice%5D=" + branchID + "&office_select%5B_token%5D=" + csrf, ParameterType.RequestBody);
} while (response.Content.Contains("error"));
}
}
public static class RequestExt
{
public static RestRequest Clone(this RestRequest req)
{
return req;
}
}
A lot of this I just intercepted from the website using Postman then copied the auto-generated code over, so if there are certain headers that can help, I probably don't know about them.
I mostly fixed the login issues (though I literally have to fail before it gives me a working CSRF token, thus the while loops). The problem is that when I try to go to {WEBSITE} in Client.switchBranch, to start actually doing stuff, it returns a JSON with a redirect to the login page. It doesn't do this in Postman, I have been able to successfully interact with the website and gather information... once or twice (logging in even sometimes fails in the browser).
It seems entirely random to me, but there MUST be some kind of common factor here. I'm guessing it's something to do with the PHPSESSID cookie or the lack of a "keep-alive" connection header (which caused RestSharp to throw an error when I tried to implement it; even if that is the problem, I still don't know how to solve it). The login request and response headers in Chrome's developer tools, intercepting from the website, look something like this.

Asynchronous HTTP POST in C# 4.0

I've used the following code to perform asynchronous HTTP request in C#.
private static Task GetUrl(string url)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent =
"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36";
request.Accept = "text/html";
return Task
.Factory
.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, url)
.ContinueWith(t =>
{
if (t.IsCompleted)
{
using (var stream = t.Result.GetResponseStream())
{
using (var reader = new StreamReader(stream))
{
//Console.WriteLine("-- Successfully downloaded {0} --", t.AsyncState);
//Console.WriteLine(reader.ReadToEnd());
}
}
}
else if (t.IsFaulted)
{
Console.WriteLine("There was an error downloading {0} - {1}", t.AsyncState, t.Exception);
}
});
}
However I'm not sure how I should modify the above code to support HTTP post as well. Any help is appreciated!
In particular I'd like to know how I should add BeginGetRequestStream and EndGetRequestStream into the current function...

Use the Method property of the HttpWebRequest:
request.Method = "POST";

Categories