We Keep Coding

cannot get page source using c# httpwebrequest

I tried to get the source of a particular site page using the code below but it failed.
I was able to get the page source in 1~2 seconds using a webbrowser or webdriver, but httpwebrequest failed.
I tried putting the actual webbrowser cookie into httpwebrequest, but it failed, too.
(Exception - The operation has timed out)
I wonder why it failed and want to learn through failure.
Thank you in advance!!.
string Html = String.Empty;
CookieContainer cc = new CookieContainer();
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://www.coupang.com/");
req.Method = "GET";
req.Host = "www.coupang.com";
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3";
req.Headers.Add("Accept-Language", "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7");
req.CookieContainer = cc;
using (HttpWebResponse res = (HttpWebResponse)req.GetResponse())
using (StreamReader str = new StreamReader(res.GetResponseStream(), Encoding.UTF8))
{
Html = str.ReadToEnd();
}

Removing req.Host from your code should do the trick.
According to the documentation:
If the Host property is not set, then the Host header value to use in an HTTP request is based on the request URI.
You already set the URI in (HttpWebRequest)WebRequest.Create("https://www.coupang.com/") so I don't think doing it again is necessary.
Result
Please let me know if it helps.

Get iCloud Contact list in C#

After searching about iCloud API, I found some example on NodeJS and Python, but unfortunately, I'm not familiar with them. I want to know how to get iCloud Contact list on C#.
Example on python: https://github.com/mindcollapse/iCloud-API/blob/master/iCloud.py
Example on NodeJS: https://www.snip2code.com/Snippet/65033/Request-Contact-List-From-iCloud
I try to parse the login code to C#:
private void iCloudLogin()
{
string guiid = Guid.NewGuid().ToString("N");
//string url = "https://p12-setup.icloud.com/setup/ws/1/login?clientBuildNumber=1P24&clientId=" + guiid;
string url = "https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=1P24&clientId=" + guiid;
using (var client = new WebClient())
{
client.Headers.Set("Origin", "https://www.icloud.com");
client.Headers.Set("Referer", "https://www.icloud.com");
client.Headers.Set("User-Agent", "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36");
var values = new NameValueCollection();
values["apple_id"] = appleId;
values["password"] = password;
values["extended_login"] = "false";
var response = client.UploadValues(url, values);
}
}
I receive 400 : Bad request with above code, please help to go give the direction where I'm wrong, I appreciate your help if there is code example.
Update:
Now I could login and get many information, include my contact server url, dsid, this is the link I used:
https://p12-setup.icloud.com/setup/ws/1/login?clientBuildNumber=1P24&clientId=MyGuid
After that, I use below url to get contact list:
https://p35-contactsws.icloud.com/co/startup?clientBuildNumber=1P24&clientId=MyGuid&clientVersion=2.1&dsid=MyDSID&locale=en-EN&order=last%2Cfirst
https://p35-contactsws.icloud.com is my contact server, it actually is https://p35-contactsws.icloud.com:443, but base on example I refer to, the port :443 need to be removed.
But I still get 421: Client Error

I know the answer
Firstly, in this case the request should be WebRequest, not WebClient.
In the first api url: https://setup.icloud.com/setup/ws/1/login?clientBuildNumber=WHATEVERNUMBER&clientId=RANDOM_GUID :
The WebRequest should be a Post and include appleid, password in data, and in header there should be Origin=https://www.icloud.com :
private void iCloudLogin()
{
string data = "{\"apple_id\":" + appleId + ", \"password\":" + password + ", \"extended_login\":false}";
byte[] dataStream = Encoding.UTF8.GetBytes(data);
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "POST";
webRequest.Headers.Set("Origin", "https://www.icloud.com");
webRequest.ContentLength = dataStream.Length;
Stream newStream=webRequest.GetRequestStream();
// Attach the data.
newStream.Write(dataStream,0,dataStream.Length);
newStream.Close();
WebResponse webResponse = webRequest.GetResponse();
// get contact server url, dsid, Cookie
}
iCloud server will response contact server url, dsid, also "X-APPLE-WEBAUTH-TOKEN" and "X-APPLE-WEBAUTH-USER" (these two values are in header "Set-Cookie" of webResponse)
When you have enough above parameters, you can get icloud contact list, follow by this way:
Make a GET request to this url:
https://p35-contactsws.icloud.com/co/startup?clientBuildNumber=1P24&clientId=MyGuid&clientVersion=2.1&dsid=MyDSID&locale=en-EN&order=last%2Cfirst
+https://p35-contactsws.icloud.com : my contact server url, yours can be different.
+clientVersion: just leave it 2.1
+MyGuid: the Guid you used in the first request.
Important: in the header, must include:
Origin:https://www.icloud.com
Cookie: X-APPLE-WEBAUTH-TOKEN=XXXXXX;X-APPLE-WEBAUTH-USER=YYYYYYYYY
After that, you will get full iCloud Contact list.
This way is web service base, so it can work in many languages, so I think this can help.

c# screen scraping and getting all the cookies for secured access to a website

I'm trying to access a website through c# program. There seems to be three cookies needed to access the website yet I only receive two in my cookie container so when I try to access other parts the website I can't. I first do a GET then a POST. The reason I programmed it this way because it seemed from the Chrome Dev tools I determined that it first used a GET for the first two and then a POST to login and get the third one. The POST shows a 302 Moved Temporarily and then right after that it's a redirect. Which I believe is the reason I can't obtain the last cookie can anyone shed any light?
cookieJar = new CookieContainer();
string formParams = string.Format("USERNAME={0}&PASSWORD={1}", username, password);
Console.Write(" \n 1st count before anything : " + cookieJar.Count + "\n"); // 0 cookies
//First go to the login page to obtain cookies
HttpWebRequest loginRequest = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
loginRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
//.Connection = "keep-alive";
loginRequest.Method = "GET";
loginRequest.UseDefaultCredentials = true;
loginRequest.CookieContainer = cookieJar;
loginRequest.AllowAutoRedirect = false;
HttpWebResponse loginResponse = (HttpWebResponse)loginRequest.GetResponse();
Console.Write(" \n 2nd count after first response : " + cookieJar.Count + "\n"); // Only 2 are recorded.
//Create another request to actually log into website
HttpWebRequest doLogin = (HttpWebRequest)HttpWebRequest.Create("https://server.com/login/login.jsp");
doLogin.Method = "POST";
doLogin.ContentType = "application/x-www-form-urlencoded";
doLogin.AllowAutoRedirect = false;
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
doLogin.ContentLength = bytes.Length;
using (Stream os = doLogin.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
oLogin.CookieContainer = cookieJar;
doLogin.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36";
doLogin.Referer = "https://server.com/login/login.jsp";
HttpWebResponse Response = (HttpWebResponse)doLogin.GetResponse();
Console.Write(" \n 3rd count after second repsonse : " + cookieJar.Count + "\n"); // still two

HttpWebRequest had a problem with cookies.
The problem was that a cookie that was assigned to "server.com" would be changed to ".server.com". However, "server.com" did not match ".server.com".
If you are using a framework older than (I think it is 3) you are probably experiencing this problem.
The work around is to use e.g. "www.server.com" in your request, this will match cookies assigned to ".server.com".

FreeAgent API (OAuth) From c#

I'm trying to connect to a API (In this case freeagent) to grab some data. I've used Googles OAuth Playground to generate me a token.
var req = (HttpWebRequest)WebRequest.Create("https://api.freeagent.com/v2/recurring_invoices");
req.Headers["Authorization"] = "Bearer " + Convert.ToBase64String(Encoding.ASCII.GetBytes(_accessToken));
req.ContentType = "application/xml";
req.Accept = "application/xml";
req.Method = "GET";
// and get the response
var resp = req.GetResponse();
var streamIn = new StreamReader(resp.GetResponseStream());
returnData = streamIn.ReadToEnd();
streamIn.Close();
return resp;
Now I'm loosly trying to conver the following : https://dev.freeagent.com/docs/using_curl
I constantly get a Bad Request HTTP 400 - Anyone have any suggestions on what could be causing this problem?

I have most of the api working using RestSharp:
https://github.com/nicwise/FreeAgent
I wrote that for my iPhone app, mobileAgent, but it's fairly basic
.NET, so it should work on anything.
My authentication injection code looks very much like yours:
protected void SetAuthentication(RestRequest request)
{
request.AddHeader("Authorization", "Bearer " +
Client.CurrentAccessToken.access_token);
}
Have you run fiddler or something similar to see exactly what is going
over the wire? that often helps.

The solution to this, is to pass the UserAgent as well.
Example...
req.UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.52 Safari/537.17";

You shouldn't need to base64 encode the token, just set the Authorization header to "Bearer " + _accessToken. If this still doesn't work, try using http://httpbin.org/get as the URL to look at your request to see if there are any other possible problems.

How do I perform a simple GET request with cookies in C#

Everything I find is either about a POST request or does not assume cookies.
I have an URL like this:
http://page.com/find/1,1,1,find.html?advanced=1&param1=val1&param2[]=val2
When put into a browser, this will direct me to a search results page. Now I would like to replicate it in a C# program. I have this so far:
WebRequest req = WebRequest.Create(url);
((HttpWebRequest)req).UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2";
req.Method = "GET";
WebResponse response = req.GetResponse();
When I run it, it returns a "please login" page, as expected. But there is a problem with one of the parameters. This is the response URL:
http://page.com/login.html?ref=find/1,1,1,find.html?advanced=1&param1=val1&param2=Array
So, two questions: what might have happened to param2? And how do I add cookies to this?
EDIT: Managed to set the cookies by casting to HttpWebRequest.

As devio said you should use HttpWebRequest. I did dirty test to check it.
Prepare cookies to send. I made available for whole localhost:
HttpWebRequest rq = (HttpWebRequest)WebRequest.Create("http://localhost/test.php");
rq.CookieContainer = new CookieContainer();
rq.CookieContainer.Add(new Cookie("test", "xxxx", "/", "localhost"));
Your script should set cookies to make them available in response. And you could use them.
HttpWebResponse resp = (HttpWebResponse)rq.GetResponse();
foreach(var c in resp.Cookies)
{
Debug("{0}: {1}", c.Name, c.Value);
}

You can use this code piece:
Cookie SessionCookie = new Cookie("{CookieName}", {Cookievalue})
{
Domain = "{domain you want to hit}", Path = "/", Expired = false, HttpOnly = true
};
CookieContainer SessionCookieHolder = new CookieContainer();
SessionCookie.Add(SessionCookie);
try
{
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create("{URL}");
WebReq.CookieContainer = SessionCookie;
WebReq.Method = "GET/POST/HEAD"; //depending on the request type//
WebReq.KeepAlive = true;
HttpWebResponse Resp = (HttpWebResponse)WebReq.GetResponse();
}
catch(Exception ex)
{
string ExceptionReader = ex.Message;
}

Cookies are stored in the HttpWebRequest.CookieContainer and HttpWebResponse.Cookies properties.
For subsequent requests, you need to add the response's cookies to the request's cookie container.