I'm trying to get the cookies from here: https://www.etoro.com/people/wesl3y/portfolio
If I call the Webpage with my Edge Browser, there are a lot of cookies, which are not flagged as "HTTP". For example the follwing cookie:
TMIS2 9a74f8b353780f2fbe59d8dc1d9cd901437be0b823f8ee60d0ab3637053d1bd9675fd2caa6404a74b4fc2a8779e1e7d486862875b11af976466a696de4a89c87e0e942f11ceed91c952d629646dc9cb71bdc2c2fd95a0a71698729f733717a775675add1a06bd9883e47c30e3bd162fabd836467a1cc16870a752373581adfd2ca .etoro.com / 2021-03-20T17:40:43.000Z 263 Medium
I want to get this cookie in my c# Program via HttpClient also, but until now I don't know how :( As far as I understood this cookies are not accessed via http, else Javascript is used to set this cookies on clientside.
Currently I used the following code for accessing the page and show the found cookies:
public void Test()
{
CookieContainer CookieJar = new CookieContainer();
var handler = new HttpClientHandler
{
CookieContainer = CookieJar,
UseCookies = true,
};
HttpClient client = new HttpClient(handler);
HttpResponseMessage response = client.GetAsync("https://www.etoro.com/people/wesl3y/portfolio").Result;
Uri uri = new Uri("https://www.etoro.com/people/wesl3y/portfolio");
IEnumerable<Cookie> responseCookies = CookieJar.GetCookies(uri).Cast<Cookie>();
foreach (Cookie cookie in responseCookies)
Console.WriteLine(cookie.Name + ": " + cookie.Value);
}
With my code I got all Cookies with Parameter HTTP=true, but miss all the other ones. Is there a way to get the missing cookies from that site with HttpClient? Or is there any other type of c# integrated Browser i have to use? The Main focus of my Tool is getting Json Information from the etoro.com API. But the API Access is very Rate limited when the right cookies are not in place :(
Thanks, Mag.
Trying to call an API from a controller using HttpClient and the API does not recognize the user as authenticated and logged in. When calling the API from JS I have no issue. I noticed the HttpClient was only sending via HTTP 1.1 and so I upgraded to 2.0 settings the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER flag but this made no difference. I have tried all combinations of the HttpClientHandler properties including UseCookies and the request is never authenticated.
using (var handler = new HttpClientHandler {UseDefaultCredentials = true})
{
using (var httpClient = new HttpClient(handler))
{
var response = httpClient.GetStringAsync(new Uri($"https://localhost:64366/api/")).Result;
}
}
Will move to token based auth in the future but for now would like to understand why there is a difference between calling the API from C# vs JS. This is all HTTPS on localhost using asp net core 2.2.
Difference between JS and C# is that browsers attach cookies automatically to requests and you have to attach cookies manually in C# as juunas mentioned.
To obtain and use authentication cookie you may use the following pattern
CookieContainer cookies = new CookieContainer(); //this container saves cookies from responses and send them in requests
var handler = new HttpClientHandler
{
CookieContainer = cookies
};
var client = new HttpClient(handler);
string authUrl = ""; //your auth url
string anyUrl = ""; //any url that requires you to be authenticated
var authContent = new FormUrlEncodedContent(
new List<KeyValuePair<string, string>> {
new KeyValuePair<string, string>("login", "log_in"),
new KeyValuePair<string, string>("password", "pass_word")
}
);
//cookies will be set on this request
HttpResponseMessage auth = await client.PostAsync(authUrl, authContent);
auth.EnsureSuccessStatusCode(); //retrieving result is not required but you will know if something goes wrong on authentication
//and here retrieved cookies will be used
string result = await client.GetStringAsync(anyUrl);
Edit: I apologize for the confusion: I'm attempting to scrape a site that I did not write; I'm not writing an ASP app. I'm only attempting to scrape one.
After making a post request to a login page, I attempt to read the cookies from another page. I do not see all the cookies I expect, however. My code is as follows:
// Downloads login cookies for subsequent requests
public async Task<CookieContainer> loginCookies()
{
var cookies = new CookieContainer();
var handler = new HttpClientHandler {
UseCookies = true,
AllowAutoRedirect = true,
CookieContainer = cookies
};
var client = new HttpClient(handler);
var loginUri = new Uri("https://connect.example.edu/login.aspx");
var credentials = new Dictionary<string, string> {
{"example$txtUsername", this.Username},
{"example$txtPassword", this.Password}
};
var formCredentials = new FormUrlEncodedContent(credentials);
await client.PostAsync(loginUri, content: formCredentials);
var pointsUri = new Uri("https://info.example.edu/");
Console.WriteLine("COOKIES:");
foreach (Cookie cookie in cookies.GetCookies(pointsUri))
Console.WriteLine($"{cookie.Name} --> {cookie.Value}");
return cookies;
}
I believe the error is a result of loginUri and pointsUri having different subdomains. The info I need to scrape exists at the pointsUri page, but the login exists at loginUri.
A cookie that I'm missing in particular is ASP.NET_SessionID.
PROBLEM:
I'm trying to make external authentication work with default Web API template project. I used the following instructions to add support for external authentication services (FB/Google/Microsoft): http://www.asp.net/web-api/overview/security/external-authentication-services
Just for the record, I was able to make external authentication work with default SPA template project.
Also, new local user creation works fine.
The problem appeared once I tried to use my client app (WPF-based) to register user using external provider (e.g., FB).
For the record, I used these two articles as the starting point: http://leastprivilege.com/2013/11/26/dissecting-the-web-api-individual-accounts-templatepart-3-external-accounts/ and thread #21065648 here at Stack Overflow.
They really helped me to understand the entire logic.
here's the short overview of the steps I've done:
two windows, main and the one for external authentication provider, with embedded WebBrowser
the flow:
2.1. user opens an app, main window appears
2.2. user clicks on a button to get the list of all supported external authentication providers,
and here's what happens in the code:
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage loginResponse = await client.GetAsync("api/Account/ExternalLogins?returnUrl=%2F&generateState=true");
if (loginResponse.IsSuccessStatusCode)
{
var externalLoginProviders = await loginResponse.Content.ReadAsAsync<IEnumerable<AuthenticationProvider>>();
// cleaning resources
client.Dispose();
loginResponse.Dispose();
// obtained data is sent to UI
This leads to the list of External Authentication Providers, with name, url, and state. URL is relative, and is used as the redirect to the actual provider (e.g., FB, or Google).
2.3. As soon as the list with external authentication providers is populated, user can enter his/her email, and click on the "Login" button, which will lead to the following:
In second window, with embedded WebBrowser, the latter is Navigated to the provided URL obtained in the previous step. If user logs successfully into the selected provider (e.g., Facebook), and agrees to give my app (Facebook app) necessary permissions (profile), user is navigated back to our website, in the form:
(our base url) /#access_token=XXX&token_type=bearer&expires=YYY
This address is then parsed and address params (token, etc.) are saved in the structured form for later use.
Right after that I go to api/UserInfo to understand if user already logged on with this external authentication provider or not:
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// setting Bearer Token obtained from Auth provider
client.SetBearerToken(result.AccessToken);
// calling /api/Account/UserInfo
var userInfoResponse = await client.GetAsync("api/Account/UserInfo");
var userInfoMessage = userInfoResponse.Content.ReadAsStringAsync().Result;
var userInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<UserInfo>(userInfoMessage);
// cleaning resources
client.Dispose();
userInfoResponse.Dispose();
if (userInfo.hasRegistered == true)
{
// going to login
}
Right after that, assuming that our user didn't create a user based on his/her login with the given external authentication provider (e.g., FB), yet, we do the following:
var data = new Dictionary<string, string>();
data.Add("Email", this.externalUserEmailTextBox.Text);
var registerExternalUrl = new Uri(string.Concat(baseAddress, #"api/Account/RegisterExternal"));
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
// setting Bearer Token obtained from External Authentication provider
client.SetBearerToken(result.AccessToken);
var response = client.PostAsync(registerExternalUrl.ToString(), new FormUrlEncodedContent(data)).Result;
At this point, user should be created (I have zero changes on the service side apart of uncommenting provider's app id and app secret lines in Startup.Auth.cs).
Unfortunately, this doesn't happen at all. Instead, I get "Internal Server Error" which means that this line,
var info = await Authentication.GetExternalLoginInfoAsync();
brings null, and so system fails to Authentication correctly with the provided Bearer Token.
I can't understand why. It seems that something is broken in the framework, or I'm doing something wrong...
SOLUTION:
Thanks to #berhir, here's the solution:
In MainWindow.xaml.cs, define cookieContainer
CookieContainer cookieContainer;
In MainWindow_Loaded event handler, instantiate it:
// we create new cookie container
cookieContainer = new CookieContainer();
As soon as user asks app to show all external login providers, instantiate HttpClient using suggested code by #berhir:
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage loginResponse = await client.GetAsync("api/Account/ExternalLogins?returnUrl=%2F&generateState=true");
As soon as list of external auth providers is obtained, the next step is to show second window, with embedded WebBrowser. There, you'll have to declare two WinAPI calls for cookies:
[DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern bool InternetSetCookie(string lpszUrlName, string lbszCookieName, string lpszCookieData);
[DllImport("wininet.dll", SetLastError = true)]
public static extern bool InternetGetCookieEx(string url, string cookieName, StringBuilder cookieData, ref int size, Int32 dwFlags, IntPtr lpReserved);
// and
private const Int32 InternetCookieHttponly = 0x2000;
/// <summary>
/// Gets the URI cookie container.
/// </summary>
/// <param name="uri">The URI.</param>
/// <returns></returns>
public static CookieContainer GetUriCookieContainer(Uri uri)
{
CookieContainer cookies = null;
// Determine the size of the cookie
int datasize = 8192 * 16;
StringBuilder cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(uri.ToString(), null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero))
{
if (datasize < 0)
return null;
// Allocate stringbuilder large enough to hold the cookie
cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(uri.ToString(), null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero))
return null;
}
if (cookieData.Length > 0)
{
cookies = new CookieContainer();
cookies.SetCookies(uri, cookieData.ToString().Replace(';', ','));
}
return cookies;
}
This window with embedded browser is instantiated, and constructor accepts several params, including start URL (to external auth provider), end URL (resulting with "#access_token=..." or "error...", callback, and, even more important, with the original cookieContainer. We use InternetSetCookie WinAPI method to pass that original cookieContainer to WebBrowser's session:
// set cookies
var cookies = cookieContainer.GetCookies(baseAddress).OfType<Cookie>().ToList();
foreach (var cookie in cookies)
{
InternetSetCookie(startUrl, cookie.Name, cookie.Value);
}
So, once user succeeds with signing into the selected external auth provider (e.g., Facebook), updated cookieContainer (which includes cookies set in the first HttpClient call, as well as right within WebBrowser right after signing in to external auth provider) obtained using InternetGetCookieEx WinAPI call, is sent back to MainWindow.xaml.cs via callback:
if (this.callback != null)
{
var cookies = GetUriCookieContainer(e.Uri);
this.callback(new AuthResult(e.Uri, this.providerName, cookies));
}
There, we issue two new requests, to api/Account/UserInfo, and then to api/Account/RegisterExternal:
this.cookieContainer = result.CookieContainer; // where result is AuthResult containing the cookies obtained from WebBrowser's session
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// setting Bearer Token obtained from Auth provider
client.SetBearerToken(result.AccessToken);
// calling /api/Account/UserInfo
var userInfoResponse = await client.GetAsync("api/Account/UserInfo");
var userInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<UserInfo>(userInfoMessage);
if (userInfo.hasRegistered == false)
{
var data = new Dictionary<string, string>();
data.Add("Email", this.externalUserEmailTextBox.Text);
var registerExternalUrl = new Uri(string.Concat(baseAddress, #"api/Account/RegisterExternal"));
var content = new FormUrlEncodedContent(data);
var response = client.PostAsync(registerExternalUrl.ToString(), content).Result;
// obtaining content
var responseContent = response.Content.ReadAsStringAsync().Result;
if (response != null && response.IsSuccessStatusCode)
{
MessageBox.Show("New user registered, with " + result.ProviderName + " account");
}
So, here we go. Cookies are used within the whole lifecycle, from first HttpClient request to the final moment when we register a new user using api/account/registerExternal.
It seems that you forgot to handle the cookies. The easiest way is to use the same HttpClient instance for all your requests or you can use the same CookieContainer.
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
}
I am trying to download daily a content of a webpage using csEXWB for my c# application usage. The site requires authentication for this. Using my browser and fiddler i got the cookie which contains my authentication information.
I have 2 questions:
1.How can i send a download request using this cookie to the webpage?
2.Can I prevent my cookie from expiration?
I am really new to cookie usage. Any pointer would be helpful.
THANKS!
Here is a method that creates an HTTP request with a session cookie:
private HttpWebRequest CreateRequest(string url, string method, string sessionCookie)
{
if (string.IsNullOrEmpty(url))
throw new ArgumentException("Empty value", "url");
HttpWebRequest result = WebRequest.Create(url) as HttpWebRequest;
if (result == null)
throw new ArgumentException("Only the HTTP protocol is supported", "url");
result.CookieContainer = new CookieContainer();
if (sessionCookie != null) {
result.CookieContainer.Add(
new Uri(url),
new Cookie(SessionCookieName, sessionCookie)
);
}
result.Method = method;
return result;
} // CreateRequest
where 'SessionCookieName' is the name of the cookie to send.
The cookie expiration is controlled by the originating Web site, you cannot prevent it from expiring.
I think that what you should really do is to write code that does the log-in to the Web site - how to do it would depend on how the login works on the Web site you want to access.