We Keep Coding

C# Get Javascript Cookies with HttpClient

I'm trying to get the cookies from here: https://www.etoro.com/people/wesl3y/portfolio
If I call the Webpage with my Edge Browser, there are a lot of cookies, which are not flagged as "HTTP". For example the follwing cookie:
TMIS2 9a74f8b353780f2fbe59d8dc1d9cd901437be0b823f8ee60d0ab3637053d1bd9675fd2caa6404a74b4fc2a8779e1e7d486862875b11af976466a696de4a89c87e0e942f11ceed91c952d629646dc9cb71bdc2c2fd95a0a71698729f733717a775675add1a06bd9883e47c30e3bd162fabd836467a1cc16870a752373581adfd2ca .etoro.com / 2021-03-20T17:40:43.000Z 263 Medium
I want to get this cookie in my c# Program via HttpClient also, but until now I don't know how :( As far as I understood this cookies are not accessed via http, else Javascript is used to set this cookies on clientside.
Currently I used the following code for accessing the page and show the found cookies:
public void Test()
{
CookieContainer CookieJar = new CookieContainer();
var handler = new HttpClientHandler
{
CookieContainer = CookieJar,
UseCookies = true,
};
HttpClient client = new HttpClient(handler);
HttpResponseMessage response = client.GetAsync("https://www.etoro.com/people/wesl3y/portfolio").Result;
Uri uri = new Uri("https://www.etoro.com/people/wesl3y/portfolio");
IEnumerable<Cookie> responseCookies = CookieJar.GetCookies(uri).Cast<Cookie>();
foreach (Cookie cookie in responseCookies)
Console.WriteLine(cookie.Name + ": " + cookie.Value);
}
With my code I got all Cookies with Parameter HTTP=true, but miss all the other ones. Is there a way to get the missing cookies from that site with HttpClient? Or is there any other type of c# integrated Browser i have to use? The Main focus of my Tool is getting Json Information from the etoro.com API. But the API Access is very Rate limited when the right cookies are not in place :(
Thanks, Mag.

Calling SSO-Enabled API From .Net Core API Using Explicit Network Credentials

I'm migrating a WinForms App to a .Net Core (3.1) API and I've run into trouble with a feature which makes a call to a SSO-protected API within our network.
In the WinForms App, I had the following:
public async void Authenticate()
{
var wi = System.Security.Principal.WindowsIdentity.GetCurrent();
var wic = wi.Impersonate();
try
{
// Initial request to grab cookies and redirect endpoint
var apiEndPoint = $"https://{_host}{_apiPath}";
var cookieRedirectRequest = new HttpRequestMessage(HttpMethod.Get, apiEndPoint);
var response = await _httpClient.SendAsync(cookieRedirectRequest);
if (response != null)
{
var cookieContainer = new CookieContainer();
foreach (var cookie in response.Headers.GetValues("Set-Cookie"))
{
var parsedCookie = cookie.ToCookie();
parsedCookie.Domain = TARGET_DOMAIN;
cookieContainer.Add(parsedCookie);
}
var redirectURL = response.Headers.GetValues("Location").FirstOrDefault();
// Add the cookies to the client to continue the authentication
_httpClient = new HttpClient(new HttpClientHandler()
{
UseDefaultCredentials = true,
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer
});
// Second request to grab code and state values
var codeAndStateRequest = new HttpRequestMessage(HttpMethod.Get, redirectURL);
response = await _httpClient.SendAsync(codeAndStateRequest);
...
The initial call to the API redirects us to the SSO-authentication page. I use the cookies from that initial response to create my own request so that I can pass in the proper credentials and get back the appropriate SSO tokens (the response the SendSync call on the last line returns back the tokens), which I then use to authenticate on future requests to the API.
I'm trying to recreate this functionality in .NET Core using a Service Account, and instead of doing the WindowsIdentity impersonation, I'm trying to explicitly set the NetworkCredentials property of my HttpClientHandler:
handler = new HttpClientHandler()
{
UseDefaultCredentials = false,
Credentials = new NetworkCredential("svc_acct_username", "svc_act_pass", "mydomain"),
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer,
};
However, the response I'm getting back from my second request is a 401.
This is consistent with what I used to get in my original app -- I would initially get a 401 with a "WWW-Authenticate: Negotiate" header, but then there would be an automatic subsequent request where the network credentials were passed in and I'd get a 200. I'm not getting this second request with the creds in .Net Core.
I tried the Credentials Cache solution suggested here:
https://github.com/dotnet/runtime/issues/24490
But this has not worked.
Thanks in advance for any help you guys can give me. I'm quite stuck!
Quick Edit:
I can recreate the .Net Core behavior in my WinForms app by simply setting the Credentials property of the HttpClientHandler explicitly and setting UseDefaultCredentials = false. So, there's some magic happening by setting UseDefaultCredentials that seemingly can't be recreated when using explicit creds.

I got this working with help from a co-worker! The issue is that when you use explicit Network Credentials, you lose the "Negotiate" authentication type, which is what our SSO-endpoint requires. By using a CredentialCache object and setting the Negotiate auth-type explicitly, the call works!
var credCache = new CredentialCache { { new Uri("https://example.com"), "Negotiate", new NetworkCredential("svc_acct_user", "svc_acct_password", "mydomain") } };
// Add the cookies to the client to continue the authentication
handler = new HttpClientHandler()
{
UseDefaultCredentials = false,
Credentials = credCache,
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer,
};

HttpClient calling API doesnt not pass cookie auth

Trying to call an API from a controller using HttpClient and the API does not recognize the user as authenticated and logged in. When calling the API from JS I have no issue. I noticed the HttpClient was only sending via HTTP 1.1 and so I upgraded to 2.0 settings the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER flag but this made no difference. I have tried all combinations of the HttpClientHandler properties including UseCookies and the request is never authenticated.
using (var handler = new HttpClientHandler {UseDefaultCredentials = true})
{
using (var httpClient = new HttpClient(handler))
{
var response = httpClient.GetStringAsync(new Uri($"https://localhost:64366/api/")).Result;
}
}
Will move to token based auth in the future but for now would like to understand why there is a difference between calling the API from C# vs JS. This is all HTTPS on localhost using asp net core 2.2.

Difference between JS and C# is that browsers attach cookies automatically to requests and you have to attach cookies manually in C# as juunas mentioned.
To obtain and use authentication cookie you may use the following pattern
CookieContainer cookies = new CookieContainer(); //this container saves cookies from responses and send them in requests
var handler = new HttpClientHandler
{
CookieContainer = cookies
};
var client = new HttpClient(handler);
string authUrl = ""; //your auth url
string anyUrl = ""; //any url that requires you to be authenticated
var authContent = new FormUrlEncodedContent(
new List<KeyValuePair<string, string>> {
new KeyValuePair<string, string>("login", "log_in"),
new KeyValuePair<string, string>("password", "pass_word")
}
);
//cookies will be set on this request
HttpResponseMessage auth = await client.PostAsync(authUrl, authContent);
auth.EnsureSuccessStatusCode(); //retrieving result is not required but you will know if something goes wrong on authentication
//and here retrieved cookies will be used
string result = await client.GetStringAsync(anyUrl);

HttpClient is sending extra cookies

Running a UWP app*
So I have an HttpClient and it's associated handler. I am making a request to a website, passing in specified headers, and using a specified CookieContainer, which is empty at the beginning of the request.
When I send the request, Fiddler shows extra cookies being sent that I have not added. Where are they coming from?
CookieContainer cookieJar = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler( );
handler.UseDefaultCredentials = false;
handler.CookieContainer = cookieJar;
handler.Proxy = null;
using (var client = new HttpClient(handler as HttpClientHandler))
{
HttpResponseMessage response = new HttpResponseMessage();
String loginUrl = Const.BUNGIE_LOGIN_URI + (provider == Const.XBOX_PLATFORM ? Const.XBOX_LOGIN : Const.PS_LOGIN);
client.BaseAddress = new Uri(loginUrl);
//client.Timeout = new TimeSpan(0, 0, 0, 0, 450);
client.DefaultRequestHeaders.Add("Referer", "http://www.bungie.net");
client.DefaultRequestHeaders.Add("User-Agent", Const.USERAGENT);
client.DefaultRequestHeaders.Add("X-API-Key", Const.X_API_KEY);
client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
handler.AutomaticDecompression = DecompressionMethods.GZip;
response = client.GetAsync("").Result;
response.ReadCookies(); //adds cookies to cookieJar
}
What fiddler shows
Now, as the associated CookieContainer is empty before the request is made, where are these cookies coming from? Are they accessible? If I wanted the values from them, how would I obtain them?
Edit: Where are they being added to my HttpClient request from? Does HttpClient have a common CookieContainer / cache? I have two separate HttpClient instances, and when Client (A) makes a request, it received a "set-cookie" header back, setting a "bungled" cookie.
Later on, a separate instance of HttpClient, Client (B), makes a request to the same website, sending the cookie set within Client (A).
I did not explicitly append this cookie to Client (B)'s request, so how is it being added?

On Windows 10 build 10240, System.Net.Http.HttpClient is a wrapper on top of Windows.Web.Http.HttpClient (more info here), and so, cookies now work little bit different.
To delete those extra cookies, you will need to delete the cookies using a HttpCookieManager:
HttpBaseProtocolFilter filter = new HttpBaseProtocolFilter();
HttpCookieManager cookieManager = filter.CookieManager;
foreach (HttpCookie cookie in cookieManager.GetCookies(uri))
{
cookieManager.DeleteCookie();
}

HttpClient not storing cookies in CookieContainer

I'm using VS2010 +.NET 4.0 + System.Net.Http (from Nuget).
For a reason which I don't manage to understand, the session cookie which I receive in my HttpResponseMessage is not automatically saved in the HttpClient CookieContainer.
Here is what my code looks like:
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
HttpClient client = new HttpClient(handler);
Uri site = new Uri("https://www.mywebsite.com");
var response1 = client.SendAsync(new HttpRequestMessage(HttpMethod.Get,site)).Result;
I can see in the response headers that I have the following:
Set-Cookie: JSESSIONID=FC8110E434C2C6DAB78B4E335024A639; Path=/member; Secure
However my cookie container remains empty ...why ?

Use this piece of code to retrieve cookies from response:
/// <summary>
/// Read web cookies
/// </summary>
public static CookieContainer ReadCookies(this HttpResponseMessage response)
{
var pageUri = response.RequestMessage.RequestUri;
var cookieContainer = new CookieContainer();
IEnumerable<string> cookies;
if (response.Headers.TryGetValues("set-cookie", out cookies))
{
foreach (var c in cookies)
{
cookieContainer.SetCookies(pageUri, c);
}
}
return cookieContainer;
}

I guess the problem is that your cookies are secure. The problem is that, CookieContainer won't send secure cookies back to the server in subsequent HTTP requests. It might be a bug, or maybe it has some reasons behind it.
A workaround is to re-add the cookie to CookieContainer manually. This way, cookie would be sent back in HTTP request header, as no secure would be defined when you send cookies back to the server.
See this article for more information.

I had problem with cookies because of case difference in path. I logged in to /mysite/login, and set cookie for mysite, but then redirect to /MySite, and HttpClient suddenly discard all the cookies! When I changed adress to /MySite/login all become ok.

Perhaps the problem is that your request Url path is to the root of the web site ("/"), but your Set-Cookie header indicates a Path of "/member".

Make sure you are automatically decompressing the response:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
Example:
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler {
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
};
var httpClient = new HttpClient(clientHandler);
var httpRequest = new HttpRequestMessage(HttpMethod.Get, "https://example.com/");
var response = httpClient.Send(httpRequest);
response.Headers.TryGetValues("set-cookie", out var cookies);
string test = (new StreamReader(response.Content.ReadAsStream()).ReadToEnd());
Edit
This also populates my cookieContainer automatically. On both a httpClient.GetAsync("https://example.com/") or SendAsync(httpRequest)