I'm trying to get cookies on the Spotify login page with C# and the HttpClient class. However, the CookieContainer is always empty when I know cookies are being set. I'm not sending any headers, but it should still give me the cookie(s) because when I send a GET request without any headers with python (requests module) I get the csrf token. Here's my code:
using System;
using System.Net;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Reflection;
using System.Runtime.InteropServices;
using System.Collections;
using System.Web;
class Program
{
static void Main()
{
Task t = new Task(MakeRequest);
t.Start();
Console.WriteLine("Getting cookies!");
Console.ReadLine();
}
static async void MakeRequest()
{
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
Uri uri = new Uri("https://accounts.spotify.com/en/login/?_locale=en-US&continue=https:%2F%2Fwww.spotify.com%2Fus%2Faccount%2Foverview%2F");
HttpClient client = new HttpClient(handler);
var response = await client.GetAsync(uri);
string res = await response.Content.ReadAsStringAsync();
Console.WriteLine(cookies.Count);
foreach (var cookie in cookies.GetCookies(uri)) {
Console.WriteLine(cookie.ToString());
}
}
}
It seems pretty simple to me, but the program always says there's 0 cookies. Anyone know what's going on?
You need to enable the use of cookies using HttpClientHandler.UseCookies Property
public bool UseCookies { get; set; }
Gets or sets a value that indicates whether the handler uses the CookieContainer property to store server cookies and uses these cookies when sending requests.
//...
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
handler.UseCookies = true; //<-- Enable the use of cookies.
//...
I tried to write the response headers to the console with Console.WriteLine(response.Headers) and a Set-Cookie header with the csrf token was printed to the console. So it seems that HttpClient doesn’t count cookies in this header as actual cookies, thus not adding these said cookies to the CookieContainer.
Make sure you are decompression the response automatically. See my answer here:
https://stackoverflow.com/a/74750572/1158313
An example of what I use:
var clientHandler = new HttpClientHandler {
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate, // <--
};
Related
Trying to call an API from a controller using HttpClient and the API does not recognize the user as authenticated and logged in. When calling the API from JS I have no issue. I noticed the HttpClient was only sending via HTTP 1.1 and so I upgraded to 2.0 settings the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER flag but this made no difference. I have tried all combinations of the HttpClientHandler properties including UseCookies and the request is never authenticated.
using (var handler = new HttpClientHandler {UseDefaultCredentials = true})
{
using (var httpClient = new HttpClient(handler))
{
var response = httpClient.GetStringAsync(new Uri($"https://localhost:64366/api/")).Result;
}
}
Will move to token based auth in the future but for now would like to understand why there is a difference between calling the API from C# vs JS. This is all HTTPS on localhost using asp net core 2.2.
Difference between JS and C# is that browsers attach cookies automatically to requests and you have to attach cookies manually in C# as juunas mentioned.
To obtain and use authentication cookie you may use the following pattern
CookieContainer cookies = new CookieContainer(); //this container saves cookies from responses and send them in requests
var handler = new HttpClientHandler
{
CookieContainer = cookies
};
var client = new HttpClient(handler);
string authUrl = ""; //your auth url
string anyUrl = ""; //any url that requires you to be authenticated
var authContent = new FormUrlEncodedContent(
new List<KeyValuePair<string, string>> {
new KeyValuePair<string, string>("login", "log_in"),
new KeyValuePair<string, string>("password", "pass_word")
}
);
//cookies will be set on this request
HttpResponseMessage auth = await client.PostAsync(authUrl, authContent);
auth.EnsureSuccessStatusCode(); //retrieving result is not required but you will know if something goes wrong on authentication
//and here retrieved cookies will be used
string result = await client.GetStringAsync(anyUrl);
Running a UWP app*
So I have an HttpClient and it's associated handler. I am making a request to a website, passing in specified headers, and using a specified CookieContainer, which is empty at the beginning of the request.
When I send the request, Fiddler shows extra cookies being sent that I have not added. Where are they coming from?
CookieContainer cookieJar = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler( );
handler.UseDefaultCredentials = false;
handler.CookieContainer = cookieJar;
handler.Proxy = null;
using (var client = new HttpClient(handler as HttpClientHandler))
{
HttpResponseMessage response = new HttpResponseMessage();
String loginUrl = Const.BUNGIE_LOGIN_URI + (provider == Const.XBOX_PLATFORM ? Const.XBOX_LOGIN : Const.PS_LOGIN);
client.BaseAddress = new Uri(loginUrl);
//client.Timeout = new TimeSpan(0, 0, 0, 0, 450);
client.DefaultRequestHeaders.Add("Referer", "http://www.bungie.net");
client.DefaultRequestHeaders.Add("User-Agent", Const.USERAGENT);
client.DefaultRequestHeaders.Add("X-API-Key", Const.X_API_KEY);
client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
client.DefaultRequestHeaders.Add("Accept", "application/json");
client.DefaultRequestHeaders.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");
handler.AutomaticDecompression = DecompressionMethods.GZip;
response = client.GetAsync("").Result;
response.ReadCookies(); //adds cookies to cookieJar
}
What fiddler shows
Now, as the associated CookieContainer is empty before the request is made, where are these cookies coming from? Are they accessible? If I wanted the values from them, how would I obtain them?
Edit: Where are they being added to my HttpClient request from? Does HttpClient have a common CookieContainer / cache? I have two separate HttpClient instances, and when Client (A) makes a request, it received a "set-cookie" header back, setting a "bungled" cookie.
Later on, a separate instance of HttpClient, Client (B), makes a request to the same website, sending the cookie set within Client (A).
I did not explicitly append this cookie to Client (B)'s request, so how is it being added?
On Windows 10 build 10240, System.Net.Http.HttpClient is a wrapper on top of Windows.Web.Http.HttpClient (more info here), and so, cookies now work little bit different.
To delete those extra cookies, you will need to delete the cookies using a HttpCookieManager:
HttpBaseProtocolFilter filter = new HttpBaseProtocolFilter();
HttpCookieManager cookieManager = filter.CookieManager;
foreach (HttpCookie cookie in cookieManager.GetCookies(uri))
{
cookieManager.DeleteCookie();
}
I'm using .NET HttpClient for PCL (2.2.15) for a common library across Windows Phone 8, Windows 8, and .NET 4.5. However, I don't see cookies being applied to requests.
I'm setting up the HttpClient and HttpClientHandler using the following code. HttpClient is a property on the wrapping class.
CookieContainer = new CookieContainer();
var handler = new HttpClientHandler
{
AllowAutoRedirect = true,
CookieContainer = CookieContainer,
Credentials = client.CredentialCache,
UseCookies = true,
AutomaticDecompression =
DecompressionMethods.Deflate | DecompressionMethods.GZip
};
HttpClient = new HttpClient(handler, false);
Requests are being sent using the following.
var response = await HttpClient.SendAsync(httpRequestMessage);
I'm able to peek into the CookieContainer on the handler when stepping through in Debug and am able to see the CookieContainer for the domain I'm working against with the expected cookie.
The cookie has the following values:
Path = '/'
Domain = '.staging3.api.com'
Value = [authentication_id]
Name = 'API_AuthId'
The subsequent request is roughly: https://staging3.api.com/api/v3/Items
I've also tried setting the cookie myself, however CookieContainer.GetCookieHeader(requestUri) returns an empty string.
Additional information:
If I modify the requestUri to https://www.staging3.api.com/api/v3/Items I'm able to get the cookie header from CookieContainer.GetCookieHeader(requestUri)
i am looking to get the data from any given domain names SSL certificate. For example I want to put in any website address e.g. "http://stackoverflow.com" and my code would firstly check if an SSL certificate exists. If it does then I want it to pull out the expiry date of the certificate. [ i am reading Domainnames from DB ]
Example :http://www.digicert.com/help/
i need to create a web service to check expiry date. how can i implement it?? - I have looked up loads of different things such as RequestCertificateValidationCallback and ClientCertificates etc.
I could be completely wrong (hence why I need help) but would I create a HTTPWebRequest and then somehow request the client certificate and specific elements that way?
i tried the example provided #SSL certificate pre-fetch .NET , but i am getting forbitten 403 error.
For this to work your project will need a reference to System.Security:
using System.Security;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
//Do webrequest to get info on secure site
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://mail.google.com");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
response.Close();
//retrieve the ssl cert and assign it to an X509Certificate object
X509Certificate cert = request.ServicePoint.Certificate;
//convert the X509Certificate to an X509Certificate2 object by passing it into the constructor
X509Certificate2 cert2 = new X509Certificate2(cert);
string cn = cert2.GetIssuerName();
string cedate = cert2.GetExpirationDateString();
string cpub = cert2.GetPublicKeyString();
//display the cert dialog box
X509Certificate2UI.DisplayCertificate(cert2);
.NET Core 2.1 - .NET 5
You can use HttpClientHandler and ServerCertificateCustomValidationCallback Property. (This class is available in .net 4.7.1 and above also).
var handler = new HttpClientHandler
{
UseDefaultCredentials = true,
ServerCertificateCustomValidationCallback = (sender, cert, chain, error) =>
{
/// Access cert object.
return true;
}
};
using (HttpClient client = new HttpClient(handler))
{
using (HttpResponseMessage response = await client.GetAsync("https://mail.google.com"))
{
using (HttpContent content = response.Content)
{
}
}
}
#cdev's solution didn't work for me on .NET Core 2.1. It seems HttpWebRequest is not completely supported on .NET Core.
Here is the function I'm using on .NET Core to get any server's X509 certificate:
// using System;
// using System.Net.Http;
// using System.Security.Cryptography.X509Certificates;
// using System.Threading.Tasks;
static async Task<X509Certificate2> GetServerCertificateAsync(string url)
{
X509Certificate2 certificate = null;
var httpClientHandler = new HttpClientHandler
{
ServerCertificateCustomValidationCallback = (_, cert, __, ___) =>
{
certificate = new X509Certificate2(cert.GetRawCertData());
return true;
}
};
var httpClient = new HttpClient(httpClientHandler);
await httpClient.SendAsync(new HttpRequestMessage(HttpMethod.Head, url));
return certificate ?? throw new NullReferenceException();
}
One thing to note is that you might need to set request.AllowAutoRedirect = False. Otherwise, if the server redirects HTTPS to HTTP, you won't be able to get the certificate from the HttpWebRequest object.
Recreating the HttpClient each time you want to make a request is very ineffective and may cause performance issues. Better to create a single readonly client for all the methods. More informations can be found here.
private readonly HttpClientHandler _handler;
private readonly HttpClient _client;
And this is my solution to getting certificate info:
Code inside constructor:
_handler = new HttpClientHandler {
ServerCertificateCustomValidationCallback = (sender, cert, chain, sslPolicyErrors) =>
{
sender.Properties.Add("Valid", sslPolicyErrors == System.Net.Security.SslPolicyErrors.None);
sender.Properties.Add("Errors", sslPolicyErrors);
return true;
}
};
_client = new HttpClient(_handler);
Then you can read all the variables by:
using var request = new HttpRequestMessage(HttpMethod.Get, "https://www.google.com/");
var response = await _client.SendAsync(request);
var isCertificateValid = (bool)request.Properties["Valid"];
I'm using VS2010 +.NET 4.0 + System.Net.Http (from Nuget).
For a reason which I don't manage to understand, the session cookie which I receive in my HttpResponseMessage is not automatically saved in the HttpClient CookieContainer.
Here is what my code looks like:
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
HttpClient client = new HttpClient(handler);
Uri site = new Uri("https://www.mywebsite.com");
var response1 = client.SendAsync(new HttpRequestMessage(HttpMethod.Get,site)).Result;
I can see in the response headers that I have the following:
Set-Cookie: JSESSIONID=FC8110E434C2C6DAB78B4E335024A639; Path=/member; Secure
However my cookie container remains empty ...why ?
Use this piece of code to retrieve cookies from response:
/// <summary>
/// Read web cookies
/// </summary>
public static CookieContainer ReadCookies(this HttpResponseMessage response)
{
var pageUri = response.RequestMessage.RequestUri;
var cookieContainer = new CookieContainer();
IEnumerable<string> cookies;
if (response.Headers.TryGetValues("set-cookie", out cookies))
{
foreach (var c in cookies)
{
cookieContainer.SetCookies(pageUri, c);
}
}
return cookieContainer;
}
I guess the problem is that your cookies are secure. The problem is that, CookieContainer won't send secure cookies back to the server in subsequent HTTP requests. It might be a bug, or maybe it has some reasons behind it.
A workaround is to re-add the cookie to CookieContainer manually. This way, cookie would be sent back in HTTP request header, as no secure would be defined when you send cookies back to the server.
See this article for more information.
I had problem with cookies because of case difference in path. I logged in to /mysite/login, and set cookie for mysite, but then redirect to /MySite, and HttpClient suddenly discard all the cookies! When I changed adress to /MySite/login all become ok.
Perhaps the problem is that your request Url path is to the root of the web site ("/"), but your Set-Cookie header indicates a Path of "/member".
Make sure you are automatically decompressing the response:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
Example:
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler {
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
};
var httpClient = new HttpClient(clientHandler);
var httpRequest = new HttpRequestMessage(HttpMethod.Get, "https://example.com/");
var response = httpClient.Send(httpRequest);
response.Headers.TryGetValues("set-cookie", out var cookies);
string test = (new StreamReader(response.Content.ReadAsStream()).ReadToEnd());
Edit
This also populates my cookieContainer automatically. On both a httpClient.GetAsync("https://example.com/") or SendAsync(httpRequest)