Edit: I apologize for the confusion: I'm attempting to scrape a site that I did not write; I'm not writing an ASP app. I'm only attempting to scrape one.
After making a post request to a login page, I attempt to read the cookies from another page. I do not see all the cookies I expect, however. My code is as follows:
// Downloads login cookies for subsequent requests
public async Task<CookieContainer> loginCookies()
{
var cookies = new CookieContainer();
var handler = new HttpClientHandler {
UseCookies = true,
AllowAutoRedirect = true,
CookieContainer = cookies
};
var client = new HttpClient(handler);
var loginUri = new Uri("https://connect.example.edu/login.aspx");
var credentials = new Dictionary<string, string> {
{"example$txtUsername", this.Username},
{"example$txtPassword", this.Password}
};
var formCredentials = new FormUrlEncodedContent(credentials);
await client.PostAsync(loginUri, content: formCredentials);
var pointsUri = new Uri("https://info.example.edu/");
Console.WriteLine("COOKIES:");
foreach (Cookie cookie in cookies.GetCookies(pointsUri))
Console.WriteLine($"{cookie.Name} --> {cookie.Value}");
return cookies;
}
I believe the error is a result of loginUri and pointsUri having different subdomains. The info I need to scrape exists at the pointsUri page, but the login exists at loginUri.
A cookie that I'm missing in particular is ASP.NET_SessionID.
Related
I'm migrating a WinForms App to a .Net Core (3.1) API and I've run into trouble with a feature which makes a call to a SSO-protected API within our network.
In the WinForms App, I had the following:
public async void Authenticate()
{
var wi = System.Security.Principal.WindowsIdentity.GetCurrent();
var wic = wi.Impersonate();
try
{
// Initial request to grab cookies and redirect endpoint
var apiEndPoint = $"https://{_host}{_apiPath}";
var cookieRedirectRequest = new HttpRequestMessage(HttpMethod.Get, apiEndPoint);
var response = await _httpClient.SendAsync(cookieRedirectRequest);
if (response != null)
{
var cookieContainer = new CookieContainer();
foreach (var cookie in response.Headers.GetValues("Set-Cookie"))
{
var parsedCookie = cookie.ToCookie();
parsedCookie.Domain = TARGET_DOMAIN;
cookieContainer.Add(parsedCookie);
}
var redirectURL = response.Headers.GetValues("Location").FirstOrDefault();
// Add the cookies to the client to continue the authentication
_httpClient = new HttpClient(new HttpClientHandler()
{
UseDefaultCredentials = true,
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer
});
// Second request to grab code and state values
var codeAndStateRequest = new HttpRequestMessage(HttpMethod.Get, redirectURL);
response = await _httpClient.SendAsync(codeAndStateRequest);
...
The initial call to the API redirects us to the SSO-authentication page. I use the cookies from that initial response to create my own request so that I can pass in the proper credentials and get back the appropriate SSO tokens (the response the SendSync call on the last line returns back the tokens), which I then use to authenticate on future requests to the API.
I'm trying to recreate this functionality in .NET Core using a Service Account, and instead of doing the WindowsIdentity impersonation, I'm trying to explicitly set the NetworkCredentials property of my HttpClientHandler:
handler = new HttpClientHandler()
{
UseDefaultCredentials = false,
Credentials = new NetworkCredential("svc_acct_username", "svc_act_pass", "mydomain"),
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer,
};
However, the response I'm getting back from my second request is a 401.
This is consistent with what I used to get in my original app -- I would initially get a 401 with a "WWW-Authenticate: Negotiate" header, but then there would be an automatic subsequent request where the network credentials were passed in and I'd get a 200. I'm not getting this second request with the creds in .Net Core.
I tried the Credentials Cache solution suggested here:
https://github.com/dotnet/runtime/issues/24490
But this has not worked.
Thanks in advance for any help you guys can give me. I'm quite stuck!
Quick Edit:
I can recreate the .Net Core behavior in my WinForms app by simply setting the Credentials property of the HttpClientHandler explicitly and setting UseDefaultCredentials = false. So, there's some magic happening by setting UseDefaultCredentials that seemingly can't be recreated when using explicit creds.
I got this working with help from a co-worker! The issue is that when you use explicit Network Credentials, you lose the "Negotiate" authentication type, which is what our SSO-endpoint requires. By using a CredentialCache object and setting the Negotiate auth-type explicitly, the call works!
var credCache = new CredentialCache { { new Uri("https://example.com"), "Negotiate", new NetworkCredential("svc_acct_user", "svc_acct_password", "mydomain") } };
// Add the cookies to the client to continue the authentication
handler = new HttpClientHandler()
{
UseDefaultCredentials = false,
Credentials = credCache,
AllowAutoRedirect = true,
ClientCertificateOptions = ClientCertificateOption.Automatic,
UseCookies = true,
CookieContainer = cookieContainer,
};
How to set cookies in the MS-Edge based Microsoft.Toolkit.Forms.UI.Controls.WebView?
I need to send an autentication token cookie to the website I'm navigating to.
What I've tried:
Passing a cookie header to the Navigate method: The header won't be passed to the website (verified by Fiddler). Other headers (like "MyCustomHeader" in the example below) are passed to the site though.
string cookieHeader = cookieContainer.GetCookieHeader(siteUri);
var headers = new Dictionary<string, string>();
headers.Add("Cookie", "MyAuthCookie=MyAuthToken; Domain=.somesite.net; Path=/");
headers.Add("MyCustomHeader", "MyCustomHeader-Value");
_browser.Navigate(siteUri, HttpMethod.Get, headers: headers);
Setting the cookie in CookieManager before calling WebView.Navigate:
var siteUri = new Uri("http://wwww.somesite.net/");
var filter = new Windows.Web.Http.Filters.HttpBaseProtocolFilter();
var cookieManager = filter.CookieManager;
var cookie = new Windows.Web.Http.HttpCookie("MyAuthCookie", siteUri.Host, "/");
cookie.Value = "MyAuthToken";
cookieManager.SetCookie(cookie);
webView.Navigate(siteUri);
This also does not work when calling NavigateWithHttpRequestMessage Microsoft.Toolkit.Win32.UI.Controls.Interop.WinRT.WebViewControlHost (via reflection) instead of WebView.Navigate.
It also does not work when requesting the same URL by HttpClient before calling WebView.Navigate:
using (var client = new Windows.Web.Http.HttpClient(filter) { })
{
var result = client.GetAsync(siteUri).AsTask().Result;
result.EnsureSuccessStatusCode();
}
webView.Navigate(siteUri);
That way, the cookie header is only sent with the HttpClient's request, but not with the subsequent WebView.Navigate's request. I guess that the reason for this could be the fact that WebView runs in it's own process.
Is there any way to pass the cookie to the website? Note that the cookie does not originate from the site. The authentication token is retrieved from some other system, and needs to be passed to the website.
Trying to call an API from a controller using HttpClient and the API does not recognize the user as authenticated and logged in. When calling the API from JS I have no issue. I noticed the HttpClient was only sending via HTTP 1.1 and so I upgraded to 2.0 settings the DOTNET_SYSTEM_NET_HTTP_USESOCKETSHTTPHANDLER flag but this made no difference. I have tried all combinations of the HttpClientHandler properties including UseCookies and the request is never authenticated.
using (var handler = new HttpClientHandler {UseDefaultCredentials = true})
{
using (var httpClient = new HttpClient(handler))
{
var response = httpClient.GetStringAsync(new Uri($"https://localhost:64366/api/")).Result;
}
}
Will move to token based auth in the future but for now would like to understand why there is a difference between calling the API from C# vs JS. This is all HTTPS on localhost using asp net core 2.2.
Difference between JS and C# is that browsers attach cookies automatically to requests and you have to attach cookies manually in C# as juunas mentioned.
To obtain and use authentication cookie you may use the following pattern
CookieContainer cookies = new CookieContainer(); //this container saves cookies from responses and send them in requests
var handler = new HttpClientHandler
{
CookieContainer = cookies
};
var client = new HttpClient(handler);
string authUrl = ""; //your auth url
string anyUrl = ""; //any url that requires you to be authenticated
var authContent = new FormUrlEncodedContent(
new List<KeyValuePair<string, string>> {
new KeyValuePair<string, string>("login", "log_in"),
new KeyValuePair<string, string>("password", "pass_word")
}
);
//cookies will be set on this request
HttpResponseMessage auth = await client.PostAsync(authUrl, authContent);
auth.EnsureSuccessStatusCode(); //retrieving result is not required but you will know if something goes wrong on authentication
//and here retrieved cookies will be used
string result = await client.GetStringAsync(anyUrl);
PROBLEM:
I'm trying to make external authentication work with default Web API template project. I used the following instructions to add support for external authentication services (FB/Google/Microsoft): http://www.asp.net/web-api/overview/security/external-authentication-services
Just for the record, I was able to make external authentication work with default SPA template project.
Also, new local user creation works fine.
The problem appeared once I tried to use my client app (WPF-based) to register user using external provider (e.g., FB).
For the record, I used these two articles as the starting point: http://leastprivilege.com/2013/11/26/dissecting-the-web-api-individual-accounts-templatepart-3-external-accounts/ and thread #21065648 here at Stack Overflow.
They really helped me to understand the entire logic.
here's the short overview of the steps I've done:
two windows, main and the one for external authentication provider, with embedded WebBrowser
the flow:
2.1. user opens an app, main window appears
2.2. user clicks on a button to get the list of all supported external authentication providers,
and here's what happens in the code:
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage loginResponse = await client.GetAsync("api/Account/ExternalLogins?returnUrl=%2F&generateState=true");
if (loginResponse.IsSuccessStatusCode)
{
var externalLoginProviders = await loginResponse.Content.ReadAsAsync<IEnumerable<AuthenticationProvider>>();
// cleaning resources
client.Dispose();
loginResponse.Dispose();
// obtained data is sent to UI
This leads to the list of External Authentication Providers, with name, url, and state. URL is relative, and is used as the redirect to the actual provider (e.g., FB, or Google).
2.3. As soon as the list with external authentication providers is populated, user can enter his/her email, and click on the "Login" button, which will lead to the following:
In second window, with embedded WebBrowser, the latter is Navigated to the provided URL obtained in the previous step. If user logs successfully into the selected provider (e.g., Facebook), and agrees to give my app (Facebook app) necessary permissions (profile), user is navigated back to our website, in the form:
(our base url) /#access_token=XXX&token_type=bearer&expires=YYY
This address is then parsed and address params (token, etc.) are saved in the structured form for later use.
Right after that I go to api/UserInfo to understand if user already logged on with this external authentication provider or not:
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// setting Bearer Token obtained from Auth provider
client.SetBearerToken(result.AccessToken);
// calling /api/Account/UserInfo
var userInfoResponse = await client.GetAsync("api/Account/UserInfo");
var userInfoMessage = userInfoResponse.Content.ReadAsStringAsync().Result;
var userInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<UserInfo>(userInfoMessage);
// cleaning resources
client.Dispose();
userInfoResponse.Dispose();
if (userInfo.hasRegistered == true)
{
// going to login
}
Right after that, assuming that our user didn't create a user based on his/her login with the given external authentication provider (e.g., FB), yet, we do the following:
var data = new Dictionary<string, string>();
data.Add("Email", this.externalUserEmailTextBox.Text);
var registerExternalUrl = new Uri(string.Concat(baseAddress, #"api/Account/RegisterExternal"));
var client = new HttpClient();
client.BaseAddress = new Uri(baseAddress);
// setting Bearer Token obtained from External Authentication provider
client.SetBearerToken(result.AccessToken);
var response = client.PostAsync(registerExternalUrl.ToString(), new FormUrlEncodedContent(data)).Result;
At this point, user should be created (I have zero changes on the service side apart of uncommenting provider's app id and app secret lines in Startup.Auth.cs).
Unfortunately, this doesn't happen at all. Instead, I get "Internal Server Error" which means that this line,
var info = await Authentication.GetExternalLoginInfoAsync();
brings null, and so system fails to Authentication correctly with the provided Bearer Token.
I can't understand why. It seems that something is broken in the framework, or I'm doing something wrong...
SOLUTION:
Thanks to #berhir, here's the solution:
In MainWindow.xaml.cs, define cookieContainer
CookieContainer cookieContainer;
In MainWindow_Loaded event handler, instantiate it:
// we create new cookie container
cookieContainer = new CookieContainer();
As soon as user asks app to show all external login providers, instantiate HttpClient using suggested code by #berhir:
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
HttpResponseMessage loginResponse = await client.GetAsync("api/Account/ExternalLogins?returnUrl=%2F&generateState=true");
As soon as list of external auth providers is obtained, the next step is to show second window, with embedded WebBrowser. There, you'll have to declare two WinAPI calls for cookies:
[DllImport("wininet.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern bool InternetSetCookie(string lpszUrlName, string lbszCookieName, string lpszCookieData);
[DllImport("wininet.dll", SetLastError = true)]
public static extern bool InternetGetCookieEx(string url, string cookieName, StringBuilder cookieData, ref int size, Int32 dwFlags, IntPtr lpReserved);
// and
private const Int32 InternetCookieHttponly = 0x2000;
/// <summary>
/// Gets the URI cookie container.
/// </summary>
/// <param name="uri">The URI.</param>
/// <returns></returns>
public static CookieContainer GetUriCookieContainer(Uri uri)
{
CookieContainer cookies = null;
// Determine the size of the cookie
int datasize = 8192 * 16;
StringBuilder cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(uri.ToString(), null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero))
{
if (datasize < 0)
return null;
// Allocate stringbuilder large enough to hold the cookie
cookieData = new StringBuilder(datasize);
if (!InternetGetCookieEx(uri.ToString(), null, cookieData, ref datasize, InternetCookieHttponly, IntPtr.Zero))
return null;
}
if (cookieData.Length > 0)
{
cookies = new CookieContainer();
cookies.SetCookies(uri, cookieData.ToString().Replace(';', ','));
}
return cookies;
}
This window with embedded browser is instantiated, and constructor accepts several params, including start URL (to external auth provider), end URL (resulting with "#access_token=..." or "error...", callback, and, even more important, with the original cookieContainer. We use InternetSetCookie WinAPI method to pass that original cookieContainer to WebBrowser's session:
// set cookies
var cookies = cookieContainer.GetCookies(baseAddress).OfType<Cookie>().ToList();
foreach (var cookie in cookies)
{
InternetSetCookie(startUrl, cookie.Name, cookie.Value);
}
So, once user succeeds with signing into the selected external auth provider (e.g., Facebook), updated cookieContainer (which includes cookies set in the first HttpClient call, as well as right within WebBrowser right after signing in to external auth provider) obtained using InternetGetCookieEx WinAPI call, is sent back to MainWindow.xaml.cs via callback:
if (this.callback != null)
{
var cookies = GetUriCookieContainer(e.Uri);
this.callback(new AuthResult(e.Uri, this.providerName, cookies));
}
There, we issue two new requests, to api/Account/UserInfo, and then to api/Account/RegisterExternal:
this.cookieContainer = result.CookieContainer; // where result is AuthResult containing the cookies obtained from WebBrowser's session
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// setting Bearer Token obtained from Auth provider
client.SetBearerToken(result.AccessToken);
// calling /api/Account/UserInfo
var userInfoResponse = await client.GetAsync("api/Account/UserInfo");
var userInfo = Newtonsoft.Json.JsonConvert.DeserializeObject<UserInfo>(userInfoMessage);
if (userInfo.hasRegistered == false)
{
var data = new Dictionary<string, string>();
data.Add("Email", this.externalUserEmailTextBox.Text);
var registerExternalUrl = new Uri(string.Concat(baseAddress, #"api/Account/RegisterExternal"));
var content = new FormUrlEncodedContent(data);
var response = client.PostAsync(registerExternalUrl.ToString(), content).Result;
// obtaining content
var responseContent = response.Content.ReadAsStringAsync().Result;
if (response != null && response.IsSuccessStatusCode)
{
MessageBox.Show("New user registered, with " + result.ProviderName + " account");
}
So, here we go. Cookies are used within the whole lifecycle, from first HttpClient request to the final moment when we register a new user using api/account/registerExternal.
It seems that you forgot to handle the cookies. The easiest way is to use the same HttpClient instance for all your requests or you can use the same CookieContainer.
var cookieContainer = new CookieContainer();
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer })
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
// send request
}
I'm using VS2010 +.NET 4.0 + System.Net.Http (from Nuget).
For a reason which I don't manage to understand, the session cookie which I receive in my HttpResponseMessage is not automatically saved in the HttpClient CookieContainer.
Here is what my code looks like:
CookieContainer cookies = new CookieContainer();
HttpClientHandler handler = new HttpClientHandler();
handler.CookieContainer = cookies;
HttpClient client = new HttpClient(handler);
Uri site = new Uri("https://www.mywebsite.com");
var response1 = client.SendAsync(new HttpRequestMessage(HttpMethod.Get,site)).Result;
I can see in the response headers that I have the following:
Set-Cookie: JSESSIONID=FC8110E434C2C6DAB78B4E335024A639; Path=/member; Secure
However my cookie container remains empty ...why ?
Use this piece of code to retrieve cookies from response:
/// <summary>
/// Read web cookies
/// </summary>
public static CookieContainer ReadCookies(this HttpResponseMessage response)
{
var pageUri = response.RequestMessage.RequestUri;
var cookieContainer = new CookieContainer();
IEnumerable<string> cookies;
if (response.Headers.TryGetValues("set-cookie", out cookies))
{
foreach (var c in cookies)
{
cookieContainer.SetCookies(pageUri, c);
}
}
return cookieContainer;
}
I guess the problem is that your cookies are secure. The problem is that, CookieContainer won't send secure cookies back to the server in subsequent HTTP requests. It might be a bug, or maybe it has some reasons behind it.
A workaround is to re-add the cookie to CookieContainer manually. This way, cookie would be sent back in HTTP request header, as no secure would be defined when you send cookies back to the server.
See this article for more information.
I had problem with cookies because of case difference in path. I logged in to /mysite/login, and set cookie for mysite, but then redirect to /MySite, and HttpClient suddenly discard all the cookies! When I changed adress to /MySite/login all become ok.
Perhaps the problem is that your request Url path is to the root of the web site ("/"), but your Set-Cookie header indicates a Path of "/member".
Make sure you are automatically decompressing the response:
HttpClientHandler handler = new HttpClientHandler()
{
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
};
Example:
var cookieContainer = new CookieContainer();
var clientHandler = new HttpClientHandler {
AllowAutoRedirect = true,
UseCookies = true,
CookieContainer = cookieContainer,
AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
};
var httpClient = new HttpClient(clientHandler);
var httpRequest = new HttpRequestMessage(HttpMethod.Get, "https://example.com/");
var response = httpClient.Send(httpRequest);
response.Headers.TryGetValues("set-cookie", out var cookies);
string test = (new StreamReader(response.Content.ReadAsStream()).ReadToEnd());
Edit
This also populates my cookieContainer automatically. On both a httpClient.GetAsync("https://example.com/") or SendAsync(httpRequest)