We Keep Coding

How to get request with JSESSIONID c#?

I've a problem with a C# HTTP GET request.
I send two requests to server:
The first request returns the value of JSESSIONID and it's work:
private string GetSessionId(){
{
string id = null;
try
{
var webRequest = WebRequest.Create("http://www.xxxxxxxxxxxxxxx.it/home") as HttpWebRequest;
var response = "";
webRequest.Date = DateTime.UtcNow;
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webRequest.Headers.Add("Accept-Language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de-DE;q=0.6,de;q=0.5");
webRequest.Connection = "keep-alive";
webRequest.Headers.Add("Cache-Control", "max-age=0");
webRequest.Host = "www.xxxxxxxxxxxxxxxx.it";
webRequest.Headers.Add("Upgrade-Insecure-Requests", "1");
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36";
if (webRequest != null)
{
webRequest.Method = "GET";
using (WebResponse resp = webRequest.GetResponse())
{
using (Stream s = resp.GetResponseStream())
{
using (StreamReader sr = new StreamReader(s))
{
id = resp.Headers["Set-Cookie"];
response = sr.ReadToEnd();
Console.WriteLine("Response1> " + resp.Headers);
}
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
return id;
}
The second request should return the server response to my search but it return an error "302 Moved Temporarily" or "Attention the session has expired".
private void RequestFromServer(string url)
{
string session = GetSessionId().Split(';')[0];
Console.WriteLine("Session> {0}",session);
try
{
HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest;
var response = "";
if (webRequest != null)
{
Uri target = new Uri(url);
webRequest.Method = "GET";
webRequest.Timeout = 20000;
webRequest.Date = DateTime.UtcNow;
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
webRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
webRequest.Headers.Add("Accept-Language", "it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,de-DE;q=0.6,de;q=0.5");
webRequest.Headers.Add("Cache-Control", "max-age=0");
webRequest.Connection = "keep-alive";
webRequest.ContentType = "application/x-www-form-urlencoded";
webRequest.Headers.Add("Cookie", session);
webRequest.Host = "www.xxxxxxxxxxxxxx.it";
webRequest.Headers.Add("Origin", "http://www.xxxxxxxxxxxx.it");
webRequest.Referer = "http://www.xxxxxxxxxxxxxxxx.it/ricerca/atto/contratti/originario?reset=true&normativi=false";
webRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36";
webRequest.Headers.Add("Upgrade-Insecure-Requests", "1");
var res = webRequest.GetResponse();
using (Stream s = res.GetResponseStream())
{
using (StreamReader sr = new StreamReader(s))
{
response = sr.ReadToEnd();
Console.Write(res.Headers);
Console.WriteLine(response);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
}
Can someone help me with this problem?

What you probably need here is a CookieContainer. Instead of
webRequest.Headers.Add("Cookie", session);
you should do something like
webRequest.CookieContainer = new CookieContainer();
Cookie cookie = new new Cookie("JSESSIONID", yourSessionIdHere);
// adjust domain accordingly
cookie.Domain = "www.xxxxxxxxxxxxxxx.it";
httpRequest.CookieContainer.Add(cookie);
Code is untested.
The MSDN site also states:
For security reasons, cookies are disabled by default. If you want to use cookies, use the CookieContainer property to enable cookies.
That means in order to use cookies you have to set the CookieContainer explicitly.

Interface-less login to login.microsoftonline.com

This issue has kept me busy for days now, so I might just as well post it here... searching in other topics has brought me far, but I'm not there yet.
Background
My job is to automate tests in the browser: I make use of Selenium and .NET to handle this. One of our clients uses Active Directory for logging in on the web application. Logging in again for every test takes a lot of time (a new session is started for each test), so I'm trying to speed this phase up by writing a class that will login beforehand, without the browser.
By using Fiddler, I managed to reproduce most of the webrequests in code. Still, I don't work with custom web requests often, so I easily overlook possible issues. Something goes wrong in the last step, where I try to retrieve the AppServiceAuthSession cookie. I'll use a picture to illustrate:
On the left, there is the web request I make using my code. On the right, there is the web request I get when using the browser.
The Nonce cookie has its value, headers are correct... the only difference here is the "Expect" header instead of the "Connection" header, but that happens because I disabled the "AllowAutoRedirect" property for the request (I don't get to see this request in Fiddler if I keep it enabled).
Next, here comes the content of both requests, since that's hard to show by using the Fiddler interface.
Code request:
code=AQABAAIAAABHh4kmS_aKT5XrjzxRAtHzZ08Y3xv2A4HxumqfAbR5iIBFEbbm6oYUq5sdHF72ybOkXFz33XVW5uXvuq-Tiz43zQ7y08bAhGH-aR_3HEpugBMAX_Op4O0Z4RjMKOLyd5b-VNVhjqv_CRmgHTEhYnxubjNyaG-5Vsuo7YRa3jXFybBheb73Erp7CDYFCQ-dFjgNAEHEbzrfE64dGVvHCCr-rvpHCTJfW55eYE9l68d7rryVfqC8dX7wplr7A3G8ySS84Tb9_hYxA61L_fQEm22NehWGNKw_MwXWNsIONtwiRd5LQD95iXYwjCw9J1p-WoZw9p6228r8YOuBVJA0k22O_zjqvKGUkVEyVWvAZZvDSO26_m-uuz7ql52ZTfBhmOfF8iXfdgeUPUqSXmTmNtL0LAOgiYKt4DyH1TbgAN2ox-sVcFp0k7bmjE81h0yiY8RgR-a2aONvuIpSR1Dx7VG-1wkllQAkYGV8zMoahs0EVRiDdOEQwwvkegALZNaU_ROKkbxj-H71h7lffkLlZoeM1lQFwgcVyB1_kV04iZKg2x5-VxHRYCDCNdeaxuaZD5dOrxIh_Y6SfQK-VBWsDB1C_i76QQpFI2c-7oxFNn7RLCU5gw7iv8k31ry2-hKSfhcU6LtN--t-eW73ADu0cW9juh1zzgzXAPXz0FqNyh07SyRawPqir9jdZ4GlgU4r-Imm41n4b7LH_nYixE1q-JNTm7TlawW3Xe7i28QMx95wPGlqyYl0PAo0VMwMPgoG3Zi-5BiEFi31qOxCMJNbXWi-YOMMcV3W5-sNbYA44mOUoYnz0hny-PlvKFwTL9RLBxkgAA&
id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyIsImtpZCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyJ9.eyJhdWQiOiI4NmJiNjljZC04OTQ4LTQ2ZmEtOWQ3NS05YWRiZWRmODI1NWYiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIvIiwiaWF0IjoxNTIxNzM2NTM1LCJuYmYiOjE1MjE3MzY1MzUsImV4cCI6MTUyMTc0MDQzNSwiYWlvIjoiQVRRQXkvOEdBQUFBNXhNTk10SUJybUI1Z2dTN0xRNWxmSUNBMnY3RGpBRXIyUzgwS3dBa0VlMlpacVhNQW1hMnE0eG5xeDJ5aGxNZSIsImFtciI6WyJ3aWEiXSwiY19oYXNoIjoiUlUxOHNDNjJldTUyVnd0ZG9CTEItUSIsImVtYWlsIjoiUnViZW4udGVyLk1hdGVuQG1hY2F3Lm5sIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvMDViYWVkNGQtZTA0OS00NWM3LWI0N2QtYzk5YzY2ZmRiNTg5LyIsImluX2NvcnAiOiJ0cnVlIiwiaXBhZGRyIjoiODIuMjAxLjExMi40IiwibmFtZSI6IlJ1YmVuIHRlciBNYXRlbiIsIm5vbmNlIjoiYmI2NDU1ZGQzODJkNDRmNGIyNDAwYThjYzY3NmNmNTZfMjAxODAzMjEwOTU3MzAiLCJvaWQiOiI2YjgzMGMyNi1hZGE4LTQyY2MtOTQ3MS1kMzU5NTc1N2IwNDciLCJyb2xlcyI6WyJDb250ZW50RWRpdG9yIl0sInN1YiI6InJIVlJfRFZsWjltN2V5LS02RVE4LUFGY1FVZ3VWa3VRZVdnR3ZVeGdmbE0iLCJ0aWQiOiIxYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIiLCJ1bmlxdWVfbmFtZSI6IlJ1YmVuLnRlci5NYXRlbkBtYWNhdy5ubCIsInV0aSI6InpoRTFLWVRhWGtxS1dWWThqQlFaQUEiLCJ2ZXIiOiIxLjAifQ.xcuSefb-pr21z53eoirQTj3nptW95rQ0RnsxkF6B436Z16bI826UK4wYLvjGl8NrXYV1-sJhlUTUuX31p90GxJthUV1tbzsvFUwNk8f9DJgKls-31X78fNbChkuNDIS9Tj6Lj0cPOlZaC6U8TXYaNsef8qURtvtXzPt7Td1t6rfvuJL29pnow0We7u-eqUR8_TDD9PzY82BdYSbfIn4X-cOw4F79O-OAce9zRebX3V1Cr53gizMUIKKDWlS8fdwKnpdluAgZup-r6RJa7c_ui0NB2REao-PxixkvO-PuCtEfLhRnbRAprVa6ggMlCrtN6Uf7EmLRGLNBGHbl379vgg&
state=redir%3d%252Flogin%252Fcheck&
session_state=4dcbbba5-e8a5-4ad7-9ba9-28f6af570ead
Browser request:
code=AQABAAIAAABHh4kmS_aKT5XrjzxRAtHzZT_hGoYPGfSq8uoVwxawNGgqfV8j3-WdO_yha1uD2rspKWZmKjbV4j3YQmg4IQOdNBKzWDYTAST-Shag2D2Ih9EHMThyu7ncE_XRSBDOb_f05q3WCKYtF5ki5VVFcd8Li029KFwEjjutGw4k-_GfPe1rym5CXRvqQzj0wGUN4Q3ndwqrcNXsdvGG6wEXbiTJ8_3aSyuAAsHBMr1l23fjJRe-CKfr7EqKDZLzm_2H6L0MODy_mWkrD715CjU2_mUb1YQ__6MZvhHss41BFPcL3zyK2oh7Z668dJMEK495MhKFt-NPUFUPXsUNwK1fP4F-tdmBIY39BL9vDxzynmcF5A63yb8q3m0CgJOZ0i1aobe1iEX75sNXjIa2385yFw08lmkVJFj9GccvIE5ak4Zi6sdjWhEtS9TyKXc5uBtKxJCYIy52avI6bUAyc5-Xq0amik0Q75wuLK0bSoeIwBofKuJnoHfl1o3_7N2uSZ_2dJx5zPW9VxRz4e6u-mkmDu9F5lK42zo3al8c03zKHhLkTVRx08JV7JZuqpLznz5JFEHp1QUq-KeYNdMXsngPZW3VEegSyu6VGYwvbmL07GYTZ3KWi-dGxej88KvobZF3O40rfISThS4gVpnFoH7UQs_RE_TEjRvUraFBUhjE-1EUd7WXl6NsxQwGO0Uw7W87ZqlY4G-JhN39MOdm4v9UZELmeZph87PWqqy9-6h1pK8wrH8S98HqV1q3DUR68Ue-zCSvV9hgxrkSnPtFDTfCgtOvpNj14GUesMi8L0XPa6DsO0hj27_A1T0STGSHMGg8QMcgAA&
id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyIsImtpZCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyJ9.eyJhdWQiOiI4NmJiNjljZC04OTQ4LTQ2ZmEtOWQ3NS05YWRiZWRmODI1NWYiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIvIiwiaWF0IjoxNTIxNzExMTIxLCJuYmYiOjE1MjE3MTExMjEsImV4cCI6MTUyMTcxNTAyMSwiYWlvIjoiQVRRQXkvOEdBQUFBT1YreEZOcjAwR3R6TG9jdkZaMVZ0SjlIRkYyUjc2VWE4d1V0eGlISkpienU4em1nN2xtRzdPT3ZCelZ6VkpyeiIsImFtciI6WyJ3aWEiXSwiY19oYXNoIjoieThoZUw5RHh3b3ZkUV9IbmtTYzlqZyIsImVtYWlsIjoiUnViZW4udGVyLk1hdGVuQG1hY2F3Lm5sIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvMDViYWVkNGQtZTA0OS00NWM3LWI0N2QtYzk5YzY2ZmRiNTg5LyIsImluX2NvcnAiOiJ0cnVlIiwiaXBhZGRyIjoiODIuMjAxLjExMi40IiwibmFtZSI6IlJ1YmVuIHRlciBNYXRlbiIsIm5vbmNlIjoiMDk5MmE5NjQ1N2JiNDA3ZTgxZTYyZjE0YzA0ODg3ZTVfMjAxODAzMjIwOTQxNTIiLCJvaWQiOiI2YjgzMGMyNi1hZGE4LTQyY2MtOTQ3MS1kMzU5NTc1N2IwNDciLCJyb2xlcyI6WyJDb250ZW50RWRpdG9yIl0sInN1YiI6InJIVlJfRFZsWjltN2V5LS02RVE4LUFGY1FVZ3VWa3VRZVdnR3ZVeGdmbE0iLCJ0aWQiOiIxYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIiLCJ1bmlxdWVfbmFtZSI6IlJ1YmVuLnRlci5NYXRlbkBtYWNhdy5ubCIsInV0aSI6IlRFUGhmbG53dGs2YTFKTVJEcHNVQUEiLCJ2ZXIiOiIxLjAifQ.eT25sB0q7a5nEPBS5z2WVFHohBdzgioj6rd5DOMlb30dp1mIAVgvRZxSJhybB6RuHOl86LTkcyH4qzXl8uQ9fq8hapZUdJ2sDDqvk-PmcIWp_Ar6Gyo--4DtUxu7quIe-JaP_HyN_lsNbm0Gxut2VkZuHmCaiBxRriosP2FRg2GyKO75P6V7r0EPmQnS_4HA0WhEUmqNBzrZnbMV4rkc4FkLoMSLgJN69XdBMCQfsnhVcDWFPGj_Q79vlSbSq17Y7EPxn5phkrLGfZGMQj6CNHSzeT76LxR4txUu16QEQLLHckw2aSsESQ8SPFXwk794jEuyFIoFPrGg75hv4XxC7Q&
state=redir%3D%252Flogin%252Fcheck&
session_state=d897cb81-a5db-41f2-8852-8c18a82d36ab
Issue
As visible in the picture, the web request gets a response without AppServiceAuthSession and is redirected to the login.windows.net location. The normal browser request is different: it is successful at this point.
Question
Does anyone have a clue on what goes wrong here, or on how to debug it successfully? I triple-checked the input of the previous requests as well, but they all seem identical to the ones given to the browser. Even just a pointer in general might help... I'm getting a bit lost here. If more info is needed, just let me know. Thanks in advance!
Extra information
The code, in case it helps (censored where needed):
class Login
{
static CookieContainer cookiesNonce = new CookieContainer();
static CookieContainer cookiesA = new CookieContainer();
static CookieContainer cookiesB = new CookieContainer();
public static void DoLogin()
{
// CLIENTNAME-api, request for login with redirect
var url = #"https://CLIENTNAME-api.CLIENTNAME-dev.nl/.auth/login/aad?post_login_redirect_url=/login/check";
var request = WebRequest.Create(url) as HttpWebRequest;
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.CookieContainer = new CookieContainer();
request.KeepAlive = true;
request.AllowAutoRedirect = false;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesNonce.Add(response.Cookies);
cookiesA.Add(response.Cookies);
request = WebRequest.Create(response.Headers[HttpResponseHeader.Location]) as HttpWebRequest;
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.KeepAlive = true;
request.Host = #"login.windows.net";
request.CookieContainer = cookiesA;
}
string clientrequestid = "";
string authorizeUrl = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesA = new CookieContainer();
cookiesA.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
var jsonRegEx = new Regex(#"\$Config=(.*?)(?<=\});");
var temp = jsonRegEx.Match(html).Groups[1].ToString() ;
var dict = JObject.Parse(temp);
var hpgrequestid = response.Headers["x-ms-request-id"];
var canary = GetValue(dict, "apiCanary");
clientrequestid = GetClientRequestId(dict);
authorizeUrl = response.Headers[HttpResponseHeader.Location];
// Request to microsoft online by credentials
url = #"https://login.microsoftonline.com/common/GetCredentialType";
request = WebRequest.Create(url) as HttpWebRequest;
request.Method = #"POST";
request.CookieContainer = cookiesB;
request.KeepAlive = true;
request.Headers.Add("hpgrequestid", hpgrequestid);
request.Headers.Add("canary", canary);
request.Headers.Add("client-request-id", clientrequestid);
request.Headers.Add("hpgact", "1800");
request.Headers.Add("hpgid", "1104");
request.Referer = response.ResponseUri.ToString(); ;
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"{""username"":""user#COMPANYNAME.nl"",""isOtherIdpSupported"":true,""checkPhones"":false,""isRemoteNGCSupported"":false,""isCookieBannerShown"":true,""isFidoSupported"":false,""originalRequest"":""rQIIAZ2Qv08UQRzFd26PDaCJxBCD3RUQE83szM7N7s1eQoGCGjEhXAwQLch35wc33rK7zK6i_Ak2YGFjZSwvVlbGP4FGav8CYyOhsvTU-A_YvPeq93l5N_0ojPqLIsmyJJUKi5QLzBMDOFW9GKegMq2MYHFs3NXZOf7jvH3x5dX68Rs2OKYlHaPesGmquk_ISBeFw1DZ8G9S-nlY5CSEZ82Q5OWeLQiAIhLyPAM5-oTQGULfEBq3yITN41iprmCKc8MzxikFIWXSS6SJk11GI0G7LKJp3OvSr60rGyuTVvZbSmeP9EVrxpRuf7cq6-atf81pZd3yErv7BztxOdRyNPav_9u656Aahoe2UOVhHRa6-egvRiBBgJE4hkRjrkDjlAPHkWBUCcplpLNTf76sdGFVp3Klsbnu6H2w-VkbfW9fpn5_enp2zlvwOt7PNno_NfmrXrp08OHJ7dXX714ObnQD73SKkFuDxxUMDF2xxc7W_c2No3zdwc7q5kPxtLmztq22WP3i0ba992BtmfWjkwCdBMF54H2e-d-rfwE1""}");
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
// Request to sts.COMPANYNAME.nl
url = #"https://sts.COMPANYNAME.nl/adfs/ls/wia?client-request-id=" + clientrequestid + #"&wa=wsignin1.0&wtrealm=urn%3afederation%3aMicrosoftOnline&wctx=LoginOptions%3D3%26estsredirect%3d2%26estsrequest%3drQIIAZ2QPU8UQRjHd26PDZwmEEOIdldATDSzOzs7uze7CYkoqBETwsUAwYLMzgs33t7uOrt4ykewAQsbK2N5sbIyfgQaqSmtjI2EytJF4xewef5P9X_53bJ9108WaZSmUcwFpDGhkESKwVj0QhgzkUqhKA5DZa515sjP8_bF19frR29x_wgVaAJ6g7ouq8TzhjLPDWSldv9-Qr5w88xz2UE98LJiX-ceY8LjLMtSxoefATgF4DsAk5bXZJMwFCKgWBCiSIoJQoxyHvUirsJoDyOfogD7KA57ATprzW6sNK748hRGH8qL1owqzGivLKr6nb1gpNBmeQnf_xPbKB9IPpzYN_513TesHLhjnYtiXLm5rD_Ziz7jjDLFYcgiCYlgEsaEEehTjARFhPsyPbHni1LmWnRLUyidya4cMZ19s2fNQSrz0Z1RYzJuZp-2wY_2VWQn09OdOeu61bV-tcGHqYZgtXTl-cend1ffvH_Vvxk41smU593u75asr9CKzne2Hm5uHGbrhu2sbj6mz-p7a9tiC1cvn2zrB4_WlnHiHzvg2HHOHevLzP_CP-vMXyKFKIDY76I4CXESkN3f0&cbcxt=&username=user%40COMPANYNAME.nl&mkt=&lc=";
request = WebRequest.Create(url) as HttpWebRequest;
cookiesB = new CookieContainer();
request.CookieContainer = cookiesB;
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Referer = response.ResponseUri.ToString(); ;
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Credentials = CredentialCache.DefaultNetworkCredentials;
request.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
var doc = new HtmlDocument();
doc.LoadHtml(html);
var node = doc.DocumentNode.SelectSingleNode("//input[#name='wresult']");
var value = node.Attributes["value"].Value;
var decode = HttpUtility.HtmlDecode(value);
var wresult = HttpUtility.UrlEncode(decode);
node = doc.DocumentNode.SelectSingleNode("//input[#name='wctx']");
value = node.Attributes["value"].Value;
decode = HttpUtility.HtmlDecode(value);
var wctx = HttpUtility.UrlEncode(decode);
reader.Close();
// Logging in to microsoft online
url = #"https://login.microsoftonline.com/login.srf?client-request-id=" + clientrequestid;
request = WebRequest.Create(url) as HttpWebRequest;
request.CookieContainer = cookiesB;
request.Referer = response.ResponseUri.ToString(); ;
request.Method = #"POST";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.KeepAlive = true;
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Origin", "https://sts.COMPANYNAME.nl");
request.Headers.Add("Cache-Control", "max-age=0");
request.ContentType = #"application/x-www-form-urlencoded";
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"wa=wsignin1.0&wresult=" + wresult + #"&wctx=" + wctx);
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB = new CookieContainer();
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
var doc = new HtmlDocument();
doc.LoadHtml(html);
reader.Close();
var state = GetValue(doc, "state");
var code = GetValue(doc, "code");
var idtoken = GetValue(doc, "id_token");
var session_state = GetValue(doc, "session_state");
// Callback to .auth/login
url = #"https://CLIENTNAME-api.CLIENTNAME-dev.nl/.auth/login/aad/callback";
request = WebRequest.Create(url) as HttpWebRequest;
request.Method = #"POST";
request.CookieContainer = new CookieContainer();
var cookie = cookiesNonce.GetCookies(new Uri("https://CLIENTNAME-api.CLIENTNAME-dev.nl"))["Nonce"];
request.CookieContainer.Add(cookie);
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
request.Headers.Add("Origin", "https://login.microsoftonline.com");
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.ContentType = #"application/x-www-form-urlencoded";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.Referer = response.ResponseUri.ToString(); ;
request.Headers.Add("Accept-Encoding", #"gzip, deflate, br");
request.Headers.Add("Accept-Language", #"en-US,en;q=0.9");
request.AllowAutoRedirect = false;
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"code=" + code + #"&id_token=" + idtoken + #"&state=" + state + "&session_state=" + session_state);
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var appServiceAuthSession = response.Cookies["AppServiceAuthSession"].ToString();
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
}
}
private static string GetValue(HtmlDocument doc, string name)
{
var node = doc.DocumentNode.SelectSingleNode("//input[#name='" + name + #"']");
var value = node.Attributes["value"].Value;
var decode = HttpUtility.HtmlDecode(value);
var result = HttpUtility.UrlEncode(decode);
return result;
}
private static string GetValue(string key, string input, string delimiter)
{
var value = RegexHelper.GetValueForKey(input, key, delimiter);
return value;
}
private static string GetValue(JObject container, string key)
{
string value = (string)container[key];
return value;
}
private static string GetClientRequestId(JObject container)
{
var obj = container["desktopSsoConfig"];
var value = (string)obj["iwaEndpointUrlFormat"];
var content = value.Split('&');
var id = content[content.Length - 1].Replace("client-request-id=", "");
return id;
}
}

OpenID connect is intentionally unspecific about the nature of the user authentication. The premise is that the authentication service will render whatever authentication it needs to the user, and if this has been satisfactory, will issue a short lived code which is redirected to the application. The application then uses that code to obtain a token.
There are other flows for authenticating non-user facing application like services. That might not help you though if you need to emulate a user logon.
As you are on Selenium already, why not have it enter the test user credentials into the ui presented in a web browser window?

Not a real solution, but at least a useful workaround in our case:
Login by using the browser.
Navigate to the auth/me page.
Example: let's say the home url I had to use was https://company.company-dev.nl. I adjusted the url to https://company-api.company-dev.nl/.auth/me and navigated there.
Retrieve the access token from the html.
To read the token, I used the following code:
private string ConvertToken(string html)
{
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
var timeout = DateTime.Now + TimeSpan.FromSeconds(5);
dynamic obj = null;
while (obj == null && DateTime.Now < timeout)
{
try
{
obj = (dynamic)JsonConvert.DeserializeObject(doc.DocumentNode.InnerText);
}
catch (Exception ex)
{
// Firefox occasionally bugs out if its JSON viewer kicks in too early, so use the according elements if needed
var element = Driver.FindElementById("rawdata-tab");
Driver.ClickElement(element);
var text = Driver.FindElementByClassName("data").Text;
obj = (dynamic)JsonConvert.DeserializeObject(text);
}
}
return obj[0]["access_token"].ToString();
}
Save the token.
When the next test starts, append the token to the home url.
{
NavigateToUrl(HomeUrl + #"login/" + Token);
}
Example: https://company.company-dev.nl/#/login/eyJ0eXAiOiandareallylonglineofcharactershere
This way, you can make only one test login, while the others will use the token generated by the first one. You have to play around with some other things as well (using the token while testing in parallel, dealing with token timeouts, etc.), but this is the core of the workaround.

C# how to Download a file with a cookie(login)

I am trying to login on this website. Currently I'm using this code:
private string Login(string username, string password)
{
string formUrl = "https://osu.ppy.sh/forum/ucp.php?mode=login";
string formParams = string.Format("username={0}&password={1}&affiliate-button=Sign%20In", username, password);
HttpWebRequest req = WebRequest.Create(formUrl) as HttpWebRequest;
req.ContentType = "application/x-www-form-urlencoded";
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36";
req.Method = "POST";
req.Referer = formUrl;
req.AllowAutoRedirect = true;
req.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
req.CookieContainer = new CookieContainer();
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream reqStream = req.GetRequestStream())
{
reqStream.Write(bytes, 0, bytes.Length);
}
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
#if DEBUG
foreach (Cookie c in resp.Cookies)
System.Diagnostics.Debug.WriteLine(c.Name + " = " + c.Value);
#endif
return resp.Headers["set-cookie"];
}
I always get 1 set-cookie back. When I login to this website with web browser(Chrome) I get back 10 + local storage. I have tried to use CookieCollection as follows:resp.Cookies. But i still got back only one cookie.
The other problem is, that i cant download this file with the cookie I get. I only get back the login page.
For downloading i use this function:
private void Download(string url, string fileName, string cookies)
{
try
{
HttpWebRequest req = WebRequest.Create(url) as HttpWebRequest;
req.Headers.Add(cookies);
req.Method = "GET";
req.Referer = "osu.ppy.sh";
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
using (Stream respStream = resp.GetResponseStream())
{
SaveToFile(respStream, fileName); //This reads the stream and writes to file
}
}
catch(Exception ex)
{
MessageBox.Show(ex.Message);
}
}
As I said before I used CookieCollection when I log in, I have also tried CookieContainer in this function like this:
req.CookieContainer = new CookieContainer();
req.CookieContainer.Add(cookies)
What am i doing wrong?

Can you try something like this to check if the cookie payload is comma separated -
public void GetAllCookies(HttpWebRequest req, HttpWebResponse resp)
{
var cookies = resp.Headers["set-cookie"];
foreach (var cookie in cookies.Split(','))
{
Match match = Regex.Match(cookie, "(.+?)=(.+?);");
if (match.Captures.Count == 0)
continue;
resp.Cookies.Add(
new Cookie(
match.Groups[1].ToString(),
match.Groups[2].ToString(),
"/",
req.Host.Split(':')[0]));
}
}

Application hangs on GetRequestStream() after first request

I've googled and searched here. Some suggest that streams were not being close, others suggested that it's a connection limit with ServicePointManager.DefaultConnectionLimit being set to 1. However, none of these seem to work.
My problem is, when i use this for the first time, it works:
using (var stream = request.GetRequestStream())
{
var data = Encoding.UTF8.GetBytes(post.ToString());
stream.Write(data, 0, data.Length);
}
When I use it a second time, it freezes. Yes, I'm disposing my stream, yes im Aborting and closing my response and requests.
Here is my entire code segment:
public string get_hash(string strUsername, string strPassword, string strUniverse)
{
// Request VAR
var request = (HttpWebRequest)WebRequest.Create("http://website.com/");
// Response VAR
var response = (HttpWebResponse)request.GetResponse();
// Cookie Var
var cookie = new CookieContainer();
ServicePointManager.DefaultConnectionLimit = 100;
request.Timeout = 10;
request = (HttpWebRequest)WebRequest.Create("http://website.com/main/login");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
request.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
request.Headers.Add("Accept-Language", "en-US,en,q=0.8");
request.Headers.Add("Cache-Control", "max-age=0");
request.ContentType = "application/x-www-form-urlencoded";
request.Host = "website.com";
request.Headers.Add("Origin", "http://website.com");
request.Referer = "http://website.com/";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.117 Safari/537.36";
request.Method = "POST";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookie.GetCookies(request.RequestUri));
// SET POST DATA HERE
var post = HttpUtility.ParseQueryString(string.Empty);
post.Add("uni", strUni);
post.Add("login", strUsername);
post.Add("pass", strPassword);
using (var stream = request.GetRequestStream())
{
var data = Encoding.UTF8.GetBytes(post.ToString());
stream.Write(data, 0, data.Length);
stream.Close();
stream.Dispose();
}
response = (HttpWebResponse)request.GetResponse();
string strSSID = "Failed";
if (response.StatusCode == HttpStatusCode.OK)
{
var data = string.Empty;
using (var sReader = new StreamReader(response.GetResponseStream()))
{
data = sReader.ReadToEnd();
sReader.Close();
sReader.Dispose();
}
string strSSIDurl = response.ResponseUri.ToString();
int intSSIDurlStart = strSSIDurl.IndexOf("PHPSESSID=") + 10;
strSSID = strSSIDurl.Substring(intSSIDurlStart);
}
request.Abort();
response.Close();
response.Dispose();
return strSSID;
}

you are not disposing the response from the first request.
var request = (HttpWebRequest)WebRequest.Create("http://website.com/");
var response = (HttpWebResponse)request.GetResponse(); // Not disposed
We had similar problem and disposing the response properly helped us.

Getting a page source after POST variables have been sent

I have some code that logs me into Facebook and it works fine as far as returning me the page you see right after you log in.
But I'm trying to request my actual profile after I log in rather than just my news feed.
So I'd have to send my email and password to the login page, then request my profile.
How can I keep the login data around while requesting my profile?
Here is what I've got
public static string logIn()
{
//get the cookies before you try to log in
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://www.facebook.com");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
response.Close();
//logging in
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create("https://www.facebook.com/login.php?login_attempt=1");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Post;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
//sending the email/password
byte[] byteArray = Encoding.ASCII.GetBytes("email=myemail#yahoo.com&pass=mypassword");
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream();
newStream.Write(byteArray, 0, byteArray.Length);
newStream.Close();
//returns the source of the page after logging in
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
StreamReader sr = new StreamReader(getResponse.GetResponseStream());
string source = sr.ReadToEnd();
cookies.Add(getResponse.Cookies);
//tries to get my profile source
//everything works fine until here
getRequest = (HttpWebRequest)WebRequest.Create("http://www.facebook.com/myprofile");
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies);
getRequest.Method = WebRequestMethods.Http.Get;
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
getResponse = (HttpWebResponse)getRequest.GetResponse();
sr = new StreamReader(getResponse.GetResponseStream());
source = sr.ReadToEnd();
getResponse.Close();
return source;
}
I've tried several ways of doing this and I have gotten it to return my profile, but it returns it as if I was not logged in and you can't actually view my profile (because it is set to private)
So I need my login info to be included somehow when requesting my profile.

This had me stumped for a while when I was making something similar. You need to make sure you're setting the proper cookies for your request after you've logged in. Try adding the new cookies to the CookieCollection object before applying them to your last request.
// ^ previous code ^
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
StreamReader sr = new StreamReader(getResponse.GetResponseStream());
string source = sr.ReadToEnd();
Uri facebookPage = new Uri("http://www.facebook.com"); // .GetCookies() only accepts a Uri
cookies.Add(request.CookieContainer.GetCookies(facebookPage));
getResponse.Close(); // Always make sure you close your responses
// V requesting your profile code V

You have to start a CookieCollection at the beginning and use it for all requests.
A CookieCollection updates itself with new cookies whenever you do a request so using cookies.Add() was also messing it up.
But I got it working fine now.