At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
If I use a regular url it works fine and if I use google domains update url I get 401 error. This is my first try on C# application.
HttpWebRequest request = WebRequest.Create("https://UUUUUUUUUUUUU:PPPPPPPPPPPPP#domains.google.com/nic/update?hostname=subdomain.example.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.102 Safari/537.36 Viv/1.97.1246.7";
request.UseDefaultCredentials = true;
request.PreAuthenticate = true;
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
//responseddns = responseText;
MessageBox.Show(responseText);
}
If I use http://example.com/getip.php it works fine I can see the output.
you cannot use
> `CredentialCache.DefaultCredentials;`
since the url is the domain.google.com's domain.
You need to enter your google credentials or else directly use
http://example.com/getip.php as u did before
This issue has kept me busy for days now, so I might just as well post it here... searching in other topics has brought me far, but I'm not there yet.
Background
My job is to automate tests in the browser: I make use of Selenium and .NET to handle this. One of our clients uses Active Directory for logging in on the web application. Logging in again for every test takes a lot of time (a new session is started for each test), so I'm trying to speed this phase up by writing a class that will login beforehand, without the browser.
By using Fiddler, I managed to reproduce most of the webrequests in code. Still, I don't work with custom web requests often, so I easily overlook possible issues. Something goes wrong in the last step, where I try to retrieve the AppServiceAuthSession cookie. I'll use a picture to illustrate:
On the left, there is the web request I make using my code. On the right, there is the web request I get when using the browser.
The Nonce cookie has its value, headers are correct... the only difference here is the "Expect" header instead of the "Connection" header, but that happens because I disabled the "AllowAutoRedirect" property for the request (I don't get to see this request in Fiddler if I keep it enabled).
Next, here comes the content of both requests, since that's hard to show by using the Fiddler interface.
Code request:
code=AQABAAIAAABHh4kmS_aKT5XrjzxRAtHzZ08Y3xv2A4HxumqfAbR5iIBFEbbm6oYUq5sdHF72ybOkXFz33XVW5uXvuq-Tiz43zQ7y08bAhGH-aR_3HEpugBMAX_Op4O0Z4RjMKOLyd5b-VNVhjqv_CRmgHTEhYnxubjNyaG-5Vsuo7YRa3jXFybBheb73Erp7CDYFCQ-dFjgNAEHEbzrfE64dGVvHCCr-rvpHCTJfW55eYE9l68d7rryVfqC8dX7wplr7A3G8ySS84Tb9_hYxA61L_fQEm22NehWGNKw_MwXWNsIONtwiRd5LQD95iXYwjCw9J1p-WoZw9p6228r8YOuBVJA0k22O_zjqvKGUkVEyVWvAZZvDSO26_m-uuz7ql52ZTfBhmOfF8iXfdgeUPUqSXmTmNtL0LAOgiYKt4DyH1TbgAN2ox-sVcFp0k7bmjE81h0yiY8RgR-a2aONvuIpSR1Dx7VG-1wkllQAkYGV8zMoahs0EVRiDdOEQwwvkegALZNaU_ROKkbxj-H71h7lffkLlZoeM1lQFwgcVyB1_kV04iZKg2x5-VxHRYCDCNdeaxuaZD5dOrxIh_Y6SfQK-VBWsDB1C_i76QQpFI2c-7oxFNn7RLCU5gw7iv8k31ry2-hKSfhcU6LtN--t-eW73ADu0cW9juh1zzgzXAPXz0FqNyh07SyRawPqir9jdZ4GlgU4r-Imm41n4b7LH_nYixE1q-JNTm7TlawW3Xe7i28QMx95wPGlqyYl0PAo0VMwMPgoG3Zi-5BiEFi31qOxCMJNbXWi-YOMMcV3W5-sNbYA44mOUoYnz0hny-PlvKFwTL9RLBxkgAA&
id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyIsImtpZCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyJ9.eyJhdWQiOiI4NmJiNjljZC04OTQ4LTQ2ZmEtOWQ3NS05YWRiZWRmODI1NWYiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIvIiwiaWF0IjoxNTIxNzM2NTM1LCJuYmYiOjE1MjE3MzY1MzUsImV4cCI6MTUyMTc0MDQzNSwiYWlvIjoiQVRRQXkvOEdBQUFBNXhNTk10SUJybUI1Z2dTN0xRNWxmSUNBMnY3RGpBRXIyUzgwS3dBa0VlMlpacVhNQW1hMnE0eG5xeDJ5aGxNZSIsImFtciI6WyJ3aWEiXSwiY19oYXNoIjoiUlUxOHNDNjJldTUyVnd0ZG9CTEItUSIsImVtYWlsIjoiUnViZW4udGVyLk1hdGVuQG1hY2F3Lm5sIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvMDViYWVkNGQtZTA0OS00NWM3LWI0N2QtYzk5YzY2ZmRiNTg5LyIsImluX2NvcnAiOiJ0cnVlIiwiaXBhZGRyIjoiODIuMjAxLjExMi40IiwibmFtZSI6IlJ1YmVuIHRlciBNYXRlbiIsIm5vbmNlIjoiYmI2NDU1ZGQzODJkNDRmNGIyNDAwYThjYzY3NmNmNTZfMjAxODAzMjEwOTU3MzAiLCJvaWQiOiI2YjgzMGMyNi1hZGE4LTQyY2MtOTQ3MS1kMzU5NTc1N2IwNDciLCJyb2xlcyI6WyJDb250ZW50RWRpdG9yIl0sInN1YiI6InJIVlJfRFZsWjltN2V5LS02RVE4LUFGY1FVZ3VWa3VRZVdnR3ZVeGdmbE0iLCJ0aWQiOiIxYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIiLCJ1bmlxdWVfbmFtZSI6IlJ1YmVuLnRlci5NYXRlbkBtYWNhdy5ubCIsInV0aSI6InpoRTFLWVRhWGtxS1dWWThqQlFaQUEiLCJ2ZXIiOiIxLjAifQ.xcuSefb-pr21z53eoirQTj3nptW95rQ0RnsxkF6B436Z16bI826UK4wYLvjGl8NrXYV1-sJhlUTUuX31p90GxJthUV1tbzsvFUwNk8f9DJgKls-31X78fNbChkuNDIS9Tj6Lj0cPOlZaC6U8TXYaNsef8qURtvtXzPt7Td1t6rfvuJL29pnow0We7u-eqUR8_TDD9PzY82BdYSbfIn4X-cOw4F79O-OAce9zRebX3V1Cr53gizMUIKKDWlS8fdwKnpdluAgZup-r6RJa7c_ui0NB2REao-PxixkvO-PuCtEfLhRnbRAprVa6ggMlCrtN6Uf7EmLRGLNBGHbl379vgg&
state=redir%3d%252Flogin%252Fcheck&
session_state=4dcbbba5-e8a5-4ad7-9ba9-28f6af570ead
Browser request:
code=AQABAAIAAABHh4kmS_aKT5XrjzxRAtHzZT_hGoYPGfSq8uoVwxawNGgqfV8j3-WdO_yha1uD2rspKWZmKjbV4j3YQmg4IQOdNBKzWDYTAST-Shag2D2Ih9EHMThyu7ncE_XRSBDOb_f05q3WCKYtF5ki5VVFcd8Li029KFwEjjutGw4k-_GfPe1rym5CXRvqQzj0wGUN4Q3ndwqrcNXsdvGG6wEXbiTJ8_3aSyuAAsHBMr1l23fjJRe-CKfr7EqKDZLzm_2H6L0MODy_mWkrD715CjU2_mUb1YQ__6MZvhHss41BFPcL3zyK2oh7Z668dJMEK495MhKFt-NPUFUPXsUNwK1fP4F-tdmBIY39BL9vDxzynmcF5A63yb8q3m0CgJOZ0i1aobe1iEX75sNXjIa2385yFw08lmkVJFj9GccvIE5ak4Zi6sdjWhEtS9TyKXc5uBtKxJCYIy52avI6bUAyc5-Xq0amik0Q75wuLK0bSoeIwBofKuJnoHfl1o3_7N2uSZ_2dJx5zPW9VxRz4e6u-mkmDu9F5lK42zo3al8c03zKHhLkTVRx08JV7JZuqpLznz5JFEHp1QUq-KeYNdMXsngPZW3VEegSyu6VGYwvbmL07GYTZ3KWi-dGxej88KvobZF3O40rfISThS4gVpnFoH7UQs_RE_TEjRvUraFBUhjE-1EUd7WXl6NsxQwGO0Uw7W87ZqlY4G-JhN39MOdm4v9UZELmeZph87PWqqy9-6h1pK8wrH8S98HqV1q3DUR68Ue-zCSvV9hgxrkSnPtFDTfCgtOvpNj14GUesMi8L0XPa6DsO0hj27_A1T0STGSHMGg8QMcgAA&
id_token=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyIsImtpZCI6IkZTaW11RnJGTm9DMHNKWEdtdjEzbk5aY2VEYyJ9.eyJhdWQiOiI4NmJiNjljZC04OTQ4LTQ2ZmEtOWQ3NS05YWRiZWRmODI1NWYiLCJpc3MiOiJodHRwczovL3N0cy53aW5kb3dzLm5ldC8xYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIvIiwiaWF0IjoxNTIxNzExMTIxLCJuYmYiOjE1MjE3MTExMjEsImV4cCI6MTUyMTcxNTAyMSwiYWlvIjoiQVRRQXkvOEdBQUFBT1YreEZOcjAwR3R6TG9jdkZaMVZ0SjlIRkYyUjc2VWE4d1V0eGlISkpienU4em1nN2xtRzdPT3ZCelZ6VkpyeiIsImFtciI6WyJ3aWEiXSwiY19oYXNoIjoieThoZUw5RHh3b3ZkUV9IbmtTYzlqZyIsImVtYWlsIjoiUnViZW4udGVyLk1hdGVuQG1hY2F3Lm5sIiwiaWRwIjoiaHR0cHM6Ly9zdHMud2luZG93cy5uZXQvMDViYWVkNGQtZTA0OS00NWM3LWI0N2QtYzk5YzY2ZmRiNTg5LyIsImluX2NvcnAiOiJ0cnVlIiwiaXBhZGRyIjoiODIuMjAxLjExMi40IiwibmFtZSI6IlJ1YmVuIHRlciBNYXRlbiIsIm5vbmNlIjoiMDk5MmE5NjQ1N2JiNDA3ZTgxZTYyZjE0YzA0ODg3ZTVfMjAxODAzMjIwOTQxNTIiLCJvaWQiOiI2YjgzMGMyNi1hZGE4LTQyY2MtOTQ3MS1kMzU5NTc1N2IwNDciLCJyb2xlcyI6WyJDb250ZW50RWRpdG9yIl0sInN1YiI6InJIVlJfRFZsWjltN2V5LS02RVE4LUFGY1FVZ3VWa3VRZVdnR3ZVeGdmbE0iLCJ0aWQiOiIxYWNhOGFmYy01YTZlLTRkYWUtOTRhNC0xODIwZDgwNGMxZWIiLCJ1bmlxdWVfbmFtZSI6IlJ1YmVuLnRlci5NYXRlbkBtYWNhdy5ubCIsInV0aSI6IlRFUGhmbG53dGs2YTFKTVJEcHNVQUEiLCJ2ZXIiOiIxLjAifQ.eT25sB0q7a5nEPBS5z2WVFHohBdzgioj6rd5DOMlb30dp1mIAVgvRZxSJhybB6RuHOl86LTkcyH4qzXl8uQ9fq8hapZUdJ2sDDqvk-PmcIWp_Ar6Gyo--4DtUxu7quIe-JaP_HyN_lsNbm0Gxut2VkZuHmCaiBxRriosP2FRg2GyKO75P6V7r0EPmQnS_4HA0WhEUmqNBzrZnbMV4rkc4FkLoMSLgJN69XdBMCQfsnhVcDWFPGj_Q79vlSbSq17Y7EPxn5phkrLGfZGMQj6CNHSzeT76LxR4txUu16QEQLLHckw2aSsESQ8SPFXwk794jEuyFIoFPrGg75hv4XxC7Q&
state=redir%3D%252Flogin%252Fcheck&
session_state=d897cb81-a5db-41f2-8852-8c18a82d36ab
Issue
As visible in the picture, the web request gets a response without AppServiceAuthSession and is redirected to the login.windows.net location. The normal browser request is different: it is successful at this point.
Question
Does anyone have a clue on what goes wrong here, or on how to debug it successfully? I triple-checked the input of the previous requests as well, but they all seem identical to the ones given to the browser. Even just a pointer in general might help... I'm getting a bit lost here. If more info is needed, just let me know. Thanks in advance!
Extra information
The code, in case it helps (censored where needed):
class Login
{
static CookieContainer cookiesNonce = new CookieContainer();
static CookieContainer cookiesA = new CookieContainer();
static CookieContainer cookiesB = new CookieContainer();
public static void DoLogin()
{
// CLIENTNAME-api, request for login with redirect
var url = #"https://CLIENTNAME-api.CLIENTNAME-dev.nl/.auth/login/aad?post_login_redirect_url=/login/check";
var request = WebRequest.Create(url) as HttpWebRequest;
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.CookieContainer = new CookieContainer();
request.KeepAlive = true;
request.AllowAutoRedirect = false;
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesNonce.Add(response.Cookies);
cookiesA.Add(response.Cookies);
request = WebRequest.Create(response.Headers[HttpResponseHeader.Location]) as HttpWebRequest;
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.KeepAlive = true;
request.Host = #"login.windows.net";
request.CookieContainer = cookiesA;
}
string clientrequestid = "";
string authorizeUrl = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesA = new CookieContainer();
cookiesA.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
var jsonRegEx = new Regex(#"\$Config=(.*?)(?<=\});");
var temp = jsonRegEx.Match(html).Groups[1].ToString() ;
var dict = JObject.Parse(temp);
var hpgrequestid = response.Headers["x-ms-request-id"];
var canary = GetValue(dict, "apiCanary");
clientrequestid = GetClientRequestId(dict);
authorizeUrl = response.Headers[HttpResponseHeader.Location];
// Request to microsoft online by credentials
url = #"https://login.microsoftonline.com/common/GetCredentialType";
request = WebRequest.Create(url) as HttpWebRequest;
request.Method = #"POST";
request.CookieContainer = cookiesB;
request.KeepAlive = true;
request.Headers.Add("hpgrequestid", hpgrequestid);
request.Headers.Add("canary", canary);
request.Headers.Add("client-request-id", clientrequestid);
request.Headers.Add("hpgact", "1800");
request.Headers.Add("hpgid", "1104");
request.Referer = response.ResponseUri.ToString(); ;
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"{""username"":""user#COMPANYNAME.nl"",""isOtherIdpSupported"":true,""checkPhones"":false,""isRemoteNGCSupported"":false,""isCookieBannerShown"":true,""isFidoSupported"":false,""originalRequest"":""rQIIAZ2Qv08UQRzFd26PDaCJxBCD3RUQE83szM7N7s1eQoGCGjEhXAwQLch35wc33rK7zK6i_Ak2YGFjZSwvVlbGP4FGav8CYyOhsvTU-A_YvPeq93l5N_0ojPqLIsmyJJUKi5QLzBMDOFW9GKegMq2MYHFs3NXZOf7jvH3x5dX68Rs2OKYlHaPesGmquk_ISBeFw1DZ8G9S-nlY5CSEZ82Q5OWeLQiAIhLyPAM5-oTQGULfEBq3yITN41iprmCKc8MzxikFIWXSS6SJk11GI0G7LKJp3OvSr60rGyuTVvZbSmeP9EVrxpRuf7cq6-atf81pZd3yErv7BztxOdRyNPav_9u656Aahoe2UOVhHRa6-egvRiBBgJE4hkRjrkDjlAPHkWBUCcplpLNTf76sdGFVp3Klsbnu6H2w-VkbfW9fpn5_enp2zlvwOt7PNno_NfmrXrp08OHJ7dXX714ObnQD73SKkFuDxxUMDF2xxc7W_c2No3zdwc7q5kPxtLmztq22WP3i0ba992BtmfWjkwCdBMF54H2e-d-rfwE1""}");
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
// Request to sts.COMPANYNAME.nl
url = #"https://sts.COMPANYNAME.nl/adfs/ls/wia?client-request-id=" + clientrequestid + #"&wa=wsignin1.0&wtrealm=urn%3afederation%3aMicrosoftOnline&wctx=LoginOptions%3D3%26estsredirect%3d2%26estsrequest%3drQIIAZ2QPU8UQRjHd26PDZwmEEOIdldATDSzOzs7uze7CYkoqBETwsUAwYLMzgs33t7uOrt4ykewAQsbK2N5sbIyfgQaqSmtjI2EytJF4xewef5P9X_53bJ9108WaZSmUcwFpDGhkESKwVj0QhgzkUqhKA5DZa515sjP8_bF19frR29x_wgVaAJ6g7ouq8TzhjLPDWSldv9-Qr5w88xz2UE98LJiX-ceY8LjLMtSxoefATgF4DsAk5bXZJMwFCKgWBCiSIoJQoxyHvUirsJoDyOfogD7KA57ATprzW6sNK748hRGH8qL1owqzGivLKr6nb1gpNBmeQnf_xPbKB9IPpzYN_513TesHLhjnYtiXLm5rD_Ziz7jjDLFYcgiCYlgEsaEEehTjARFhPsyPbHni1LmWnRLUyidya4cMZ19s2fNQSrz0Z1RYzJuZp-2wY_2VWQn09OdOeu61bV-tcGHqYZgtXTl-cend1ffvH_Vvxk41smU593u75asr9CKzne2Hm5uHGbrhu2sbj6mz-p7a9tiC1cvn2zrB4_WlnHiHzvg2HHOHevLzP_CP-vMXyKFKIDY76I4CXESkN3f0&cbcxt=&username=user%40COMPANYNAME.nl&mkt=&lc=";
request = WebRequest.Create(url) as HttpWebRequest;
cookiesB = new CookieContainer();
request.CookieContainer = cookiesB;
request.KeepAlive = true;
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Referer = response.ResponseUri.ToString(); ;
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Credentials = CredentialCache.DefaultNetworkCredentials;
request.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
var doc = new HtmlDocument();
doc.LoadHtml(html);
var node = doc.DocumentNode.SelectSingleNode("//input[#name='wresult']");
var value = node.Attributes["value"].Value;
var decode = HttpUtility.HtmlDecode(value);
var wresult = HttpUtility.UrlEncode(decode);
node = doc.DocumentNode.SelectSingleNode("//input[#name='wctx']");
value = node.Attributes["value"].Value;
decode = HttpUtility.HtmlDecode(value);
var wctx = HttpUtility.UrlEncode(decode);
reader.Close();
// Logging in to microsoft online
url = #"https://login.microsoftonline.com/login.srf?client-request-id=" + clientrequestid;
request = WebRequest.Create(url) as HttpWebRequest;
request.CookieContainer = cookiesB;
request.Referer = response.ResponseUri.ToString(); ;
request.Method = #"POST";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.KeepAlive = true;
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Headers.Add("Origin", "https://sts.COMPANYNAME.nl");
request.Headers.Add("Cache-Control", "max-age=0");
request.ContentType = #"application/x-www-form-urlencoded";
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"wa=wsignin1.0&wresult=" + wresult + #"&wctx=" + wctx);
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
cookiesB = new CookieContainer();
cookiesB.Add(response.Cookies);
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
var doc = new HtmlDocument();
doc.LoadHtml(html);
reader.Close();
var state = GetValue(doc, "state");
var code = GetValue(doc, "code");
var idtoken = GetValue(doc, "id_token");
var session_state = GetValue(doc, "session_state");
// Callback to .auth/login
url = #"https://CLIENTNAME-api.CLIENTNAME-dev.nl/.auth/login/aad/callback";
request = WebRequest.Create(url) as HttpWebRequest;
request.Method = #"POST";
request.CookieContainer = new CookieContainer();
var cookie = cookiesNonce.GetCookies(new Uri("https://CLIENTNAME-api.CLIENTNAME-dev.nl"))["Nonce"];
request.CookieContainer.Add(cookie);
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
request.Headers.Add("Origin", "https://login.microsoftonline.com");
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.ContentType = #"application/x-www-form-urlencoded";
request.UserAgent = #"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36";
request.Accept = #"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
request.Referer = response.ResponseUri.ToString(); ;
request.Headers.Add("Accept-Encoding", #"gzip, deflate, br");
request.Headers.Add("Accept-Language", #"en-US,en;q=0.9");
request.AllowAutoRedirect = false;
var reqStream = request.GetRequestStream();
var buffer = new ASCIIEncoding().GetBytes(#"code=" + code + #"&id_token=" + idtoken + #"&state=" + state + "&session_state=" + session_state);
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
var appServiceAuthSession = response.Cookies["AppServiceAuthSession"].ToString();
var reader = new StreamReader(response.GetResponseStream());
string html = reader.ReadToEnd();
reader.Close();
}
}
private static string GetValue(HtmlDocument doc, string name)
{
var node = doc.DocumentNode.SelectSingleNode("//input[#name='" + name + #"']");
var value = node.Attributes["value"].Value;
var decode = HttpUtility.HtmlDecode(value);
var result = HttpUtility.UrlEncode(decode);
return result;
}
private static string GetValue(string key, string input, string delimiter)
{
var value = RegexHelper.GetValueForKey(input, key, delimiter);
return value;
}
private static string GetValue(JObject container, string key)
{
string value = (string)container[key];
return value;
}
private static string GetClientRequestId(JObject container)
{
var obj = container["desktopSsoConfig"];
var value = (string)obj["iwaEndpointUrlFormat"];
var content = value.Split('&');
var id = content[content.Length - 1].Replace("client-request-id=", "");
return id;
}
}
OpenID connect is intentionally unspecific about the nature of the user authentication. The premise is that the authentication service will render whatever authentication it needs to the user, and if this has been satisfactory, will issue a short lived code which is redirected to the application. The application then uses that code to obtain a token.
There are other flows for authenticating non-user facing application like services. That might not help you though if you need to emulate a user logon.
As you are on Selenium already, why not have it enter the test user credentials into the ui presented in a web browser window?
Not a real solution, but at least a useful workaround in our case:
Login by using the browser.
Navigate to the auth/me page.
Example: let's say the home url I had to use was https://company.company-dev.nl. I adjusted the url to https://company-api.company-dev.nl/.auth/me and navigated there.
Retrieve the access token from the html.
To read the token, I used the following code:
private string ConvertToken(string html)
{
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
var timeout = DateTime.Now + TimeSpan.FromSeconds(5);
dynamic obj = null;
while (obj == null && DateTime.Now < timeout)
{
try
{
obj = (dynamic)JsonConvert.DeserializeObject(doc.DocumentNode.InnerText);
}
catch (Exception ex)
{
// Firefox occasionally bugs out if its JSON viewer kicks in too early, so use the according elements if needed
var element = Driver.FindElementById("rawdata-tab");
Driver.ClickElement(element);
var text = Driver.FindElementByClassName("data").Text;
obj = (dynamic)JsonConvert.DeserializeObject(text);
}
}
return obj[0]["access_token"].ToString();
}
Save the token.
When the next test starts, append the token to the home url.
{
NavigateToUrl(HomeUrl + #"login/" + Token);
}
Example: https://company.company-dev.nl/#/login/eyJ0eXAiOiandareallylonglineofcharactershere
This way, you can make only one test login, while the others will use the token generated by the first one. You have to play around with some other things as well (using the token while testing in parallel, dealing with token timeouts, etc.), but this is the core of the workaround.
Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}
I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...
The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();
First thing, the site and post.
https://saplic.receita.pb.gov.br/sintegra/SINf_ConsultaSintegra.jsp
Fullfil Field CNPJ with value For instance 34151100004209.
Works fine in Chrome and .Net 4.0 Httpwebrequest. But I cant debug in fiddler and cant make it work on .Net 3.5
I'm already using
System.Net.ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(customXertificateValidation);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
System.Net.ServicePointManager.Expect100Continue = false;
This is driving me insane. Any help will be greatly appreciated
To make things more clear, i'm having one problem that is happening on 2 differente places (Fiddler and .net 3.5)
The code i'm trying to run is.
CookieContainer CookCon = new CookieContainer();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb");
Request.Timeout = Configuration.TimeOut;
Request.Proxy = Configuration.Proxy;
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
Request.CookieContainer = new CookieContainer();
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
foreach (Cookie C in Response.Cookies)
{
CookCon.Add(C);
}
Response.Close();
}
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://saplic.receita.pb.gov.br/sintegra/");
Request.Referer = "http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb";
Request.Headers.Add("X-DevTools-Emulate-Network-Conditions-Client-Id", "C5E5561B-11F5-4D9E-A9D4-54A18E660D11");
Request.CookieContainer = CookCon;
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36";
string HTML = "";
string Encoding = "ISO-8859-1";
Request.Timeout = 20000;
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
Stream S = Response.GetResponseStream();
if (Response.ContentEncoding.ToLower().Contains("gzip"))
{
S = new GZipStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else if (Response.ContentEncoding.ToLower().Contains("deflate"))
{
S = new DeflateStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else
{
using (System.IO.StreamReader ResStream = new StreamReader(Response.GetResponseStream(), System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
S.Close();
S.Dispose();
Response.Close();
}
The error i'm getting is the "unexpeted EOF" excpetion when i try to make the post(GetResponse part of the code above).
If i try to surf the site on Chrome with Fiddler on, it wont return northing, just keeps on hold, hold, hold and nothing.
If I copy the same code to a new 4.0 project, it runs without problem.
If its not clear, please post the doubts, I'm trying to be as clear as possible.
Since no one helped, i just made a webservice using the .net 4.0 framework and I'm calling it from the 3.5 app.
Still no clue why this happens..