Trying to use C# to POST HttpWebRequest into airframes.org for aircraft information. This is the code I use for many other POST request with no problems (used with other urls), but it / I am not able to load the airframes.org page using ICAO24 number (A64294).
var cookies = new CookieContainer();
ServicePointManager.Expect100Continue = false;
var request = (HttpWebRequest)WebRequest.Create("http://www.airframes.org/");
request.CookieContainer = cookies;
request.Method = "POST";
request.KeepAlive = true;
request.ContentType = "application/x-www-form-urlencoded";
using (var requestStream = request.GetRequestStream())
using (var writer = new StreamWriter(requestStream))
{
writer.Write("reg=&selcal=&icao24=A64294&submit=submit");
}
using (var responseStream = request.GetResponse().GetResponseStream())
using (var reader = new StreamReader(responseStream))
{
var result = reader.ReadToEnd();
Console.WriteLine(result);
}
Note that the site has a no-bots policy, which is why your request won't work.
That being said, if you still wish to request the page, adding a user-agent string (so your request looks like it came from a browser) does the trick:
request.UserAgent = "Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1";
It is usually a good idea to respect the policies of a site. The code above is merely for educational purposes.
Related
I'm sending a request but the cookies are not being stored in my container for the response?
Example of code -
string uri = "https://www.localhost.com/"
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.ContentType = "application/x-www-form-urlencoded";
CookieContainer cookies = new CookieContainer();
request.CookieContainer = cookies;
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
using (Stream stream = request.GetRequestStream())
{
byte[] bytes = new UTF8Encoding().GetBytes(s);
stream.Write(bytes, 0, bytes.Length);
}
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream2 = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream2, Encoding.UTF8))
{
str6 = reader.ReadToEnd();
}
}
}
as you can see I've implemented
CookieContainer cookies = new CookieContainer();
request.CookieContainer = cookies;
Which should store the cookies from the request within the container for further usage right within the response right? Like if the response needs the request cookies to load the page.
I suggest using HttpClient as it handles cookies for you and is just generally easier to work with. Also note that even though it’s disposable you generally should use the same HttpClient throughout your application. See: https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/
I have a C# code (it's a Web Application, that's hosted on IIS) where I use HttpWebRequest to get HttpWebResponse. There I make request to any website & get the response as string, then I analysis the response string. But recently I get the response where JavaScript performs data fetching after page is loaded in browser.
I tried to debug this in firebug & saw that at bottom of response there's a JavaScript function that updates the dom elements after pageload. Is there any way that I could do the same in my C# code. I have searched on net about this found not solution till now.
Following is the code I am using:
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
cookieList.Add(cookie);
}
string postData = string.Format("username=" + txtUserID.Text + "&password=" + txtPwd.Text + "&url=https://example.com/&game=");
byte[] postBytes = Encoding.UTF8.GetBytes(postData);
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("https://login.example.com/Login/authenticate");
req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0";
req.KeepAlive = true;
req.AutomaticDecompression = DecompressionMethods.GZip;
////set the cookie
req.CookieContainer = new CookieContainer();
foreach (Cookie cook in cookieList)
{
Cookie cookie = new Cookie();
cookie.Name = cook.Name;
cookie.Value = cook.Value;
cookie.Domain = cook.Domain;
cookie.Expires = DateTime.Now.AddDays(10);
req.CookieContainer.Add(cookie);
}
req.Headers.Add("Accept-Encoding", "gzip, deflate");
req.Headers.Add("Accept-Language", "en-GB,en-US;q=0.8,en;q=0.6");//en-GB,en-US;q=0.8,en;q=0.6
req.Method = "POST";
req.Host = "login.example.com";
req.Referer = "https://login.example.com/Login/logout";
req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
req.ContentType = "application/x-www-form-urlencoded;";
req.ContentLength = postBytes.Length;
//getting the request stream and posting data
StreamWriter requestwriter = new StreamWriter(req.GetRequestStream(), System.Text.Encoding.ASCII);
requestwriter.Write(postData);
requestwriter.Close();
HttpWebResponse myHttpWebResponse = (HttpWebResponse)req.GetResponse();
Stream responseStream = myHttpWebResponse.GetResponseStream();
StreamReader myStreamReader = new StreamReader(responseStream, Encoding.ASCII);
string responseString = myStreamReader.ReadToEnd();
myStreamReader.Close();
responseStream.Close();
myHttpWebResponse.Close();
I finally got the easy solution to my need. following is the link that I followed:
Link to tutorial
Following is the code that'll get the results:
First you'll need to import following:
using System.Drawing;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;
using System.Text.RegularExpressions;
using System.IO;
using HtmlAgilityPack;
Now the code:
var options = new PhantomJSOptions();
var driver = new PhantomJSDriver(options);
driver.Manage().Window.Size = new Size(1360, 728);
var size = driver.Manage().Window.Size;
driver.Navigate().GoToUrl("https://example.com/");
string url = driver.Url;
//the driver can now provide you with what you need (it will execute the script)
//get the source of the page
var source = driver.PageSource;
//fully navigate the dom
var pathElement1 = driver.FindElementByName("username");
var pathElement2 = driver.FindElementByName("password");
var pathElement3 = driver.FindElementByXPath("//button[#class='SubmitButton']");
pathElement1.Clear();
pathElement1.SendKeys("username");
pathElement2.Clear();
pathElement2.SendKeys("password");
pathElement3.Click();
//Now get the response after login button click
source = driver.PageSource;
I am trying to write some code to connect to an HTTPS site that uses Siteminder authentication.
I keep getting a 401. Any ideas?
I have read a few different things on here but none have really seemed all that helpful. I am also using Fiddler/Firefox Tamper to snoop what's going on.
Here is what I've got so far in regards to code:
try
{
Uri uri = new Uri("https://websiteaddresshere");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri) as HttpWebRequest;
request.Accept = "text/html, application/xhtml+xml, */*";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
// request.Connection = "Keep-Alive";
// request.Method = "Get";
// request.Accept = "text";
request.AllowAutoRedirect = true;
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
Cookie emersoncookie = new Cookie("SMCHALLENGE", "YES");
emersoncookie.Domain = "mydomain";
emersoncookie.Path = "/";
// authentication
var cache = new CredentialCache();
cache.Add(uri, "False", new NetworkCredential("myusername", "mypassword"));
request.Credentials = cache;
// response.
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
using (Stream stream = response.GetResponseStream())
{
XmlTextReader reader = new XmlTextReader(stream);
MessageBox.Show(stream.ToString());
}
}
}
catch (WebException exception)
{
string responseText;
using (var reader = new StreamReader(exception.Response.GetResponseStream()))
{
responseText = reader.ReadToEnd();
MessageBox.Show(responseText.ToString());
}
}
After doing some more reading on the MSDN website I decided to go a different route.
I ended up making this a service since it will need to be a service at the end of the day and did the following:
CookieContainer emersoncookie = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)
WebRequest.Create("https://websiteaddress");
request.Credentials = new NetworkCredential("username", "password");
request.CookieContainer = emersoncookie;
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)
request.GetResponse();
Stream resStream = response.GetResponseStream();
using (Stream output = File.OpenWrite(#"c:\\somefolder\\somefile.someextention"))
using (Stream input = resStream)
{
input.CopyTo(output);
}
To anyone that might be running into Siteminder authentication issues, this piece of code works pretty well.
I couldn't get Jasen's code to work. Maybe your SM setup is different from mine. But with SiteMinder it's generally a two step authentication process. The code block below works for me:
//Make initial request
RestClient client = new RestClient("http://theResourceDomain/myApp");
client.CookieContainer = new CookieContainer();
IRestResponse response = client.Get(new RestRequest("someProduct/orders"));
//Now add credentials.
client.Authenticator = new HttpBasicAuthenticator("username", "password");
//Get resource from the SiteMinder URI which will redirect to the correct API URI upon authentication.
response = client.Get(new RestRequest(response.ResponseUri));
Although this uses RestSharp, it can be easily replicated using HttpClient or even HttpWebRequest.
I have a webbrowser control which loads a page.
I then hit a button to call this method:
public void get(Uri myUri)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myUri);
CookieContainer cookieJar = new CookieContainer();
cookieJar.SetCookies(webBrowser1.Document.Url,webBrowser1.Document.Cookie.Replace(';', ','));
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
int cookieCount = cookieJar.Count;
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
txt.Text = readStream.ReadToEnd();
txt2.Text = cookieCount.ToString();
}
As from the cookieCount int i can see that if i call the method before logging in on the page in the web browser control i would get 6 cookies, and after i log in i get 7.
However, even with the cookies the response i get is the same as if i wouldnt have been logged in.
So i am guessing that the cookies isnt being sent with the request?
Thanks!
You're recreating your CookieContainer every time you call this method, you need to use the same CookieContainer in all requests
you can use this code, to handle your requests:
static CookieContainer cookies = new CookieContainer();
static HttpWebRequest GetNewRequest(string targetUrl, CookieContainer SessionCookieContainer)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(targetUrl);
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
return request;
}
public static HttpWebResponse MakeRequest(HttpWebRequest request, CookieContainer SessionCookieContainer, Dictionary<string, string> parameters = null)
{
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
if (parameters != null)
{
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = "";
foreach (KeyValuePair<String, String> parametro in parameters)
{
if (postData.Length == 0)
{
postData += String.Format("{0}={1}", parametro.Key, parametro.Value);
}
else
{
postData += String.Format("&{0}={1}", parametro.Key, parametro.Value);
}
}
byte[] postBuffer = UTF8Encoding.UTF8.GetBytes(postData);
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBuffer, 0, postBuffer.Length);
}
}
else
{
request.Method = "GET";
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
SessionCookieContainer.Add(response.Cookies);
while (response.StatusCode == HttpStatusCode.Found)
{
response.Close();
request = GetNewRequest(response.Headers["Location"], SessionCookieContainer);
response = (HttpWebResponse)request.GetResponse();
SessionCookieContainer.Add(response.Cookies);
}
return response;
}
and to request a page,
HttpWebRequest request = GetNewRequest("http://www.elitepvpers.com/forum/login.php?do=login", cookies);
Dictionary<string,string> parameters = new Dictionary<string,string>{{"your params","as key value"};
HttpWebResponse response = MakeRequest(request, cookies, parameters);
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
if(!reader.EndOfStream)
{
Console.Write(reader.ReadToEnd());
}
}
When matching the session, your web server may be taking into account some other HTTP request headers, besides cookies. To name a few: User-Agent, Authorization, Accept-Language.
Because WebBrowser control and WebRequest do not share sessions, you'd need to replicate all headers from the WebBrowser session. This would be hard thing to do, as you'd need to intercept WebBrowser trafic, in a way similar to what Fiddler does.
A more feasible solution might be to stay on the same session with WebBrowser by using Windows UrlMon APIs like URLOpenStream, URLDownloadToFile etc., instead of WebRequest. That works, because WebBrowser uses UrlMon library behind the scene.
I've recently answered some related questions:
The notorious yet unaswered issue of downloading a file when windows security is required
Upload a file to a website programmatically?
I am trying to authenticate google with the following code but google sent me back to the login page again.
//STEP# 1
string loginURL = "https://www.google.com/accounts/ServiceLoginBox?service=analytics&nui=1&hl=en-US&continue=https%3A%2F%2Fwww.google.com%2Fanalytics%2Fsettings%2F%3Fet%3Dreset%26hl%3Den%26et%3Dreset%26hl%3Den-US";
request = (HttpWebRequest)WebRequest.Create(loginURL);
request.CookieContainer = cookieJar;
request.Method = "GET";
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
sr.Close();
}
galx = ExtractValue(serverResponse,"GALX","name=\"GALX\" value=\"");
Console.WriteLine(galx);
//Request# 2
string uriWithData = "https://www.google.com/accounts/ServiceLoginBoxAuth";
request = (HttpWebRequest)WebRequest.Create(uriWithData);
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.4) Gecko/2008111217 Fedora/3.0.4-1.fc10 Firefox/3.0.4";
request.Method = "POST";
request.CookieContainer = cookieJar;
string param = string.Format("Email={0}&Passwd={1}&continue={2}&service=analytics&nui=1&dsh=8209101995200094904&GALX={3}&hl=en-US&PersistentCookie=yes","**my email address**",p,"",galx);
byte[] postArr = StrToByteArray(param);
request.ContentType = #"application/x-www-form-urlencoded";
request.ContentLength = param.Length;
Stream reqStream = request.GetRequestStream();
reqStream.Write(postArr,0,postArr.Length);
reqStream.Close();
response = (HttpWebResponse)request.GetResponse();
foreach (Cookie cook in response.Cookies)
{
cookieJar.Add(cook);
}
using (StreamReader sr = new StreamReader(response.GetResponseStream()) )
{
serverResponse = sr.ReadToEnd();
Console.WriteLine(serverResponse);
// Close and clean up the StreamReader
sr.Close();
}
I have no idea an I am pretty sure not many people are going to want to sift through that much code.
One thing that I see at a glance that may be causing problems is that you are playing with the cookies too much.
Create one CookieContainer and just pass it in with each request.
No need to 'transfer' or 'recreate' the container.
Try looking into google accounts api and GBaseService. I believe you will have to set HOSTED_OR_GOOGLE in accountType as Marty has done here in this post.