Is HttpWebResponse::GetResponseStream() guaranteed to get all data contained in an HTTP response? Or do I need to create some kind of loop and wait to make sure all data is being sent from the server to which I'm connected? The code below successfully grabs a response about 50% of the time.
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://JohnDoeServerSite.com");
req.Method = "POST";
req.ContentType = #"text/xml; charset=utf-8";
req.Host = "http://JohnDoeServerSite.com";
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.1)";
using (StreamWriter sw = new StreamWriter(req.GetRequestStream()))
{
sw.Write(xml);
}
string result;
using (HttpWebResponse res = (HttpWebResponse)req.GetResponse())
using (Stream st = res.GetResponseStream())
{
Thread.Sleep(10000); // Added to see if additional data would be sent (perhaps?)
using (StreamReader sr = new StreamReader(st, Encoding.UTF8))
{
st.Flush();
result = sr.ReadToEnd();
}
}
After looking into this more, it turns out that the server was not sending all of the data in its entirety. It seems that Michael Yoon is right in saying that everything should be returned.
Related
I'm almost certain that I'm just misunderstanding how query strings should be passed in using C# HTTPWebRequest/Response, but I can't figure out why when I call (HTTPwebResponse)request.GetResponse(); it strips out the query portion and only gives the resulting HTML of the base URL. What am I doing wrong? Note that I generalized the URL for simplicity but kept the same query string just in case the structure of that is the problem.
string urlQueryString = "https://baseUrl.com/folder1/folder2?status_filter_mode=approved";
var targetUri = Uri.EscapeUriString(urlQueryString);
request = HttpWebRequest.Create(targetUri) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
request.Method = "GET";
//set the cookie in the request header
request.Headers.Add("Cookie", cookiedata);
response = (HttpWebResponse)request.GetResponse();
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
htmlResult = reader.ReadToEnd();
}
}
response.Close();
I would like to download an image from this url http://squirlytraffic.com/surficon.php?ts=1491591235
I tried this code but I don't see the image when I open it.
using (WebClient client = new WebClient())
{
client.DownloadFile("http://squirlytraffic.com/surficon.php?ts=1491591235", #"D:\image.jpg");
}
You need to set your credentials using the the WebClient Credentials property. You can do this by assigning it an instance of NetworkCredential. See below:
using (WebClient client = new WebClient()){
client.Credentials = new NetworkCredential("user-name", "password");
client.DownloadFile("url", #"file-location");
}
EDIT
If you don't want to hard code a username and password, you can set the UseDefaultCredentials property of the WebClient to true. This will use the credentials of the currently logged in user. From the documentation.
The Credentials property contains the authentication credentials used to access a resource on a host. In most client-side scenarios, you should use the DefaultCredentials, which are the credentials of the currently logged on user. To do this, set the UseDefaultCredentials property to true instead of setting this property.
Which would mean you could amend the above code to:
using (WebClient client = new WebClient()){
client.UseDefaultCredentials = true;
client.DownloadFile("url", #"file-location");
}
Try this way, those parameters are passed when login
StringBuilder postData = new StringBuilder();
postData.Append("login=" + HttpUtility.UrlEncode("username") + "&");
postData.Append("password=" + HttpUtility.UrlEncode("password") + "&");
postData.Append("Submit=" + HttpUtility.UrlEncode("Login"));
ASCIIEncoding ascii = new ASCIIEncoding();
byte[] postBytes = ascii.GetBytes(postData.ToString());
CookieContainer cc = new CookieContainer();
HttpWebRequest webReq = (HttpWebRequest)WebRequest.Create("http://squirlytraffic.com/members.php");
webReq.Method = "POST";
webReq.ContentType = "application/x-www-form-urlencoded";
webReq.ContentLength = postBytes.Length;
webReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
webReq.CookieContainer = cc;
Stream postStream = webReq.GetRequestStream();
postStream.Write(postBytes, 0, postBytes.Length);
postStream.Flush();
postStream.Close();
HttpWebResponse res = (HttpWebResponse)webReq.GetResponse();
HttpWebRequest ImgReq = (HttpWebRequest)WebRequest.Create("http://squirlytraffic.com/surficon.php?ts=1491591235");
ImgReq.Method = "GET";
ImgReq.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
ImgReq.CookieContainer = cc;
HttpWebResponse ImgRes = (HttpWebResponse)ImgReq.GetResponse();
Stream Img = ImgRes.GetResponseStream();
How to make postback request to ASP.NET page? I caught data by the Fiddler and tried to make the same request with HttpWebRequest class, but I receive short answer in wrong encoding. (Various encoding tried also, no result)
Here is the part of code:
var req = (HttpWebRequest)WebRequest.Create("http://mySite.aspx?test=test");
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded; charset=utf-8";
req.Accept="*/*";
req.Headers.Add("Accept-Language", "en-us");
req.Referer="http://mysiterefer.com";
req.Headers.Add("x-microsoftajax", "Delta=true");
req.Headers.Add("Cache-Control", "no-cache");
req.Headers.Add("Accept-Encoding", "gzip, deflate");
req.UserAgent="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; AskTbFXTV5/5.15.1.22229)";
req.Headers.Add("Pragma", "no-cache");
req.Headers.Add("Cookie", "ASP.NET_SessionId=blablablaxur445; vehicle=1=2011##VAZ##2101##; s_cc=true; s_p_s_prop8=typed-bookmarked; ev_36_getval=20120501%2000; s_sq=%5B%5BB%5D%5D");
string postData = "__EVENTTARGET=lalalala&__EVENTARGUMENT=&__LASTFOCUS=&__VIEWSTATE=veryLongBase64string&__ASYNCPOST=true&";
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
req.ContentLength = byteArray.Length;
Stream dataStream = req.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();
using (var response = (HttpWebResponse)req.GetResponse())
{
StreamReader objReader = new StreamReader(response.GetResponseStream());
string str = objReader.ReadToEnd();
...
}
So, who can provide example with postback request?
I am trying to Screen Scrape a WebSite that uses ASP.Net Forms Authentication. I have the following code which makes a Get request to get the cookies and then I want to post the Login info with the cookies I just got to the login.aspx page.
When I watch what is submitted to the login.aspx page with Fiddler I see the Posted data but for the submitted cookies I see the message " but it says "This request did not send any cookie data."
If I login into the app using Internet Explorer I can see that the cookies and the posted data are submitted to the login.aspx page and everything is working fine.
But if I loop through I can print out the cookies that I am assuming should be sent along with the request using this block
foreach (System.Net.Cookie cook in getCookies.CookieContainer.GetCookies(new Uri("https://app.example.com")))
{
Console.WriteLine(cook.Name + ": " + cook.Value);
}
What is is that I am doing wrong that is resulting in the cookies not being sent with my request?
public void Login()
{
HttpWebRequest getCookies = (HttpWebRequest)WebRequest.Create("https://app.example.com");
CookieContainer cookieJar = new CookieContainer();
getCookies.Accept = "image/gif, image/jpeg, image/pjpeg, image/pjpeg, application/vnd.ms-xpsdocument, application/x-ms-application, application/x-ms-xbap, application/xaml+xml, */*";
getCookies.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0E; .NET4.0C)";
getCookies.Headers.Add("Accept-Encoding", "gzip, deflate");
getCookies.AllowAutoRedirect = true;
getCookies.CookieContainer = cookieJar;
using (HttpWebResponse cookieResponse = getCookies.GetResponse() as HttpWebResponse)
{
StreamReader responseReader = new StreamReader(cookieResponse.GetResponseStream());
string responseData = responseReader.ReadToEnd();
string ViewState = this.GetViewStateFromHtml(responseData);
getCookies = (HttpWebRequest)WebRequest.Create("https://app.example.com/Membership/Login.aspx?ReturnUrl=%2fHome%2fHomeSummary.aspx");
getCookies.Method = "Post";
getCookies.ContentType = "application/x-www-form-urlencoded";
getCookies.Accept = "text/html, application/xhtml+xml. */*";
getCookies.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET4.0E; .NET4.0C)";
getCookies.Headers.Add("Accept-Encoding", "gzip, deflate");
getCookies.AllowAutoRedirect = true;
ASCIIEncoding encoding = new ASCIIEncoding();
byte[] byte1 = encoding.GetBytes(this.GetPostData(Username, Password));
getCookies.ContentLength = byte1.Length;
StreamWriter requestWriter = new StreamWriter(getCookies.GetRequestStream());
requestWriter.Write(this.GetPostData(Username, Password));
requestWriter.Close();
getCookies.CookieContainer = cookieJar;
foreach (System.Net.Cookie cook in getCookies.CookieContainer.GetCookies(new Uri("https://app.example.com")))
{
Console.WriteLine(cook.Name + ": " + cook.Value);
}
using (HttpWebResponse postResponse = (HttpWebResponse)getCookies.GetResponse())
{
StreamReader postLoginResponseReader = new StreamReader(postResponse.GetResponseStream());
string postLoginResponseData = postLoginResponseReader.ReadToEnd();
File.WriteAllText(#"C:\postLoginResponse.txt", postLoginResponseData);
}
Console.WriteLine(#"Check C:\postLoginResponse.txt");
Console.ReadLine();
}
I don't see where you are adding your forms auth cookie to the HttpWebRequest cookie container.
In the past I have done this
var request = (HttpWebRequest) WebRequest.Create(remoteFilename);
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Cookie(".ASPXAUTH", "71AE9C8F6CFDC86BD7FD3AD7B214C4E1", "/", "build.mercaridirect.com.au"));
minor point You should change the name of your variable getCookies to webRequest or similar, improves readibility.
This is a portion of a larger code snippet, so please let me know if you need more of it. Essentially, you can code directly against the MembershipProvider and the FormsAuthentication classes to simulate the login:
bool validated = Membership.ValidateUser(uname, pwd);
if (validated)
{
if (Request.QueryString["ReturnUrl"] != null)
{
FormsAuthentication.RedirectFromLoginPage(uname, false);
}
else
{
FormsAuthentication.SetAuthCookie(uname, false);
}
return;
}
//Response.Write("Failed to authenticate, invalid credentials.");
Hope this helps.
After authorisation on www.vkontakte.ru through ie8 me spans on page: www.vkontakte.ru/MyPage. But I cannot receive www.vkontakte.ru/MyPage through a code
HttpWebRequest authRequest = (HttpWebRequest)WebRequest.Create(new Uri("http://vkontakte.ru/login.php", UriKind.Absolute));
authRequest.CookieContainer = new CookieContainer();
authRequest.AllowAutoRedirect = false;
string param = string.Format("email={0}&pass={1}&expire=1", HttpUtility.UrlEncode("---"), HttpUtility.UrlEncode("---"));
authRequest.Method = "POST";
authRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)";
authRequest.ContentType = "application/x-www-form-urlencoded";
authRequest.ContentLength = param.Length;
authRequest.GetRequestStream().Write(Encoding.GetEncoding(1251).GetBytes(param), 0, param.Length);
HttpWebResponse authResponse = (HttpWebResponse)authRequest.GetResponse();
listBox1.Items.Add(authRequest.Address);
Returns http://vkontakte.ru/ instead of www.vkontakte.ru/MyPage =(
HttpContext.Current.Request.Url.AbsoluteUri - can help?
help me!
You forgot to close the request stream.
You should write the following:
using (Stream requestStream = authRequest.GetRequestStream())
using (StreamWriter writer = new StreamWriter(requestStream, Encoding.GetEncoding(1251))
writer.Write(param);
Also, you should run Fiddler check what the request and response look like.