We Keep Coding

Problem with getting response of a HTTP Web request

You can see my code down there.This is going to show user id of a instagram user as the response but it is showing "�".
private void button18_Click(object sender, EventArgs e)
{
string username = ""; // your username
string password = ""; // your password
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://i.instagram.com/api/v1/accounts/login/");
httpWebRequest.Headers.Add("X-IG-Connection-Type", "WiFi");
httpWebRequest.Method = "POST";
httpWebRequest.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
httpWebRequest.Headers.Add("X-IG-Capabilities", "AQ==");
httpWebRequest.Accept = "*/*";
httpWebRequest.UserAgent = "Instagram 10.9.0 Android (23/6.0.1; 944dpi; 915x1824; samsung; SM-T185; gts210velte; qcom; en_GB)";
httpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
httpWebRequest.Headers.Add("Accept-Language", "en;q=1, ru;q=0.9, ar;q=0.8");
httpWebRequest.Headers.Add("Cookie", "mid=qzejldb8ph9eipis9e6nrd1n457b;csrftoken=a7nd2ov9nbxgqy473aonahi58y21i8ee");
httpWebRequest.Host = "i.instagram.com";
httpWebRequest.KeepAlive = true;
byte[] bytes = new ASCIIEncoding().GetBytes("ig_sig_key_version=5&signed_body=5128c31533802ff7962073bb1ebfa9972cfe3fd9c5e3bd71fe68be1d02aa92c8.%7B%22username%22%3A%22"+ username + "%22%2C%22password%22%3A%22"+ password +"%22%2C%22_uuid%22%3A%22D26E6E86-BDB7-41BE-8688-1D60DE60DAF6%22%2C%22_uid%22%3A%22%22%2C%22device_id%22%3A%22android-a4d01b84202d6018%22%2C%22_csrftoken%22%3A%22a7nd2ov9nbxgqy473aonahi58y21i8ee%22%2C%22login_attempt_count%22%3A%220%22%7D");
httpWebRequest.ContentLength = (long)bytes.Length;
using (Stream requestStream = httpWebRequest.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
string result = new StreamReader(((HttpWebResponse)httpWebRequest.GetResponse()).GetResponseStream()).ReadToEnd();
textBox1.Text = result;
}

It's answered in a comment by #Dour, but I'm going to add a bit of detail that isn't mentioned.
This following line tells the server: Hey server! Please send me a compressed response (to reduce size).
httpWebRequest.Headers.Add("Accept-Encoding", "gzip, deflate");
so, the response you're getting is a compressed response.
Does that mean you just remove this line ?
Well, Removing it will fix your problem but that isn't the correct way to do that.
It's really better to get a reduced size response because that will need lower time to download it from the server.
What you really need to do is to make HttpWebRequest handles the decompression process for you.
You can do that by setting AutomaticDecompression property.
httpWebRequest.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
Note: With the previous line, you do NOT need to set Accept-Encoding yourself. HttpWebRequest will send it automatically for you. And when you call GetResponse, HttpWebRequest will handle the decompression process for you.
Besides of your problem:
I suggest that you use using statement for getting the response, because with your current code, I think it won't get disposed.

Cookies and C# HttpWebRequest

I have been trying to log in to a server to grab the authentication cookie (a session cookie), which I can then use for further calls to the server. Every example I have read follows the same pattern:
HttpWebRequest request = WebRequest.Create(loginURL) as HttpWebRequest;
var response = request.GetResponse() as HttpWebResponse;
var cookies = response.Cookies;
This didn't work for me, as the cookies variable ended up empty, and a debug analysis showed response.Cookies was empty. The server is mine, and I can see, through debugging, the cookie is being set. I can also see the cookie in Firefox if I log in to my site with it. So I know the cookie is being set.
After some messing around, I discovered the cookie was being set in the request, not the response. So the code below worked. My question is: Why? Why is the request being populated, but not the response? Is it something to do with being a post, not a get? I am totally baffled.
private void Login()
{
string userName = UserNameText.Text;
string password = PasswordText.Password;
string baseURL = URLText.Text;
string loginURL = baseURL + "/Authentication/LoginAction";
HttpWebRequest request = WebRequest.Create(loginURL) as HttpWebRequest;
request.Method = "POST";
string formContent =
"UserName=" + userName +
"&Password=" + password;
var byteArray = Encoding.UTF8.GetBytes(formContent);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
request.CookieContainer = new CookieContainer();
try
{
using (var dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
using (var response = request.GetResponse() as HttpWebResponse)
{
var cookies = request.CookieContainer;
if (cookies.Count != 0)
{
cookies_ = cookies;
}
}
}
}
catch(Exception ex)
{
// don't bother too much
Debug.WriteLine(ex.Message);
}
}

The CookieContainer should be considered similar to a browser's cookie cache for a particular site. The idea is that you supply the container as part of the request, and then it's populated by the cookies you receive and you can reuse that container for subsequent requests. When you make a request, the cookies in the container are sent with the request (just like the browser would with stored cookies).
So, for example, if you have a page that uses cookies to store an authentication token, you can pass the cookie container with the login request, and then pass it with subsequent requests which require an authenticated cookie.
As to why you can't simply extract it from the request, I guess Microsoft just didn't want to duplicate things when you can pass in a reference to a mutable cookie container in the request.

Login to website which requires Cookies using c#

I'm trying to write a script in c# capable to retrieve some information from a website. These information are protected so I need to login before I can read them. That's what I think should be my procedure:
First of all use a POST request to login into the website. Here my first problem: the page where I find the login form is this https://idp.kk-abcdefg.com/idp/Authn/UserPassword . Should I submit the POST request to this page or should I use a different address?
I've tested the headers using some tools of Firefox or Chrome but I can't understand which is the right procedure. I have noticed that If I open this login page I receive some cookies. If I delete these cookies and try to login by inserting user and password (via browser) I get an error as a response from the website .. it says that I need to activate cookies to be able to login. So it seems like when I open the login page for the first time I receive some cookies and then I need to send them together with the first POST request for login. Does it make sense for any of you?
That's the code I'm using right now:
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword”);
string cookieHeader;
string pageSource;
WebRequest req = WebRequest.Create(formUrl);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
And I believe this is working because the result of pageSource changes if I use correct user / pass or I write them wrong. But when user / pass are correct, I'm still not able to login because I get the following error message: "This application requires cookies. Please make sure cookies are enabled in the settings of your browser. Please reload the login page and try logging in again".
This is the same error that I get if I disable cookies in my browser or if I delete cookies that I get when I load the login page for the first time.
Can you help me with all that? My idea is that I need to save the cookies received when I open the login page for the first time and then send them together with following requests but I don't know how to do ..
Thanks a lot!

In Web Application, once the user is logged in successfully a cookie is sent back to the browser to track the user session and to determine if the user is logged in or not during further requests. Furthermore login process of your application requires cookies to be sent from client along with username and password. So when you are trying to perform login without browser it complains about missing cookies.
If you know what cookies and their values need to be sent along with username and password for login, you can send them using cookieContainer in WebRequest as following.
string formUrl = "https://idp.kk-abcdefg.de/idp/Authn/UserPassword";
string formParams = string.Format("j_username=MyUserName&j_password=MyPassword");
string cookieHeader;
string pageSource;
CookieContainer cookieContainer = new CookieContainer();
Cookie cookie1 = new Cookie("<<cookiename>>", "<<cookievalue>>","/", "<<yourdomainname>>");
Cookie cookie2 = new Cookie("<<cookiename>>", "<<cookievalue>>", "/", "<<yourdomainname>>");
cookieContainer.Add(cookie1);
cookieContainer.Add(cookie2);
// You can keep adding all required cookies this way.
var req = (HttpWebRequest)WebRequest.Create(formUrl);
req.CookieContainer = cookieContainer;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(formParams);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
// You can access the cookies coming as part of response as following.
HttpWebResponse response = resp as HttpWebResponse;
if(response != null)
{
var cookiesCollections = response.Cookies;
}
using (StreamReader sr = new StreamReader(resp.GetResponseStream()))
{
pageSource = sr.ReadToEnd();
}
If you don't know the cookies and you need to get the cookies first by requesting login page before posting username and password then use following.
var loginPageUrl = "<<Your Login Page url>>";
CookieContainer cookieContainer = new CookieContainer();
var req = (HttpWebRequest)WebRequest.Create(loginPageUrl);
req.CookieContainer = cookieContainer;
req.Method = "GET";
WebResponse resp = req.GetResponse();
HttpWebResponse response = resp as HttpWebResponse;
CookieCollection cookies;
if (response != null)
{
cookies = response.Cookies; //Use this cookies in above code to send with username and password.
}

You send the request login twice and use single CookieContainer object!

Writing a cookie locally

I let my program (c#) log in to website, i get correct buffered cookie information. Yet, i get a 401, session timed out, when i want to retrieve the correct data behind my login.
So i figured, the website must not be able to retrieve that cookie info. Can't figure out how to store it so that the website can retrieve it.
WebRequest req = WebRequest.Create(Url);
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(Gegevens);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
WebResponse resp = req.GetResponse();
cookieHeader = resp.Headers["Set-cookie"];
cookieHeader, contains the correct info. Thanks in advance.

You need to assign a CookieContainer to your web request and use this very same CookieContainer for the following requests, too.
See MSDN for reference.
You could (if you want to persist the cookies upon closing your application) get the list of Cookies from the CookieContainer and serialize these. Upon opening the application you could deserialize and rebuild the CookieContainer.

From the comment you provided, I'm going to hazard a guess and say you aren't properly adding your login cookies to your next WebRequest. Cookie handling with a WebRequest object is a bit difficult, so I recommend using HttpWebRequest and HttpWebResponse which has built-in cookie parsing. You only have to change a couple lines of code here and there:
Building a request (using the same example in your question)
CookieContainer cookies = new CookieContainer();
// When using HttpWebRequest or HttpWebResponse, you need to cast for it to work properly
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.CookieContainer = cookies;
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
byte[] bytes = Encoding.ASCII.GetBytes(Gegevens);
req.ContentLength = bytes.Length;
using (Stream os = req.GetRequestStream())
{
os.Write(bytes, 0, bytes.Length);
}
// Cast here as well
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
{
// Code related to the web response goes in here
}
Right now, your cookie information is saved in the CookieContainer object. It can be reused later in your code to validate your login. You don't need to write this cookie information to the disk if you don't need to.
Building a request with cookie information
(Pretty much the same as above, but you're not adding POST data and you're using a GET request)
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.CookieContainer = cookies; // This is where you add your login cookies to the current request
req.Method = "GET";
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
{
// Code related to the web response goes here
}
Hopefully this will get you on the right track :)

C# : How to sign in and keep it valid and make another webrequest?

private void button1_Click(object sender, EventArgs e)
{
string userName = textBox1.Text;
string password = textBox2.Text;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.youmint.com/LoginVerification.php?name="+userName+"&pass="+password+"&agreement=true&checkvalue=true");
request.Method = "GET";
request.KeepAlive = true;
request.Headers.Add("Keep-Alive: 300");
WebResponse response = request.GetResponse();
Stream dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();
if (responseFromServer.Equals(""))
MessageBox.Show("Successfully logged in!!");
else if (responseFromServer.Equals("1"))
MessageBox.Show("Login failed!");
request = (HttpWebRequest)WebRequest.Create("http://www.youmint.com/FreeSms.html");
response = request.GetResponse();
dataStream = response.GetResponseStream();
reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
/*secret code :P */
reader.Close();
dataStream.Close();
response.Close();
}
So, that's my code... The first webrequest logs into the website. It works fine in the browser and returns 1 if the login is not correct. Then the second one is a normal webrequest to a webpage of the same website. But the login is already gone and the response I get is what I get if I'm not logged in! Can someone please tell me what I'm doing wrong? How do I keep it alive? Do I have to use an invisible webbroswer control or something like that?

This has nothing to do with "keep alive".
You need to preserve session cookie between requests. For that you first need to enable cookies for your login request (read HttpWebRequest docs - it is a bit unobvious). Then you need to pass that cookie with all the following reuests.
Also please make use of the using()

#liho1eye is correct. Here's some more info from the HttpWebRequest page:
For security reasons, cookies are
disabled by default. If you want to
use cookies, use the CookieContainer
property to enable cookies.
You'll need to reference the HttpWebResponse.Cookies property to get the initial session token cookie.
Edit:
Here's a quick and dirty sample of making a request to a page, and transferring response cookies to the next request. Didn't do much testing or validation (so beware!) - just to give you the idea of the approach.
//this only has login/password info, you may need other parameters to trigger the appropriate action:
const string Parameters = "Login1$username=pfadmin&Login1$Password=password";
System.Net.HttpWebRequest req = (HttpWebRequest)System.Net.WebRequest.Create("http://[WebApp]/Login.aspx");
req.Method = "GET";
req.CookieContainer = new CookieContainer();
System.Net.HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
//Create POST request and transfer session cookies from initial request
req = (HttpWebRequest)System.Net.WebRequest.Create("http://localhost/AdminWeb/Login.aspx");
req.CookieContainer = new CookieContainer();
foreach (Cookie c in resp.Cookies)
{
req.CookieContainer.Add(c);
}
req.ContentType = "application/x-www-form-urlencoded";
//...continue on with your form POST