We Keep Coding

HttpWebReponse Cookies Not Setting For Redirect

I'm trying to do the following:
send a GET request to fetch a login page (which prompts for username, password, and sets a cookie)
build a POST request that sends the cookie from #1 and a body of the username/password (this returns a Set-Cookie and redirects to the website's landing page for logged in users)
My trouble is with the 302 redirect. The webserver is returning a 302 with a Set-Cookie, but when the HttpWebRequests auto-redirects, it doesn't pass along the now updated cookie. To get around that, I'm trying to set .AllowAutoRedirect = false, saving the cookies in a CookieCollection, then building a 3rd HTTP request: a GET to the final 302 location. Unfortunately, I can't set the cookies on this request. I'm not sure why and it's driving me mad.
The HTTP requests are, in order, named request, postRequest, redirectRequest.
string loginGetUrl = "https://<..>/signin.htm";
string loginPostUrl = "https://<..>/j_acegi_security_check";
string loginRedirectUrl = "https://<..>/centraladmin/poslinks.htm";
string postData = String.Format("j_username={0}&j_password={1}", username, password);
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginGetUrl);
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
HttpWebRequest postRequest = (HttpWebRequest)WebRequest.Create(loginPostUrl);
postRequest.CookieContainer = new CookieContainer();
// Add the received Cookies from the HTTP Get
postRequest.CookieContainer.Add(cookies);
postRequest.Method = WebRequestMethods.Http.Post;
postRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
postRequest.AllowWriteStreamBuffering = false;
postRequest.ProtocolVersion = HttpVersion.Version11;
postRequest.AllowAutoRedirect = false;
postRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
postRequest.ContentLength = byteArray.Length;
Stream newStream = postRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse postResponse = (HttpWebResponse)postRequest.GetResponse();
// Save the cookies from the POST login request, then send them on to the redirected URL
cookies = postResponse.Cookies;
HttpWebRequest redirectRequest = (HttpWebRequest)WebRequest.Create(loginRedirectUrl);
redirectRequest.CookieContainer = new CookieContainer();
// add cookies from POST
redirectRequest.CookieContainer.Add(cookies);
HttpWebResponse redirectResponse = (HttpWebResponse)redirectRequest.GetResponse();
At redirectRequest.CookieContainer.Add(cookies);, the cookies object contains the correct cookie. But when I look with Fiddler, I only see this info:
GET https://<...>/centraladmin/poslinks.htm HTTP/1.1
Host: host:port
I'm kind of banging my head on the wall at this point. Any suggestions? Am I referencing something wrong? Beware, I don't usually write C# code

I wasn't able to resolve this on my own, but did find a useful code snippet from this blog post by #malte-clasen. The code is on Github and I've attached it here for retention.
I removed the async components as it wasn't necessary in my code.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading.Tasks;
namespace DracWake.Core
{
public class WebClient : IWebClient
{
private readonly CookieContainer _cookies = new CookieContainer();
private HttpWebRequest CreateRequest(Uri uri)
{
var request = HttpWebRequest.CreateHttp(uri);
request.AllowAutoRedirect = false;
request.CookieContainer = _cookies;
SetHeaders(request);
var defaultValidator = System.Net.ServicePointManager.ServerCertificateValidationCallback;
request.ServerCertificateValidationCallback =
(sender, certificate, chain, sslPolicyErrors) =>
certificate.Subject.Contains("O=DO_NOT_TRUST, OU=Created by http://www.fiddler2.com")
|| (certificate.Subject == "CN=DRAC5 default certificate, OU=Remote Access Group, O=Dell Inc., L=Round Rock, S=Texas, C=US")
|| (defaultValidator != null && defaultValidator(request, certificate, chain, sslPolicyErrors));
return request;
}
private async Task<string> DecodeResponse(HttpWebResponse response)
{
foreach (System.Net.Cookie cookie in response.Cookies)
{
_cookies.Add(new Uri(response.ResponseUri.GetLeftPart(UriPartial.Authority)), cookie);
}
if (response.StatusCode == HttpStatusCode.Redirect)
{
var location = response.Headers[HttpResponseHeader.Location];
if (!string.IsNullOrEmpty(location))
return await Get(new Uri(location));
}
var stream = response.GetResponseStream();
var buffer = new System.IO.MemoryStream();
var block = new byte[65536];
var blockLength = 0;
do{
blockLength = stream.Read(block, 0, block.Length);
buffer.Write(block, 0, blockLength);
}
while(blockLength == block.Length);
return Encoding.UTF8.GetString(buffer.GetBuffer());
}
public async Task<string> Get(Uri uri)
{
var request = CreateRequest(uri);
var response = (HttpWebResponse) await request.GetResponseAsync();
return await DecodeResponse(response);
}
private void SetHeaders(HttpWebRequest request)
{
request.Accept = "text/html, application/xhtml+xml, */*";
request.UserAgent = "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)";
request.ContentType = "application/x-www-form-urlencoded";
request.Headers[HttpRequestHeader.AcceptLanguage] = "en-US,en;q=0.8,de-DE;q=0.5,de;q=0.3";
request.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
request.Headers[HttpRequestHeader.CacheControl] = "no-cache";
}
public async Task<string> Post(Uri uri, byte[] data)
{
var request = CreateRequest(uri);
request.Method = "POST";
request.GetRequestStream().Write(data, 0, data.Length);
var response = (HttpWebResponse) await request.GetResponseAsync();
return await DecodeResponse(response);
}
}
}
The DecodeResponse resolved my problem.

HTTPwebrequest, Json, populate online textbox

I am working with HTTPWebRequests for the first time now and i'm trying to wrap my head around it all. I am trying to populate the "Start Date" text box on this website http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx. I used Fiddler to find the incredibly long Json query that comes back from the POST. ( i will post the entire thing if needed )...but i noticed the last part is what I'm after
"............,"StartDate":"1/1/2013","EndDate":"12/31/2013"}}
So far, this is the function i am using but the result comes back without any changes (so it's as if you just went to the website i first posted)
public static void json(string url)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "text/json";
httpWebRequest.Method = "POST";
httpWebRequest.CookieContainer = cookieJar;
//httpWebRequest.Accept = "text/html, application/xhtml+xml, */*";
httpWebRequest.Headers.Add("Accept-Language: en-US");
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW65; Trident/7.0; MAM5; rv:11.0) like Gecko";
httpWebRequest.Headers.Add("Accept-Encoding: gzip, deflate");
//httpWebRequest.Referer = url4;
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = #"{StartDate: 1/1/2013}, {EndDate: 12/13/2013}";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
}

c# web browser cookies in httpwebrequest

I have a webbrowser control which loads a page.
I then hit a button to call this method:
public void get(Uri myUri)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(myUri);
CookieContainer cookieJar = new CookieContainer();
cookieJar.SetCookies(webBrowser1.Document.Url,webBrowser1.Document.Cookie.Replace(';', ','));
request.CookieContainer = cookieJar;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
int cookieCount = cookieJar.Count;
Stream receiveStream = response.GetResponseStream();
StreamReader readStream = new StreamReader(receiveStream, Encoding.UTF8);
txt.Text = readStream.ReadToEnd();
txt2.Text = cookieCount.ToString();
}
As from the cookieCount int i can see that if i call the method before logging in on the page in the web browser control i would get 6 cookies, and after i log in i get 7.
However, even with the cookies the response i get is the same as if i wouldnt have been logged in.
So i am guessing that the cookies isnt being sent with the request?
Thanks!

You're recreating your CookieContainer every time you call this method, you need to use the same CookieContainer in all requests
you can use this code, to handle your requests:
static CookieContainer cookies = new CookieContainer();
static HttpWebRequest GetNewRequest(string targetUrl, CookieContainer SessionCookieContainer)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(targetUrl);
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
return request;
}
public static HttpWebResponse MakeRequest(HttpWebRequest request, CookieContainer SessionCookieContainer, Dictionary<string, string> parameters = null)
{
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.CookieContainer = SessionCookieContainer;
request.AllowAutoRedirect = false;
if (parameters != null)
{
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
string postData = "";
foreach (KeyValuePair<String, String> parametro in parameters)
{
if (postData.Length == 0)
{
postData += String.Format("{0}={1}", parametro.Key, parametro.Value);
}
else
{
postData += String.Format("&{0}={1}", parametro.Key, parametro.Value);
}
}
byte[] postBuffer = UTF8Encoding.UTF8.GetBytes(postData);
using (Stream postStream = request.GetRequestStream())
{
postStream.Write(postBuffer, 0, postBuffer.Length);
}
}
else
{
request.Method = "GET";
}
HttpWebResponse response = request.GetResponse() as HttpWebResponse;
SessionCookieContainer.Add(response.Cookies);
while (response.StatusCode == HttpStatusCode.Found)
{
response.Close();
request = GetNewRequest(response.Headers["Location"], SessionCookieContainer);
response = (HttpWebResponse)request.GetResponse();
SessionCookieContainer.Add(response.Cookies);
}
return response;
}
and to request a page,
HttpWebRequest request = GetNewRequest("http://www.elitepvpers.com/forum/login.php?do=login", cookies);
Dictionary<string,string> parameters = new Dictionary<string,string>{{"your params","as key value"};
HttpWebResponse response = MakeRequest(request, cookies, parameters);
using (StreamReader reader = new StreamReader(response.GetResponseStream()))
{
if(!reader.EndOfStream)
{
Console.Write(reader.ReadToEnd());
}
}

When matching the session, your web server may be taking into account some other HTTP request headers, besides cookies. To name a few: User-Agent, Authorization, Accept-Language.
Because WebBrowser control and WebRequest do not share sessions, you'd need to replicate all headers from the WebBrowser session. This would be hard thing to do, as you'd need to intercept WebBrowser trafic, in a way similar to what Fiddler does.
A more feasible solution might be to stay on the same session with WebBrowser by using Windows UrlMon APIs like URLOpenStream, URLDownloadToFile etc., instead of WebRequest. That works, because WebBrowser uses UrlMon library behind the scene.
I've recently answered some related questions:
The notorious yet unaswered issue of downloading a file when windows security is required
Upload a file to a website programmatically?

Login in website Exception

Hi ever body I am new in c# webscraping.
I want to login to http://telelisting.ca/ but I cant. I have used WebClient, WebBrowser and now HttpWebRequest, but the same errors occured all the time which is server exception 500.
Can any body help me what i am doing wrong??
CookieCollection cookies = new CookieCollection();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://telelisting.ca/");
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(cookies);
//Get the response from the server and save the cookies from the first request..
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
cookies = response.Cookies;
string getUrl = "http://telelisting.ca/";
string postData = String.Format("ctl00_LoginPopup1_Login1_UserName={0}&ctl00_LoginPopup1_Login1_Password={1}&__LASTFOCUS={2}&__EVENTTARGET={3}&__EVENTARGUMENT={4}&__VIEWSTATE={5}&__EVENTVALIDATION={6}&ctl00_LoginPopup1_Login1_LoginButton={7}&ctl00_LoginPopup1_hiddenImagePath={8}&ctl00_cphContent_hiddenVideoDialogTitle={9}&ctl00_cphContent_hiddenShowDefaultVideo={10}", "fg#xyz.com", "abcede", "", "", "", "/wEPDwUKMTYzMzk2MDQwNQ9kFgJmD2QWAgIDDxYEHgVjbGFzc2UeCG9udW5sb2FkBQ9PbkVuZFNlc3Npb24oKTsWAgIBD2QWDAIHD2QWBAIDD2QWAgIDDw8WBB4EVGV4dAUVRm9yZ290IHlvdXIgcGFzc3dvcmQ/HgtOYXZpZ2F0ZVVybAUWfi9wYXNzd29yZHJlY292ZXIuYXNweGRkAgUPDxYCHg9WYWxpZGF0aW9uR3JvdXAFGGN0bDAwJExvZ2luUG9wdXAxJExvZ2luMWRkAgkPZBYCAgEPFgIeBGhyZWYFHmphdmFzY3JpcHQ6U2hvd1ZpZGVvQ29udGVudCgpO2QCCw9kFgICAQ9kFgYCCw8PFgIfAwURL0RlZmF1bHQuYXNweD9sPTBkZAIND2QWAgIBD2QWAmYPFgIfAAURRW1wdHlTaG9wcGluZ0NhcnQWAgIBDw8WAh8CBXgmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtkZAIZDw8WAh4HVmlzaWJsZWhkZAIND2QWAgIBD2QWAgIBDw8WAh8CBTM8c3Ryb25nPjxhIGhyZWY9IkRlZmF1bHQuYXNweCI+SG9tZTwvYT4gPiA8L3N0cm9uZz5kZAIPD2QWBAIJDw8WAh8CBdgJPG1hcnF1ZWUgU0NST0xMREVMQVk9JzE1MCcgPkF1ZyAxOCAtIE5ldyBDbGllbnQgaW4gPGI+TWFnb2cnPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMTYgLSBOZXcgQ2xpZW50IGluIDxiPkxvbmRvbic8L2I+Li4uLiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO0F1ZyAxNiAtIE5ldyBDbGllbnQgaW4gPGI+U3QuIE1hcnlzJzwvYj4uLi4uJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7QXVnIDE2IC0gTmV3IENsaWVudCBpbiA8Yj5CYXJyaWUnPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMTQgLSBOZXcgQ2xpZW50IGluIDxiPkJhcnJpZSc8L2I+Li4uLiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO0F1ZyAxNCAtIE5ldyBDbGllbnQgaW4gPGI+VmFuY291dmVyJzwvYj4uLi4uJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7QXVnIDEzIC0gTmV3IENsaWVudCBpbiA8Yj52YXVnaGFuJzwvYj4uLi4uJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7QXVnIDEzIC0gTmV3IENsaWVudCBpbiA8Yj5Mb25ndWV1aWwnPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMTMgLSBOZXcgQ2xpZW50IGluIDxiPlRvcm9udG8nPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMTIgLSBOZXcgQ2xpZW50IGluIDxiPlN0ZS1DYXRoZXJpbmUtZGUtbGEtSi4tQy4nPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMDkgLSBOZXcgQ2xpZW50IGluIDxiPk5ld21hcmtldCAnPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMDggLSBOZXcgQ2xpZW50IGluIDxiPkJyYW1wdG9uJzwvYj4uLi4uJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7QXVnIDA3IC0gTmV3IENsaWVudCBpbiA8Yj5WYW5jb3V2ZXInPC9iPi4uLi4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtBdWcgMDcgLSBOZXcgQ2xpZW50IGluIDxiPk90dGF3YSc8L2I+Li4uLiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwO0F1ZyAwNyAtIE5ldyBDbGllbnQgaW4gPGI+VHJvaXMtUml2acOocmVzJzwvYj4uLi4uJm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7PC9tYXJxdWVlPmRkAhEPZBYCZg8PFgIfAgXyAzxoMyBjbGFzcz0idGl0bGVuYXYiPg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBSZWZlcnJhbCBQcm9ncmFtPC9oMz4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDxwIGlkPSJyMjUiPjEgTW9udGggRnJlZSEgPC9wPiAgDQogICAgICAgICAgICAgICAgICAgICAgICAgICAgPHAgaWQ9InRlc3RpbW9uaWFsbmF2Ij4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUmVmZXIgY29sbGVhZ3VlcyBhbmQgZnJpZW5kcyB0byBUZWxlbGlzdGluZyBhbmQgcmVjZWl2ZSAxIE1vbnRoIEZyZWUgYWNjZXNzIHRvIG91ciBwaG9uZWJvb2sgc2VydmljZSBmb3IgZXZlcnkgcmVmZXJyYWwgdGhhdCByZXN1bHRzIGluIGEgcGFpZCBzdWJzY3JpcHRpb24uIA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvcD4NCjxhIHJ1bmF0PSJzZXJ2ZXIiIGhyZWY9IlJlZmVycmFsTG9naW4uYXNweCIgY2xhc3M9ImJ1eW5vdyI+UmVmZXIgTm93PC9hPiAgICAgICANCmRkAhEPZBYCAgEPZBYMAgMQPCsACQEADxYEHghEYXRhS2V5cxYAHgtfIUl0ZW1Db3VudGZkZGQCBQ8PZA8QFgFmFgEWAh4OUGFyYW1ldGVyVmFsdWUFBWVuLVVTFgECBmRkAgkQPCsACQEADxYEHwcWAB8IZmRkZAILDw9kDxAWAmYCARYCFgIfCWQWAh8JBQVlbi1VUxYCAgMCBmRkAg8QPCsACQEADxYEHwcWAB8IAgFkZBYCZg9kFgICAQ8PFgIfAgW1AUkndmUgYmVlbiB1c2luZyBUZWxlbGlzdGluZyBzZXJ2aWNlcyAgZm9yIHRoZSBsYXN0IDMgeWVhcnMgbm93IGFuZCB0aGV5IHJlYWxseSBoZWxwIG1lIGNvbmNlbnRyYXRlIG9uIGJ1aWxkaW5nIG15IGJ1c2luZXNzIGFuZCBpbiBjbG9zaW5nIG1vcmUgb3Bwb3J0dW5pdGllcyBldmVyeSB5ZWFyLg0KDQpNLiBTbWl0aC5kZAIRDw9kDxAWAWYWARYCHwkFBWVuLVVTFgECBmRkGAEFHl9fQ29udHJvbHNSZXF1aXJlUG9zdEJhY2tLZXlfXxYCBSNjdGwwMCRMb2dpblBvcHVwMSRMb2dpbjEkUmVtZW1iZXJNZQUpY3RsMDAkTG9naW5Qb3B1cDEkTG9naW4xJExvZ2luSW1hZ2VCdXR0b25RUp74UUPL72rYSQL3cfYMy9DWZU4l5e0kF3vCJf5fkQ==", "/wEWCQL9vYfeCAKVyobIBgKB94O0CwKz7JHYDQLsu7CWDgL31sOaAQLmjZ/bDQLto+ilAwLC4KCXAqOJh1kiYkswzxstW/o4u1wgJbA0/EnFO2L8sDvV8fVl", "Log In", "/HSIControl", "Telelisting Instruction Video", "");
HttpWebRequest getRequest = (HttpWebRequest)WebRequest.Create(getUrl);
getRequest.CookieContainer = new CookieContainer();
getRequest.CookieContainer.Add(cookies); //recover cookies First request
getRequest.Method = WebRequestMethods.Http.Post;
//Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
getRequest.ContentType = "text/html";
getRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.121 Safari/535.2";
getRequest.AllowWriteStreamBuffering = true;
getRequest.ProtocolVersion = HttpVersion.Version11;
getRequest.AllowAutoRedirect = true;
getRequest.ContentType = "application/x-www-form-urlencoded";
byte[] byteArray = Encoding.ASCII.GetBytes(postData);
getRequest.ContentLength = byteArray.Length;
Stream newStream = getRequest.GetRequestStream(); //open connection
newStream.Write(byteArray, 0, byteArray.Length); // Send the data.
newStream.Close();
HttpWebResponse getResponse = (HttpWebResponse)getRequest.GetResponse();
using (StreamReader sr = new StreamReader(getResponse.GetResponseStream()))
{
string sourceCode = sr.ReadToEnd();
Debug.WriteLine(sourceCode);
}

A server error in the 500 range means something gone wrong server side. From the looks of your request, I think it might be related to the Viewstate variable you are posting. The server will try to reconstruct the page according to your submitted state. It's not something you just copy and paste. Have you tried only sending the parameters that relate to the login itself (name, password, the name of the button clicked)?

For your work see fiddler run the same request and see the cookies match and other post parameters.
And also try :
postparameter= UrlEncode(postparameter)
Before converting them to byte and see if that works.
Maybe it works that way.

Why dont you go to the exact page you want to scrape in your browser and then export the full COOKIE set for manual attachment to your script's header?

Truncated response received using HttpWebRequest

I'm using the following code to make HttpWebRequests to a web site:
public static HttpWebResponse SendGETRequest(string url, string agent, CookieContainer cookieContainer)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = agent;
request.Method = "GET";
request.ContentType = "text/html";
request.CookieContainer = cookieContainer;
return (HttpWebResponse)request.GetResponse();
}
Everything worked fine with several web pages until I tried with a new one and only received the last part of the page. This is the received response:
<tr>
<td colspan="2" height="5"><spacer type="block" width="100%" height="5"></td>
</tr>
</table>
</td>
</tr>
</table>
</body>
</html>
The header is correct and says that only the received data is sent. The following are the headers of the request and response:
Request:
GET /Broker/Ops/FichaContratoJS.asp?nc=815044&IP=5&YY=2012&M=6&St=0&CC=FESX201206 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.168 Safari/535.19
Content-Type: text/html
Host: www.xxxx.com
Cookie: ASPSESSIONIDACBDCDBT=MGDNMNABOANDMILHBNCIDFCH;Autenticacion=Sid=230fae3d%2De0e2%2D4df1%2D8aa8%2D000fb352eaef&IdUsuarioWeb=xxxx; ASPSESSIONIDACBCCDAT=AFDJMNABAFJDDHABLOLAINDK; ASPSESSIONIDCADCBCAT=CEBJGNABLCALPJLDJFPBMLDE
Response:
HTTP/1.1 200 OK
Date: Wed, 09 May 2012 07:25:03 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Pragma: no-cache
**Content-Length: 155**
Content-Type: text/html
Expires: Wed, 09 May 2012 07:24:02 GMT
Set-Cookie: Autenticacion=Sid=230fae3d%2De0e2%2D4df1%2D8aa8%2D000fb352eaef&IdUsuarioWeb=xxxx; path=/
Cache-control: no-cache
Doing the same with a web browser works fine and returns a content length of about 4000 bytes.
Any ideas?
PD: Just in case it matters I'm doing several calls to the SendGETRequest from different threads to the same site but as there are no shared variables I think it shouldn't make a difference.
EDIT: This is the extension I use to extract the text from the Stream:
public static string ReadTextResponse(this Stream stream)
{
int count;
Encoding enconding = System.Text.Encoding.GetEncoding(1252);
System.Text.StringBuilder stringBuilder = new StringBuilder();
byte[] buffer = new byte[1023];
do
{
count = stream.Read(buffer, 0, buffer.Length);
if (count != 0)
{
string tempString = enconding.GetString(buffer, 0, count);
stringBuilder.Append(tempString);
}
}
while (count > 0);
return stringBuilder.ToString();
}
As far as I know it's correct. Also, note that the response header from the server contains the length of the truncated data

I think that you are not using right the HttpWebResponse object.
Maybe you are not closing the request or reading all the response strem.
http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.getresponse.aspx
Your method should be:
public static string SendGETRequest(string url, string agent, CookieContainer cookieContainer)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = agent;
request.Method = "GET";
request.ContentType = "text/html";
request.CookieContainer = cookieContainer;
string result;
using (var myResponse = (HttpWebResponse) request.GetResponse())
{
using (var stream = myResponse.GetResponseStream())
{
result = null;
if (stream != null)
{
using (var sr = new StreamReader(stream, System.Text.Encoding.UTF8))
{
result = sr.ReadToEnd();
sr.Close();
}
stream.Close();
}
}
myResponse.Close();
}
return result;
}

Incredible ... I was sending the URL /Broker/Ops/FichaContratoJS.asp?nc=815044&IP=5&YY=2012&M=6 and the browser was sending /Broker/Ops/FichaContratoJS.asp?nc=815044&IP=5&YY=2012&M=06& (note the extra 0 on the M parameter (it's a month). Putting there that 0 returned the full page. Sounds like a defect to me

I have run into a similar situation and found that copying the response stream into a MemoryStream seemed to fix my problems.
public static string SendGETRequest(string url, string agent, CookieContainer cookieContainer)
{
var request = (HttpWebRequest)WebRequest.Create(url);
request.UserAgent = agent;
request.Method = "GET";
request.ContentType = "text/html";
request.CookieContainer = cookieContainer;
string result;
using (var myResponse = (HttpWebResponse) request.GetResponse())
{
using (var stream = myResponse.GetResponseStream())
{
result = null;
if (stream != null)
{
MemoryStream memStream = new MemoryStream();
stream.CopyTo(memStream);
memStream.Flush();
stream.Close();
using (var sr = new StreamReader(memStream, System.Text.Encoding.UTF8))
{
result = sr.ReadToEnd();
sr.Close();
}
memStream.Close();
}
}
myResponse.Close();
}
return result;
}