We Keep Coding

C# WebRequest returns intermittent 500 errors from valid URLs

I have an application that was written to read and parse a large quantity of XML from a web service. It's been working fine for months. Quite suddenly one particular section of the code is throwing 500 errors when trying to get responses from the server.
On investigation, hitting one of these problematic URLs in a browser gets a valid response and the expected XML. To further add to the mystery, this section iterates through a number of requests varying the parameters in the URL, and the failures are intermittent - there's not one particular call that's causing the issue.
I have tried hitting the failure, requesting via the browser, then proceeding with the url from the code in debug mode: this still results in a 500 error.
Here's the code - there's some notes on attempted fixes in the comments. Essentially I've seen a number of questions on this already, and the suggested fix is to add a user-agent. It didn't work in this case, and previous questions haven't had intermittent problems.
private static HttpWebResponse RetryGetResponse(WebRequest request)
{
int failCounter = 1;
while (failCounter < 20)
{
try
{
// I tried this as a fix for the issue
// the code functions but it didn't fix the bug
// HttpWebRequest req = (HttpWebRequest)request;
// req.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
// return (HttpWebResponse)req.GetResponse();
// this is original code
return (HttpWebResponse)request.GetResponse();
}
catch (WebException ex)
{
if (ex.Status != WebExceptionStatus.ReceiveFailure &&
ex.Status != WebExceptionStatus.ConnectFailure &&
ex.Status != WebExceptionStatus.KeepAliveFailure)
{
LogError(string.Format("Failed to contact server, retrying {0} times", failCounter), ex);
Thread.Sleep(failCounter * 10000);
failCounter++;
}
}
}
throw new Exception("Fatal error - unable to connect");
}
public static void PopulateActions()
{
// this fetches data to iterate through
DataTable customers = GetTableData("Customers");
foreach (DataRow row in customers.Rows)
{
string url = string.Format(baseUrlTemplate, "subscribers", "actions.xml?email=" + HttpUtility.UrlEncode(row["EmailAddress"].ToString()));
WebRequest request = WebRequest.Create(url);
request.Timeout = 999999;
request.Credentials = new NetworkCredential(networkKey, "");
using (WebResponse response = RetryGetResponse(request))
{
XDocument doc = XDocument.Load(response.GetResponseStream());
IEnumerable<XElement> emails = doc.Root.Elements("Email");
foreach (XElement email in emails)
{
// parse the xml and record to Db
}
}
}
}
What else can I do to investigate and attempt to fix this infuriating intermittent problem?
EDIT: Response details
StatusCode: InternalServerError
StatusDescription: "Internal Server Error"
Headers:
{
Vary: Accept-Encoding
X-Cache: MISS
X-Code: 500
X-Error-Type: WS-Failure
X-Request-Duration: 30457ms
X-Request-Id: 979b6ee7-59c0-41e8-8acc-a38ec9b4087b
transfer-encoding: chunked
Connection: keep-alive
Cache-Control: private, s-maxage=0
Content-Type: application/xml
Date: Thu, 05 Jan 2017 11:02:32 GMT
P3P: CP="OTI DSP COR CUR IVD CONi OTPi OUR IND UNI STA PRE"
Server: cm-api-server
}

C# WebClient HTTP Basic Authentication Failing 401 with Correct Credentials

I'm trying to automate configuring a wireless router's SSID and Password via c# webclient. The router has no API that I know of. It's an unbranded chinese router. The web config seems to be the only option for configuration. It uses http-basic-authentication (you browse to the IP address of the router and get a generic dialog asking for username and password).
I've used Wireshark to get the headers and form fields that the http-post requests use when I manually update the SSID and Password (two separate forms). I then attempted to use webclient to emulate those post requests.
Here is a snippet of code that I am using to attempt to save a new SSID (NameValueCollection is defined elsewhere):
private const string FORM_SSID = "http://192.168.1.2/formWlanSetup.htm";
private const string REF_SSID = "http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0";
private NameValueCollection mFields = HttpUtility.ParseQueryString(string.Empty, Encoding.ASCII);
public string SaveConfigResponse()
{
try
{
using (WebClient wc = new WebClient())
{
wc.Headers[HttpRequestHeader.Accept] = "text/html, application/xhtml+xml, */*";
wc.Headers[HttpRequestHeader.Referer] = REF_SSID;
wc.Headers[HttpRequestHeader.AcceptLanguage] = "en-US";
wc.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
wc.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
wc.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
wc.Headers[HttpRequestHeader.Host] = "192.168.1.2";
wc.Headers[HttpRequestHeader.Connection] = "Keep-Alive";
wc.Headers[HttpRequestHeader.ContentLength] = Encoding.ASCII.GetBytes(mFields.ToString()).Length.ToString();
wc.Headers[HttpRequestHeader.CacheControl] = "no-cache";
string credentials = Convert.ToBase64String(Encoding.ASCII.GetBytes(config_user + ":" + config_pass));
wc.Headers[HttpRequestHeader.Authorization] = string.Format("Basic {0}", credentials);
//wc.Credentials = new NetworkCredential("admin", "admin");
return Encoding.ASCII.GetString(wc.UploadValues(FORM_SSID, "POST", mFields));
}
}
catch (Exception ex)
{
return ex.Message;
}
}
This results in an http-status-code-401 not authorized response. Is what I'm trying to do just impossible?
UPDATE
Here are the HTTP headers of both the browser post/response and the WebClient post/response. Again, I tried to match what I saw the browser posting as well as I could with my WebClient post.
Browser:
POST /formWlanSetup.htm HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Referer: http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0
Accept-Language: en-US
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: 192.168.1.2
Content-Length: 524
Connection: Keep-Alive
Cache-Control: no-cache
Authorization: Basic YWRtaW46YWRtaW4=
HTTP/1.1 302 Found
Location: wlbasic.htm
Content-Length: 183
Date: Thu, 23 Oct 2014 18:18:27 GMT
Server: eCos Embedded Web Server
Connection: close
Content-Type: text/html
Transfer-Encoding: chunked
Cache-Control: no-cache
WebClient:
POST /formWlanSetup.htm HTTP/1.1
Accept-Language: en-US
Accept-Encoding: gzip, deflate
Cache-Control: no-cache
Authorization: Basic YWRtaW46YWRtaW4=
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
Referer: http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Host: 192.168.1.2
Content-Length: 524
Connection: Keep-Alive
HTTP/1.1 401 Not Authorized
WWW-Authenticate: Basic realm="AP"
Date: Thu, 23 Oct 2014 18:18:41 GMT
Server: eCos Embedded Web Server
Connection: close
Content-Type: text/html
Transfer-Encoding: chunked
Cache-Control: no-cache
Again, that was all gleaned from Wireshark. I'm not very familiar with Wireshark, but I was able to get this far. If I knew how to properly extract the raw packet data and pastebin it, I would.
Important New Observations
The Wireshark captures of the post packets from both Browser and WebClient obviously differ in the order of the headers. I don't know how significant that might or might not be, though, as the data for each header is clearly the same.
One stark difference between the packets that I noticed is that Wireshark reports the Browser packet to be significantly larger than the WebClient packet. Looking at the itemized view, I couldn't find any obvious differences. I assume posting raw data for comparison would reveal a lot, but again, I don't really know how to do that.
I had a bewildering revelation. Despite the response clearly stating '(401) Unauthorized', the post is in fact being accepted by the router! Driving in to the router's web config after my WebClient post shows that the settings were accepted and saved.
That last one is a biggie. I find myself in a situation where I can get my config to save with a WebClient post, but I have to ignore a 401 response in order to do so. Obviously, this is far from ideal. So close, yet so far!
FINAL UPDATE (RESOLUTION)
I've solved the issue of failing basic authentication, though not with WebClient. I used the suggestion from #caesay and went with HttpWebRequest (together with WebResponse). My form posts result in redirects, so I had to allow for that.
This is essentially what I went with:
private bool ConfigureRouter()
{
bool passed = false;
string response = "";
HttpWebRequest WEBREQ = null;
WebResponse WEBRESP = null;
// Attempt to POST form to router that saves a new SSID.
try
{
var uri = new Uri(FORM_SSID); // Create URI from URL string.
WEBREQ = HttpWebRequest.Create(uri) as HttpWebRequest;
// If POST will result in redirects, you won't see an "OK"
// response if you don't allow those redirects
WEBREQ.AllowAutoRedirect = true;
// Basic authentication will first send the request without
// creds. This is protocol standard.
// When the server replies with 401, the HttpWebRequest will
// automatically send the request again with the creds when
// when PreAuthenticate is set.
WEBREQ.PreAuthenticate = true;
WEBREQ.AuthenticationLevel = System.Net.Security.AuthenticationLevel.MutualAuthRequested;
// Mimic all headers known to satisfy the request
// as discovered with a tool like Wireshark or Fiddler
// when the form was submitted from a browser.
WEBREQ.Method = "POST";
WEBREQ.Accept = "text/html, application/xhtml+xml, */*";
WEBREQ.Headers.Add("Accept-Language", "en-US"); // No AcceptLanguage property built-in to HttpWebRequest
WEBREQ.UserAgent = USER_AGENT;
WEBREQ.Referer = REF_SSID;
WEBREQ.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
WEBREQ.KeepAlive = true;
WEBREQ.Headers.Add("Pragma", "no-cache"); // No Pragma property built-in to HttpWebRequest
// Use a cached credential so that the creds are properly
// submitted with subsequent redirect requests.
CredentialCache creds = new CredentialCache();
creds.Add(uri, "Basic", new NetworkCredential(config_user, config_pass));
WEBREQ.Credentials = creds;
// Submit the form.
using (Stream stream = WEBREQ.GetRequestStream())
{
SSID ssid = new SSID(ssid_scanned); // Gets predefined form fields with new SSID inserted (NameValueCollection PostData)
stream.Write(ssid.PostData, 0, ssid.PostData.Length);
}
// Get the response from the final redirect.
WEBRESP = WEBREQ.GetResponse();
response = ((HttpWebResponse)WEBRESP).StatusCode.ToString();
if (response == "OK")
{
StatusUpdate("STATUS: SSID save was successful.");
passed = true;
}
else
{
StatusUpdate("FAILED: SSID save was unsuccessful.");
passed = false;
}
WEBRESP.Close();
}
catch (Exception ex)
{
StatusUpdate("ERROR: " + ex.Message);
return false;
}
return passed;
}

Is what I'm trying to do just impossible?
No, its not impossible. I have had many headaches with web scraping like this over the years because some web servers are picky, and your router interface is likely a custom web server implementation that isnt as forgiving as apache or iis.
I would do a wireshark capture and get the raw packet data that chrome sends (w/ payload etc), and then do the same capture for your application. Make sure the packets are as similar as you can get them. If you still have issues, post the packet captures to pastebin or something so we can have a look.
EDIT::
Instead of using the limited WebClient API, try using some lower level items, I wonder if the following code will work for you:
var uri = new Uri("http://192.168.1.2/formWlanSetup.htm");
var cookies = new CookieContainer();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);
request.CookieContainer = cookies;
request.ServicePoint.Expect100Continue = false;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko";
request.Referer = "http://192.168.1.2/formRedirect.htm?redirect-url=wlbasic.htm&wlan_id=0";
request.Credentials = new NetworkCredential(config_user, config_pass);
request.PreAuthenticate = true;
var response = request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
string htmlResponse = reader.ReadToEnd();

HttpWebRequest throws exception in code but not browser

I'm currently writing an API that gets data from a Point of Sale System's web interface. So far, I haven't had any problems logging in and generating reports to get data from until this situation.
In general, I can use the follow method to return an HttpWebRequest object that does the trick for most request to the web server.
private HttpWebRequest DefaultRequestObject(string path)
{
var request = (HttpWebRequest)WebRequest.Create(_baseUrl + path);
request.Method = "GET";
request.Host = _baseUrl.Substring(8); // Cut off the https://
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Accept-Language", "en-US,en;q=0.5");
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.SendChunked = false;
request.AllowAutoRedirect = true;
request.ProtocolVersion = HttpVersion.Version11;
var sp = request.ServicePoint;
sp.Expect100Continue = false;
var prop = sp.GetType().GetProperty("HttpBehaviour", BindingFlags.Instance | BindingFlags.NonPublic);
prop.SetValue(sp, (byte)0, null);
request.CookieContainer = _cookieJar;
if (!String.IsNullOrEmpty(_cookieString))
request.Headers.Add(HttpRequestHeader.Cookie, _cookieString);
return request;
}
When I send a GET request, I use the following method:
public MgrngResponse GetContent(string serverPath, Dictionary<string, string> args)
{
if (!serverPath.StartsWith("/"))
serverPath = "/~pos/mgrng/" + serverPath;
var requestString = serverPath + "?" + HttpDataFormat(args);
var request = DefaultRequestObject(requestString);
try
{
var response = (HttpWebResponse)request.GetResponse();
var mgrngResponse = new MgrngResponse(response);
if (!String.IsNullOrEmpty(mgrngResponse.HttpResponse.GetResponseHeader("Set-Cookie")))
SaveMgrngResponseCookies(mgrngResponse);
_sessionExpiration = DateTime.Now.AddMinutes(15);
UpdateStatus(mgrngResponse.Content);
return mgrngResponse;
}
catch (WebException webException)
{
using (WebResponse response = webException.Response)
{
var httpResponse = (HttpWebResponse)response;
Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
using (Stream data = response.GetResponseStream())
using (var reader = new StreamReader(data))
{
string text = reader.ReadToEnd();
Console.WriteLine(text);
}
}
var eventArgs = new SessionUpdateEventArgs(SessionStatus.ConnectionError, "Unable to GET data");
RaiseStatusChangeEvent(eventArgs);
return null;
}
}
This works well for all of the pages I've attempted so far, but now I'm running into a new problem where when I try to get the response for a particular page, the method throw a WebException with a 500 Internal Server Error.
I used Fiddler to match a browser's request exactly with mine (with the exception of the order of the headers and the cookie values obviously) but I'm still getting the Internal Server Error.
Below are the Raw Requests from Fiddler. The first one is from Firefox and the second one is from my program.
GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
Host: location.posprovider.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: USER=af182fda473354eb3199522726ca61c9d5516c95f165cffd63b9522726ca61c9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9dc714cb52a46278e4399720706ea41e9fc516c950a6607ae63b9522726ca61c9; PHPSESSID=9d7f54f9a1769a3e0572745fe0db3d97
Connection: keep-alive
GET https://location.posprovider.com/~pos/mgrng/Report.php?boc_brand=7&csv_delimeter=csv_delimeter_comma&format=text&format1=csv&format1=txt&format1=pdf&format1=html HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Host: location.posprovider.com
Cookie: PHPSESSID=16ad21c9d69fe85b3d185ab284f8620b; USER=30dca66f355e0ba89b6eda3c3e822ea24a95e5209e0f90bec94eda3c3e822ea243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e8243b5c500582b78cde96efa1c1ea20e826395e520780b58edc94eda3c3e822ea2
Connection: Keep-Alive
I even tried logging into the web interface and then copy and pasting my generated request string into the browser and I got the desired data.
Any ideas?

Pull your programs request into Fiddlers composer and gradually eliminate the remaining differences. What do you find doing that?
Great advice! Apparently there's something wrong with my cookie line.
I copy and pasted the cookie line from the browser's version into mine
and it worked. Now I just need to resolve why my cookies aren't
correct...
I agree with that diagnosis. Who knows what's going on inside of the server. Probably some fragile code that expect a very exact cookie string format.

Malformed string exception while sending "&" in Json to AppEngine

I am trying to send Facebook graph link to the AppEngine server. I receive "Malformed string exception". Here is my method sending json to server:
public async Task<string> SendJSONData(string urlToCall, string JSONData)
{
// server to POST to
string url = urlToCall;
// HTTP web request
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "application/x-www-form-urlencoded";
httpWebRequest.Method = "POST";
// Write the request Asynchronously
using (var stream = await Task.Factory.FromAsync<Stream>(httpWebRequest.BeginGetRequestStream,
httpWebRequest.EndGetRequestStream, null))
{
//create some json string
string json = "action=" + JSONData;
// convert json to byte array
byte[] jsonAsBytes = Encoding.UTF8.GetBytes(json);
// Write the bytes to the stream
await stream.WriteAsync(jsonAsBytes, 0, jsonAsBytes.Length);
}
WebResponse response = await httpWebRequest.GetResponseAsync();
StreamReader requestReader = new StreamReader(response.GetResponseStream());
String webResponse = requestReader.ReadToEnd();
return webResponse; }
Here is what I sniff using Fiddler:
POST http://x.appspot.com/register HTTP/1.1
Accept: */*
Content-Length: 376
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
User-Agent: NativeHost
Host: x.appspot.com
Connection: Keep-Alive
Pragma: no-cache
action={
"mailFb": "mail#gmail.com",
"userName": "Michael",
"userSurname": "w00t",
"nickname": "Michael w00t",
"userSex": "male",
"userAvatar": "https://graph.facebook.com/myperfectid/picture?type=large&access_token=BlahblahblahblahToken"
}
So everything looks fine, but the problem is that i receive the following error in AppEngine log:
2013-03-02 17:52:10.431 /register 500 56ms 0kb NativeHost
W 2013-03-02 17:52:10.427 /register com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 7 column 79 at com.google.g
C 2013-03-02 17:52:10.429 Uncaught exception from servlet com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string at line 7 colu
I managed to narrow the problem to its source, which is "&" character. So my question is, how to fix the code, so that it works with AppEngine.
Oh, here is how i read the received data on the server:
gson.fromJson(reader, User.class);

The problem is highlighted by the fact you're claiming you are sending "Content-Type: application/x-www-form-urlencoded"
But it isn't. Hence the error.
The correct encoding for & is &.

submitting form programmatically

Im trying to submit a specific form programatically, but I allways get the initial page back.
I must be doing something wrong or missing something here.
Im sending the session cookie and some POST data like viewState (that I parse from the initial request), and SessionID (this is the value i change in the form toget data from other years). But in the second request I allways get data for Session 899, instead of the one i request: 875.
Here is the code used:, any help is greatly apreciated
retrieveEdmIndexForSession(875);
protected string retrieveEdmIndexForSession(int sessionId) {
CookieContainer cookies;
HttpWebRequest oRequest;
HttpWebResponse oResponse;
Stream sw;
StreamReader sr;
string pageData;
string PathRemote = #"http://edmi.parliament.uk/EDMi/EDMList.aspx";
/*
* download the index page so we can get Cookies and ViewState from it.
*/
oRequest = (HttpWebRequest)WebRequest.Create(PathRemote);
oRequest.Method = "GET";
oRequest.AllowAutoRedirect = true;
oRequest.CookieContainer = new CookieContainer();
oRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
oRequest.Referer = "http://edmi.parliament.uk/EDMi/EDMList.aspx";
oResponse =(HttpWebResponse) oRequest.GetResponse();
sr = new StreamReader(oResponse.GetResponseStream());
pageData = sr.ReadToEnd();
/*
* extract view state from pageData.
*/
string viewState = this.ExtractViewState(pageData);
/*
* lets submit the form with the parameters we want
*/
oRequest = (HttpWebRequest)WebRequest.Create(PathRemote);
oRequest.Method = "POST";
oRequest.AllowAutoRedirect = true;
oRequest.ContentType = "application/x-www-form-urlencoded";
oRequest.CookieContainer = new CookieContainer();
oRequest.CookieContainer.Add(oResponse.Cookies);
oRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
oRequest.Referer = "http://edmi.parliament.uk/EDMi/EDMList.aspx";
string postdata = "__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=" + viewState + "&_MenuCtrl%3AddlSession=" + sessionId + "&_MenuCtrl%3A_GoTo.x=57&_MenuCtrl%3A_GoTo.y=14&ddlStatus=1&ddlSortedBy=1";
byte[] buffer = Encoding.UTF8.GetBytes(postdata);
oRequest.ContentLength = buffer.Length;
/*
* Send post data into request stream first
*/
sw = oRequest.GetRequestStream();
sw.Write(buffer, 0, buffer.Length);
sw.Flush();
sw.Close();
/*
* Connect, send and get response
*/
oResponse = (HttpWebResponse)oRequest.GetResponse();
sr = new StreamReader(oResponse.GetResponseStream());
//OnLogUpdated(1, "\r\nStatus Code: " + oResponse.StatusCode);
//OnLogUpdated(1, "\r\nServer: " + oResponse.Server);
pageData = sr.ReadToEnd();
string result = getSessionId(pageData);
//OnLogUpdated(1, "\r\nRestuls: [" + result + "]");
//OnLogUpdated(1, "\r\nPage: [" + pageData + "]");
return pageData;
}
private string ExtractViewState(string str)
{
string viewState = "";
string pattern = "(?<=__VIEWSTATE\" value=\")(?<val>.*?)(?=\")";
Match match = Regex.Match(str, pattern);
if (match.Success)
{
viewState = match.Groups["val"].Value;
viewState = HttpUtility.UrlEncodeUnicode(viewState);
}
return viewState;
}
protected string getSessionId(string str)
{
string sessionId = string.Empty;
str = str.Trim();
string pattern = #"&SESSION=([^']+)'";
Match match = Regex.Match(str, pattern, RegexOptions.IgnoreCase);
if (match.Success)
{
sessionId = match.Groups[1].ToString(); ;
}
return sessionId;
}
This is the RAW Request being sent by the .NET script.
POST /EDMi/EDMList.aspx HTTP/1.1
Content-Type:
application/x-www-form-urlencoded
Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Referer:
http://edmi.parliament.uk/EDMi/EDMList.aspx
User-Agent: .NET Framework Client
Host: edmi.parliament.uk Cookie:
ASP.NET_SessionId=k55fqarvx2oszp2wxhtrol45
Content-Length: 2431 Expect:
100-continue
__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=dDwxMDgyNzIxNDQ2O3Q8O2w8aTwzPjs%2bO2w8dDw7bDxpPDE%2bO2k8Mz47aTw1PjtpPDExPjs%2bO2w8dDw7bDxpPDEzPjtpPDE3Pjs%2bO2w8dDx0PHA8cDxsPERhdGFWYWx1ZUZpZWxkO0RhdGFUZXh0RmllbGQ7PjtsPFNFU1NJT05JRDtJVEVNX1ZBTFVFOz4%2bOz47dDxpPDIwPjtAPDA4LTA5OzA3LTA4OzA2LTA3OzA1LTA2OzA0LTA1OzAzLTA0OzAyLTAzOzAxLTAyOzAwLTAxOzk5LTAwOzk4LTk5Ozk3LTk4Ozk2LTk3Ozk1LTk2Ozk0LTk1OzkzLTk0OzkyLTkzOzkxLTkyOzkwLTkxOzg5LTkwOz47QDw4OTk7ODkxOzg4NTs4NzU7ODczOzY4Mjs2ODE7NjgwOzY3OTs3MDM7NzAyOzcwMTs3MDA7Njk5OzY5ODs2OTc7Njk2OzY5NTs2OTQ7NjkzOz4%2bOz47Oz47dDxwPGw8VGV4dDs%2bO2w8TGlzdCBPZiBFYXJseSBEYXkgTW90aW9uczs%2bPjs7Pjs%2bPjt0PDtsPGk8MT47aTwzPjs%2bO2w8dDx0PDs7bDxpPDA%2bOz4%2bOzs%2bO3Q8dDw7O2w8aTwwPjs%2bPjs7Pjs%2bPjt0PDtsPGk8MT47aTwzPjs%2bO2w8dDw7bDxpPDE%2bO2k8Mz47aTw1PjtpPDc%2bOz47bDx0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztFbmFibGVkO18hU0I7PjtsPDA7UGFnZUZpcnN0RGlzYWJsZWQ7bzxmPjtpPDI%2bOz4%2bOz47Oz47dDxwPHA8bDxDb21tYW5kQXJndW1lbnQ7Q3NzQ2xhc3M7RW5hYmxlZDtfIVNCOz47bDwtMTtQYWdlUHJldkRpc2FibGVkO288Zj47aTwyPjs%2bPjs%2bOzs%2bO3Q8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO18hU0I7PjtsPDE7UGFnZU5leHRFbmFibGVkO2k8Mj47Pj47Pjs7Pjt0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztfIVNCOz47bDw0MjtQYWdlTGFzdEVuYWJsZWQ7aTwyPjs%2bPjs%2bOzs%2bOz4%2bO3Q8O2w8aTwxPjtpPDM%2bO2k8NT47aTw3Pjs%2bO2w8dDxwPHA8bDxUZXh0Oz47bDwyMTA5Oz4%2bOz47Oz47dDxwPHA8bDxUZXh0Oz47bDxFRE1zIGFuZCBBbWVuZG1lbnRzOz4%2bOz47Oz47dDxwPHA8bDxUZXh0Oz47bDwxOz4%2bOz47Oz47dDxwPHA8bDxUZXh0Oz47bDw1MDs%2bPjs%2bOzs%2bOz4%2bOz4%2bO3Q8O2w8aTwxPjtpPDM%2bOz47bDx0PDtsPGk8MT47aTwzPjtpPDU%2bO2k8Nz47PjtsPHQ8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO0VuYWJsZWQ7XyFTQjs%2bO2w8MDtQYWdlRmlyc3REaXNhYmxlZDtvPGY%2bO2k8Mj47Pj47Pjs7Pjt0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztFbmFibGVkO18hU0I7PjtsPC0xO1BhZ2VQcmV2RGlzYWJsZWQ7bzxmPjtpPDI%2bOz4%2bOz47Oz47dDxwPHA8bDxDb21tYW5kQXJndW1lbnQ7Q3NzQ2xhc3M7XyFTQjs%2bO2w8MTtQYWdlTmV4dEVuYWJsZWQ7aTwyPjs%2bPjs%2bOzs%2bO3Q8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO18hU0I7PjtsPDQyO1BhZ2VMYXN0RW5hYmxlZDtpPDI%2bOz4%2bOz47Oz47Pj47dDxwPHA8bDxWaXNpYmxlOz47bDxvPGY%2bOz4%2bOz47bDxpPDE%2bO2k8Mz47aTw1PjtpPDc%2bOz47bDx0PHA8cDxsPFRleHQ7PjtsPDIxMDk7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPEVETXMgYW5kIEFtZW5kbWVudHM7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPDE7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPDUwOz4%2bOz47Oz47Pj47Pj47Pj47Pj47bDxfTWVudUN0cmw6X0dvVG87Pj5NHcFbPBNzNuwxs7sYLdUE2omkjw%3d%3d&_MenuCtrl%3AddlSession=875&_MenuCtrl%3A_GoTo.x=57&_MenuCtrl%3A_GoTo.y=14&ddlStatus=1&ddlSortedBy=1
This is the RAW request sent by IE:
POST /EDMi/EDMList.aspx HTTP/1.1
Accept: image/gif, image/jpeg,
image/pjpeg, image/pjpeg,
application/x-shockwave-flash,
application/xaml+xml,
application/vnd.ms-xpsdocument,
application/x-ms-xbap,
application/x-ms-application,
application/vnd.ms-excel,
application/vnd.ms-powerpoint,
application/msword, / Referer:
http://edmi.parliament.uk/EDMi/EDMList.aspx
Accept-Language: en-gb User-Agent:
Mozilla/4.0 (compatible; MSIE 8.0;
Windows NT 5.1; Trident/4.0; .NET CLR
1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; InfoPath.1; .NET CLR 3.0.04506.648; OfficeLiveConnector.1.3;
OfficeLivePatch.0.0; .NET CLR
3.0.4506.2152; .NET CLR 3.5.30729) Content-Type:
application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate Host:
edmi.parliament.uk Content-Length:
2431 Connection: Keep-Alive Pragma:
no-cache Cookie:
WT_FPC=id=83.217.99.254-2364242496.30021299:lv=1249572414567:ss=1249572414567;
ASP.NET_SessionId=vwxgo4rlex1j5m55l0bivrqo
__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE=dDwxMDgyNzIxNDQ2O3Q8O2w8aTwzPjs%2BO2w8dDw7bDxpPDE%2BO2k8Mz47aTw1PjtpPDExPjs%2BO2w8dDw7bDxpPDEzPjtpPDE3Pjs%2BO2w8dDx0PHA8cDxsPERhdGFWYWx1ZUZpZWxkO0RhdGFUZXh0RmllbGQ7PjtsPFNFU1NJT05JRDtJVEVNX1ZBTFVFOz4%2BOz47dDxpPDIwPjtAPDA4LTA5OzA3LTA4OzA2LTA3OzA1LTA2OzA0LTA1OzAzLTA0OzAyLTAzOzAxLTAyOzAwLTAxOzk5LTAwOzk4LTk5Ozk3LTk4Ozk2LTk3Ozk1LTk2Ozk0LTk1OzkzLTk0OzkyLTkzOzkxLTkyOzkwLTkxOzg5LTkwOz47QDw4OTk7ODkxOzg4NTs4NzU7ODczOzY4Mjs2ODE7NjgwOzY3OTs3MDM7NzAyOzcwMTs3MDA7Njk5OzY5ODs2OTc7Njk2OzY5NTs2OTQ7NjkzOz4%2BOz47Oz47dDxwPGw8VGV4dDs%2BO2w8TGlzdCBPZiBFYXJseSBEYXkgTW90aW9uczs%2BPjs7Pjs%2BPjt0PDtsPGk8MT47aTwzPjs%2BO2w8dDx0PDs7bDxpPDA%2BOz4%2BOzs%2BO3Q8dDw7O2w8aTwwPjs%2BPjs7Pjs%2BPjt0PDtsPGk8MT47aTwzPjs%2BO2w8dDw7bDxpPDE%2BO2k8Mz47aTw1PjtpPDc%2BOz47bDx0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztFbmFibGVkO18hU0I7PjtsPDA7UGFnZUZpcnN0RGlzYWJsZWQ7bzxmPjtpPDI%2BOz4%2BOz47Oz47dDxwPHA8bDxDb21tYW5kQXJndW1lbnQ7Q3NzQ2xhc3M7RW5hYmxlZDtfIVNCOz47bDwtMTtQYWdlUHJldkRpc2FibGVkO288Zj47aTwyPjs%2BPjs%2BOzs%2BO3Q8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO18hU0I7PjtsPDE7UGFnZU5leHRFbmFibGVkO2k8Mj47Pj47Pjs7Pjt0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztfIVNCOz47bDw0MjtQYWdlTGFzdEVuYWJsZWQ7aTwyPjs%2BPjs%2BOzs%2BOz4%2BO3Q8O2w8aTwxPjtpPDM%2BO2k8NT47aTw3Pjs%2BO2w8dDxwPHA8bDxUZXh0Oz47bDwyMTA5Oz4%2BOz47Oz47dDxwPHA8bDxUZXh0Oz47bDxFRE1zIGFuZCBBbWVuZG1lbnRzOz4%2BOz47Oz47dDxwPHA8bDxUZXh0Oz47bDwxOz4%2BOz47Oz47dDxwPHA8bDxUZXh0Oz47bDw1MDs%2BPjs%2BOzs%2BOz4%2BOz4%2BO3Q8O2w8aTwxPjtpPDM%2BOz47bDx0PDtsPGk8MT47aTwzPjtpPDU%2BO2k8Nz47PjtsPHQ8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO0VuYWJsZWQ7XyFTQjs%2BO2w8MDtQYWdlRmlyc3REaXNhYmxlZDtvPGY%2BO2k8Mj47Pj47Pjs7Pjt0PHA8cDxsPENvbW1hbmRBcmd1bWVudDtDc3NDbGFzcztFbmFibGVkO18hU0I7PjtsPC0xO1BhZ2VQcmV2RGlzYWJsZWQ7bzxmPjtpPDI%2BOz4%2BOz47Oz47dDxwPHA8bDxDb21tYW5kQXJndW1lbnQ7Q3NzQ2xhc3M7XyFTQjs%2BO2w8MTtQYWdlTmV4dEVuYWJsZWQ7aTwyPjs%2BPjs%2BOzs%2BO3Q8cDxwPGw8Q29tbWFuZEFyZ3VtZW50O0Nzc0NsYXNzO18hU0I7PjtsPDQyO1BhZ2VMYXN0RW5hYmxlZDtpPDI%2BOz4%2BOz47Oz47Pj47dDxwPHA8bDxWaXNpYmxlOz47bDxvPGY%2BOz4%2BOz47bDxpPDE%2BO2k8Mz47aTw1PjtpPDc%2BOz47bDx0PHA8cDxsPFRleHQ7PjtsPDIxMDk7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPEVETXMgYW5kIEFtZW5kbWVudHM7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPDE7Pj47Pjs7Pjt0PHA8cDxsPFRleHQ7PjtsPDUwOz4%2BOz47Oz47Pj47Pj47Pj47Pj47bDxfTWVudUN0cmw6X0dvVG87Pj5NHcFbPBNzNuwxs7sYLdUE2omkjw%3D%3D&_MenuCtrl%3AddlSession=885&ddlStatus=0&ddlSortedBy=1&_MenuCtrl%3A_GoTo.x=37&_MenuCtrl%3A_GoTo.y=12
The IE Header seems to have an extra cookie(WT_FPC=id=83.217.99.254-2364242496.30021299:lv=1249572414567:ss=1249572414567;) witch appers to track visitors using cookies via the WebTrends Cookie Plug-In. Both POST Requests Return HTTP Status Code 302 and redirect to a GET request that returns status 200.
Any ideas ?

You are creating a new CookieContainer each time. Try using the CookieContainer of the index page request as the container for the form submit, it should contain the extra cookie. Alternatively, something other than the form could be passing the extra cookie. Check when the page loads using IEinspector's HttpAnalyzer or Firefox's Firebug to make sure other form elements like images aren't passing the cookie.

I have cracked it. It seems that the .NET server does not like me changing the Status and the Session at the same time.
It works if i change the "ddlStatus" from:
string postdata =
"__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE="
+ viewState + "&_MenuCtrl%3AddlSession=" + sessionId
+ "&_MenuCtrl%3A_GoTo.x=57&_MenuCtrl%3A_GoTo.y=14&ddlStatus=1&ddlSortedBy=1";
to this:
string postdata =
"__EVENTTARGET=&__EVENTARGUMENT=&__VIEWSTATE="
+ viewState + "&_MenuCtrl%3AddlSession=" + sessionId
+ "&_MenuCtrl%3A_GoTo.x=57&_MenuCtrl%3A_GoTo.y=14&ddlStatus=0&ddlSortedBy=1";
I will have to change the ddlStatus in a second POST request, and so on for each change in the form. .NET seems to be very well educated, It can only take one sweet at a time.
Thanks for all the help :)

Maybe this is not related to your problem, but sometimes when you scrape a site the remote server recognize that the one who's browsing is a script and block some operations. I usually workaround this via setting UserAgent property to the request. It's a poor security measure, but often used in my experience.
Here you can find some sample user agent strings.

The only thing that struck me as odd in that request is the Expect header. Apparently it's caused problems for people in the past. To stop that you can use the following line:
System.Net.ServicePointManager.Expect100Continue = false;
If that is the problem it's probably worth setting that property back to whatever it was before, since it's a global variable and may well affect other parts of your code.