I have realy strange problem with Headers collection in WebClient class.
Here is my example:
WebClient client = new WebClient();
client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0");
client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
client.Headers.Add("Accept-Language", "pl,en-us;q=0.7,en;q=0.3");
Console.WriteLine("Before request:");
foreach (string key in client.Headers)
{
Console.WriteLine(key + ": " + client.Headers[key]);
}
client.DownloadString("http://www.google.com");
Console.WriteLine();
Console.WriteLine("After request:");
foreach (string key in client.Headers)
{
Console.WriteLine(key + ": " + client.Headers[key]);
}
Console.ReadLine();
My result of running this simple program:
Before request:
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: pl,en-us;q=0.7,en;q=0.3
After request:
Accept-Language: pl,en-us;q=0.7,en;q=0.3
Why my headers are disappearing?
because headers are sent and webclient did its job. if you want same headers for next request, you should add them again.
Related
I am trying to download a pdf file from a web server over https in c# and getting HTTP OK response but file stream is not coming with response. What am I doing wrong?
Here is my code behind:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, ch, sec) => { return true; };
HttpWebRequest myRequest = (HttpWebRequest)HttpWebRequest.Create("https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport");
myRequest.Method = "POST";
myRequest.CookieContainer = cookies;
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.Accept = myRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
myRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
myRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
myRequest.Referer = "https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search";
myRequest.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36";
byte[] bytes = Encoding.Default.GetBytes(content);
myRequest.ContentLength = bytes.Length;
Stream dataStream = myRequest.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
byte[] result = null;
byte[] buffer = new byte[2048];
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
Downloading file from chrome is working fine and here is fiddler RAW http packet from browser:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Host: services.dps.ohio.gov
Connection: keep-alive
Content-Length: 177
Cache-Control: max-age=0
Origin: https://services.dps.ohio.gov
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=t1nddo51g0jotl51cqev15rp; __RequestVerificationToken_L0NyYXNoT25saW5l0=iMy0lJ3OviGs0iXame8ebp8IdS6K9wpc8tFinYd0nhNnySTLlii-yHYU5tDJjv49Fqi29M_31mPiPdKWWtJJPWSmp4qGGvna9lW2rkDHUAs1
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
This is my application's http packet:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Language: en-GB,es-ES;q=0.8,es;q=0.6,en-US;q=0.4,en;q=0.2
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: services.dps.ohio.gov
Content-Length: 177
Connection: Keep-Alive
Cache-Control: no-cache
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
I am trying to get info from a website that has no public API. I can't give the website's name because you literally can only sign in with a work account created on their end, so there's no way for other people to test this without my credentials (which I am not giving out).
class Client
{
public string PHPSESSID { get; set; }
public Client()
{
this.Login();
}
public void Login()
{
string csrf;
RestClient client;
RestRequest csrfRequest = new RestRequest(Method.GET);
csrfRequest.AddHeader("accept-language", "en-US,en;q=0.8");
csrfRequest.AddHeader("accept-encoding", "gzip, deflate, br");
csrfRequest.AddHeader("dnt", "1");
csrfRequest.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
csrfRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
csrfRequest.AddHeader("upgrade-insecure-csrfRequests", "1");
RestRequest request = new RestRequest(Method.POST);
request.AddHeader("x-requested-with", "XMLHttpRequest");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36");
request.AddHeader("referer", "{WEBSITE}/login");
request.AddHeader("origin", "{WEBSITE}");
request.AddHeader("host", "{WEBSITE}");
request.AddHeader("dnt", "1");
request.AddHeader("content-type", "application/x-www-form-urlencoded");
request.AddHeader("content-length", "118");
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
do
{
client = new RestClient("{WEBSITE}/login");
response = client.Execute(csrfRequest);
if (String.IsNullOrEmpty(this.PHPSESSID))
this.setPHPID(response);
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/login_check");
RestRequest bak = request.Clone();
request.AddCookie("PHPSESSID", this.PHPSESSID);
request.AddParameter("application/x-www-form-urlencoded", "form%5Busername%5D={USERNAME}&form%5Bpassword%5D={PASSWORD}&_csrf_token=" + csrf, ParameterType.RequestBody);
client.FollowRedirects = true;
response = client.Execute(request);
request = bak.Clone();
} while (response.Content.Contains("error"));
}
public string FindCSRF(string input)
{
int find = input.IndexOf("value=", input.IndexOf("_token")) + 8;
int find2 = input.IndexOf('"', find);
--find;
return input.Substring(find, find2-find);
}
public void setPHPID(IRestResponse response)
{
try
{
string cookie = response.Headers.ToList().Find(p => p.Name.Contains("Set-Cookie")).Value.ToString();
if(String.IsNullOrEmpty(cookie))
this.PHPSESSID = response.Cookies[0].Value;
else
{
int cookieStart = cookie.IndexOf("PHPSESSID=") + 10;
this.PHPSESSID = cookie.Substring(cookieStart, cookie.IndexOf(';', cookieStart) - cookieStart);
}
}
catch
{
this.PHPSESSID = response.Cookies[0].Value;
}
}
public void switchBranch(string branchID)
{
RestRequest request = new RestRequest(Method.GET);
request.AddHeader("accept-language", "en-US,en;q=0.8");
request.AddHeader("accept-encoding", "gzip, deflate, br");
request.AddHeader("referer", "{WEBSITE}/message-board");
request.AddHeader("dnt", "1");
request.AddHeader("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
request.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
request.AddHeader("upgrade-insecure-requests", "1");
RestRequest branchRequest = new RestRequest(Method.POST);
branchRequest.AddHeader("accept-language", "en-US,en;q=0.8");
branchRequest.AddHeader("accept-encoding", "gzip, deflate, br");
branchRequest.AddHeader("referer", "{WEBSITE}/message-board");
branchRequest.AddHeader("dnt", "1");
branchRequest.AddHeader("content-type", "application/x-www-form-urlencoded");
branchRequest.AddHeader("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");
branchRequest.AddHeader("x-branchRequested-with", "XMLHttpbranchRequest");
branchRequest.AddHeader("origin", "{WEBSITE}");
branchRequest.AddHeader("accept", "application/json, text/javascript, */*; q=0.01");
IRestResponse response;
RestClient client;
string csrf;
do
{
client = new RestClient("{WEBSITE}/");
client.FollowRedirects = true;
RestRequest acc = request.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
response = client.Execute(acc);
if (response.Content.Contains("login"))
{
this.Login();
Console.WriteLine("Login Required");
continue;
}
csrf = this.FindCSRF(response.Content);
client = new RestClient("{WEBSITE}/branch-switch");
acc = branchRequest.Clone();
acc.AddCookie("PHPSESSID", this.PHPSESSID);
acc.AddParameter("application/x-www-form-urlencoded", "office_select%5Boffice%5D=" + branchID + "&office_select%5B_token%5D=" + csrf, ParameterType.RequestBody);
} while (response.Content.Contains("error"));
}
}
public static class RequestExt
{
public static RestRequest Clone(this RestRequest req)
{
return req;
}
}
A lot of this I just intercepted from the website using Postman then copied the auto-generated code over, so if there are certain headers that can help, I probably don't know about them.
I mostly fixed the login issues (though I literally have to fail before it gives me a working CSRF token, thus the while loops). The problem is that when I try to go to {WEBSITE} in Client.switchBranch, to start actually doing stuff, it returns a JSON with a redirect to the login page. It doesn't do this in Postman, I have been able to successfully interact with the website and gather information... once or twice (logging in even sometimes fails in the browser).
It seems entirely random to me, but there MUST be some kind of common factor here. I'm guessing it's something to do with the PHPSESSID cookie or the lack of a "keep-alive" connection header (which caused RestSharp to throw an error when I tried to implement it; even if that is the problem, I still don't know how to solve it). The login request and response headers in Chrome's developer tools, intercepting from the website, look something like this.
I am trying to capture the return html after a button click. I've used fiddler to see what request gets sent when the user clicks a button. Below is the header information.
GET http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS HTTP/1.1
Host: www.nseindia.com
Proxy-Connection: keep-alive
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Referer: http://www.nseindia.com/products/content/equities/equities/bulk.htm
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I have the below code which is not working. Can someone please point me to the correct solution ?
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
client.Headers.Add(values);
client.Proxy = WebRequest.DefaultWebProxy;
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
You have to fill enough information for headers.
It works with this code:
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
values.Add("Accept", "*/*");
//values.Add("Accept-Encoding", "gzip, deflate, sdch");
values.Add("Accept-Language", "en-US,en;q=0.8");
client.Proxy = WebRequest.DefaultWebProxy;
//client.UploadValues(nseBulkDealsUrl, "GET", values);
client.Headers.Add(values);
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
I'm setting up some HTTP request headers like this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
WebResponse response = http.GetResponse();
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
I am capturing them like this:
for (count = 0; count < http.Headers.Keys.Count; count++)
{
headerKey = http.Headers.Keys[count];
headerValue = http.Headers[headerKey];
if (headerValue != null)
{
if (headerKey == null)
{
requestbuffer.Append(headerValue);
requestbuffer.Append(Newline);
}
else
{
requestbuffer.Append(headerKey + ": " + headerValue);
requestbuffer.Append(Newline);
}
}
}
When I run the testing tool everything seems good:
Host: domain.com
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0)Gecko/20100101 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.9
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
However in Wireshark and Fiddler only the following header is sent:
GET / HTTP/1.1
Host: domain.com
Any idea why that may be?
You're trying to add the headers after you've called http.GetResponse(). That's after the request has been sent. Change it to this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
using (WebResponse response = http.GetResponse())
{
// Do whatever
}
(Note that you really should be disposing of the response.)
Yes, you are setting the header after you have sent the request.
Try this:
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
//Added diesposal of response too
using (WebResponse response = http.GetResponse())
{
}
This code
using (WebClient webClient = new WebClient())
{
string address = "https://www.any.com/name.htm";
byte[] postData = Encoding.ASCII.GetBytes("login=123");
webClient.Headers[HttpRequestHeader.Accept] = "text/html, application/xhtml+xml, */*";
webClient.Headers[HttpRequestHeader.AcceptLanguage] = "ru-RU";
webClient.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)";
webClient.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
webClient.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
webClient.UploadData(address, postData);
}
produces this request (postData doubles):
POST /name.htm HTTP/1.1
Accept-Language: ru-RU
Accept-Encoding: gzip, deflate
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Host: [url]www.any.com[/url]
Content-Length: 9
Connection: Keep-Alive
login=123login=123
If I change
string address = "https://www.any.com/name.htm"
to
string address = "http://www.any.com/name.htm"
then data don't doubles.
How avoid data doubling when send request by HTTPS ?
It is bug of HttpAnalizer.
Many thanks to Hardrada.