I am trying to download a pdf file from a web server over https in c# and getting HTTP OK response but file stream is not coming with response. What am I doing wrong?
Here is my code behind:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback = (s, cert, ch, sec) => { return true; };
HttpWebRequest myRequest = (HttpWebRequest)HttpWebRequest.Create("https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport");
myRequest.Method = "POST";
myRequest.CookieContainer = cookies;
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.Accept = myRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8";
myRequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate, br");
myRequest.Headers.Add(HttpRequestHeader.AcceptLanguage, "en-US,en;q=0.9");
myRequest.Referer = "https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search";
myRequest.UserAgent = "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36";
byte[] bytes = Encoding.Default.GetBytes(content);
myRequest.ContentLength = bytes.Length;
Stream dataStream = myRequest.GetRequestStream();
dataStream.Write(bytes, 0, bytes.Length);
dataStream.Close();
byte[] result = null;
byte[] buffer = new byte[2048];
using (HttpWebResponse response = (HttpWebResponse)myRequest.GetResponse())
Downloading file from chrome is working fine and here is fiddler RAW http packet from browser:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Host: services.dps.ohio.gov
Connection: keep-alive
Content-Length: 177
Cache-Control: max-age=0
Origin: https://services.dps.ohio.gov
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: ASP.NET_SessionId=t1nddo51g0jotl51cqev15rp; __RequestVerificationToken_L0NyYXNoT25saW5l0=iMy0lJ3OviGs0iXame8ebp8IdS6K9wpc8tFinYd0nhNnySTLlii-yHYU5tDJjv49Fqi29M_31mPiPdKWWtJJPWSmp4qGGvna9lW2rkDHUAs1
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
This is my application's http packet:
POST https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/GetReport HTTP/1.1
Accept: text/html, application/xhtml+xml, image/jxr, */*
Referer: https://services.dps.ohio.gov/CrashOnline/CrashRetrieval/OhioCrashReportRetrieval/Search
Accept-Language: en-GB,es-ES;q=0.8,es;q=0.6,en-US;q=0.4,en;q=0.2
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko
Content-Type: application/x-www-form-urlencoded
Accept-Encoding: gzip, deflate
Host: services.dps.ohio.gov
Content-Length: 177
Connection: Keep-Alive
Cache-Control: no-cache
__RequestVerificationToken=eTJN31araRPeB-bhQaeU3awBDsCwR9BpkIYNqpsakTPRT1w1nXtbGFcfVRZS776ZW0NGv1aKahx3WJv5PNQ199iM7GF9Y9f7ws5KTG4gI3I1&id=NjgzMTIwOQ%3D%3D&btn_NjgzMTIwOQ%3D%3D=
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 days ago.
Improve this question
I want to do this request with C#
POST /ImageDownloadPro HTTP/1.1
Host: access.imgdownloader.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0
Accept: application/json, text/plain, */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Content-Type: application/json;charset=utf-8
Content-Length: 58
Origin: https://imgdownloader.com
DNT: 1
Connection: keep-alive
Referer: https://imgdownloader.com/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
{"url":"https://i.ytimg.com/vi/yMDfF_dNctE/hqdefault.jpg"}
output with firefox
{"code":200,"data":[{"originalurl":"https://access.imgdownloader.com/original_20230213020654271378/imgdownloader.com-20230213020654303205.jpg","thumburl":"https://access.imgdownloader.com/original_20230213020654271378/imgdownloader.com-20230213020654303205.jpg"}],"msg":null}
I tested this
System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
byte[] bytes = Encoding.ASCII.GetBytes($"{{"url":"https://i.ytimg.com/vi/yMDfF_dNctE/hqdefault.jpg"}}");
HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://test.com");
httpWebRequest.Method = "POST";
httpWebRequest.Host = "access.test.com";
httpWebRequest.KeepAlive = true;
httpWebRequest.AutomaticDecompression = DecompressionMethods.GZip;
httpWebRequest.Headers.Add("Accept-Language", "en-US,en;q=0.5");
httpWebRequest.Headers.Add("sec-fetch-dest", "document");
httpWebRequest.Headers.Add("sec-fetch-mode", "navigate");
httpWebRequest.Headers.Add("sec-fetch-site", "cross-site");
httpWebRequest.ContentType = "application/json;charset=utf-8";
httpWebRequest.ContentLength = bytes.LongLength;
httpWebRequest.Referer = "https://test.com/TestTe";
httpWebRequest.Accept = "\ttext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,/;q=0.8";
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/109.0";
using (Stream requestStream = httpWebRequest.GetRequestStream())
{
requestStream.Write(bytes, 0, bytes.Length);
}
string responseText = null;
using (WebResponse response = httpWebRequest.GetResponse())
{
using (Stream responseStream = response.GetResponseStream())
{
using (StreamReader streamReader = new StreamReader(responseStream))
{
responseText = streamReader.ReadToEnd();
}
}
}
return responseText;
But the result is very big and it seems to download the source of the site
I am trying to capture the return html after a button click. I've used fiddler to see what request gets sent when the user clicks a button. Below is the header information.
GET http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS HTTP/1.1
Host: www.nseindia.com
Proxy-Connection: keep-alive
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36
Referer: http://www.nseindia.com/products/content/equities/equities/bulk.htm
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
I have the below code which is not working. Can someone please point me to the correct solution ?
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
client.Headers.Add(values);
client.Proxy = WebRequest.DefaultWebProxy;
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
You have to fill enough information for headers.
It works with this code:
var nseBulkDealsUrl = #"http://www.nseindia.com/products/dynaContent/equities/equities/bulkdeals.jsp?symbol=&segmentLink=13&symbolCount=&dateRange=day&fromDate=&toDate=&dataType=DEALS";
var client = new WebClient();
client.Encoding = Encoding.UTF8;
var values = new NameValueCollection();
values.Add("Referer", "http://www.nseindia.com/products/content/equities/equities/bulk.htm");
values.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36");
values.Add("X-Requested-With", "XMLHttpRequest");
values.Add("Accept", "*/*");
//values.Add("Accept-Encoding", "gzip, deflate, sdch");
values.Add("Accept-Language", "en-US,en;q=0.8");
client.Proxy = WebRequest.DefaultWebProxy;
//client.UploadValues(nseBulkDealsUrl, "GET", values);
client.Headers.Add(values);
var htmlPageSource = client.DownloadString(nseBulkDealsUrl);
I have realy strange problem with Headers collection in WebClient class.
Here is my example:
WebClient client = new WebClient();
client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0");
client.Headers.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
client.Headers.Add("Accept-Language", "pl,en-us;q=0.7,en;q=0.3");
Console.WriteLine("Before request:");
foreach (string key in client.Headers)
{
Console.WriteLine(key + ": " + client.Headers[key]);
}
client.DownloadString("http://www.google.com");
Console.WriteLine();
Console.WriteLine("After request:");
foreach (string key in client.Headers)
{
Console.WriteLine(key + ": " + client.Headers[key]);
}
Console.ReadLine();
My result of running this simple program:
Before request:
User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: pl,en-us;q=0.7,en;q=0.3
After request:
Accept-Language: pl,en-us;q=0.7,en;q=0.3
Why my headers are disappearing?
because headers are sent and webclient did its job. if you want same headers for next request, you should add them again.
I'm setting up some HTTP request headers like this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
WebResponse response = http.GetResponse();
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
I am capturing them like this:
for (count = 0; count < http.Headers.Keys.Count; count++)
{
headerKey = http.Headers.Keys[count];
headerValue = http.Headers[headerKey];
if (headerValue != null)
{
if (headerKey == null)
{
requestbuffer.Append(headerValue);
requestbuffer.Append(Newline);
}
else
{
requestbuffer.Append(headerKey + ": " + headerValue);
requestbuffer.Append(Newline);
}
}
}
When I run the testing tool everything seems good:
Host: domain.com
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0)Gecko/20100101 Firefox/17.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Encoding: gzip,deflate,sdch Accept-Language: en-US,en;q=0.9
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
However in Wireshark and Fiddler only the following header is sent:
GET / HTTP/1.1
Host: domain.com
Any idea why that may be?
You're trying to add the headers after you've called http.GetResponse(). That's after the request has been sent. Change it to this:
this.Url = new Uri(u);
HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
using (WebResponse response = http.GetResponse())
{
// Do whatever
}
(Note that you really should be disposing of the response.)
Yes, you are setting the header after you have sent the request.
Try this:
//headers
http.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20100101 Firefox/17.0\r\n";
http.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
http.Headers.Add("Accept-Encoding", "gzip,deflate,sdch'r'n");
http.Headers.Add("Accept-Language", "en-US,en;q=0.9\r\n");
http.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n");
//Added diesposal of response too
using (WebResponse response = http.GetResponse())
{
}
This code
using (WebClient webClient = new WebClient())
{
string address = "https://www.any.com/name.htm";
byte[] postData = Encoding.ASCII.GetBytes("login=123");
webClient.Headers[HttpRequestHeader.Accept] = "text/html, application/xhtml+xml, */*";
webClient.Headers[HttpRequestHeader.AcceptLanguage] = "ru-RU";
webClient.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)";
webClient.Headers[HttpRequestHeader.AcceptEncoding] = "gzip, deflate";
webClient.Headers[HttpRequestHeader.ContentType] = "application/x-www-form-urlencoded";
webClient.UploadData(address, postData);
}
produces this request (postData doubles):
POST /name.htm HTTP/1.1
Accept-Language: ru-RU
Accept-Encoding: gzip, deflate
Accept: text/html, application/xhtml+xml, */*
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)
Host: [url]www.any.com[/url]
Content-Length: 9
Connection: Keep-Alive
login=123login=123
If I change
string address = "https://www.any.com/name.htm"
to
string address = "http://www.any.com/name.htm"
then data don't doubles.
How avoid data doubling when send request by HTTPS ?
It is bug of HttpAnalizer.
Many thanks to Hardrada.