I'm almost certain that I'm just misunderstanding how query strings should be passed in using C# HTTPWebRequest/Response, but I can't figure out why when I call (HTTPwebResponse)request.GetResponse(); it strips out the query portion and only gives the resulting HTML of the base URL. What am I doing wrong? Note that I generalized the URL for simplicity but kept the same query string just in case the structure of that is the problem.
string urlQueryString = "https://baseUrl.com/folder1/folder2?status_filter_mode=approved";
var targetUri = Uri.EscapeUriString(urlQueryString);
request = HttpWebRequest.Create(targetUri) as HttpWebRequest;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko";
request.Method = "GET";
//set the cookie in the request header
request.Headers.Add("Cookie", cookiedata);
response = (HttpWebResponse)request.GetResponse();
using (Stream stream = response.GetResponseStream())
{
using (StreamReader reader = new StreamReader(stream))
{
htmlResult = reader.ReadToEnd();
}
}
response.Close();
Related
I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}
A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!
At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
I spent several hours debuging the code, playing with Fiddler and googling, but still no luck, so hopefully you will help me.
I am trying to get the source of http://www.finishline.com. The catch is, the HttpWebRequest works in some regions (like here in Slovakia), but doesn't work in USA what I need to achieve.
For USA the request.GetResponse() just time outs. I have tried countless headers combinations, but without success. Can you please help? Thank you
var request = (HttpWebRequest)WebRequest.Create("http://www.finishline.com");
request.CookieContainer = new CookieContainer();
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36";
request.AutomaticDecompression = (DecompressionMethods.GZip | DecompressionMethods.Deflate);
request.Headers.Add("Accept-Encoding", "gzip, deflate");
request.Accept = " text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.Headers.Add("Upgrade-Insecure-Requests", "1");
request.Headers.Add("Accept-Language", "sk,cs;q=0.8,en-US;q=0.5,en;q=0.3");
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "max-age=0");
var responseText = "";
using (var response = request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
There are two kind of timeouts. Client timeout and server timeout. Have you tried doing something like this:
request.Timeout = Timeout.Infinite;
request.KeepAlive = true;
So, your GetResponse never get timedout you can check with it.
Or
using (HttpWebResponse response = (HttpWebResponse) request.GetResponse())
{
var httpWebResponse = response.GetResponseStream();
using (var sr = new StreamReader(httpWebResponse))
{
responseText = sr.ReadToEnd();
}
}
Try this method to fix your issue.
Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}
I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...
The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();
I am working with HTTPWebRequests for the first time now and i'm trying to wrap my head around it all. I am trying to populate the "Start Date" text box on this website http://treasurer.maricopa.gov/Parcel/TaxReceipt.aspx. I used Fiddler to find the incredibly long Json query that comes back from the POST. ( i will post the entire thing if needed )...but i noticed the last part is what I'm after
"............,"StartDate":"1/1/2013","EndDate":"12/31/2013"}}
So far, this is the function i am using but the result comes back without any changes (so it's as if you just went to the website i first posted)
public static void json(string url)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "text/json";
httpWebRequest.Method = "POST";
httpWebRequest.CookieContainer = cookieJar;
//httpWebRequest.Accept = "text/html, application/xhtml+xml, */*";
httpWebRequest.Headers.Add("Accept-Language: en-US");
httpWebRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW65; Trident/7.0; MAM5; rv:11.0) like Gecko";
httpWebRequest.Headers.Add("Accept-Encoding: gzip, deflate");
//httpWebRequest.Referer = url4;
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
string json = #"{StartDate: 1/1/2013}, {EndDate: 12/13/2013}";
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
}
}