I have a download method for HTTP protocol. But it seems it doesnt work correctly, something is wrong. I tested it with some url sources and it was correct except the last one. The ContentLength property is wrong for the url. It is shown as 210 kb in runtime, but it is 8 MB in fact. I will show it by sharing my code. How to fix it?
Code:
void TestMethod(string fileUrl)
{
HttpWebRequest req = WebRequest.Create(fileUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
long contentSize = resp.ContentLength;
MessageBox.Show(contentSize.ToString());
}
private void TestButton_Click(object sender, EventArgs e)
{
string url1 = "http://www.calprog.com/Sounds/NealMorseDowney_audiosample.mp3";
string url2 = "http://www.stephaniequinn.com/Music/Canon.mp3";
TestMethod(url1); //This file size must be 8 MB, but it shows up as 210 kb. This is the problem
TestMethod(url2); //This file size is correct here, about 2.1 MB
}
I think you are not allowed to access this url this way (with HttpWebRequest).
if you try to get the response text :
HttpWebRequest req = WebRequest.Create(fileUrl) as HttpWebRequest;
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
using (var streamreader = new StreamReader(resp.GetResponseStream()))
{
var r = streamreader.ReadToEnd();
long contentSize = r.Length;
Console.WriteLine(contentSize.ToString());
}
You ll get this response :
<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. If you think this is an error, please contact the webmaster. <br><br>Your support ID is: 2719994757208255263</body></html>
You must set the UserAgent to be able to get the full response. Like this :
req.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
By setting this value the server will think your program is a Firefox browser.
So these few lines of code should do the trick :
void TestMethod(string fileUrl)
{
HttpWebRequest req = WebRequest.Create(fileUrl) as HttpWebRequest;
req.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Firefox/31.0";
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
long contentSize = resp.ContentLength;
Console.WriteLine(contentSize.ToString());
}
Have a good day !
Related
I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}
A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!
At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}
I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...
The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();
I'm stuck on authentication to web site with webrequest on windows phone 8.
I have the following code:
PostMessage = string.Concat("job=LOGIN&password=&giftCode=&language=ua&login=%D0%9D%D0%BE%D0%BC%D0%B5%D1%80+%D0%BA%D0%B0%D1%80%D1%82%D0%BA%D0%B8&actn=&custId=&dateFrom=&dateTo=&showGift=&type=&card=&trnId=&catId=&subCatId=&awrId=&ordId=&qstId=&acqId=&", "crdNo=", cardNumber, "&PIN=", pin);
this.postDataBytes = Encoding.UTF8.GetBytes(PostMessage);
public void Load()
{
HttpWebRequest loginRequest = HttpWebRequest.CreateHttp(loginUrl);
loginRequest.ContentLength = this.postDataBytes.Length;
loginRequest.Method = "POST";
loginRequest.Accept = #"text/html, application/xhtml+xml, */*";
loginRequest.ContentType = "application/x-www-form-urlencoded";
loginRequest.Headers["Host"] = "*host*";
loginRequest.Headers["Referer"] = "*refer*";
loginRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0) like Gecko";
loginRequest.CookieContainer = new CookieContainer();
loginRequest.BeginGetRequestStream(OnBeginGetRequestStream, loginRequest);
}
private void OnBeginGetRequestStream(IAsyncResult result)
{
HttpWebRequest loginRequest = result.AsyncState as HttpWebRequest;
using (Stream stream = loginRequest.EndGetRequestStream(result))
{
stream.Write(this.postDataBytes, 0, this.postDataBytes.Length);
}
loginRequest.BeginGetResponse(OnAuthenticated, loginRequest);
}
private void OnAuthenticated(IAsyncResult result)
{
HttpWebRequest request = result.AsyncState as HttpWebRequest;
using (WebResponse response = request.EndGetResponse(result))
{
Stream responseStream = response.GetResponseStream();
using (StreamReader reader = new StreamReader(responseStream))
{
string page = reader.ReadToEnd();
}
}
}
It works sometimes... I mean this code do successfully authentication and return correct response. But sometimes it doesn't. Just ran this code few times and at some point I got correct response.
I tried to use fiddler but he doesn't want track requests from emulator.
Maybe someone know the reason of this strange behavior?
Did you look at this answer? I had the same problem, and the reason was cached response data.This answer is about win phone 7 but works for me on win phone 8.
Httpwebrequest caching in windows phone 7
I'm trying to get simple gzip encoded html response from a website and it keeps getting time out, following is my code:
HttpWebRequest httpClient = (HttpWebRequest)WebRequest.Create(url);
httpClient.Method = "GET";
httpClient.Accept = "text/html, application/xhtml+xml, */*";
httpClient.Headers.Add("Accept-Encoding: gzip, deflate");
httpClient.Headers.Add("Accept-Language: en-US");
httpClient.Headers.Add("DNT: 1");
httpClient.ProtocolVersion = HttpVersion.Version10;
httpClient.KeepAlive = true;
httpClient.Timeout = System.Threading.Timeout.Infinite;
httpClient.CookieContainer = cookieJar;
String responseAsText;
using (HttpWebResponse response = (HttpWebResponse)httpClient.GetResponse())
{
System.IO.StreamReader sr;
if (response.ContentEncoding.Equals("gzip"))
{
sr = new StreamReader(new GZipStream(response.GetResponseStream(), CompressionMode.Decompress));
}
else
{
sr = new System.IO.StreamReader(response.GetResponseStream());
}
responseAsText = sr.ReadToEnd();
}
The url I'm trying to hit is "https client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx"
This works perfectly fine in the Browser, using Fiddler I viewed the browser's Request header and since its Transfer-Encoding: chunked, I have used HttpVersion10
I have also tried setting httpClient.Timeout = System.Threading.Timeout.Infinite, but it never gets back with a response, however in browser the response gets in few seconds.
Please someone help me in achieving this.
probably you can try setting Agent property, so it doesn't recognize you as a bot.
I think Nero has answered your question ..
Try adding these Lines in your code..
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0";