Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}
I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...
The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();
Related
I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}
A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!
At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
If I use a regular url it works fine and if I use google domains update url I get 401 error. This is my first try on C# application.
HttpWebRequest request = WebRequest.Create("https://UUUUUUUUUUUUU:PPPPPPPPPPPPP#domains.google.com/nic/update?hostname=subdomain.example.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.102 Safari/537.36 Viv/1.97.1246.7";
request.UseDefaultCredentials = true;
request.PreAuthenticate = true;
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
//responseddns = responseText;
MessageBox.Show(responseText);
}
If I use http://example.com/getip.php it works fine I can see the output.
you cannot use
> `CredentialCache.DefaultCredentials;`
since the url is the domain.google.com's domain.
You need to enter your google credentials or else directly use
http://example.com/getip.php as u did before
I'm trying to make a POST request to a website using C#.
I tried this code first :
byte[] buffer = Encoding.UTF8.GetBytes(postData);
HttpWebRequest WebReq = (HttpWebRequest)WebRequest.Create("https://www.foo.com");
WebReq.Method = "POST";
WebReq.Accept = "*/*";
WebReq.Headers.Add("Accept-Encoding","gzip, deflate");
WebReq.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36";
WebReq.ContentType = "application/x-www-form-urlencoded";
WebReq.ContentLength = buffer.Length;
Stream PostData = WebReq.GetRequestStream();
PostData.Write(buffer, 0, buffer.Length);
PostData.Close();
HttpWebResponse WebResp = (HttpWebResponse)WebReq.GetResponse();
Stream Answer = WebResp.GetResponseStream();
StreamReader _Answer = new StreamReader(Answer);
String reply = _Answer.ReadToEnd();
//do some tasks on reply
Turns out that this always returned this error : "underlying connection was closed..."
I found out that it's because the site only accepts TLS. So I added this at the beggining of the code:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls11;
The site accepted the request, but now I can't read the response I'm getting, it's encrypted to something like this :
��;ks�J���+&�$�� zYK)�NN�{������TJ5� � ��`���o� H �؎]��(K0��~� >~|����}NbL?��ϾG^='�/�����~<�k$��|��P�
I need to perform some tasks on the response, so I need to decrypt it within the code... I'm open to ideas ! Thanks in advance !
I'm trying to get cookies form a loginform, saved it on my cookie container cookieJar, and using in the next request.
The cookies are saved correctly(at least,the count shows an appropiate quantity,but when doing the webrequest3, I not getting the content , getting the page as not logged in.
P.D: I read the related posts, but the major are not completly implemented(obviously),and the others are doing exactly as I do, so I'm in a loss.
CookieContainer cookieJar = new CookieContainer();
//The First Req
HttpWebRequest webRequest1 = (HttpWebRequest)WebRequest.Create("url1");
webRequest1.Method = "GET";
webRequest1.ContentType = "text/html";
webRequest1.KeepAlive = true;
webRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36";
webRequest1.Host = "url1";
webRequest1.CookieContainer = cookieJar;
webRequest1.ContentType = "text/html";
HttpWebResponse webResponse;
webResponse = (HttpWebResponse)webRequest1.GetResponse();
Console.WriteLine(cookieJar.Count.ToString());
StreamReader reader = new StreamReader(webResponse.GetResponseStream());
// Read the content fully up to the end.
string responsereq = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
webResponse.Close();
Console.ReadKey();
//Second Request
HttpWebRequest webRequest3 = (HttpWebRequest)WebRequest.Create("url2");
webRequest3.Method = "GET";
webRequest3.KeepAlive = true;
webRequest3.UserAgent="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36";
webRequest3.Host = "url2";
webRequest3.CookieContainer = cookieJar;
webRequest3.ContentType = "text/html";
Console.WriteLine(cookieJar.Count.ToString() +"CookieJar");
Console.ReadKey();
webResponse = (HttpWebResponse)webRequest3.GetResponse();
StreamReader reader3 = new StreamReader(webResponse.GetResponseStream());
// Read the content fully up to the end.
string responseFromServer = reader3.ReadToEnd();
Console.WriteLine(responseFromServer);
// Clean up the streams.
webResponse.Close();
Console.ReadKey();
EDIT:
I get with fiddler that when loggin from the explorer,enter to a page automatically after the webrequest1, don't save any cookie,but seems to use some server-side check, that if you don't enter to that page before the webrequest3, the webrequest2 didn't recognize your login.
So, creating another webrequest before webrequest3, do the trick.
Try with following.
First get cookies from the response of first request.
HttpWebResponse webResponse;
webResponse = (HttpWebResponse)webRequest1.GetResponse();
CookieContainer cookieJar= new CookieContainer();
foreach (Cookie cook in webResponse .Cookies)
{
cookieJar.Add(cook);
}
Pass it to subsequent request.
webRequest3.CookieContainer = cookieJar;