If I use a regular url it works fine and if I use google domains update url I get 401 error. This is my first try on C# application.
HttpWebRequest request = WebRequest.Create("https://UUUUUUUUUUUUU:PPPPPPPPPPPPP#domains.google.com/nic/update?hostname=subdomain.example.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.102 Safari/537.36 Viv/1.97.1246.7";
request.UseDefaultCredentials = true;
request.PreAuthenticate = true;
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
//responseddns = responseText;
MessageBox.Show(responseText);
}
If I use http://example.com/getip.php it works fine I can see the output.
you cannot use
> `CredentialCache.DefaultCredentials;`
since the url is the domain.google.com's domain.
You need to enter your google credentials or else directly use
http://example.com/getip.php as u did before
Related
I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}
A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!
At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;
The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).
Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}
I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...
The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();
First thing, the site and post.
https://saplic.receita.pb.gov.br/sintegra/SINf_ConsultaSintegra.jsp
Fullfil Field CNPJ with value For instance 34151100004209.
Works fine in Chrome and .Net 4.0 Httpwebrequest. But I cant debug in fiddler and cant make it work on .Net 3.5
I'm already using
System.Net.ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(customXertificateValidation);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
System.Net.ServicePointManager.Expect100Continue = false;
This is driving me insane. Any help will be greatly appreciated
To make things more clear, i'm having one problem that is happening on 2 differente places (Fiddler and .net 3.5)
The code i'm trying to run is.
CookieContainer CookCon = new CookieContainer();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb");
Request.Timeout = Configuration.TimeOut;
Request.Proxy = Configuration.Proxy;
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
Request.CookieContainer = new CookieContainer();
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
foreach (Cookie C in Response.Cookies)
{
CookCon.Add(C);
}
Response.Close();
}
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://saplic.receita.pb.gov.br/sintegra/");
Request.Referer = "http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb";
Request.Headers.Add("X-DevTools-Emulate-Network-Conditions-Client-Id", "C5E5561B-11F5-4D9E-A9D4-54A18E660D11");
Request.CookieContainer = CookCon;
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36";
string HTML = "";
string Encoding = "ISO-8859-1";
Request.Timeout = 20000;
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
Stream S = Response.GetResponseStream();
if (Response.ContentEncoding.ToLower().Contains("gzip"))
{
S = new GZipStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else if (Response.ContentEncoding.ToLower().Contains("deflate"))
{
S = new DeflateStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else
{
using (System.IO.StreamReader ResStream = new StreamReader(Response.GetResponseStream(), System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
S.Close();
S.Dispose();
Response.Close();
}
The error i'm getting is the "unexpeted EOF" excpetion when i try to make the post(GetResponse part of the code above).
If i try to surf the site on Chrome with Fiddler on, it wont return northing, just keeps on hold, hold, hold and nothing.
If I copy the same code to a new 4.0 project, it runs without problem.
If its not clear, please post the doubts, I'm trying to be as clear as possible.
Since no one helped, i just made a webservice using the .net 4.0 framework and I'm calling it from the 3.5 app.
Still no clue why this happens..
I'm trying to get cookies form a loginform, saved it on my cookie container cookieJar, and using in the next request.
The cookies are saved correctly(at least,the count shows an appropiate quantity,but when doing the webrequest3, I not getting the content , getting the page as not logged in.
P.D: I read the related posts, but the major are not completly implemented(obviously),and the others are doing exactly as I do, so I'm in a loss.
CookieContainer cookieJar = new CookieContainer();
//The First Req
HttpWebRequest webRequest1 = (HttpWebRequest)WebRequest.Create("url1");
webRequest1.Method = "GET";
webRequest1.ContentType = "text/html";
webRequest1.KeepAlive = true;
webRequest1.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36";
webRequest1.Host = "url1";
webRequest1.CookieContainer = cookieJar;
webRequest1.ContentType = "text/html";
HttpWebResponse webResponse;
webResponse = (HttpWebResponse)webRequest1.GetResponse();
Console.WriteLine(cookieJar.Count.ToString());
StreamReader reader = new StreamReader(webResponse.GetResponseStream());
// Read the content fully up to the end.
string responsereq = reader.ReadToEnd();
// Clean up the streams.
reader.Close();
webResponse.Close();
Console.ReadKey();
//Second Request
HttpWebRequest webRequest3 = (HttpWebRequest)WebRequest.Create("url2");
webRequest3.Method = "GET";
webRequest3.KeepAlive = true;
webRequest3.UserAgent="Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36";
webRequest3.Host = "url2";
webRequest3.CookieContainer = cookieJar;
webRequest3.ContentType = "text/html";
Console.WriteLine(cookieJar.Count.ToString() +"CookieJar");
Console.ReadKey();
webResponse = (HttpWebResponse)webRequest3.GetResponse();
StreamReader reader3 = new StreamReader(webResponse.GetResponseStream());
// Read the content fully up to the end.
string responseFromServer = reader3.ReadToEnd();
Console.WriteLine(responseFromServer);
// Clean up the streams.
webResponse.Close();
Console.ReadKey();
EDIT:
I get with fiddler that when loggin from the explorer,enter to a page automatically after the webrequest1, don't save any cookie,but seems to use some server-side check, that if you don't enter to that page before the webrequest3, the webrequest2 didn't recognize your login.
So, creating another webrequest before webrequest3, do the trick.
Try with following.
First get cookies from the response of first request.
HttpWebResponse webResponse;
webResponse = (HttpWebResponse)webRequest1.GetResponse();
CookieContainer cookieJar= new CookieContainer();
foreach (Cookie cook in webResponse .Cookies)
{
cookieJar.Add(cook);
}
Pass it to subsequent request.
webRequest3.CookieContainer = cookieJar;