We Keep Coding

HTTPWebResponse is returning an error number: 500

I am trying to call my API reference but when it gets to the HttpWebResponse response = (HttpWebResponse)request.GetResponse()) section my code just returns the error 500. I'm unsure on what my code is actually missing as im pretty new to APIs
any help is appreciated.
var request = (HttpWebRequest)WebRequest.Create($"####");
request.Method = "GET";
request.ContentType = "application/json";
request.Headers.Add($"USERNAME: {Username}, PASSWORD: {Password}");
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36";
request.ServicePoint.Expect100Continue = false;
request.ProtocolVersion = HttpVersion.Version11;
string responseFromServer = "";
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
//Console.WriteLine(((HttpWebResponse)response).StatusDescription);
using (var dataStream = response.GetResponseStream())
{
StreamReader reader = new StreamReader(dataStream);
responseFromServer = reader.ReadToEnd();
//Console.WriteLine(responseFromServer);
}

A 500 HTTP response code is a server based error, your code is not the issue. However, some "poorly made" APIs do not handle insufficient keys with the proper HTTP codes, resulting in a server error due to the front end not sending the required keys/values in the request.
Good luck!

HTTP POST not displaying results

Trying to POST a webrequest but not getting the actual results.. webrequest not getting submit. Previously this code was working fine in 2018. I'm not sure if they changed something at their end. I copied all the latest parameters, and other stuff from fiddler. Is there anyone please look into it as my deadline for the actual project is very tight and this code is just an integral part of the main project and was assuming it is functional already.. already spent few hours on this..
CookieContainer _cookies = new CookieContainer();
string urls = "https://ww3.freddiemac.com/loanlookup?Phone=&Email=&Contact=&lang=en&fname=TEST&lname=PERSON&strt_nbr=123&unit_num=&strt_name=FAKE STREET&strt_suffix=&addr_city=CHICAGO&addr_state=IL&addr_zip=60628&ssn=1234&Verify=Yes";
var request = (HttpWebRequest)WebRequest.Create(urls);
request.Host = "ww3.freddiemac.com";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36 Edg/83.0.478.61";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9";
WebHeaderCollection myWebHeaderCollection = request.Headers;
myWebHeaderCollection.Add("Accept-Language", "en-US,en;q=0.9");
request.Headers["Accept-Encoding"] = "gzip, deflate, br";
request.ContentType = "application/x-www-form-urlencoded";
request.Method = "POST";
request.Referer = "https://ww3.freddiemac.com/loanlookup";
var encoding = new ASCIIEncoding();
var postBytes = encoding.GetBytes(urls);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip | DecompressionMethods.None;
request.ContentLength = postBytes.Length;
request.CookieContainer = _cookies;
request.KeepAlive = true;
var requestStream = request.GetRequestStream();
requestStream.Write(postBytes, 0, postBytes.Length);
requestStream.Close();
var response = (HttpWebResponse)request.GetResponse();
string html;
using (var reader = new StreamReader(response.GetResponseStream()))
{
html = reader.ReadToEnd();
}
if (html.Contains("No. Our records show"))
{
_qualified = false;
}

C# using HttpWebRequest doesn't work

At the time of opening, using HttpWebRequest. Why are visiting
https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc
Can't get the data inside?
string strResult = "";
try
{
string url = "https://web4.sa8888.net/sport/Games.aspx?lang=3&device=pc";
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Tls12;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.GZip;
request.Method = "GET";
request.Timeout = 30000;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.Headers.Set("Pragma", "no-cache");
//request.ContentLength = bs.Length;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
Stream streamReceive = response.GetResponseStream();
Encoding encoding = Encoding.GetEncoding("UTF-8");
StreamReader streamReader = new StreamReader(streamReceive, encoding);
strResult = streamReader.ReadToEnd();
}
catch (Exception e)
{
MessageBox.Show(e.Message);
}
textBox1.Text=strResult;

The page you've linked to is built using javascript so what you are seeing is not raelly part of the source. If you look at for example https://web4.sa8888.net/js/v1000/sport/games/IceHockey.js you will see that they build alot of the table data there.
HttpWebRequest will only fetch a file for you, it won't run javascript and such like a real browser would. Thats why you are not getting the data you want.
After looking at it a bit more it seems like they are streaming information in a binary format via websockets so this is a very non-trivial setup to get the data out (and probably at least partially built to avoid people scraping their data).

Disable image download for HttpWebRequest

Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}

I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...

The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();

Why GetResponseStream() hangs in this code? How to debug it?

Run this code inside a very simple Console application:
try
{
var request = WebRequest.Create("some url here") as HttpWebRequest;
byte[] bytes = Encoding.ASCII.GetBytes("some JSON string here");
request.Method = "POST";
request.Host = "some host here";
request.ContentLength = bytes.Length;
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Pragma", "no-cache");
request.Headers.Add("Origin", "some host here");
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
request.UserAgent =
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36";
request.Accept = "*/*";
request.Referer = "host here";
request.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
request.Headers.Add("Accept-Language", "en-US,en;q=0.8");
using (Stream newStream = request.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
var response = request.GetResponse();
var result = new StreamReader(response.GetResponseStream()).ReadToEnd();
}
catch (Exception)
{
throw;
}
Why request.GetRequestStream() hangs?

GetRequestStream of the HttpWebRequest ensures that the connection to the remote endpoint is open and that the headers are sent, before returning and allowing you to write content. If the method hangs, that likely means a network issue.
MSDN states the following:
This member outputs trace information when you enable network tracing
in your application. For more information, see Network Tracing.
For debugging, you can use both network tracing ( http://msdn.microsoft.com/en-us/library/hyb3xww8.aspx ) as well as a packet sniffer like wireshark.

Remove the line request.Host = "some host here"; from your code. It must solve your problem.