We Keep Coding

How to overcome from 503 Server unavailable error?

I am doing an http request to the unknown server. When i make request from browser then it works like a charm but whenever i make the same request from my application then i get 503 Server unavailable.
I searched for this error and found that this is caused due to server temporary unavailable and another answered as the server believe that this call is made from a bot application.
I made same kind of request to the same server and there is only a small change in the uri. All previous calls are working except this one.
private static Root_BoardingStationList Request_BoardingStation(string uri, string auth,string greq, string sp_csr)
{
Root_BoardingStationList rootBoaringStationEnquiry = null;
string captchaImageURL = string.Empty;
string html = string.Empty;
try
{
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Referer = "https://www.irctc.co.in/nget/train-list";
request.Headers.Set("Authorization", "Bearer " + login_authorization_bearer);
request.Headers.Set("greq", greq);
request.Headers.Set("spa-csrf-token", sp_csr);
request.Proxy = null;
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.None;
request.CookieContainer = new CookieContainer();
request.Method = method;
request.ContentType = contentType;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
request.Connection = "keepalive";
using (var response_ = (HttpWebResponse)request.GetResponse())
{
var responseString = new StreamReader(response_.GetResponseStream()).ReadToEnd();
}
}
catch (WebException ex)
{
ExceptionLog.HandleException(ex);
using (var sr = new StreamReader(ex.Response.GetResponseStream()))
html = sr.ReadToEnd();
}
return rootBoaringStationEnquiry;
}
Please check and help.

I have found the solution for above as
csrf-token was mismatched.
I have corrected csrf-token and this works like a charm.
I was using
request.Headers.Set("spa-csrf-token", sp_csr);
and sent wrong sp_csr value to server.

Why I keep getting 401 from my webrequest code in C#

If I use a regular url it works fine and if I use google domains update url I get 401 error. This is my first try on C# application.
HttpWebRequest request = WebRequest.Create("https://UUUUUUUUUUUUU:PPPPPPPPPPPPP#domains.google.com/nic/update?hostname=subdomain.example.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.102 Safari/537.36 Viv/1.97.1246.7";
request.UseDefaultCredentials = true;
request.PreAuthenticate = true;
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
//responseddns = responseText;
MessageBox.Show(responseText);
}
If I use http://example.com/getip.php it works fine I can see the output.

you cannot use
> `CredentialCache.DefaultCredentials;`
since the url is the domain.google.com's domain.
You need to enter your google credentials or else directly use
http://example.com/getip.php as u did before

c# The underlying connection was closed: An unexpected error occurred on a send

First of all I think my issue is pretty different than the other topics in stackoverflow since I've tried the solutions in them.
I'm using .NET 4.5 :
HttpWebRequest MainRequest = (HttpWebRequest)WebRequest.Create(url);
WebHeaderCollection myWebHeaderCollection = MainRequest.Headers;
MainRequest.Method = "GET";
MainRequest.Host = "aro.example.com";
MainRequest.Timeout = 20000;
MainRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
myWebHeaderCollection.Add("Upgrade-Insecure-Requests", "1");
MainRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.103 Safari/537.36";
myWebHeaderCollection.Add("Accept-Encoding", "gzip, deflate, sdch");
myWebHeaderCollection.Add("Accept-Language", "en-US,en;q=0.8");
MainRequest.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
MainRequest.ServicePoint.Expect100Continue = false;
MainRequest.CookieContainer = new CookieContainer();
HttpWebResponse MainResponse = (HttpWebResponse)MainRequest.GetResponse();
This throws the exception the underlying ...
I added :
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
I get no exception now, but also no response until the timeout occurs.

Disable image download for HttpWebRequest

Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}

I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...

The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();

Why GetResponseStream() hangs in this code? How to debug it?

Run this code inside a very simple Console application:
try
{
var request = WebRequest.Create("some url here") as HttpWebRequest;
byte[] bytes = Encoding.ASCII.GetBytes("some JSON string here");
request.Method = "POST";
request.Host = "some host here";
request.ContentLength = bytes.Length;
request.KeepAlive = true;
request.Headers.Add("Cache-Control", "no-cache");
request.Headers.Add("Pragma", "no-cache");
request.Headers.Add("Origin", "some host here");
request.Headers.Add("X-Requested-With", "XMLHttpRequest");
request.UserAgent =
"Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36";
request.Accept = "*/*";
request.Referer = "host here";
request.Headers.Add("Accept-Encoding", "gzip,deflate,sdch");
request.Headers.Add("Accept-Language", "en-US,en;q=0.8");
using (Stream newStream = request.GetRequestStream())
{
newStream.Write(bytes, 0, bytes.Length);
}
var response = request.GetResponse();
var result = new StreamReader(response.GetResponseStream()).ReadToEnd();
}
catch (Exception)
{
throw;
}
Why request.GetRequestStream() hangs?

GetRequestStream of the HttpWebRequest ensures that the connection to the remote endpoint is open and that the headers are sent, before returning and allowing you to write content. If the method hangs, that likely means a network issue.
MSDN states the following:
This member outputs trace information when you enable network tracing
in your application. For more information, see Network Tracing.
For debugging, you can use both network tracing ( http://msdn.microsoft.com/en-us/library/hyb3xww8.aspx ) as well as a packet sniffer like wireshark.

Remove the line request.Host = "some host here"; from your code. It must solve your problem.