We Keep Coding

How to overcome from 503 Server unavailable error?

I am doing an http request to the unknown server. When i make request from browser then it works like a charm but whenever i make the same request from my application then i get 503 Server unavailable.
I searched for this error and found that this is caused due to server temporary unavailable and another answered as the server believe that this call is made from a bot application.
I made same kind of request to the same server and there is only a small change in the uri. All previous calls are working except this one.
private static Root_BoardingStationList Request_BoardingStation(string uri, string auth,string greq, string sp_csr)
{
Root_BoardingStationList rootBoaringStationEnquiry = null;
string captchaImageURL = string.Empty;
string html = string.Empty;
try
{
var request = (HttpWebRequest)WebRequest.Create(uri);
request.Referer = "https://www.irctc.co.in/nget/train-list";
request.Headers.Set("Authorization", "Bearer " + login_authorization_bearer);
request.Headers.Set("greq", greq);
request.Headers.Set("spa-csrf-token", sp_csr);
request.Proxy = null;
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.None;
request.CookieContainer = new CookieContainer();
request.Method = method;
request.ContentType = contentType;
request.KeepAlive = true;
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36";
request.Connection = "keepalive";
using (var response_ = (HttpWebResponse)request.GetResponse())
{
var responseString = new StreamReader(response_.GetResponseStream()).ReadToEnd();
}
}
catch (WebException ex)
{
ExceptionLog.HandleException(ex);
using (var sr = new StreamReader(ex.Response.GetResponseStream()))
html = sr.ReadToEnd();
}
return rootBoaringStationEnquiry;
}
Please check and help.

I have found the solution for above as
csrf-token was mismatched.
I have corrected csrf-token and this works like a charm.
I was using
request.Headers.Set("spa-csrf-token", sp_csr);
and sent wrong sp_csr value to server.

Why I keep getting 401 from my webrequest code in C#

If I use a regular url it works fine and if I use google domains update url I get 401 error. This is my first try on C# application.
HttpWebRequest request = WebRequest.Create("https://UUUUUUUUUUUUU:PPPPPPPPPPPPP#domains.google.com/nic/update?hostname=subdomain.example.com") as HttpWebRequest;
//request.Accept = "application/xrds+xml";
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.102 Safari/537.36 Viv/1.97.1246.7";
request.UseDefaultCredentials = true;
request.PreAuthenticate = true;
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
WebHeaderCollection header = response.Headers;
var encoding = ASCIIEncoding.ASCII;
using (var reader = new System.IO.StreamReader(response.GetResponseStream(), encoding))
{
string responseText = reader.ReadToEnd();
//responseddns = responseText;
MessageBox.Show(responseText);
}
If I use http://example.com/getip.php it works fine I can see the output.

you cannot use
> `CredentialCache.DefaultCredentials;`
since the url is the domain.google.com's domain.
You need to enter your google credentials or else directly use
http://example.com/getip.php as u did before

Disable image download for HttpWebRequest

Is it possible to say a webrequest to only get text-based data from a site? And if it is how should I do this?
The only thing I can imagine is to search in the response string and remove all the image-tags. But this is a very bad way to do this...
EDIT: this is my code snippet:
string baseUrl = kvPair.Value[0];
string loginUrl = kvPair.Value[1];
string notifyUrl = kvPair.Value[2];
cc = new CookieContainer();
string loginDetails = DataCollector.GetLoginDetails(baseUrl, ref cc);
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUrl);
request.Method = "POST";
request.Accept = "text/*";
request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
request.CookieContainer = cc;
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
Byte[] data = Encoding.ASCII.GetBytes(loginDetails);
request.ContentLength = data.Length;
using (Stream s = request.GetRequestStream())
{
s.Write(data, 0, data.Length);
}
HttpWebResponse res = (HttpWebResponse)request.GetResponse();
request = (HttpWebRequest)WebRequest.Create(notifyUrl);
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36";
request.CookieContainer = cc;
res = (HttpWebResponse)request.GetResponse();
Stream streamResponse = res.GetResponseStream();
using (StreamReader sr = new StreamReader(streamResponse))
{
ViewData["data"] += "<div style=\"float: left; margin-bottom: 50px;\">" + sr.ReadToEnd() + "</div>";
}

I found myself a good coding solution:
public static string StripImages(string input)
{
return Regex.Replace(input, "<img.*?>", String.Empty);
}
this kills all images but only as soon as you have loaded all the images so there is no savings in transfered data in this solution...

The HTTP/1.1 Header Field Definitions' section 14.1 contains the Accept header definition. It states the following:
... If an Accept header field is present, and if the server cannot send a response which is acceptable according to the combined Accept field value, then the server SHOULD send a 406 (not acceptable) response.
So it is up to the server if it respects the client's request.
I have found that most of the servers ignore the Accept header. So far I have found only one exceptoin: it is GitHub. I requested the GitHub homepage with audio as the Accept parameter. And it responded appropriately with response code 406.
Try the following snippet for a demo, you should get System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
HttpWebRequest request = (HttpWebRequest) WebRequest.Create("https://github.com/");
request.Method = "GET";
request.Accept = "audio/*";
var response = request.GetResponse();

Fiddler and .Net 3.5 SSL error. Works fine in .Net 4.0

First thing, the site and post.
https://saplic.receita.pb.gov.br/sintegra/SINf_ConsultaSintegra.jsp
Fullfil Field CNPJ with value For instance 34151100004209.
Works fine in Chrome and .Net 4.0 Httpwebrequest. But I cant debug in fiddler and cant make it work on .Net 3.5
I'm already using
System.Net.ServicePointManager.ServerCertificateValidationCallback += new System.Net.Security.RemoteCertificateValidationCallback(customXertificateValidation);
System.Net.ServicePointManager.SecurityProtocol = System.Net.SecurityProtocolType.Ssl3;
System.Net.ServicePointManager.Expect100Continue = false;
This is driving me insane. Any help will be greatly appreciated
To make things more clear, i'm having one problem that is happening on 2 differente places (Fiddler and .net 3.5)
The code i'm trying to run is.
CookieContainer CookCon = new CookieContainer();
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb");
Request.Timeout = Configuration.TimeOut;
Request.Proxy = Configuration.Proxy;
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36";
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
Request.CookieContainer = new CookieContainer();
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
foreach (Cookie C in Response.Cookies)
{
CookCon.Add(C);
}
Response.Close();
}
HttpWebRequest Request = (HttpWebRequest)WebRequest.Create("https://saplic.receita.pb.gov.br/sintegra/");
Request.Referer = "http://sintegra.receita.pb.gov.br/sintegra/sintegra.asp?estado=pb";
Request.Headers.Add("X-DevTools-Emulate-Network-Conditions-Client-Id", "C5E5561B-11F5-4D9E-A9D4-54A18E660D11");
Request.CookieContainer = CookCon;
Request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
Request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36";
string HTML = "";
string Encoding = "ISO-8859-1";
Request.Timeout = 20000;
Request.Headers.Add("Accept-Encoding", "gzip, deflate");
using (HttpWebResponse Response = Request.GetResponse() as HttpWebResponse)
{
Stream S = Response.GetResponseStream();
if (Response.ContentEncoding.ToLower().Contains("gzip"))
{
S = new GZipStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else if (Response.ContentEncoding.ToLower().Contains("deflate"))
{
S = new DeflateStream(S, CompressionMode.Decompress);
using (System.IO.StreamReader ResStream = new StreamReader(S, System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
else
{
using (System.IO.StreamReader ResStream = new StreamReader(Response.GetResponseStream(), System.Text.Encoding.GetEncoding(Encoding)))
{
HTML = ResStream.ReadToEnd();
ResStream.Close();
}
}
S.Close();
S.Dispose();
Response.Close();
}
The error i'm getting is the "unexpeted EOF" excpetion when i try to make the post(GetResponse part of the code above).
If i try to surf the site on Chrome with Fiddler on, it wont return northing, just keeps on hold, hold, hold and nothing.
If I copy the same code to a new 4.0 project, it runs without problem.
If its not clear, please post the doubts, I'm trying to be as clear as possible.

Since no one helped, i just made a webservice using the .net 4.0 framework and I'm calling it from the 3.5 app.
Still no clue why this happens..

Transform methods with AsyncCallbacks to Task<Result> methods

would like to change my methods from HttpWebRequest.BeginGetRequestStream and HttpWebRequest.BeginGetResponse to methods which use async and Tasks. I need to use these methods because I work on Windows Phone and other methods like GetRequestStream which is not async are not available on this platform.
This is my method DownloadString:
private static Action<string> completed;
public static void DownloadString(Action<string> c)
{
completed = c;
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(new Uri("linkhere", UriKind.Absolute));
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.BeginGetRequestStream(RequestCallback, request);
}
private static void RequestCallback(IAsyncResult ar)
{
HttpWebRequest request = (HttpWebRequest)ar.AsyncState;
Stream postStream = request.EndGetRequestStream(ar);
//some post data
request.BeginGetResponse(ResponseCallback, request);
}
private static void ResponseCallback(IAsyncResult ar)
{
HttpWebRequest webRequest = (HttpWebRequest)ar.AsyncState;
HttpWebResponse response = (HttpWebResponse)webRequest.EndGetResponse(ar);
Stream streamResponse = response.GetResponseStream();
StreamReader streamReaders = new StreamReader(streamResponse);
string content = streamReaders.ReadToEnd();
streamResponse.Close();
streamReaders.Close();
response.Close();
completed.Invoke(content);
}
For the next step I used the following link:
Converting Callback Functions to Task Pattern
Now I want to have a wrapper method like this:
public async static Task<string> DownloadWithoutAction()
{
TaskCompletionSource<string> tcs = new TaskCompletionSource<string>();
var task = tcs.Task;
Action<string> callback = tcs.SetResult;
await Task.Factory.StartNew(() =>
{
DownloadString(callback);
}, TaskCreationOptions.AttachedToParent);
return await Task.FromResult(task.Result);
}
The Problem:
This does not work because the callbacks are async and so the thread stops before the callbacks are fired. This is only my guess. How can I work around?
Thanks in advance

First, I strongly recommend that you upgrade to HttpClient.
But if you want to do it the harder way, then you would do something like this:
public static Task<string> DownloadStringAsync()
{
TaskCompletionSource<string> tcs = new TaskCompletionSource<string>();
DownloadString(result => tcs.SetResult(result));
return tcs.Task;
}

You could use FromAsync method from TaskFactory:
var request = HttpWebRequest.CreateHttp("http://url");
var stream = await Task.Factory.FromAsync<Stream>(request.BeginGetRequestStream, request.EndGetRequestStream, TaskCreationOptions.None);
var response = (HttpWebResponse)(await Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, TaskCreationOptions.None));
The method using Tasks could be:
private async Task<string> DownloadString()
{
var request = (HttpWebRequest)WebRequest.Create(new Uri("linkhere", UriKind.Absolute));
request.Method = "POST";
request.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.52 Safari/536.5Accept: */*";
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
using (var response = (HttpWebResponse)(await Task.Factory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, TaskCreationOptions.None)))
using (var streamResponse = new StreamReader(response.GetResponseStream()))
{
return await streamResponse.ReadToEndAsync();
}
}