We Keep Coding

What is the role of my machine's Application pool in Windows service development when consuming third party rest services [duplicate]

I am working on Windows Service in visual studio 2017. In the rest api's call, getting exceptions while debugging code. Sometimes first 2 3 calls working after that getting exceptions.
System.Net.WebException: 'The remote server returned an error: (503)
Server Unavailable.'
The remote server returned an error: (429)
Unable to connect to the remote server
When calling same api's from Postman, getting response successfully.
This is my code
private void timer1_Tick(object sender, ElapsedEventArgs e)
{
WriteToFile("timer1_Tick method called..");
try
{
string jsonString = "";
string jsonstring2 = "";
string prodfetchurl = HOST;
var req = WebRequest.Create(prodfetchurl) as HttpWebRequest;
req.Method = "GET";
InitializeRequest(req);
req.Accept = MIME_TYPE;
//System.Threading.Thread.Sleep(5000);
var response = (HttpWebResponse)req.GetResponse();
WriteToFile("First service called...");
if (response.StatusCode == HttpStatusCode.OK)
{
Stream responseStream = response.GetResponseStream();
StreamReader responseReader = new StreamReader(responseStream);
jsonString = responseReader.ReadToEnd();
}
var deserialsseobj = JsonConvert.DeserializeObject<ProductList>(jsonString).Products.Where(i => i.Failed > 0).ToList();
foreach (var a in deserialsseobj)
{
var pid = a.ID;
string url = FailedDevicesUrl + pid.Value + "/failed";
var req2 = WebRequest.Create(url) as HttpWebRequest;
req2.Method = "GET";
InitializeRequest(req2);
req2.Timeout = 300000;
req2.Accept = MIME_TYPE;
var response1 = (HttpWebResponse)req2.GetResponse();
Stream responsestream2 = response1.GetResponseStream();
WriteToFile("Second service called...");
if (response1.StatusCode == HttpStatusCode.OK)
{
StreamReader responsereader1 = new StreamReader(responsestream2);
jsonstring2 = responsereader1.ReadToEnd();
}
var output = JsonConvert.DeserializeObject<List<FailedDeviceList>>(jsonstring2); // Will get List of the Failed devices
List<int> deviceids = new List<int>();
Reprocessdata reproc = new Reprocessdata();
Reprocessdata.DeviceId rprod = new Reprocessdata.DeviceId();
reproc.ForceFlag = true;
reproc.ProductID = pid.Value;
foreach (var dd in output)
{
rprod.ID = dd.DeviceId;
reproc.DeviceIds.Add(rprod);
}
// Reprocess the Product in Devices
var req3 = WebRequest.Create(ReprocessUrl) as HttpWebRequest;
req3.Method = "POST";
InitializeRequest(req3);
req3.Accept = MIME_TYPE;
req3.Timeout = 300000;
req3.ContentType = "application/json";
using (StreamWriter writer = new StreamWriter(req3.GetRequestStream()))
{
string json = new JavaScriptSerializer().Serialize(reproc);
writer.Write(json);
writer.Close();
}
System.Threading.Thread.Sleep(5000);
var response5 = (HttpWebResponse)req3.GetResponse();
WriteToFile("Third service called...");
if (response5.StatusCode == HttpStatusCode.OK)
{
string result;
using (StreamReader rdr = new StreamReader(response5.GetResponseStream()))
{
result = rdr.ReadToEnd();
}
}
}
response.Close();
}
catch (Exception ex)
{
WriteToFile("Simple Service Error on: {0} " + ex.Message + ex.StackTrace);
}
}
Methods used in above code
protected override void OnStart(string[] args)
{
base.OnStart(args);
timer1 = new System.Timers.Timer();
timer1.Interval = 60000; //every 1 min
timer1.Elapsed += new System.Timers.ElapsedEventHandler(timer1_Tick);
timer1.Enabled = true;
WriteToFile("Service has started..");
}
public void InitializeRequest(HttpWebRequest request)
{
request.Headers.Add("aw-tenant-code", API_TENANT_CODE);
request.Credentials = new NetworkCredential(USER_NAME, PASSWORD);
request.KeepAlive = false;
request.AddRange(1024);
}
When I contacted service provide they said everything fine from there side. Is this my code is buggy or windows service not reliable? How can I fix this issue?
Note: All APIS are working fine from Angular application using Visual Studio Code. It means my code is not working.
Edit1: Three below services I am using from this document of VMware.
private const string HOST = "https:host/api/mdm/products/search?";
private const string FailedDevicesUrl = "https:host/api/mdm/products/";
private const string ReprocessUrl = "https:host/api/mdm/products/reprocessProduct";

Response http code 429 indicates that you sending too many requests on target web service.
This means service you trying to send requests has a policies that blocks some requests by request-per-time limit.
Also I admit that external service can be manually configured to throw 403 code in specific cases that you can't know about. If that, this information can be explained in external service documentation... or not :)
What you can do with this?
Fit in limitations
You can make detailed research what limits target webservice has and set up your code to fit in this limitations. For example if service has limitation for receiving only one request per 10 minutes - you must set up your timer to send one request each 10 or more minutes. If documentation not provide such information - you can test it manually by finding some patterns with external service responses.
Use proxy
Every limitation policy based on information about requests senders. Usually this information consists of IP address of sender only. This means if you send 2 requests from two different IP addresses - limitation policy will perceive that like 2 different computers sending these requests. So you can find/buy/rent some proxy IP addresses and send requests through there on target web server.
How to connect through proxy in C# using WebRequest you can see in this answer.
Negotiate with external service provider
If you have possibility to communicate with external service developers or help center, you can ask their to reduce limitations for your IP address (if it static) or provide some mechanisms to avoid limitation policy for you. If for some reason they cannot provide this opportunity, at least you can ask detailed information about limitations.
Repetition mechanism
Some times 503 error code that is outer exception you received may be caused by service unavailable. It means that server can be under maintenance or temporary overloaded. So you can write repetition mechanism to make continious sending requests to server until it'll be accessible.
Polly library may help you with repetition mechanism creation

The inner error of that 503 is:
The remote server returned an error: (429)
HTTP 429 indicates too many requests. Maybe your upstream server can’t process all requests sent.
This can happen when you reached rate limiting / throttling value if you’re calling a third party API.
UPDATE
As per page 28 in the API docs, you could configure throttling when creating a new API. Check if the throttling is too small or maybe turn off the throttling and see if that could fix the error?

WebClient UploadStringTaskAsync fails with no exception

I am calling UploadStringTaskAsync on a restful Web API 2 post method I wrote and it is failing with no exceptions. If I change the call to be UploadString, it works as expected. I've tried a number of different approaches. With UploadStringTaskAsync attempt Fiddler does show the post being issued but with a content-length mismatch (see below). I am running this call from a class library included in a test console app. .Net 4.5.2 So far just running in debug mode from VS 2015. here's my code:
private async Task PostLoggingItem(SmgLoggingItem loggingItem)
{
try
{
//using (WebClient client = new WebClient())
//{
//WebClient client = new WebClient();
const string authToken = "mytoken";
loggingItem.AuthToken = Encryptor.GenerateSecurityToken(authToken);
client.Encoding = Encoding.UTF8;
//client.Credentials = CredentialCache.DefaultNetworkCredentials;
//client.UseDefaultCredentials = true;
client.Credentials = new NetworkCredential("user","mypwd","mydom");
// set content type to JSon
client.Headers.Add("Content-Type", "application/json");
var jsonItem = JsonConvert.SerializeObject(loggingItem);
var response = await client.UploadStringTaskAsync(new Uri(ConfigurationManager.AppSettings["WebLogAPI"]), jsonItem);
//string response = client.UploadString(new Uri(ConfigurationManager.AppSettings["SMGWebLogAPI"]), "POST", jsonItem);
string result = JsonConvert.DeserializeObject<string>(response);
if (result != "ok")
{
await SMTPSendEmailAsync.SendEmail("brownp#spectrummg.com", "logging failed WebAPI call",
"Error return from WebAPI call in PostLoggingItem");
}
return;
//}
}
catch (Exception e)
{
await SMTPSendEmailAsync.SendEmail("brownp#spectrummg.com", "logging failed WebAPI call",
"Exception in PostLoggingItem" + e.Message);
return;
}
}
You can see where I have commented out the working UploadString call. Also, I have a theory problem related to "lifetime" of the WebClient object, so played around with creating it in the method (see commented using), but now create it with the object instantiation to which method PostLoggingItem belongs.
here's fiddler:
I'd sure like to know why the Async does not work. Also, I have used aync methods and awaits all the way up the call tree - to no avail.

Use Webclient to download several files

I'm trying to download files from webserver via a NUnit-testcase like this:
[TestCase("url_to_test_server/456.pdf")]
[TestCase("url_to_test_server/457.pdf")]
[TestCase("url_to_test_server/458.pdf")]
public void Test(string url)
{
using (WebClient client = new WebClient())
{
client.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0)");
client.DownloadFile(new Uri(url), #"C:\Temp\" + Path.GetFileName(url));
}
}
This code works, but when i'm trying to get the file size, it hangs.
[TestCase("url_to_test_server/456.pdf")]
[TestCase("url_to_test_server/457.pdf")]
[TestCase("url_to_test_server/458.pdf")]
public void Test(string url)
{
using (WebClient client = new WebClient())
{
client.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 8.0)");
client.OpenRead(url);
Int64 bytes_total = Convert.ToInt64(client.ResponseHeaders["Content-Length"]);
client.DownloadFile(new Uri(url), #"C:\Temp\" + Path.GetFileName(url));
}
}
How to solve this?

Waking up dead post but here is the answer...
This issue happens when server is not providing Content-Length in the response header. You have to fix that on the server side.
Another reason when it happens is when we have reached the connection limit to the server.
So I am assuming that your issue was similar and it was hanging on second or third try in a loop.
When we call OpenRead, it opens up a stream. We just need to close this stream after getting our file size to make it work properly.
Here is the code I use to get the size:
/// <summary>
/// Gets file size from a url using WebClient and Stream classes
/// </summary>
/// <param name="address">url</param>
/// <param name="useHeaderOnly">requests only headers instead of full file</param>
/// <returns>File size or -1 if their is an issue.</returns>
static Int64 GetFileSize(string address, bool useHeaderOnly = false)
{
Int64 retVal = 0;
try
{
if(useHeaderOnly)
{
WebRequest request = WebRequest.Create(address);
request.Method = "HEAD";
// WebResponse also has to be closed otherwise we get the same issue of hanging on the connection limit. Using statement closes it automatically.
using (WebResponse response = request.GetResponse())
{
if (response != null)
{
retVal = response.ContentLength;
//retVal = Convert.ToInt64(response.Headers["Content-Length"]);
}
}
request = null;
}
else
{
using (WebClient client = new WebClient())
{
// Stream has to be closed otherwise we get the issue of hanging on the connection limit. Using statement closes it automatically.
using (Stream response = client.OpenRead(address))
{
retVal = Convert.ToInt64(client.ResponseHeaders["Content-Length"]);
}
}
}
}
catch (Exception ex)
{
Console.WriteLine(ex.Message);
retVal = -1;
}
return retVal;
}

You have to make sure that your server supports this header too. It doesn't seem to be a client problem.
I'd download the file in a browser and check the communications using firebug or some similar program. You have to see Content-length explicitly returned in the response. If not, you need to check the server, otherwise your problem is on the client side.
I actually can't imagine a reason why the client can't read the header if it is indeed beeing returned.

C# WebRequest Check if page requires HTTP Authentication

Does anyone know how to check if a webpage is asking for HTTP Authentication via C# using the WebRequest class? I'm not asking how to post Credentials to the page, just how to check if the page is asking for Authentication.
Current Snippet to get HTML:
WebRequest wrq = WebRequest.Create(address);
wrs = wrq.GetResponse();
Uri uri = wrs.ResponseUri;
StreamReader strdr = new StreamReader(wrs.GetResponseStream());
string html = strdr.ReadToEnd();
wrs.Close();
strdr.Close();
return html;
PHP Server side source:
<?php
if (!isset($_SERVER['PHP_AUTH_USER'])) {
header('WWW-Authenticate: Basic realm="Secure Sign-in"');
header('HTTP/1.0 401 Unauthorized');
echo 'Text to send if user hits Cancel button';
exit;
} else {
echo "<p>Hello {$_SERVER['PHP_AUTH_USER']}.</p>";
echo "<p>You entered {$_SERVER['PHP_AUTH_PW']} as your password.</p>";
}
?>

WebRequest.GetResponse returns an object of type HttpWebResponse. Just cast it and you can retrieve StatusCode.
However, .Net will give you an exception if it receives a response of status 4xx or 5xx (thanks for your feedback).
There is a little workaround, check it out:
HttpWebRequest wrq = (HttpWebRequest)WebRequest.Create(#"http://webstrand.comoj.com/locked/safe.php");
HttpWebResponse wrs = null;
try
{
wrs = (HttpWebResponse)wrq.GetResponse();
}
catch (System.Net.WebException protocolError)
{
if (((HttpWebResponse)protocolError.Response).StatusCode == HttpStatusCode.Unauthorized)
{
//do something
}
}
catch (System.Exception generalError)
{
//run to the hills
}
if (wrs.StatusCode == HttpStatusCode.OK)
{
Uri uri = wrs.ResponseUri;
StreamReader strdr = new StreamReader(wrs.GetResponseStream());
string html = strdr.ReadToEnd();
wrs.Close();
strdr.Close();
}
Hope this helps.
Regards

Might want to try
WebClient wc = new WebClient();
CredentialCache credCache = new CredentialCache();
If you can work with WebClient instead of WebRequest, you should it's a bit higher level, easier to handle headers etc.
Also, might want to check this thread:
System.Net.WebClient fails weirdly

Sending a http request in C# and catching network issues

I previously had a small VBScript that would test if a specific website was accessible by sending a GET request. The script itself was extremely simple and did everything I needed:
Function GETRequest(URL) 'Sends a GET http request to a specific URL
Dim objHttpRequest
Set objHttpRequest = CreateObject("MSXML2.XMLHTTP.3.0")
objHttpRequest.Open "GET", URL, False
On Error Resume Next 'Error checking in case access is denied
objHttpRequest.Send
GETRequest = objHttpRequest.Status
End Function
I now want to include this sort of functionality in an expanded C# application. However I've been unable to get the same results my previous script provided.
Using code similar to what I've posted below sort of gets me a proper result, but fails to run if my network connection has failed.
public static void GETRequest()
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url");
request.Method = "GET";
HttpStatusCode status;
HttpWebResponse response;
try
{
response = (HttpWebResponse)request.GetResponse();
status = response.StatusCode;
Console.WriteLine((int)response.StatusCode);
Console.WriteLine(status);
}
catch (WebException e)
{
status = ((HttpWebResponse)e.Response).StatusCode;
Console.WriteLine(status);
}
}
But as I said, I need to know if the site is accessible, not matter the reason: the portal could be down, or the problem might reside on the side of the PC that's trying to access it. Either way: I don't care.
When I used MSXML2.XMLHTTP.3.0 in the script I was able to get values ranging from 12000 to 12156 if I was having network problems. I would like to have the same functionality in my C# app, that way I could at least write a minimum of information to a log and let the computer act accordingly. Any ideas?

A direct translation of your code would be something like this:
static void GetStatusCode(string url)
{
dynamic httpRequest = Activator.CreateInstance(Type.GetTypeFromProgID("MSXML2.XMLHTTP.3.0"));
httpRequest.Open("GET", url, false);
try { httpRequest.Send(); }
catch { }
finally { Console.WriteLine(httpRequest.Status); }
}
It's as small and simple as your VBScript script, and uses the same COM object to send the request.
This code happily gives me error code like 12029 ERROR_WINHTTP_CANNOT_CONNECT or 12007 ERROR_WINHTTP_NAME_NOT_RESOLVED etc.

If the code is failing only when you don't have an available network connection, you can use GetIsNetworkAvailable() before executing your code. This method will return a boolean indicating if a network connection is available or not. If it returns false, you could execute an early return / notify the user, and if not, continue.
System.Net.NetworkInformation.NetworkInterface.GetIsNetworkAvailable()
using the code you provided above:
public static void GETRequest()
{
if (!System.Net.NetworkInformation.NetworkInterface.GetIsNetworkAvailable())
return; //or alert the user there is no connection
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://url");
request.Method = "GET";
HttpStatusCode status;
HttpWebResponse response;
try
{
response = (HttpWebResponse)request.GetResponse();
status = response.StatusCode;
Console.WriteLine((int)response.StatusCode);
Console.WriteLine(status);
}
catch (WebException e)
{
status = ((HttpWebResponse)e.Response).StatusCode;
Console.WriteLine(status);
}
}

This should work for you, i've used it many times before, cut it down a bit for your needs: -
private static string GetStatusCode(string url)
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
req.Method = WebRequestMethods.Http.Get;
req.ProtocolVersion = HttpVersion.Version11;
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)";
try
{
HttpWebResponse response = (HttpWebResponse)req.GetResponse();
StringBuilder sb = new StringBuilder();
foreach (string header in response.Headers)
{
sb.AppendLine(string.Format("{0}: {1}", header, response.GetResponseHeader(header)));
}
return string.Format("Response Status Code: {0}\nServer:{1}\nProtocol: {2}\nRequest Method: {3}\n\n***Headers***\n\n{4}", response.StatusCode,response.Server, response.ProtocolVersion, response.Method, sb);
}
catch (Exception e)
{
return string.Format("Error: {0}", e.ToString());
}
}
Feel free to ignore the section that gets the headers