We Keep Coding

Adding SSL cert causes 404 only in browser calls

I am working in an internal corporate environment. We have created a webapi installed on iis on port 85. We call this from another MVC HelperApp on port 86. It all works as expected. Now we want to tighten security and add an SSL cert to iis on port 444 and bind it to our API.
Initially we test it with Postman, SoapUI, and a C# console app and it all works. Now we try calling it from our MVC HelperApp and it returns a 404 sometimes.
Deeper debugging; I put the code into a C# DLL (see below). Using the console app I call the Dll.PostAPI and it works as expected. Now I call that same Dll.PostAPI from the MVC HelperApp and it won't work. When I step through the code I make it as far as this line await client.PostAsync(url, data); and the code bizarrely ends, it doesn't return and it doesn't throw an exception. Same for Post and Get. I figure it makes the call and nothing is returned, no response and no error.
Also, if I change the url to "https://httpbin.org/post" or to the open http port85 on iss it will work. I have concluded that the C# code is not the problem (but I'm open to being wrong).
Therefore I have come to the conclusion that for some reason the port or cert is refusing calls from browsers.
We are looking at:
the "Subject Alternative Name" but all the examples show
WWW.Addresses which we are not using.
the "Friendly Name" on the cert creation.
and CORS Cross-Origin Resource Sharing.
These are all subjects we lack knowledge in.
This is the calling code used exactly the same in the console app and the web app:
var lib = new HttpsLibrary.ApiCaller();
lib.makeHttpsCall();
This is what's in the DLL that gets called:
public async Task<string> makeHttpsCall()
{
try
{
List<Quote> quotes = new List<Quote>();
quotes.Add(CreateDummyQuote());
var json = JsonConvert.SerializeObject(quotes);
var data = new StringContent(json, Encoding.UTF8, "application/json");
var url = "https://httpbin.org/post"; //this works in Browser
//url = "https://thepath:444//api/ProcessQuotes"; //444 DOES NOT WORK in browsers only. OK in console app.
//url = "http://thepath:85/api/ProcessQuotes"; //85 works.
var client = new HttpClient();
var response = await client.PostAsync(url, data); //<<<this line never returns when called from browser.
//var response = await client.GetAsync(url); //same outcome for Get or Post
var result = await response.Content.ReadAsStringAsync();
return result;
}
catch (Exception ex)
{
throw;
}
}

Problem with C# Proxy in test console script

I have a project that has to get 100's of pages of data from a site each day. I use a paid for proxy with login details and I wait 5 seconds between requests so I don't hammer their site and pass a referer, user-agent and it is a simple GET request.
However I tried to make just a little C# Console script to test various ways of adding proxies e.g with or without credentials and got a working IP:Port from the web > http://www.freeproxylists.net/ to test it with, as my own details in this test didn't work. I am at a loss to why this test script isn't working when my main project is.
I am accessing an old site I own anyway so I am not blocking my own home IP as I can access it on the web (or any other page or site) in a browser easily.
Without using a proxy I just get a 30 second wait (the timeout length) then a "Timeout Error", with the proxy I get NO wait at all (free proxy OR one I own with credentials) before a "Timeout Error" - so whether I use a proxy or not it fails to return a response.
I am probably just sleep drained but would like to know what I am doing wrong as I just copied my "MakeHTTPGetRequest" method from my main projects Scraper class and just removed all the case statements in the try/catch to check for Connection/Timeout/404/Service/Server errors etc and put it into one simple Main method here...
public static void Main(string[] args)
{
string url = "https://www.strictly-software.com"; // a site I own
//int port = ????; // working in main project crawler
int port = 3128; // from a list of working free proxies
string proxyUser = "????"; // working in main project crawler
string proxyPassword = "????"; // working in main project crawler
string proxyIP = "167.99.230.151"; // from a list of working proxies
ShowDebug("Make a request to: " + url + " with proxy:" + proxyIP + ":" + port.ToString());
// user basic IP and Port proxy with no login
WebProxy proxy = new WebProxy(proxyIP, port);
/*
// use default port, username and password to login
// get same error with correct personal proxy and login but not
// in main project
WebProxy proxy = new WebProxy(proxyIP, port)
{
Credentials = new NetworkCredential(proxyUser, proxyPassword)
};
*/
ShowDebug("Use Proxy: " + proxy.Address.ToString());
HttpWebRequest client = (HttpWebRequest)WebRequest.Create(url);
client.Referer = "https://www.strictly-software.com";
client.Method = "GET";
client.ContentLength = 0;
client.ContentType = "application/x-www-form-urlencoded;charset=UTF-8";client.Proxy = proxy;
client.UserAgent = "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0";
client.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
client.Headers.Add("Accept-Encoding", "gzip,deflate");
client.KeepAlive = true;
client.Timeout = 30;
ShowDebug("make request with " + client.UserAgent.ToString());
try
{
// tried adding this to see if it would help but didn't
//ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
// get the response
HttpWebResponse response = (HttpWebResponse)client.GetResponse();
ShowDebug("response.ContentEncoding = " + response.ContentEncoding.ToString());
ShowDebug("response.ContentType = " + response.ContentType.ToString());
ShowDebug("Status Desc: " + response.StatusDescription.ToString());
ShowDebug("HTTP Status Code: " + response.StatusCode.ToString());
ShowDebug("Now get the full response back");
// old method not working with £ signs
StreamReader ResponseStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
string ResponseContent = ResponseStream.ReadToEnd().Trim();
ShowDebug("content from response == " + Environment.NewLine + ResponseContent);
ResponseStream.Close();
response.Close();
}
catch (WebException ex)
{
ShowDebug("An error occurred");
ShowDebug("WebException " + ex.Message.ToString());
ShowDebug(ex.Status.ToString());
}
catch(Exception ex)
{
ShowDebug("An error occurred");
ShowDebug("Exception " + ex.Message.ToString());
}
finally
{
ShowDebug("At the end");
}
}
The error messages from the console (ShowDebug is just a wrapper for the time + message)...
02/08/2020 00:00:00: Make a request to: https://www.strictly-software.com with proxy:167.99.230.151:3128
02/08/2020 00:00:00: Use Proxy: http://167.99.230.151:3128/
02/08/2020 00:00:00: make request with Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0
02/08/2020 00:00:00: An error occurred
02/08/2020 00:00:00: WebException The operation has timed out
02/08/2020 00:00:00: Timeout
02/08/2020 00:00:00: At the end
I am sure it is just a something I have missed but I know that this code was copied from my main project that is currently crawling through 100s of pages with the same code and using a proxy with my credentials that work as I am getting data back at the moment from the main projects code.
I can ping the IP address of the proxy, but going to it in a browser returns a connection error, this is despite my big project using the same proxy to ripple through tons of pages and return HTML all night long...
I just wanted to update my main project by adding new methods to pass in custom proxies, or not use a proxy for the 1st attempt but if it fails then use one for a final attempt, or use a default proxy:port etc.

You set your timeout to be 30 milliseconds: client.Timeout = 30;
That could be causing your timeouts.
More info here.

Not sure if this solves your problem, but:
The documentation for HttpWebRequest states the following:
The local computer or application config file may specify that a
default proxy be used. If the Proxy property is specified, then the
proxy settings from the Proxy property override the local computer or
application config file and the HttpWebRequest instance will use the
proxy settings specified. If no proxy is specified in a config file
and the Proxy property is unspecified, the HttpWebRequest class uses
the proxy settings inherited from Internet Explorer on the local
computer. If there are no proxy settings in Internet Explorer, the
request is sent directly to the server.
Maybe there is a proxy in the IE settings configured? This does not explain why the request fails using the custom proxy, but it may be worth a shot.
I also suggest, that you test the free proxy using something like postman. From my experience, these free proxies don't work at least half of the time.
My best guess is, that when not using the proxy, the request fails because of the IE stuff and when using the proxy, the proxy itself is simply not working.
Hopefully this solves the problem...

Why is my Windows CE device unable to successfully invoke my server's REST methods?

This is related/sort of a followup to this question but contains more and more specific information.
I have a REST app running on my PC. I need to call some of the methods on the server from a handheld WindowsCE device that uses the compact framework.
I can contact the server's methods from Postman, so it's not a problem that way. In Postman, I use:
http://localhost:21609/api/inventory/sendXML/duckbill/platypus/bloo
...and with this I reach the breakpoint in this REST method on my server app:
[HttpPost]
[Route("api/inventory/sendxml/{userId}/{pwd}/{filename}")]
public async Task SendInventoryXML(String userId, String pwd, String fileName)
However, although using "PPP_PEER" is the way to contact the PC from the handheld device (127.0.0.1 doesn't fly, because the handheld sees that as being itself, and commits egregious acts of cannibalism when trying to contact that address) - as can be seen by this TCP code that does work from the handheld device:
string pingString = "PING|";
TcpClient client = new TcpClient("PPP_PEER", 7727);
try
{
try
{
Stream s = client.GetStream();
StreamReader sr = new StreamReader(s);
StreamWriter sw = new StreamWriter(s) { AutoFlush = true };
String response = String.Empty;
if (firstRecord)
{
sw.WriteLine(pingString);
. . .
...trying to use PPP_PEER to call the REST method fails.
Here is the code I use to try to do that:
//HHSConsts
public static string BASE_REST_URL = "http://PPP_PEER:21609/api/";
. . .
//frmMain
private void SendInventories()
{
try
{
foreach (String tblname in listBoxWork.Items)
{
String xmlData = hhsdbutils.GetINVDataAsXMLFromTable(tblname, fileName);
String uri = String.Format("{0}inventory/sendXML/duckbill/platypus/{1}",
HHSConsts.BASE_REST_URL, fileName);
fileXferImp = HHSConsts.GetFileTransferMethodology();
fileXferImp.SendDataContentsAsXML(uri, xmlData, tblname);
. . .
// FileXferREST.cs
public void SendDataContentsAsXML(String destinationPath, String data, String fileName,
String siteNumber, bool firstRecord, bool lastRecord)
{
SendHTTPRequestNoCredentials(destinationPath, HttpMethods.POST, data, "application/xml");
}
public static HttpWebRequest SendHTTPRequestNoCredentials(string uri, HttpMethods method,
string data, string contentType)
{
WebRequest request = WebRequest.Create(uri);
try
{
request.Method = Enum.ToObject(typeof(HttpMethods), method).ToString();
request.ContentType = contentType;
((HttpWebRequest)request).Accept = contentType;
((HttpWebRequest)request).KeepAlive = false;
((HttpWebRequest)request).ProtocolVersion = HttpVersion.Version10;
if (method != HttpMethods.GET && method != HttpMethods.DELETE)
{
byte[] arrData = Encoding.UTF8.GetBytes(data);
request.ContentLength = arrData.Length;
using (Stream oS = request.GetRequestStream())
{
oS.Write(arrData, 0, arrData.Length);
}
}
else
{
request.ContentLength = 0;
}
}
catch (WebException webex)
{
HttpWebResponse hwr = (HttpWebResponse)webex.Response;
HttpStatusCode hsc = hwr.StatusCode;
String webExMsgAndStatusCode = String.Format("{0} Status code == {1}", webex.Message,
hsc.ToString());
ExceptionLoggingService.Instance.WriteLog(String.Format("From
FileXferREST.SendHTTPRequestNoCredentials: {0}", webExMsgAndStatusCode));
}
return request as HttpWebRequest;
}
No exception is thrown; it simply doens't work.
When attempting this call, I run rawcap to capture the packages being sent over the network with these command line args:
rawcap 127.0.0.1 [fileName].pcap
I then open the .pcap file in Wireshark, search for "21609" and get this TCP stream:
...I added the screen shot above to show the red/blue coloration for the request/response, but here is the entire rawcap/Wireshark conversation involving port 21609:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
POST /api//inventory/sendXML/duckbill/platypus/INV_3_20090307181658000.xml HTTP/1.0
Content-Type: application/xml
Accept: application/xml
Connection: Close
Content-Length: 388
Host: ppp_peer:21609
112209003343742SOME DESC2.2testVendorID]6161.51.995.58HTTP/1.1 404 Not Found
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/8.0
X-SourceFiles: =?UTF-8?B?QzpccHJvamVjdFxnaXRcQ1N0b3JlXEhIUy5BUElcYXBpXGludmVudG9yeVxzZW5kWE1MXGR1Y2tiaWxsXHBsYXR5cHVzXElOVl8zXzIwMDkwMzA3MTgxNjU4MDAwLnhtbA==?=
X-Powered-By: ASP.NET
Date: Thu, 18 Dec 2014 17:23:01 GMT
Connection: close
Content-Length: 5016
IIS 8.0 Detailed Error - 404.0 - Not Found
HTTP Error 404.0 - Not Found
The resource you are looking for has been removed, had its name changed, or is temporarily unavailable.
Most likely causes:
.The directory or file specified does not exist on the Web server. .The URL contains a typographical error. .A custom filter or module, such as URLScan, restricts access to the file.
Things you can try:
.Create the content on the Web server. .Review the browser URL. .Check the failed request tracing log and see which module is calling SetStatus. For more information, click here.
Detailed Error Information:
Module IIS Web Core
Notification MapRequestHandler
Handler StaticFile
Error Code 0x80070002
Requested URL http://ppp_peer:21609/api/inventory/sendXML/duckbill/platypus/INV_3_20090307181658000.xml
Physical Path C:\project\git\CStore\HHS.API\api\inventory\sendXML\duckbill\platypus\INV_3_20090307181658000.xml
Logon Method Anonymous
Logon User Anonymous
Request Tracing Directory C:\Users\clay\Documents\IISExpress\TraceLogFiles\HHS.API
More Information:
This error means that the file or directory does not exist on the server. Create the file or directory and try the request again.
View more information »
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So to recap, "PPP_PEER" does work as a way for the handheld device to describe/contact the PC (it works with the TCP code), and the REST method is reachable from another process (such as Postman), but trying to call the method using PPP_PEER from the handheld device fail with the "404.0 - Not Found" error.
Why?
UPDATE
When I replace "PPP_PEER" with the machine name of the PC, the error changes from "404 - Not Found" to "503 - Service Unavailable":
But that is a bit of a rompecabeza, because the service is obviously available, as it is running and its breakpoint is hit when I call it from Postman.
UPDATE 2
Now this is odd: If I use the full machine name instead of just the "truncated" name, the error goes back to being 404 instead of 503:
So if I use PPP_PEER as the host name, I get a 404; if I use shannon2, I get a 503; if I use shannon2.sscs.ad, I again get 404. Should I blame this on Tim Berners-Lee, Al Gore, Andy Warhol, or someone else?
SO...those are my results on trying to follow Eric Law's first suggestion; there are two others, but: what would I edit the Host header to that might make a difference? Or overwrite it to what?

It seems as if the basic problem is trying to access a "server" app running locally on my PC (even though the handheld != the PC, it is "sort of" the same thing, after a fashion).
Now that the server app is running on a different machine on the network altogether, I'm able to hit it; albeit not yet totally unexceptionally, as can be seen here.

Getting 403 error while scraping a website page for data

I have a price comparison website which scrapes prices from various websites. For all websites the code is working fine but one is returning 403 forbidden error. The website is developed in Asp.net MVC3 framework. Following is my code.
public static decimal? GetSpanFromWebSite(string url, string identification)
{
var baseUrl = new Uri(url);
HtmlAgilityPack.HtmlDocument document = new HtmlDocument();
try
{
WebClient client = new WebClient();
document.Load(client.OpenRead(baseUrl));
var div = document.DocumentNode.SelectNodes(identification).FirstOrDefault();
return Convert.ToDecimal(div.InnerHtml);
}
catch (Exception)
{
return null;
}
}
What is the workaround and how can i continue scrapping the website?

It is likely a scraping countermeasure implemented by the site.
Try to mimick the browser request as closely as possible (especially headers - user agent, referer, content-type etc.)

403 Forbidden
Actually server is understanding and accepting your request at the same time the server is denying your request, so check your HttpRequest Headers and Cookie values
you can use web debugging tool like fiddler http://www.telerik.com/fiddler/web-debugging
to debug the request and response

Failing to retrieve access token in .NET desktop app

I'm writing a .NET app that runs on a Windows computer. It is not accessible through the browser. The problem is, I can't authenticate like I should. I'm currently coding in C# .NET, more specific in C#.
I have a webbrowser control on my form.
The user logs on to facebook through this webbrowser control.
After the logon, I start the authentication procedure.
I then retreive a code.
Here's where it goes wrong. With this code I want to obtain an access token.
The generated request URL looks like: https://graph.facebook.com/oauth/access_token?client_id=____MY_APP_ID______&redirect_uri=http://localhost/&client_secret=_____MY_APP_SECRET_____&code=____MY_RETREIVED_CODE_____ and is made through the code below.
Please note that my redirect URL is http://localhost. This should be okay, right?
Also, in my App Settings, I have the following information.
Site URL: http://localhost/
Site Domain: localhost
private String ExchangeCodeForToken(String code, Uri redirectUrl)
{
var TokenEndpoint = new Uri("https://graph.facebook.com/oauth/access_token");
var url = TokenEndpoint + "?" +
"client_id=" + _AppID + "&" +
"redirect_uri=" + redirectUrl + "&" +
"client_secret=" + _AppSecret + "&" +
"code=" + code;
var request = WebRequest.CreateDefault(new Uri(url));
using (var response = request.GetResponse())
{
using (var responseStream = response.GetResponseStream())
{
using (var responseReader = new StreamReader(responseStream))
{
var responseText = responseReader.ReadToEnd();
var token = responseText.Replace("access_token=", "");
return token;
}
}
}
}
When I execute this, I get this error:
error http://www.imageupload.org/getfile.php?id=50131&a=447f6fcc0ebd4d3f8e8a59a3a6e36ac3&t=4de0841c&o=0889D68FDC35508BA2C6F2689FCBAB7C30A8670CC9647EE598701D8BEC13ED278F0989D393&n=autherror.png&i=1
Webexception was unhandled by user code
The remote server returned an error: (400) Bad Request.
Here's where I think I might be going wrong:
Are my app settings correct?
Should my redirect url be http://localhost, even if there isn't actually a service listening there?
Most importantly:
how do I get rid of this error and retreive the access token?
Thanks in advance!

You get this error because you are not supposed to call this URL from a Desktop app : as far as I know, you can not use the token endpoint for Desktop app authentication. Also, you can get the access token directly (no need to ask for a code first). Here is what you have to do.
Load the following URL in your embedded web browser :
https://www.facebook.com/dialog/oauth?
client_id=YOUR_APP_ID&
redirect_uri=https://www.facebook.com/connect/login_success.html
The user will be asked to log in and will be redirected to this URL with the access token in the URL :
https://www.facebook.com/connect/login_success.html#access_token=...
So you have to detect the redirect and retrieve the access token from the URL.

Thanks quinten!
However, I've managed to solve my own problem by using the C# Facebook SDK.
This software development kit has been a really great help!
There are a lot of samples included (including authorisation)
Anyone who programs in .NET with facebook should check it out! Coding for facebook is now much easier.
http://facebooksdk.codeplex.com/