We Keep Coding

A simple reverse proxy using ASP.NET, C# with authentication

I am attempting to forward custom parameters to a RESTful API server and return the proxied response to the client-facing server. I don't want the client to have access to or be able to read the API HTTP request/response interactions, so I decided to perform this action using a reverse proxy. I have no problem forwarding the request and returning a response. The problem lies in the authentication. The client-facing server always wants to redirect to the login page because it doesn't believe the client is authenticated. I have tried using HTTPS and HTTP with similar results.
I have been researching this problem for quite some time and found quite a variety of answers, none of which seem to quite encompass my specific use case. I am following this example, which is the closest to what I specifically need. However, the credentials portion the author commented out (//request.Credentials = CredentialCache.DefaultCredentials;) doesn't seem to cover the authentication portion I am attempting to implement. Please help me understand this problem and solution.
Here is the code I am using from the controller:
public ActionResult ProxyEndpoint(string custom_string, string another_custom_string)
{
//Bunch of code here to grab the remoteUrl from AppConfig and do stuff to the parameters and store them in queryString, unnecessary to show here.
//Here's the important bits:
remoteUrl = remoteUrl + "?" + queryString; // create my remoteUrl
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(remoteUrl);
request.Credentials = CredentialCache.DefaultCredentials;
// Also tried this to no avail:
request.Credentials = CredentialCache.DefaultNetworkCredentials;
return ProxyActionResult(request.GetResponse());
}
Here is the ProxyActionResult class:
public class ProxyActionResult : ActionResult
{
WebResponse _response;
public ProxyActionResult(WebResponse response)
{
_response = response;
}
public override void ExecuteResult(ControllerContext controllerContext)
{
HttpContextBase httpContext = controllerContext.HttpContext;
WebResponse response = _response;
// Read the byte stream from the response:
Stream responseStream = response.GetResponseStream();
// Pulled this next piece from http://www.codeproject.com/Articles/7135/Simple-HTTP-Reverse-Proxy-with-ASP-NET-and-IIS
// Seemed to fit our use case.
if ((response.ContentType.ToLower().IndexOf("html") >= 0) || (response.ContentType.ToLower().IndexOf("javascript") >= 0))// || (response.ContentType.ToLower().IndexOf("image") >= 0))
{
//If the response is HTML Content, parse it like HTML:
StreamReader readStream = new StreamReader(responseStream, Encoding.Default);
String content;
content = ParseHtmlResponse(readStream.ReadToEnd(), httpContext.Request.ApplicationPath);
//Write the updated HTML to the client(and then close the response):
httpContext.Response.Write(content);
httpContext.Response.ContentType = response.ContentType;
response.Close();
httpContext.Response.End();
}
else
{
// If the response is not HTML Content, write the stream directly to the client:
var buffer = new byte[1024];
int bytes = 0;
while ((bytes = responseStream.Read(buffer, 0, 1024)) > 0)
{
httpContext.Response.OutputStream.Write(buffer, 0, bytes);
}
// from http://www.dotnetperls.com/response-binarywrite
httpContext.Response.ContentType = response.ContentType; // Set the appropriate content type of the response stream.
// and close the stream:
response.Close();
httpContext.Response.End();
}
//throw new NotImplementedException();
}
// Debating whether we need this:
public string ParseHtmlResponse(string html, string appPath)
{
html = html.Replace("\"/", "\"" + appPath + "/");
html = html.Replace("'/", "'" + appPath + "/");
html = html.Replace("=/", "=" + appPath + "/");
return html;
}

It turns out that nothing is wrong with the reverse proxy code. The remote server was an ArcGIS OpenLayers API and it had a setting that said crossOrigin: anonymous. I commented out this setting and it worked perfectly.
Check out the documentation if you have this particular ArcGIS OpenLayers problem:
http://openlayers.org/en/v3.14.2/apidoc/ol.source.ImageWMS.html

Get web data by using webclient or httpRequest

I want to use webclient or httpRequest to make request on some URL. It ok if the web don't need authetication, But if website need credentical then the respond not ok. I use example code of Microsoft like below:
public class Test
{
// Specify the URL to receive the request.
public static void Main (string[] args)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create (args[0]);
// Set some reasonable limits on resources used by this request
request.MaximumAutomaticRedirections = 4;
request.MaximumResponseHeadersLength = 4;
// Set credentials to use for this request.
request.Credentials = CredentialCache.DefaultCredentials;
HttpWebResponse response = (HttpWebResponse)request.GetResponse ();
Console.WriteLine ("Content length is {0}", response.ContentLength);
Console.WriteLine ("Content type is {0}", response.ContentType);
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream ();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
Console.WriteLine ("Response stream received.");
Console.WriteLine (readStream.ReadToEnd ());
response.Close ();
readStream.Close ();
}
}
Who have experience about this please help me in more detail.

HttpResponseCode has a property called StatusCode which specifies if the request has failed or succeeded. There are various status code which you can find online. But the one you are looking for is 401 which means unauthorized and 200 is for request that is successful. You can add following line after request.GetResponse() to check whether your request was successful or not.
if (response.StatusCode == HttpStatusCode.OK){
// you can continue parsing your response here.
}elseif(response.StatusCode == HttpStatusCode.Unauthorized){
// this means your request has failed due to authentication issue.
}

how to read content from webpage

I want to access a webpage & store the contents of the webpage into a database
this is the code I have tried for reading the contents of the webpage
public static WebClient wClient = new WebClient();
public static TextWriter textWriter;
public static String readFromLink()
{
string url = "http://www.ncedc.org/cgi-bin/catalog-search2.pl";
HttpWebRequest webRequest = WebRequest.Create(url) as HttpWebRequest;
webRequest.Method = "POST";
System.Net.WebClient client = new System.Net.WebClient();
byte[] data = client.DownloadData(url);
string html = System.Text.Encoding.UTF8.GetString(data);
return html;
}
public static bool WriteTextFile(String fileName, String t)
{
try
{
textWriter = new StreamWriter(fileName);
}
catch (Exception)
{
return false;
Console.WriteLine("Data Save Unsuccessful: Could Not create File");
}
try
{
textWriter.WriteLine(t);
}
catch (Exception)
{
return false;
Console.WriteLine("Data Save UnSuccessful: Could Not Save Data");
}
textWriter.Close();
return true;
Console.WriteLine("Data Save Successful");
}
static void Main(string[] args)
{
String saveFile = "E:/test.txt";
String reSultString = readFromLink();
WriteTextFile(saveFile, reSultString);
Console.ReadKey();
}
but this code gives me an o/p as- This script should be referenced with a METHOD of POST. REQUEST_METHOD=GET
please tell me how to resolve this

You are mixing HttpWebRequest with System.Net.WebClient code. They are a different. You can use WebClient.UploadValues to send a POST with WebClient. You will also need to provide some POST data:
System.Net.WebClient client = new System.Net.WebClient();
NameValueCollection postData = new NameValueCollection();
postData.Add("format","ncread");
postData.Add("mintime","2002/01/01,00:00:00");
postData.Add("minmag","3.0");
postData.Add("etype","E");
postData.Add("outputloc","web");
postData.Add("searchlimit","100000");
byte[] data = client.UploadValues(url, "POST", postData);
string html = System.Text.Encoding.UTF8.GetString(data);
You can find out what parameters to pass by inspecting the POST message in Fiddler. And yes, as commented by #Chris Pitman, use File.WriteAllText(path, html);

I'm not sure if it's a fault on your side as I get the same message just by opening the page. The page source does not contain any html so I don't think you can do webRequest.Method = "POST". Have you spoken to the administrators of the site?

The .NET framework provides a rich set of methods to access data stored on the web. First you will have to include the right namespaces:
using System.Text;
using System.Net;
using System.IO;
The HttpWebRequest object allows us to create a request to the URL, and the WebResponse allows us to read the response to the request.
We’ll use a StreamReader object to read the response into a string variable.
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(URL);
myRequest.Method = "GET";
WebResponse myResponse = myRequest.GetResponse();
StreamReader sr = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
string result = sr.ReadToEnd();
sr.Close();
myResponse.Close();
In this code sample, the URL variable should contain the URL that you want to get, and the result variable will contain the contents of the web page. You may want to add some error handling as well for a real application.

As far as I see, the URL you're requesting is a perl script. I think it demands POST to get search arguments and therefore delivers search results.

GetRequestStream() is throwing time out exception when posting data to HTTPS url

I'm calling an API hosted on Apache server to post data. I'm using HttpWebRequest to perform POST in C#.
API has both normal HTTP and secure layer (HTTPS) PORT on the server. When I call HTTP URL it works perfectly fine. However, when I call HTTPS it gives me time-out exception (at GetRequestStream() function). Any insights? I'm using VS 2010, .Net framework 3.5 and C#. Here is the code block:
string json_value = jsonSerializer.Serialize(data);
HttpWebRequest request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;
System.IO.Stream reqStream = request.GetRequestStream();
reqStream.Write(buffer, 0, buffer.Length);
reqStream.Close();
EDIT:
The console program suggested by Peter works fine. But when I add data (in JSON format) that needs to be posted to the API, it throws out operation timed out exception. Here is the code that I add to console based application and it throws error.
byte[] buffer = Encoding.ASCII.GetBytes(json_value);
request.ContentLength = buffer.Length;

I ran into the same issue. It seems like it is solved for me. I went through all my code making sure to invoke webResponse.Close() and/or responseStream.Close() for all my HttpWebResponse objects. The documentation indicates that you can close the stream or the HttpWebResponse object. Calling both is not harmful, so I did. Not closing the responses may cause the application to run out of connections for reuse, and this seems to affect the HttpWebRequest.GetRequestStream as far as I can observe in my code.

I don't know if this will help you with your specific problem but you should consider Disposing some of those objects when you are finished with them. I was doing something like this recently and wrapping stuff up in using statements seems to clean up a bunch of timeout exceptions for me.
using (var reqStream = request.GetRequestStream())
{
if (reqStream == null)
{
return;
}
//do whatever
}
also check these things
Is the server serving https in your local dev environment?
Have you set up your bindings *.443 (https) properly?
Do you need to set credentials on the request?
Is it your application pool account accessing the https resources or is it your account being passed through?
Have you thought about using WebClient instead?
using (WebClient client = new WebClient())
{
using (Stream stream = client.OpenRead("https://server-url-xxxx.com"))
using (StreamReader reader = new StreamReader(stream))
{
MessageBox.Show(reader.ReadToEnd());
}
}
EDIT:
make a request from console.
internal class Program
{
private static void Main(string[] args)
{
new Program().Run();
Console.ReadLine();
}
public void Run()
{
var request = (HttpWebRequest)System.Net.WebRequest.Create("https://server-url-xxxx.com");
request.Method = "POST";
request.ProtocolVersion = System.Net.HttpVersion.Version10;
request.ContentType = "application/x-www-form-urlencoded";
using (var reqStream = request.GetRequestStream())
{
using(var response = new StreamReader(reqStream )
{
Console.WriteLine(response.ReadToEnd());
}
}
}
}

Try this:
WebRequest req = WebRequest.Create("https://server-url-xxxx.com");
req.Method = "POST";
string json_value = jsonSerializer.Serialize(data); //Body data
ServicePointManager.Expect100Continue = false;
using (var streamWriter = new StreamWriter(req.GetRequestStream()))
{
streamWriter.Write(json_value);
streamWriter.Flush();
streamWriter.Close();
}
HttpWebResponse resp = req.GetResponse() as HttpWebResponse;
Stream GETResponseStream = resp.GetResponseStream();
StreamReader sr = new StreamReader(GETResponseStream);
var response = sr.ReadToEnd(); //Response
resp.Close(); //Close response
sr.Close(); //Close StreamReader
And review the URI:
Reserved characters. Send reserved characters by the URI can bring
problems ! * ' ( ) ; : # & = + $ , / ? # [ ]
URI Length: You should not exceed 2000 characters

I ran into this, too. I wanted to simulate hundreds of users with a Console app. When simulating only one user, everything was fine. But with more users came the Timeout exception all the time.
Timeout occurs because by default the ConnectionLimit=2 to a ServicePoint (aka website).
Very good article to read: https://venkateshnarayanan.wordpress.com/2013/04/17/httpwebrequest-reuse-of-tcp-connections/
What you can do is:
1) make more ConnectionGroups within a servicePoint, because ConnectionLimit is per ConnectionGroups.
2) or you just simply increase the connection limit.
See my solution:
private HttpWebRequest CreateHttpWebRequest<U>(string userSessionID, string method, string fullUrl, U uploadData)
{
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(fullUrl);
req.Method = method; // GET PUT POST DELETE
req.ConnectionGroupName = userSessionID; // We make separate connection-groups for each user session. Within a group connections can be reused.
req.ServicePoint.ConnectionLimit = 10; // The default value of 2 within a ConnectionGroup caused me always a "Timeout exception" because a user's 1-3 concurrent WebRequests within a second.
req.ServicePoint.MaxIdleTime = 5 * 1000; // (5 sec) default was 100000 (100 sec). Max idle time for a connection within a ConnectionGroup for reuse before closing
Log("Statistics: The sum of connections of all connectiongroups within the ServicePoint: " + req.ServicePoint.CurrentConnections; // just for statistics
if (uploadData != null)
{
req.ContentType = "application/json";
SerializeToJson(uploadData, req.GetRequestStream());
}
return req;
}
/// <summary>Serializes and writes obj to the requestStream and closes the stream. Uses JSON serialization from System.Runtime.Serialization.</summary>
public void SerializeToJson(object obj, Stream requestStream)
{
DataContractJsonSerializer json = new DataContractJsonSerializer(obj.GetType());
json.WriteObject(requestStream, obj);
requestStream.Close();
}

You may want to set timeout property, check it here http://www.codeproject.com/Tips/69637/Setting-timeout-property-for-System-Net-WebClient

Reading remote file [C#]

I am trying to read a remote file using HttpWebRequest in a C# Console Application. But for some reason the request is empty - it never finds the URL.
This is my code:
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create("http://uo.neverlandsreborn.org:8000/botticus/status.ecl");
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
How come this is not possible?
The file only contains a string. Nothing more!

How are you reading the response data? Does it come back as successful but empty, or is there an error status?
If that doesn't help, try Wireshark, which will let you see what's happening at the network level.
Also, consider using WebClient instead of WebRequest - it does make it incredibly easy when you don't need to do anything sophisticated:
string url = "http://uo.neverlandsreborn.org:8000/botticus/status.ecl";
WebClient wc = new WebClient();
string data = wc.DownloadString(url);

You have to get the response stream and read the data out of that. Here's a function I wrote for one project that does just that:
private static string GetUrl(string url)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
if (response.StatusCode != HttpStatusCode.OK)
throw new ServerException("Server returned an error code (" + ((int)response.StatusCode).ToString() +
") while trying to retrieve a new key: " + response.StatusDescription);
using (var sr = new StreamReader(response.GetResponseStream()))
{
return sr.ReadToEnd();
}
}
}