How to one proxy server [duplicate] - c#

I have downloaded Privoxy few weeks ago and for the fun I was curious to know how a simple version of it can be done.
I understand that I need to configure the browser (client) to send request to the proxy. The proxy send the request to the web (let say it's a http proxy). The proxy will receive the answer... but how can the proxy send back the request to the browser (client)?
I have search on the web for C# and http proxy but haven't found something that let me understand how it works behind the scene correctly. (I believe I do not want a reverse proxy but I am not sure).
Does any of you have some explication or some information that will let me continue this small project?
Update
This is what I understand (see graphic below).
Step 1 I configure the client (browser) for all request to be send to 127.0.0.1 at the port the Proxy listen. This way, request will be not sent to the Internet directly but will be processed by the proxy.
Step2 The proxy see a new connection, read the HTTP header and see the request he must executes. He executes the request.
Step3 The proxy receive an answer from the request. Now he must send the answer from the web to the client but how???
Useful link
Mentalis Proxy : I have found this project that is a proxy (but more that I would like). I might check the source but I really wanted something basic to understand more the concept.
ASP Proxy : I might be able to get some information over here too.
Request reflector : This is a simple example.
Here is a Git Hub Repository with a Simple Http Proxy.

I wouldn't use HttpListener or something like that, in that way you'll come across so many issues.
Most importantly it'll be a huge pain to support:
Proxy Keep-Alives
SSL won't work (in a correct way, you'll get popups)
.NET libraries strictly follows RFCs which causes some requests to fail (even though IE, FF and any other browser in the world will work.)
What you need to do is:
Listen a TCP port
Parse the browser request
Extract Host connect to that host in TCP level
Forward everything back and forth unless you want to add custom headers etc.
I wrote 2 different HTTP proxies in .NET with different requirements and I can tell you that this is the best way to do it.
Mentalis doing this, but their code is "delegate spaghetti", worse than GoTo :)

I have recently written a light weight proxy in c# .net using TcpListener and TcpClient.
https://github.com/titanium007/Titanium-Web-Proxy
It supports secure HTTP the correct way, client machine needs to trust root certificate used by the proxy. Also supports WebSockets relay. All features of HTTP 1.1 are supported except pipelining. Pipelining is not used by most modern browsers anyway. Also supports windows authentication (plain, digest).
You can hook up your application by referencing the project and then see and modify all traffic. (Request and response).
As far as performance, I have tested it on my machine and works without any noticeable delay.

You can build one with the HttpListener class to listen for incoming requests and the HttpWebRequest class to relay the requests.

Proxy can work in the following way.
Step1, configure client to use proxyHost:proxyPort.
Proxy is a TCP server that is listening on proxyHost:proxyPort.
Browser opens connection with Proxy and sends Http request.
Proxy parses this request and tries to detect "Host" header. This header will tell Proxy where to open connection.
Step 2: Proxy opens connection to the address specified in the "Host" header. Then it sends HTTP request to that remote server. Reads response.
Step 3: After response is read from remote HTTP server, Proxy sends the response through an earlier opened TCP connection with browser.
Schematically it will look like this:
Browser Proxy HTTP server
Open TCP connection
Send HTTP request ----------->
Read HTTP header
detect Host header
Send request to HTTP ----------->
Server
<-----------
Read response and send
<----------- it back to the browser
Render content

If you are just looking to intercept the traffic, you could use the fiddler core to create a proxy...
http://fiddler.wikidot.com/fiddlercore
run fiddler first with the UI to see what it does, it is a proxy that allows you to debug the http/https traffic. It is written in c# and has a core which you can build into your own applications.
Keep in mind FiddlerCore is not free for commercial applications.

Agree to dr evil
if you use HTTPListener you will have many problems, you have to parse requests and will be engaged to headers and ...
Use tcp listener to listen to browser requests
parse only the first line of the request and get the host domain and port to connect
send the exact raw request to the found host on the first line of browser request
receive the data from the target site(I have problem in this section)
send the exact data received from the host to the browser
you see you dont need to even know what is in the browser request and parse it, only get the target site address from the first line
first line usually likes this
GET http://google.com HTTP1.1
or
CONNECT facebook.com:443 (this is for ssl requests)

Things have become really easy with OWIN and WebAPI. In my search for a C# Proxy server, I also came across this post http://blog.kloud.com.au/2013/11/24/do-it-yourself-web-api-proxy/ . This will be the road I'm taking.

Socks4 is a very simple protocol to implement. You listen for the initial connection, connect to the host/port that was requested by the client, send the success code to the client then forward the outgoing and incoming streams across sockets.
If you go with HTTP you'll have to read and possibly set/remove some HTTP headers so that's a little more work.
If I remember correctly, SSL will work across HTTP and Socks proxies. For a HTTP proxy you implement the CONNECT verb, which works much like the socks4 as described above, then the client opens the SSL connection across the proxied tcp stream.

For what it's worth, here is a C# sample async implementation based on HttpListener and HttpClient (I use it to be able to connect Chrome in Android devices to IIS Express, that's the only way I found...).
And If you need HTTPS support, it shouldn't require more code, just certificate configuration: Httplistener with HTTPS support
// define http://localhost:5000 and http://127.0.0.1:5000/ to be proxies for http://localhost:53068
using (var server = new ProxyServer("http://localhost:53068", "http://localhost:5000/", "http://127.0.0.1:5000/"))
{
server.Start();
Console.WriteLine("Press ESC to stop server.");
while (true)
{
var key = Console.ReadKey(true);
if (key.Key == ConsoleKey.Escape)
break;
}
server.Stop();
}
....
public class ProxyServer : IDisposable
{
private readonly HttpListener _listener;
private readonly int _targetPort;
private readonly string _targetHost;
private static readonly HttpClient _client = new HttpClient();
public ProxyServer(string targetUrl, params string[] prefixes)
: this(new Uri(targetUrl), prefixes)
{
}
public ProxyServer(Uri targetUrl, params string[] prefixes)
{
if (targetUrl == null)
throw new ArgumentNullException(nameof(targetUrl));
if (prefixes == null)
throw new ArgumentNullException(nameof(prefixes));
if (prefixes.Length == 0)
throw new ArgumentException(null, nameof(prefixes));
RewriteTargetInText = true;
RewriteHost = true;
RewriteReferer = true;
TargetUrl = targetUrl;
_targetHost = targetUrl.Host;
_targetPort = targetUrl.Port;
Prefixes = prefixes;
_listener = new HttpListener();
foreach (var prefix in prefixes)
{
_listener.Prefixes.Add(prefix);
}
}
public Uri TargetUrl { get; }
public string[] Prefixes { get; }
public bool RewriteTargetInText { get; set; }
public bool RewriteHost { get; set; }
public bool RewriteReferer { get; set; } // this can have performance impact...
public void Start()
{
_listener.Start();
_listener.BeginGetContext(ProcessRequest, null);
}
private async void ProcessRequest(IAsyncResult result)
{
if (!_listener.IsListening)
return;
var ctx = _listener.EndGetContext(result);
_listener.BeginGetContext(ProcessRequest, null);
await ProcessRequest(ctx).ConfigureAwait(false);
}
protected virtual async Task ProcessRequest(HttpListenerContext context)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
var url = TargetUrl.GetComponents(UriComponents.SchemeAndServer, UriFormat.Unescaped);
using (var msg = new HttpRequestMessage(new HttpMethod(context.Request.HttpMethod), url + context.Request.RawUrl))
{
msg.Version = context.Request.ProtocolVersion;
if (context.Request.HasEntityBody)
{
msg.Content = new StreamContent(context.Request.InputStream); // disposed with msg
}
string host = null;
foreach (string headerName in context.Request.Headers)
{
var headerValue = context.Request.Headers[headerName];
if (headerName == "Content-Length" && headerValue == "0") // useless plus don't send if we have no entity body
continue;
bool contentHeader = false;
switch (headerName)
{
// some headers go to content...
case "Allow":
case "Content-Disposition":
case "Content-Encoding":
case "Content-Language":
case "Content-Length":
case "Content-Location":
case "Content-MD5":
case "Content-Range":
case "Content-Type":
case "Expires":
case "Last-Modified":
contentHeader = true;
break;
case "Referer":
if (RewriteReferer && Uri.TryCreate(headerValue, UriKind.Absolute, out var referer)) // if relative, don't handle
{
var builder = new UriBuilder(referer);
builder.Host = TargetUrl.Host;
builder.Port = TargetUrl.Port;
headerValue = builder.ToString();
}
break;
case "Host":
host = headerValue;
if (RewriteHost)
{
headerValue = TargetUrl.Host + ":" + TargetUrl.Port;
}
break;
}
if (contentHeader)
{
msg.Content.Headers.Add(headerName, headerValue);
}
else
{
msg.Headers.Add(headerName, headerValue);
}
}
using (var response = await _client.SendAsync(msg).ConfigureAwait(false))
{
using (var os = context.Response.OutputStream)
{
context.Response.ProtocolVersion = response.Version;
context.Response.StatusCode = (int)response.StatusCode;
context.Response.StatusDescription = response.ReasonPhrase;
foreach (var header in response.Headers)
{
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
foreach (var header in response.Content.Headers)
{
if (header.Key == "Content-Length") // this will be set automatically at dispose time
continue;
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
var ct = context.Response.ContentType;
if (RewriteTargetInText && host != null && ct != null &&
(ct.IndexOf("text/html", StringComparison.OrdinalIgnoreCase) >= 0 ||
ct.IndexOf("application/json", StringComparison.OrdinalIgnoreCase) >= 0))
{
using (var ms = new MemoryStream())
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(ms).ConfigureAwait(false);
var enc = context.Response.ContentEncoding ?? Encoding.UTF8;
var html = enc.GetString(ms.ToArray());
if (TryReplace(html, "//" + _targetHost + ":" + _targetPort + "/", "//" + host + "/", out var replaced))
{
var bytes = enc.GetBytes(replaced);
using (var ms2 = new MemoryStream(bytes))
{
ms2.Position = 0;
await ms2.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
else
{
ms.Position = 0;
await ms.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
else
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
}
}
public void Stop() => _listener.Stop();
public override string ToString() => string.Join(", ", Prefixes) + " => " + TargetUrl;
public void Dispose() => ((IDisposable)_listener)?.Dispose();
// out-of-the-box replace doesn't tell if something *was* replaced or not
private static bool TryReplace(string input, string oldValue, string newValue, out string result)
{
if (string.IsNullOrEmpty(input) || string.IsNullOrEmpty(oldValue))
{
result = input;
return false;
}
var oldLen = oldValue.Length;
var sb = new StringBuilder(input.Length);
bool changed = false;
var offset = 0;
for (int i = 0; i < input.Length; i++)
{
var c = input[i];
if (offset > 0)
{
if (c == oldValue[offset])
{
offset++;
if (oldLen == offset)
{
changed = true;
sb.Append(newValue);
offset = 0;
}
continue;
}
for (int j = 0; j < offset; j++)
{
sb.Append(input[i - offset + j]);
}
sb.Append(c);
offset = 0;
}
else
{
if (c == oldValue[0])
{
if (oldLen == 1)
{
changed = true;
sb.Append(newValue);
}
else
{
offset = 1;
}
continue;
}
sb.Append(c);
}
}
if (changed)
{
result = sb.ToString();
return true;
}
result = input;
return false;
}
}

The browser is connected to the proxy so the data that the proxy gets from the web server is just sent via the same connection that the browser initiated to the proxy.

Related

How to change base url of Swagger in .core depend on the request

In the old version of Swagger (.Net framework) I've used this function to change the URL:
RootUrl(req => ComputeHostAsSeenByOriginalClient(req))
Actually I calculate the base URL from the request (my app is behind a load balancer)
The new .core way is to use this RoutePrefix
The problem:
RoutePrefix is Property and not an Action, so I don't have the HttpRequestMessage
This is the ComputeHostAsSeenByOriginalClient full code:
public static string ComputeHostAsSeenByOriginalClient(HttpRequestMessage req)
{
var authority = req.RequestUri.Authority;
var scheme = req.RequestUri.Scheme;
if (req.Headers.Contains("X-Forwarded-Host"))
{
//we are behind a reverse proxy, use the host that was used by the client
var xForwardedHost = req.Headers.GetValues("X-Forwarded-Host").First();
//when multiple apache httpd are chained, each proxy append to the header
//with a comma (see //https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#x-headers).
//so we need to take only the first host because it is the host that was
//requested by the original client.
//note that other reverse proxies may behave differently but
//we are not taking care of them...
var firstForwardedHost = xForwardedHost.Split(',')[0];
authority = firstForwardedHost;
}
if (req.Headers.Contains("X-Forwarded-Proto"))
{
//now that we have the host, we also need to determine the protocol used by the
//original client.
//if present, we are using the de facto standard header X-Forwarded-Proto
//otherwise, we fallback to http
//note that this is extremely brittle, either because the first proxy
//can "forget" to set the header or because another proxy can rewrite it...
var xForwardedProto = req.Headers.GetValues("X-Forwarded-Proto").First();
if (xForwardedProto.IndexOf(",") != -1)
{
//when multiple apache, X-Forwarded-Proto is also multiple ...
xForwardedProto = xForwardedProto.Split(',')[0];
}
scheme = xForwardedProto;
}
//no reverse proxy mean we can directly use the RequestUri
return scheme + "://" + authority;
}
Any idea how to workaround this?
Try to use this.Request. You could access the request object from Controller.
public string ComputeHostAsSeenByOriginalClient()
{
var req = this.Request;
var scheme = req.Scheme;
var authority = this.Request.Host.Value.ToString();
if (req.Headers.ContainsKey("X-Forwarded-Host"))
{
//we are behind a reverse proxy, use the host that was used by the client
StringValues xForwardedHost = "";
if (req.Headers.TryGetValue("X-Forwarded-Host", out xForwardedHost))
{
//when multiple apache httpd are chained, each proxy append to the header
//with a comma (see //https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#x-headers).
//so we need to take only the first host because it is the host that was
//requested by the original client.
//note that other reverse proxies may behave differently but
//we are not taking care of them...
var firstForwardedHost = xForwardedHost.First().ToString().Split(',')[0];
authority = firstForwardedHost;
}
}
if (req.Headers.ContainsKey("X-Forwarded-Proto"))
{
//now that we have the host, we also need to determine the protocol used by the
//original client.
//if present, we are using the de facto standard header X-Forwarded-Proto
//otherwise, we fallback to http
//note that this is extremely brittle, either because the first proxy
//can "forget" to set the header or because another proxy can rewrite it...
StringValues xForwardedProto = "";
if (req.Headers.TryGetValue("X-Forwarded-Proto", out xForwardedProto))
{
if (xForwardedProto.First().ToString().IndexOf(",") != -1)
{
//when multiple apache, X-Forwarded-Proto is also multiple ...
xForwardedProto = xForwardedProto.First().ToString().Split(',')[0];
}
scheme = xForwardedProto;
}
}
//no reverse proxy mean we can directly use the RequestUri
return scheme + "://" + authority;
}

ZeroMQ architecture for cache-like service

I made a service which accepts requests on a 0MQ router socket in NetMQ, and acts mostly like a cache for other apps, i.e. it can receive a "GET" request and send a response, or it can receive a "PUT" request which doesn't need a response.
I have two issues though:
I tried using a RequestSocket in client code, but it still fails if I don't wait for a response after a sent message (I guess this is how REQ/REP works in 0MQ). What is the appropriate 0MQ socket type if I only want to get a response for some message types?
Currently my client code creates a new RequestSocket instance whenever it needs to communicate with the service. Is there a thread-safe way in NetMQ where I could have a single socket that can be reused over the client code when I need to send a request and get a response, without the need to create a new connection each time?
The service code is currently basically:
var socket = new RouterSocket(endpoint);
socket.ReceiveReady += (sender, e) =>
{
var msg = e.Socket.ReceiveMultipartMessage();
var action = msg[3].ConvertToString();
switch (action)
{
case "GET":
{
var key = msg[4].ConvertToString();
var data = _cache.Get(key);
var response = new NetMQMessage();
response.Append(msg[0]); // msg id
response.Append(NetMQFrame.Empty);
response.Append(msg[4]);
response.Append(data);
e.Socket.SendMultipartMessage(response);
}
break;
case "SET":
{
var key = msg[4].ConvertToString();
var data = msg[5].Buffer;
_cache.Set(key, data);
// no response needed here, but needed for RequestSocket clients
}
break;
default: throw new NotImplementedException();
}
};

OAuth 2.0 Authorization for windows desktop application using HttpListener

I am writing a windows desktop application with External Authentication(Google, Facebook) in C#.
I'm using HttpListener to allow a user to get Barer token by External Authentication Service with ASP.NET Web API, but administrator privileges are required for that and I want run without admin mode.
My reference was Sample Desktop Application for Windows.
Is this the best practice for external authentication provider from C#? Or is there another way to do that?
This is my code to get Barer token by external provider:
public static async Task<string> RequestExternalAccessToken(string provider)
{
// Creates a redirect URI using an available port on the loopback address.
string redirectURI = string.Format("http://{0}:{1}/", IPAddress.Loopback, GetRandomUnusedPort());
// Creates an HttpListener to listen for requests on that redirect URI.
var http = new HttpListener();
http.Prefixes.Add(redirectURI);
http.Start();
// Creates the OAuth 2.0 authorization request.
string authorizationRequest = Properties.Settings.Default.Server
+ "/api/Account/ExternalLogin?provider="
+ provider
+ "&response_type=token&client_id=desktop"
+ "&redirect_uri="
+ redirectURI + "?";
// Opens request in the browser.
System.Diagnostics.Process.Start(authorizationRequest);
// Waits for the OAuth authorization response.
var context = await http.GetContextAsync();
// Sends an HTTP response to the browser.
var response = context.Response;
string responseString = string.Format("<html><head></head><body></body></html>");
var buffer = System.Text.Encoding.UTF8.GetBytes(responseString);
response.ContentLength64 = buffer.Length;
var responseOutput = response.OutputStream;
Task responseTask = responseOutput.WriteAsync(buffer, 0, buffer.Length).ContinueWith((task) =>
{
responseOutput.Close();
http.Stop();
Console.WriteLine("HTTP server stopped.");
});
// Checks for errors.
if (context.Request.QueryString.Get("access_token") == null)
{
throw new ApplicationException("Error connecting to server");
}
var externalToken = context.Request.QueryString.Get("access_token");
var path = "/api/Account/GetAccessToken";
var client = new RestClient(Properties.Settings.Default.Server + path);
RestRequest request = new RestRequest() { Method = Method.GET };
request.AddParameter("provider", provider);
request.AddParameter("AccessToken", externalToken);
request.AddHeader("Content-Type", "application/x-www-form-urlencoded");
var clientResponse = client.Execute(request);
if (clientResponse.StatusCode == HttpStatusCode.OK)
{
var responseObject = JsonConvert.DeserializeObject<dynamic>(clientResponse.Content);
return responseObject.access_token;
}
else
{
throw new ApplicationException("Error connecting to server", clientResponse.ErrorException);
}
}
I don't know about Facebook, but usually (I am experienced with Google OAuth2 and Azure AD as well as Azure AD B2C), the authentication provider allows you to use a custom URI scheme for the authentication callback, something like badcompany://auth
To acquire an authentication token I ended up implementing the following scheme (All code is presented without warranty and not to be copied thoughtlessly.)
1. Register an URI-handler when the app is started
You can register an URI-Handler by creating a key in the HKEY_CURRENT_USER/Software/Classes (hence no admin privileges needed) key in the Windows registry
The name of the key equals the URI prefix, badcompany in our case
The key contains an empty string value named URL Protocol
The key contains a subkey DefaultIcon for the icon (actually I do not know whether this is necessary), I used the path of the current executable
There is a subkey shell/open/command, whose default value determines the path of the command to execute when the URI is tried to be opened, **please note*, that the "%1" is necessary to pass the URI to the executable
this.EnsureKeyExists(Registry.CurrentUser, "Software/Classes/badcompany", "URL:BadCo Applications");
this.SetValue(Registry.CurrentUser, "Software/Classes/badcompany", "URL Protocol", string.Empty);
this.EnsureKeyExists(Registry.CurrentUser, "Software/Classes/badcompany/DefaultIcon", $"{location},1");
this.EnsureKeyExists(Registry.CurrentUser, "Software/Classes/badcompany/shell/open/command", $"\"{location}\" \"%1\"");
// ...
private void SetValue(RegistryKey rootKey, string keys, string valueName, string value)
{
var key = this.EnsureKeyExists(rootKey, keys);
key.SetValue(valueName, value);
}
private RegistryKey EnsureKeyExists(RegistryKey rootKey, string keys, string defaultValue = null)
{
if (rootKey == null)
{
throw new Exception("Root key is (null)");
}
var currentKey = rootKey;
foreach (var key in keys.Split('/'))
{
currentKey = currentKey.OpenSubKey(key, RegistryKeyPermissionCheck.ReadWriteSubTree)
?? currentKey.CreateSubKey(key, RegistryKeyPermissionCheck.ReadWriteSubTree);
if (currentKey == null)
{
throw new Exception("Could not get or create key");
}
}
if (defaultValue != null)
{
currentKey.SetValue(string.Empty, defaultValue);
}
return currentKey;
}
2. Open a pipe for IPC
Since you'll have to pass messages from one instance of your program to another, you'll have to open a named pipe that can be used for that purpose.
I called this code in a loop in a background Task
private async Task<string> ReceiveTextFromPipe(CancellationToken cancellationToken)
{
string receivedText;
PipeSecurity ps = new PipeSecurity();
System.Security.Principal.SecurityIdentifier sid = new System.Security.Principal.SecurityIdentifier(System.Security.Principal.WellKnownSidType.WorldSid, null);
PipeAccessRule par = new PipeAccessRule(sid, PipeAccessRights.ReadWrite, System.Security.AccessControl.AccessControlType.Allow);
ps.AddAccessRule(par);
using (var pipeStream = new NamedPipeServerStream(this._pipeName, PipeDirection.InOut, 1, PipeTransmissionMode.Message, PipeOptions.Asynchronous, 4096, 4096, ps))
{
await pipeStream.WaitForConnectionAsync(cancellationToken);
using (var streamReader = new StreamReader(pipeStream))
{
receivedText = await streamReader.ReadToEndAsync();
}
}
return receivedText;
}
3. Make sure that the application is started only once
This can be acquired using a Mutex.
internal class SingleInstanceChecker
{
private static Mutex Mutex { get; set; }
public static async Task EnsureIsSingleInstance(string id, Action onIsSingleInstance, Func<Task> onIsSecondaryInstance)
{
SingleInstanceChecker.Mutex = new Mutex(true, id, out var isOnlyInstance);
if (!isOnlyInstance)
{
await onIsSecondaryInstance();
Application.Current.Shutdown(0);
}
else
{
onIsSingleInstance();
}
}
}
When the mutex has been acquired by another instance, the application is not fully started, but
4. Handle being called with the authentication redirect URI
If it's the only (first) instance, it may handle the authentication redirect URI itself
Extract the token from the URI
Store the token (if necessary and/or wanted)
Use the token for requests
If it's a further instance
Pass the redirect URI to the first instance by using pipes
The first instance now performs the steps under 1.
Close the second instance
The URI is sent to the first instance with
using (var client = new NamedPipeClientStream(this._pipeName))
{
try
{
var millisecondsTimeout = 2000;
await client.ConnectAsync(millisecondsTimeout);
}
catch (Exception)
{
onSendFailed();
return;
}
if (!client.IsConnected)
{
onSendFailed();
}
using (StreamWriter writer = new StreamWriter(client))
{
writer.Write(stringToSend);
writer.Flush();
}
}
To add to Paul's excellent answer:
Identity Model Libraries are worth looking at - one of the things they'll do for you is Authorization Code Flow (PKCE) which is recommended for native apps
My preference is the same as Paul's - to use custom URI schemes - usability is better I think
Having said that, a loopback solution should work without admin rights for ports greater than 1024
If it helps there is some stuff on my blog about this - including a Nodejs / Electron sample you can run from here to see what a finished solution looks like.

HTTPS traffic not captured using FiddlerCore on Linux

Trying to capture all traffic from browser using FiddlerCore using the .NetStandard libraries.
Below is what I have:
public class ProxyConfig
{
private const string SecureEndpointHostname = "localhost";
private readonly int _secureEndpointPort = 18888;
private static readonly ICollection<Session> AllSessions = new List<Session>();
private static Fiddler.Proxy _secureEndpoint;
private static readonly LoggerCnx Logger = new LoggerCnx();
public void SetupProxyListener()
{
// This is a workaround for known issue in .NET Core - https://github.com/dotnet/coreclr/issues/12668
CultureInfo.DefaultThreadCurrentUICulture = new CultureInfo("en-US");
FiddlerApplication.BeforeRequest += session =>
{
// In order to enable response tampering, buffering mode MUST
// be enabled; this allows FiddlerCore to permit modification of
// the response in the BeforeResponse handler rather than streaming
// the response to the client as the response comes in.
session.bBufferResponse = false;
lock (AllSessions)
{
AllSessions.Add(session);
}
};
Logger.Info($"Starting {FiddlerApplication.GetVersionString()}...");
CONFIG.IgnoreServerCertErrors = true;
CONFIG.bCaptureCONNECT = true;
FiddlerApplication.Prefs.SetBoolPref("fiddler.network.streaming.abortifclientaborts", true);
FiddlerCoreStartupFlags startupFlags = FiddlerCoreStartupFlags.Default;
startupFlags = (startupFlags | FiddlerCoreStartupFlags.DecryptSSL);
FiddlerApplication.Startup(BaseConfiguration.ProxyPort, startupFlags);
Logger.Info("Created endpoint listening on port {0}", BaseConfiguration.ProxyPort);
Logger.Info("Starting with settings: [{0}]", startupFlags);
Logger.Info("Gateway: {0}", CONFIG.UpstreamGateway.ToString());
// Create a HTTPS listener, useful for when FiddlerCore is masquerading as a HTTPS server
// instead of acting as a normal CERN-style proxy server.
_secureEndpoint = FiddlerApplication.CreateProxyEndpoint(_secureEndpointPort, true, SecureEndpointHostname);
if (null != _secureEndpoint)
{
Logger.Info("Created secure endpoint listening on port {0}, using a HTTPS certificate for '{1}'", _secureEndpointPort, SecureEndpointHostname);
}
}
}
Everything seems to work fine on Windows, all traffic is captured, HTTP and HTTPS.
The problem arises when I'm trying to do the same when running the code on Linux (tried it on both VM and Linux conainer, Ubuntu 16.04). HTTP traffic is captured but HTTPS is not.
Any idea what I'm I'm missing?

How to create a simple proxy in C#?

I have downloaded Privoxy few weeks ago and for the fun I was curious to know how a simple version of it can be done.
I understand that I need to configure the browser (client) to send request to the proxy. The proxy send the request to the web (let say it's a http proxy). The proxy will receive the answer... but how can the proxy send back the request to the browser (client)?
I have search on the web for C# and http proxy but haven't found something that let me understand how it works behind the scene correctly. (I believe I do not want a reverse proxy but I am not sure).
Does any of you have some explication or some information that will let me continue this small project?
Update
This is what I understand (see graphic below).
Step 1 I configure the client (browser) for all request to be send to 127.0.0.1 at the port the Proxy listen. This way, request will be not sent to the Internet directly but will be processed by the proxy.
Step2 The proxy see a new connection, read the HTTP header and see the request he must executes. He executes the request.
Step3 The proxy receive an answer from the request. Now he must send the answer from the web to the client but how???
Useful link
Mentalis Proxy : I have found this project that is a proxy (but more that I would like). I might check the source but I really wanted something basic to understand more the concept.
ASP Proxy : I might be able to get some information over here too.
Request reflector : This is a simple example.
Here is a Git Hub Repository with a Simple Http Proxy.
I wouldn't use HttpListener or something like that, in that way you'll come across so many issues.
Most importantly it'll be a huge pain to support:
Proxy Keep-Alives
SSL won't work (in a correct way, you'll get popups)
.NET libraries strictly follows RFCs which causes some requests to fail (even though IE, FF and any other browser in the world will work.)
What you need to do is:
Listen a TCP port
Parse the browser request
Extract Host connect to that host in TCP level
Forward everything back and forth unless you want to add custom headers etc.
I wrote 2 different HTTP proxies in .NET with different requirements and I can tell you that this is the best way to do it.
Mentalis doing this, but their code is "delegate spaghetti", worse than GoTo :)
I have recently written a light weight proxy in c# .net using TcpListener and TcpClient.
https://github.com/titanium007/Titanium-Web-Proxy
It supports secure HTTP the correct way, client machine needs to trust root certificate used by the proxy. Also supports WebSockets relay. All features of HTTP 1.1 are supported except pipelining. Pipelining is not used by most modern browsers anyway. Also supports windows authentication (plain, digest).
You can hook up your application by referencing the project and then see and modify all traffic. (Request and response).
As far as performance, I have tested it on my machine and works without any noticeable delay.
You can build one with the HttpListener class to listen for incoming requests and the HttpWebRequest class to relay the requests.
Proxy can work in the following way.
Step1, configure client to use proxyHost:proxyPort.
Proxy is a TCP server that is listening on proxyHost:proxyPort.
Browser opens connection with Proxy and sends Http request.
Proxy parses this request and tries to detect "Host" header. This header will tell Proxy where to open connection.
Step 2: Proxy opens connection to the address specified in the "Host" header. Then it sends HTTP request to that remote server. Reads response.
Step 3: After response is read from remote HTTP server, Proxy sends the response through an earlier opened TCP connection with browser.
Schematically it will look like this:
Browser Proxy HTTP server
Open TCP connection
Send HTTP request ----------->
Read HTTP header
detect Host header
Send request to HTTP ----------->
Server
<-----------
Read response and send
<----------- it back to the browser
Render content
If you are just looking to intercept the traffic, you could use the fiddler core to create a proxy...
http://fiddler.wikidot.com/fiddlercore
run fiddler first with the UI to see what it does, it is a proxy that allows you to debug the http/https traffic. It is written in c# and has a core which you can build into your own applications.
Keep in mind FiddlerCore is not free for commercial applications.
Agree to dr evil
if you use HTTPListener you will have many problems, you have to parse requests and will be engaged to headers and ...
Use tcp listener to listen to browser requests
parse only the first line of the request and get the host domain and port to connect
send the exact raw request to the found host on the first line of browser request
receive the data from the target site(I have problem in this section)
send the exact data received from the host to the browser
you see you dont need to even know what is in the browser request and parse it, only get the target site address from the first line
first line usually likes this
GET http://google.com HTTP1.1
or
CONNECT facebook.com:443 (this is for ssl requests)
Things have become really easy with OWIN and WebAPI. In my search for a C# Proxy server, I also came across this post http://blog.kloud.com.au/2013/11/24/do-it-yourself-web-api-proxy/ . This will be the road I'm taking.
Socks4 is a very simple protocol to implement. You listen for the initial connection, connect to the host/port that was requested by the client, send the success code to the client then forward the outgoing and incoming streams across sockets.
If you go with HTTP you'll have to read and possibly set/remove some HTTP headers so that's a little more work.
If I remember correctly, SSL will work across HTTP and Socks proxies. For a HTTP proxy you implement the CONNECT verb, which works much like the socks4 as described above, then the client opens the SSL connection across the proxied tcp stream.
For what it's worth, here is a C# sample async implementation based on HttpListener and HttpClient (I use it to be able to connect Chrome in Android devices to IIS Express, that's the only way I found...).
And If you need HTTPS support, it shouldn't require more code, just certificate configuration: Httplistener with HTTPS support
// define http://localhost:5000 and http://127.0.0.1:5000/ to be proxies for http://localhost:53068
using (var server = new ProxyServer("http://localhost:53068", "http://localhost:5000/", "http://127.0.0.1:5000/"))
{
server.Start();
Console.WriteLine("Press ESC to stop server.");
while (true)
{
var key = Console.ReadKey(true);
if (key.Key == ConsoleKey.Escape)
break;
}
server.Stop();
}
....
public class ProxyServer : IDisposable
{
private readonly HttpListener _listener;
private readonly int _targetPort;
private readonly string _targetHost;
private static readonly HttpClient _client = new HttpClient();
public ProxyServer(string targetUrl, params string[] prefixes)
: this(new Uri(targetUrl), prefixes)
{
}
public ProxyServer(Uri targetUrl, params string[] prefixes)
{
if (targetUrl == null)
throw new ArgumentNullException(nameof(targetUrl));
if (prefixes == null)
throw new ArgumentNullException(nameof(prefixes));
if (prefixes.Length == 0)
throw new ArgumentException(null, nameof(prefixes));
RewriteTargetInText = true;
RewriteHost = true;
RewriteReferer = true;
TargetUrl = targetUrl;
_targetHost = targetUrl.Host;
_targetPort = targetUrl.Port;
Prefixes = prefixes;
_listener = new HttpListener();
foreach (var prefix in prefixes)
{
_listener.Prefixes.Add(prefix);
}
}
public Uri TargetUrl { get; }
public string[] Prefixes { get; }
public bool RewriteTargetInText { get; set; }
public bool RewriteHost { get; set; }
public bool RewriteReferer { get; set; } // this can have performance impact...
public void Start()
{
_listener.Start();
_listener.BeginGetContext(ProcessRequest, null);
}
private async void ProcessRequest(IAsyncResult result)
{
if (!_listener.IsListening)
return;
var ctx = _listener.EndGetContext(result);
_listener.BeginGetContext(ProcessRequest, null);
await ProcessRequest(ctx).ConfigureAwait(false);
}
protected virtual async Task ProcessRequest(HttpListenerContext context)
{
if (context == null)
throw new ArgumentNullException(nameof(context));
var url = TargetUrl.GetComponents(UriComponents.SchemeAndServer, UriFormat.Unescaped);
using (var msg = new HttpRequestMessage(new HttpMethod(context.Request.HttpMethod), url + context.Request.RawUrl))
{
msg.Version = context.Request.ProtocolVersion;
if (context.Request.HasEntityBody)
{
msg.Content = new StreamContent(context.Request.InputStream); // disposed with msg
}
string host = null;
foreach (string headerName in context.Request.Headers)
{
var headerValue = context.Request.Headers[headerName];
if (headerName == "Content-Length" && headerValue == "0") // useless plus don't send if we have no entity body
continue;
bool contentHeader = false;
switch (headerName)
{
// some headers go to content...
case "Allow":
case "Content-Disposition":
case "Content-Encoding":
case "Content-Language":
case "Content-Length":
case "Content-Location":
case "Content-MD5":
case "Content-Range":
case "Content-Type":
case "Expires":
case "Last-Modified":
contentHeader = true;
break;
case "Referer":
if (RewriteReferer && Uri.TryCreate(headerValue, UriKind.Absolute, out var referer)) // if relative, don't handle
{
var builder = new UriBuilder(referer);
builder.Host = TargetUrl.Host;
builder.Port = TargetUrl.Port;
headerValue = builder.ToString();
}
break;
case "Host":
host = headerValue;
if (RewriteHost)
{
headerValue = TargetUrl.Host + ":" + TargetUrl.Port;
}
break;
}
if (contentHeader)
{
msg.Content.Headers.Add(headerName, headerValue);
}
else
{
msg.Headers.Add(headerName, headerValue);
}
}
using (var response = await _client.SendAsync(msg).ConfigureAwait(false))
{
using (var os = context.Response.OutputStream)
{
context.Response.ProtocolVersion = response.Version;
context.Response.StatusCode = (int)response.StatusCode;
context.Response.StatusDescription = response.ReasonPhrase;
foreach (var header in response.Headers)
{
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
foreach (var header in response.Content.Headers)
{
if (header.Key == "Content-Length") // this will be set automatically at dispose time
continue;
context.Response.Headers.Add(header.Key, string.Join(", ", header.Value));
}
var ct = context.Response.ContentType;
if (RewriteTargetInText && host != null && ct != null &&
(ct.IndexOf("text/html", StringComparison.OrdinalIgnoreCase) >= 0 ||
ct.IndexOf("application/json", StringComparison.OrdinalIgnoreCase) >= 0))
{
using (var ms = new MemoryStream())
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(ms).ConfigureAwait(false);
var enc = context.Response.ContentEncoding ?? Encoding.UTF8;
var html = enc.GetString(ms.ToArray());
if (TryReplace(html, "//" + _targetHost + ":" + _targetPort + "/", "//" + host + "/", out var replaced))
{
var bytes = enc.GetBytes(replaced);
using (var ms2 = new MemoryStream(bytes))
{
ms2.Position = 0;
await ms2.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
else
{
ms.Position = 0;
await ms.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
else
{
using (var stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false))
{
await stream.CopyToAsync(context.Response.OutputStream).ConfigureAwait(false);
}
}
}
}
}
}
public void Stop() => _listener.Stop();
public override string ToString() => string.Join(", ", Prefixes) + " => " + TargetUrl;
public void Dispose() => ((IDisposable)_listener)?.Dispose();
// out-of-the-box replace doesn't tell if something *was* replaced or not
private static bool TryReplace(string input, string oldValue, string newValue, out string result)
{
if (string.IsNullOrEmpty(input) || string.IsNullOrEmpty(oldValue))
{
result = input;
return false;
}
var oldLen = oldValue.Length;
var sb = new StringBuilder(input.Length);
bool changed = false;
var offset = 0;
for (int i = 0; i < input.Length; i++)
{
var c = input[i];
if (offset > 0)
{
if (c == oldValue[offset])
{
offset++;
if (oldLen == offset)
{
changed = true;
sb.Append(newValue);
offset = 0;
}
continue;
}
for (int j = 0; j < offset; j++)
{
sb.Append(input[i - offset + j]);
}
sb.Append(c);
offset = 0;
}
else
{
if (c == oldValue[0])
{
if (oldLen == 1)
{
changed = true;
sb.Append(newValue);
}
else
{
offset = 1;
}
continue;
}
sb.Append(c);
}
}
if (changed)
{
result = sb.ToString();
return true;
}
result = input;
return false;
}
}
The browser is connected to the proxy so the data that the proxy gets from the web server is just sent via the same connection that the browser initiated to the proxy.

Categories