I'm trying to write a basic rotating proxy implementation for HttpClient via the IWebProxy interface, problem is when changing the IWebProxy.Credentials property inside GetProxy the request doesn't actually use the credentials set and therefore the proxy returns error 407:
namespace ProxyTestMin
{
public record WebProxyEntry(Uri Uri, ICredentials Credentials);
public class MyWebProxy : IWebProxy
{
private List<WebProxyEntry> _proxies;
private int _position = 0;
public MyWebProxy(List<WebProxyEntry> proxies) => _proxies = proxies;
public ICredentials? Credentials { get; set; }
public bool IsBypassed(Uri host) => false;
public Uri? GetProxy(Uri destination)
{
_position++;
_position %= _proxies.Count;
Credentials = _proxies[_position].Credentials;
return _proxies[_position].Uri;
}
}
public class Program
{
static List<WebProxyEntry> proxies = new()
{
new(new("http://XX.XX.XX.XX:3128"), new NetworkCredential("user1", "pass1")),
new(new("http://YY.YY.YY.YY:3128"), new NetworkCredential("user2", "pass2")),
};
static async Task Main(string[] args)
{
var httpClientHandler = new HttpClientHandler() { Proxy = new MyWebProxy(proxies) };
using var httpClient = new HttpClient(httpClientHandler, true);
for (int i = 0; i < proxies.Count; i++)
{
var response = await httpClient.GetAsync("https://api.ipify.org");
Console.WriteLine($"{i + 1}:" + await response.Content.ReadAsStringAsync());
}
}
}
}
If I disable authentication on the proxy it works OK, and if I set the user/pass to be the same on all proxies and just set Credentials in the MyWebProxy constructor it also works fine. Unfortunately though I need to be able to have different credentials for each proxy, and it seems that if Credentials is set anywhere other than the constructor the GetAsync call uses the wrong credentials for the proxy.
Does anyone know if there's a way to workaround this without creating multiple HttpClientHandler/HttpClient instances?
Related
I have Asp.Net Core WebApi. I am making Http requests according to HttpClientFactory pattern. Here is my sample code:
public void ConfigureServices(IServiceCollection services)
{
...
services.AddHttpClient<IMyInterface, MyService>();
...
}
public class MyService: IMyInterface
{
private readonly HttpClient _client;
public MyService(HttpClient client)
{
_client = client;
}
public async Task CallHttpEndpoint()
{
var request = new HttpRequestMessage(HttpMethod.Get, "www.customUrl.com");
var response = await _client.SendAsync(request);
...
}
}
I want to implement sending requests through dynamic proxy. This basically means that I might need to change proxy with each request. As for now I find out 2 approuces, non of which seems good to me:
1.Have a static proxy like this:
public void ConfigureServices(IServiceCollection services)
{
...
services.AddHttpClient<IMyInterface, MyService>().ConfigurePrimaryHttpMessageHandler(() =>
{
return new HttpClientHandler
{
Proxy = new WebProxy("http://127.0.0.1:8888"),
UseProxy = true
};
});
...
}
But I can only have single proxy per service in this approach.
2.Dispose HttpClient with each request:
HttpClientHandler handler = new HttpClientHandler()
{
Proxy = new WebProxy("http://127.0.0.1:8888"),
UseProxy = true,
};
using(var client = new HttpClient(handler))
{
var request = new HttpRequestMessage(HttpMethod.Get, "www.customUrl.com");
var response = await client.SendAsync(request);
...
}
But in this way I violate HttpClientFactory pattern and it might cause issues to application performance as stated in following article
Is there a third way where I could change proxy dinamically without re-creating HttpClient?
There is no way to change the any of the properties of HttpClientHandler or to assign a new version of HttpClientHandler to an existing HttpClient after it is instantiated. As such, it is then impossible to have a dynamic proxy for a particular HttpClient: you can only specify one proxy.
The correct way to achieve this is to use named clients, instead, and define a client for each proxy endpoint. Then, you'll need to inject IHttpClientFactory and pick one of the proxies to use, requesting the named client that implements that.
services.AddHttpClient("MyServiceProxy1").ConfigurePrimaryHttpMessageHandler(() =>
{
return new HttpClientHandler
{
Proxy = new WebProxy("http://127.0.0.1:8888"),
UseProxy = true
};
});
services.AddHttpClient("MyServiceProxy2").ConfigurePrimaryHttpMessageHandler(() =>
{
return new HttpClientHandler
{
Proxy = new WebProxy("http://127.0.0.1:8889"),
UseProxy = true
};
});
...
Then:
public class MyService : IMyInterface
{
private readonly HttpClient _client;
public MyService(IHttpClientFactory httpClientFactory)
{
_client = httpClientFactory.CreateClient("MyServiceProxy1");
}
public async Task CallHttpEndpoint()
{
var request = new HttpRequestMessage(HttpMethod.Get, "www.customUrl.com");
var response = await _client.SendAsync(request);
...
}
}
I can do that by inheriting from HttpClientHandler:
public class ProxyHttpHandler : HttpClientHandler
{
private int currentProxyIndex = 0;
private ProxyOptions proxyOptions;
public ProxyHttpHandler(IOptions<ProxyOptions> options)
{
proxyOptions = options != null ? options.Value : throw new ArgumentNullException(nameof(options));
UseProxy = true;
}
protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
var proxy = proxyOptions.Proxies[currentProxyIndex];
var proxyResolver = new WebProxy(proxy.Host, proxy.Port)
{
Credentials = proxy.Credentials
};
Proxy = proxyResolver;
currentProxyIndex++;
if(currentProxyIndex >= proxyOptions.Proxies.Count)
currentProxyIndex = 0;
return base.SendAsync(request, cancellationToken);
}
}
Then I register my ProxyHttpHandler and ProxyOptions in IoC:
public IForksCoreConfigurationBuilder ConfigureProxy(Action<ProxyOptions> options)
{
Services.AddOptions<ProxyOptions>().Configure(options);
Services.AddTransient<ProxyHttpHandler>();
Services.AddHttpClient<IService, MyService>()
.ConfigurePrimaryHttpMessageHandler<ProxyHttpHandler>();
return this;
}
I create an interface called INetClient which supports two clients: HttpClient and WebClient ( I want that my app to be unit-testable)
The interface looks like:
public interface INetClient
{
...
void SetCredentials(ICredentials credentials);
void SetHeaders(Dictionary<HttpRequestHeader, string> headers);
void SetHeaders(Dictionary<string, string> headers);
...
}
For each of them, I created a wrapper class which implements that interface.
For web client, I did like:
public sealed class WebClientEx : WebClient, INetClient
{
public WebClientEx()
{
CookieContainer = new CookieContainer();
Encoding = Encoding.UTF8;
}
public void SetCredentials(ICredentials credentials)
{
Credentials = credentials;
}
}
and for http client:
public sealed class HttpClientEx : INetClient, IDisposable
{
private HttpClient _client;
public HttpClientEx()
{
var handler = new HttpClientHandler();
_client = new HttpClient(handler);
}
public void SetCredentials(ICredentials credentials)
{
// here, how I can set credentials ?
}
}
Later on I will use dependency injection like RegisterType<INetClient, WebClientEx> or I can change easily to RegisterType<INetClient, HttpClientEx>
How can set credentials for http client without creating new instance for it ?
If your web service is using HTTP Basic authentication, you could set the header using the HttpClient.DefaultRequestHeaders property
I have a simple post request using the Flurl client, and I was wondering how to make this request using a proxy using information like the IP, port, username, and password.
string result = await atc.Request(url)
.WithHeader("Accept", "application/json")
.WithHeader("Content-Type", "application/x-www-form-urlencoded")
.WithHeader("Host", "www.website.com")
.WithHeader("Origin", "http://www.website.com")
.PostUrlEncodedAsync(new { st = colorID, s = sizeID, qty = 1 })
.ReceiveString();
I was looking for a similar answer and found this:
https://github.com/tmenier/Flurl/issues/228
Here is a copy of the contents of that link. It worked for me!
You can do this with a custom factory:
using Flurl.Http.Configuration;
public class ProxyHttpClientFactory : DefaultHttpClientFactory {
private string _address;
public ProxyHttpClientFactory(string address) {
_address = address;
}
public override HttpMessageHandler CreateMessageHandler() {
return new HttpClientHandler {
Proxy = new WebProxy(_address),
UseProxy = true
};
}
}
To register it globally on startup:
FlurlHttp.Configure(settings => {
settings.HttpClientFactory = new ProxyHttpClientFactory("http://myproxyserver");
});
We have a singleton service fabric service that needs to communicate to a partitioned service, both of which are running in a secure cluster with certificate-based authentication. We are using ServicePartitionClient to do the talking. What follows is a simplified version of our implementations of ICommunicationClient and ICommunicationClientFactory as required by the ServicePartitionClient.
The client:
public class HttpCommunicationClient : ICommunicationClient
{
// Lots of fields omitted for simplicity.
public HttpCommunicationClient(HttpClientWrapper client, Uri baseAddress)
{
this.HttpClient = client;
this.BaseAddress = baseAddress;
}
public Uri BaseAddress { get; set; }
// Wraps System.Net.Http.HttpClient to do stuff like add default headers
public HttpClientWrapper HttpClient { get; }
public ResolvedServicePartition ResolvedServicePartition { get; set; }
public string ListenerName { get; set; }
public ResolvedServiceEndpoint Endpoint { get; set; }
public Uri GetUri(string relativeUri)
{
return new Uri(this.BaseAddress, relativeUri);
}
}
The factory:
// Note that this base class is under the
// Microsoft.ServiceFabric.Services.Communication.Client namespace,
// it's not something we wrote
public class HttpCommunicationClientFactory :
CommunicationClientFactoryBase<HttpCommunicationClient>
{
// Lots of fields omitted for simplicity.
private readonly HttpClientWrapper client;
public HttpCommunicationClientFactory(
ServicePartitionResolver resolver,
IEnumerable<IExceptionHandler> exceptionHandlers)
: base(resolver, exceptionHandlers, null)
{
// There's a bunch of other args that are omitted for clarity.
this.client = new HttpClientWrapper();
}
protected override Task<HttpCommunicationClient> CreateClientAsync(
string endpoint,
CancellationToken cancellationToken)
{
HttpCommunicationClient client = new HttpCommunicationClient(
this.client,
new Uri(endpoint));
if (this.ValidateClient(endpoint, client))
{
return Task.FromResult(client);
}
else
{
throw new ArgumentException();
}
}
protected override bool ValidateClient(HttpCommunicationClient client)
{
// Not much to validate on httpclient
return true;
}
protected override bool ValidateClient(
string endpoint,
HttpCommunicationClient client)
{
return !string.IsNullOrWhiteSpace(endpoint);
}
protected override void AbortClient(HttpCommunicationClient client)
{
}
}
And here's an example of how everything is being constructed and used:
internal class FooFabricClient : IDisposable
{
// Lots of fields omitted for simplicity.
private readonly HttpCommunicationClientFactory clientFactory;
private readonly ServicePartitionClient<HttpCommunicationClient> partitionClient;
public FooFabricClient(Uri fabricUri, ServicePartitionKey partitionKey = null)
{
this.clientFactory = new HttpCommunicationClientFactory(
ServicePartitionResolver.GetDefault());
this.partitionClient = new ServicePartitionClient<HttpCommunicationClient>(
this.clientFactory,
fabricUri,
partitionKey,
retrySettings: new OperationRetrySettings());
}
public async Task<Foo> GetFooAsync()
{
return await this.partitionClient.InvokeWithRetryAsync(async (client) =>
{
// Note: See above for actual GetUri() implementation and context,
// but this will give us a Uri composed of the client's BaseAddress
// (passed in during HttpCommunicationClientFactory.CreateClientAsync())
// followed by "/foo"
Uri requestUri = client.GetUri("foo");
return await client.HttpClient.GetAsync<Foo>(requestUri);
});
}
Now, the issue is that when I call GetFooAsync(), it throws an exception saying this:
Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
After some debugging, we found that this is most likely due to the fact that we get the internal service fabric IP address (e.g., 10.0.0.4) as HttpCommunicationClient.BaseAddress, so when we make our API call the server cert doesn't validate against the domain in the request. As a temporary fix we've done the following in the calling service:
ServicePointManager.ServerCertificateValidationCallback += (a, b, c, d) => true;
Of course, we'd rather not just blindly say "yup, looks good" when validating server certs, so how should we go about resolving this issue? Is there another way to set up the client(s) so that they can properly validate the server cert on the request without needing our own callback? Or do we just need to put acceptable thumbprints in our config and compare against those in the ServicePointManager callback or whatever?
I am using this test method (and helper class) to verify the response from an external web service:
[TestMethod]
public void WebServiceReturnsSuccessResponse()
{
using (var provider = new Provider(new Info()))
using (var result = provider.GetHttpResponseMessage())
{
Assert.IsTrue(result.IsSuccessStatusCode);
}
}
private class Info : IInfo
{
public string URL { get; set; } =
"https://notreallythe.website.com:99/service/";
public string User { get; set; } = "somename";
public string Password { get; set; } = "password1";
}
I can't get this test to pass; I always get a 500 - Internal Server Error result. I have connected via an external utility (Postman) - so the web service is up and I can connect with the url & credentials that I have.
I think the problem is in my instantiation of the HttpClient class, but I can't determine where. I am using Basic authentication:
public class Provider : IProvider, IDisposable
{
private readonly HttpClient _httpClient;
public Provider(IInfo config){
if (config == null)
throw new ArgumentNullException(nameof(config));
var userInfo = new UTF8Encoding().GetBytes($"{config.User}:{config.Password}");
_httpClient = new HttpClient
{
BaseAddress = new Uri(config.URL),
DefaultRequestHeaders =
{
Accept = { new MediaTypeWithQualityHeaderValue("application/xml")},
Authorization = new AuthenticationHeaderValue(
"Basic", Convert.ToBase64String(userInfo)),
ExpectContinue = false,
},
};
}
public HttpResponseMessage GetHttpResponseMessage()
{
return _httpClient.GetAsync("1234").Result;
}
}
The response I get back appears to go to the correct endpoint; the RequestUri in the response looks exactly like I expect, https://notreallythe.website.com:99/service/1234.
You need to load up Fiddler and do a recording of the HTTP traffic when this operation succeeds (through the browser).
Then, load up your code, stand up another instance (or window) of Fiddler, and do the same thing with your code. Now, compare the two Fiddler windows to see what is different.
You only need to compare those things in Fiddler that are highlighted in blue. You can ignore the other communications.