We Keep Coding

.NET HttpClient do not persist authentication between reqeusts to IIS when using NTLM Negotiate

IIS site configured to use windows authentication with default options. The client is written in C# and uses single HttpClient instance to perform requests. Requests success, but every request triggers 401 Challenge:
Traffic captured with Wireshark. We performed loud test, and noticed, that with anonymous authentication client performs 5000 reqeusts per second, but with windows authentication - 800. So, looks like wireshark does not impact to authentication, performance slowdown indicates, that 401 Challenge also occurs without wireshark.
Wirehshark log: https://drive.google.com/file/d/1vDNZMjiKPDisFLq6ZDhASQZJJKuN2cpj/view?usp=sharing
Code of client is here:
var httpClientHandler = new HttpClientHandler();
httpClientHandler.UseDefaultCredentials = true;
var httpClient = new HttpClient(httpClientHandler);
while (working)
{
var response = await httpClient.GetAsync(textBoxAddress.Text + "/api/v1/cards/" + cardId);
var content = await response.Content.ReadAsStringAsync();
}
IIS site settings:
How to make HttpClient to persist authentication between requests, to prevent waste negotiate handshakes on every request?
UPD: Code of client: https://github.com/PFight/httpclientauthtest
Steps to reproduce:
Create folder with simple file index.html, create application 'testsite' in IIS for this folder. Enable anonymous authentication.
Run client (https://github.com/PFight/httpclientauthtest/blob/main/TestDv5/bin/Debug/TestDv5.exe), press start button - see count of requests per second. Press stop.
Disable anonymous atuhentication, enable windows authentication.
Press start button in client, see count of reqeusts per second.
On my computer I see ~1000 requests per second on anonymous, and ~180 on windows. Wireshark shows 401 challenges on every request for windows authentication. keep-alive header enabled in IIS.
IIS version: 10.0.18362.1 (windows 10)
Version of System.Net.Http.dll loaded to process: 4.8.3752.0

Firstly I tried to save the Authorization header for re-use it with every new request.
using System.Net.Http.Headers;
Requester requester = new Requester();
await requester.MakeRequest("http://localhost/test.txt");
await Task.Delay(100);
await requester.MakeRequest("http://localhost/test.txt");
class Requester
{
private readonly HttpClientHandler _httpClientHandler;
private readonly HttpClient _httpClient;
private AuthenticationHeaderValue _auth = null;
public Requester()
{
_httpClientHandler = new HttpClientHandler();
_httpClientHandler.UseDefaultCredentials = true;
_httpClient = new HttpClient(_httpClientHandler);
_httpClient.DefaultRequestHeaders.Add("User-Agent", Guid.NewGuid().ToString("D"));
}
public async Task<string> MakeRequest(string url)
{
HttpRequestMessage message = new HttpRequestMessage(HttpMethod.Get, url);
message.Headers.Authorization = _auth;
HttpResponseMessage resp = await _httpClient.SendAsync(message);
_auth = resp.RequestMessage?.Headers?.Authorization;
resp.EnsureSuccessStatusCode();
string responseText = await resp.Content.ReadAsStringAsync();
return responseText;
}
}
But it didn't work. Every time there was http code 401 asking for authentication despite of Authorization header.
The IIS logs is listed below.
2021-12-23 15:07:47 ::1 GET /test.txt - 80 - ::1 c75eeab7-a0ea-4ebd-91a8-21f5cd59c10f - 401 2 5 127
2021-12-23 15:07:47 ::1 GET /test.txt - 80 MicrosoftAccount\account#domain.com ::1 c75eeab7-a0ea-4ebd-91a8-21f5cd59c10f - 200 0 0 4
2021-12-23 15:07:47 ::1 GET /test.txt - 80 - ::1 c75eeab7-a0ea-4ebd-91a8-21f5cd59c10f - 401 1 2148074248 0
2021-12-23 15:07:47 ::1 GET /test.txt - 80 MicrosoftAccount\account#domain.com ::1 c75eeab7-a0ea-4ebd-91a8-21f5cd59c10f - 200 0 0 0
IIS's Failed Requests Tracing reports the following when receiving re-used Authentication Header:
Property
Value
ModuleName
WindowsAuthenticationModule
Notification
AUTHENTICATE_REQUEST
HttpStatus
401
HttpReason
Unauthorized
HttpSubStatus
1
ErrorCode
The token supplied to the function is invalid (0x80090308)
I've made a research and I can say that this is not possible without alive connection.
Every time the connection is closed there will be new handshake.
According this and this answers NTLM authenticates a connection, so you need to keep your connection open.
NTLM over http is using HTTP persistent connection or http keep-alive.
A single connection is created and then kept open for the rest of the session.
If using the same authenticated connection, it is not necessary to send the authentication headers anymore.
This is also the reason why NTLM doesn't work with certain proxy servers that don't support keep-alive connections.
UPDATE:
I found the key point using your example.
First: You must enable keep-alive at your IIS
Second: You must set authPersistSingleRequest flag to false. Setting this flag to True specifies that authentication persists only for a single request on a connection. IIS resets the authentication at the end of each request, and forces re-authentication on the next request of the session. The default value is False.
Third: You can force the HttpClient to send keep-alive headers:
httpClient.DefaultRequestHeaders.Add("Connection", "keep-alive");
httpClient.DefaultRequestHeaders.Add("Keep-Alive", "600");
Using this three key points I have achieved only one NTLM handshake during connection lifetime.
Also it's important which version of .NET \ .NET Framework do you use.
Because HttpClient hides different realizations dependent on framework version.
Framework
Realization of HttpClient
.Net Framework
Wrapper around WebRequest
.Net Core < 2.1
Native handlers (WinHttpHandler / CurlHandler)
.Net Core >= 2.1
SocketsHttpHandler
I tried it on .NET 6 and it works great, but it didn't work on .Net Framework as I can see, so here is the question: which platform do you use?
UPDATE 2:
Found the solution for .Net Framework.
CredentialCache myCache = new CredentialCache();
WebRequestHandler handler = new WebRequestHandler()
{
UseDefaultCredentials = true,
AllowAutoRedirect = true,
UnsafeAuthenticatedConnectionSharing = true,
Credentials = myCache,
};
var httpClient = new HttpClient(handler);
httpClient.DefaultRequestHeaders.Add("Connection", "keep-alive");
httpClient.DefaultRequestHeaders.Add("Keep-Alive", "600");
var from = DateTime.Now;
var countPerSecond = 0;
working = true;
while (working)
{
var response = await httpClient.GetAsync(textBoxAddress.Text);
var content = await response.Content.ReadAsStringAsync();
countPerSecond++;
if ((DateTime.Now - from).TotalSeconds >= 1)
{
this.labelRPS.Text = countPerSecond.ToString();
countPerSecond = 0;
from = DateTime.Now;
}
Application.DoEvents();
}
The key point is to use WebRequestHandler with UnsafeAuthenticatedConnectionSharing enabled option and use credential cache.
If this property is set to true, the connection used to retrieve the response remains open after the authentication has been performed. In this case, other requests that have this property set to true may use the connection without re-authenticating. In other words, if a connection has been authenticated for user A, user B may reuse A's connection; user B's request is fulfilled based on the credentials of user A.
Caution
Because it is possible for an application to use the connection without being authenticated, you need to be sure that there is no administrative vulnerability in your system when setting this property to true. If your application sends requests for multiple users (impersonates multiple user accounts) and relies on authentication to protect resources, do not set this property to true unless you use connection groups as described below.
Big thanks to this article for solution.

How to correctly set up AspNet Core 2 authentication behind a load balancer?

I've set up AspNet Core 2 authentication successfully, but now would like to get it working behind a load balancer.
Because the load balancer address is different from my app address I'm changing the redirect Uri in my startup.cs ConfigureServices like this...
options.Events.OnRedirectToIdentityProvider = async n =>
{
n.ProtocolMessage.RedirectUri = "https://frontfacingaddress.com";
await Task.FromResult(0);
};
This works fine and I successfully authenticate and the callback from the identity server calls https://frontfacingaddress.com/signin-oidc. That is correctly handled and handling OnTokenResponseReceived shows that I successfully recieve the token.
The problem is: it is then making another call to the identity server but this time to the app's actual (not load balancing) address. When that comes back it gives an error of: AspNetCore.Correlation.OpenIdConnect cookie not found.
So the Fiddler trace looks like this:
302 HTTPS frontfacingaddress.com /account/signin
200 HTTPS identity.serveraddress.com /connect/authorize/callback etc...
302 HTTPS frontfacingaddress.com /signin-oidc
-- this is where I successfully receive the code, but then:
302 HTTPS actualwebaddress.com /account/signin
200 HTTPS identity.serveraddress.com /connect/authorize/callback etc...
400 HTTPS frontfacingaddress.com /signin-oidc
-- this is the 400 cookie not found error
Why, after successfully authenticating, is it then firing again from the actual address and failing?

The solution was to modify the ReturnUri to use the front-facing address when the ticket was received:
options.Events.OnTicketReceived = async context =>
{
var host = context.HttpContext.Request.Host.Host;
var forwardedHost = context.HttpContext.Request.Headers["X-Forwarded-Host"].ToString();
context.ReturnUri = context.ReturnUri.Replace(host, forwardedHost);
await Task.FromResult(0);
};

WebPush {"Received unexpected response code"}

How to send notification using Webpush library. I was trying but its throws error msg like {"Received unexpected response code"}
****Now i have created web API to send notification & calling you through fiddler,but did n't get exception it's stuck somewhere
here is my code sample****
public void Sendnotification()
{
try
{
WebPush.VapidDetails vapidKeys = apidHelper.GenerateVapidKeys();
string subject =#"mailto:xyz.com";
string publicKey = Convert.ToString(vapidKeys.PublicKey);
string privateKey = Convert.ToString(vapidKeys.PrivateKey);
var subscription = new PushSubscription(pushEndpoint, p256dh, auth);
var vapidDetails = new VapidDetails(subject, publicKey, privateKey);
client.SendNotification(subscription, "payload", vapidDetails);
}
catch (WebPushException e)
{
}
}
I have configured Https enabled to call api using fidder. Please have look. also its throws error, it stuck somewhere
now it got the error please have look it's showing error HTTP/1.1 410 NotRegistered
See the full screen of Fiddler response error details

If you are getting the error 410 (to check the error use fiddler to intercept the https call), probably what you have is an error in the subscription data of the user probably the keys stored in your database doesn't match the subscription in the browser an easy fix could be to re-subscribe and re-save the subscription data and try again.
to setup fiddler, you have to use it as a proxy visual studio to intercept the https calls and also you have to enable https decryption.
EDIT
you can set up fiddler just by adding this configuration in your web.config or app.config:
<system.net>
<defaultProxy
enabled = "true"
useDefaultCredentials = "true">
<proxy autoDetect="false" bypassonlocal="false" proxyaddress="http://127.0.0.1:8888" usesystemdefault="false" />
</defaultProxy>
</system.net>
if in any case, you get unauthorized registration check this questions:
Web Push API Chrome, returning "Unauthorized Registration"
WebPushError and UnauthorizedRegistration when try to send push notification to Chrome and Opera, FF is OK

Logging in to HTTPS interface of Serv-U server

I am trying to connect to a corporate ftp server via code (C#/VB). It has been set up to run in a browser using SolarWinds Serv-U software so that users can access it via a browser. The address is in the form:
https://ftp.example.com
From here they are presented with a login form (part of Serv-U) in which they enter their u/p and log in.
I have been trying to use HttpWebRequest class to log in, but each time I get an '401 Unauthorised - not logged in' error. In the web request I set the credentials:
Dim loginUri = New Uri("https://ftp.example.com")
Dim loginRequest As HttpWebRequest = DirectCast(WebRequest.Create(loginUri), HttpWebRequest)
With loginRequest
.Accept = "*/*"
.ContentType = "application/x-www-form-urlencoded"
.CookieContainer = New CookieContainer()
.Credentials = New NetworkCredential("user", "pass")
.Method = WebRequestMethods.Http.Get
End With
Dim loginResponse As HttpWebResponse = loginRequest.GetResponse()
I'm not even sure if this approach is possible; there are quite a number of cookies set by the browser during the login process which is not a desirable thing to replicate in code.
I've done a fair bit of searching on the subject and haven't found any definitive answers. Should I just push back on the sysadmin to set up a proper ftp server over SSL? It is a requirement that we use :443 as many firewalls block 21 (& 22).
Thanks - Z

HttpClient / WebClient - Trouble Interacting With Websites via Proxy

I'd like to use an HttpClient to interact with a website. This is in a corporate-type environment, all web access goes through a web proxy. By default, both the HttpClient and WebClient seem to 'just work' with the proxy here - but I've also specified the proxy details in code.
My problem is that some URLs will load correctly, others will not. The following code shows what I mean.
var webc = new WebClient();
var x1 = webc.DownloadString("http://www.google.com"); // Works
var x2 = webc.DownloadString("http://www.google.ie"); // Works
var x3 = webc.DownloadString("http://maps.google.com"); // Works
var x4 = webc.DownloadString("http://maps.google.ie"); // 403 Forbidden exception
I see the same behaviour with the HttpClient, but the code is more verbose. If I fetch the HTML returned in the 403 error it indicates that I have not authenticated and shows my username as empty.
Using Chrome/FF/IE - I can browse to all four of the sample URLs. The proxy doesn't prevent me, or show the same error message.
It seems like the code only fails on sites that have a non 'www' subdomain - when it is a non .com site. As crazy as that sounds.
I've tried running Fiddler locally to see if anything was different between the requests - from what I can see - it looks identical, except for the URL:
GET http://maps.google.ie/ HTTP/1.1 Host: maps.google.ie
Proxy-Connection: Keep-Alive
GET http://www.google.com/ HTTP/1.1 Host: www.google.com
Proxy-Connection: Keep-Alive
In the 'Auth' tab fiddler shows:
No Proxy-Authorization Header is present.
No Authorization Header is present.
For both. But the .com example works; and the .ie example fails. I tried pulling up the same maps.google.ie URL from within Chrome - which works great and I can see that it has a Proxy-Authorization in it's GET
GET http://maps.google.ie/ HTTP/1.1 Host: maps.google.ie
Proxy-Connection: keep-alive Proxy-Authorization: NTLM
T3RMTVNTUAAFAAACB4IBogQABAAzAAAACwALACgAAAAGAbFdAAAAD1BBVUxTT01xOTlEU1UTUR==
Can anyone tell me what's going on here? If that Proxy-Authorization is what I need, how do I get the HttpClient/WebClient to include it? I've tried creating a WebProxy and setting the Credentials on it - with the CredentialCache and with supplying the username/pass/domain (and every variation of the domain name I could think of). When I get it 'wrong' - all the sites seem to return 403. But when I get it right - the top 3 work and the 4th doesn't. In Fiddler, I'm never able to see that Proxy-Authorization in any of the requests I make - but it still works for the 3 first three sites.
I'm sure I've missed something, but I'm at a loss. Any help would be much appreciated.

There are two ways:
var webc = new WebClient();
webc.UseDefaultCredentials = true;
var x4 = webc.DownloadString("http://maps.google.ie");
or, put this in your app.config
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<system.net>
<defaultProxy useDefaultCredentials="true" />
</system.net>
</configuration>
See tomfanning's answer here: Proxy Basic Authentication in C#: HTTP 407 error
I don't understand why "UseDefaultCredentials" does not default to true. If you work in a corporation that uses a proxy, any app that doesn't do this cannot get out of the LAN.

Moby Disk and Aron are both correct, in the sense that those are ways of specifying the proxy. But as mentioned in my question, using them didn't help.
For whatever reason, the web proxy required a User-Agent to be set. Once set, everything worked.
_client.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0");

var webc = new WebClient
{
Proxy = new WebProxy
{
Credentials = new NetworkCredential(...),
}
};
var x1 = webc.DownloadString("http://www.google.com"); // Works
var x2 = webc.DownloadString("http://www.google.ie"); // Works
var x3 = webc.DownloadString("http://maps.google.com"); // Works
var x4 = webc.DownloadString("http://maps.google.ie"); // 403 Forbidden exception
Unfortunately .net is really annoying for programmatically setting the Proxy credentials. You expect you should be able to do this all in config, but it doesn't work out of the box. You can only set the Proxy address in config and not the credentials.