We Keep Coding

DefaultAzureCredetials can't authenticate via Visual Studio - Can't find AzureServiceAuth\tokenProvider.json

I'm attempting to connect to an Azure Key Vault instance from a .NET 4.7 application running locally under IIS and the debugger (Visual Studio 2022 17.4.4) but am encountering the below exception(s) from the Azure.Identity package when it attempts to retrieve a token to authenticate to Azure when calling to perform a KeyVault action such as GetSecretAsync().
DefaultAzureCredential failed to retrieve a token from the included
credentials. See the troubleshooting guide for more information.
https://aka.ms/azsdk/net/identity/defaultazurecredential/troubleshoot
ManagedIdentityCredential authentication unavailable. Multiple attempts failed to obtain a token from the managed identity endpoint.
Visual Studio Token provider can't be accessed at C:\WINDOWS\system32\config\systemprofile\AppData\Local.IdentityService\AzureServiceAuth\tokenprovider.json
I need to connect to the KeyVault instance via a User Assigned Managed Identity in cloud environments such as production, whereas in development environments, we therefore need to connect via the developer's Visual Studio account to authenticate them to access the service, similarly. Perhaps I have misunderstood, but I believed this is possible via the DefaultAzureCredential option, which will try various methods of authentication in order (such as environment variables, managed identities, then Visual Studio credentials, etc) until one succeeds.
When inspecting the inner exception(s) relating to the Visual Studio Credentials flow, I see the System.Exception {System.IO.DirectoryNotFoundException} exception message states...
"Could not find a part of the path 'C:\WINDOWS\system32\config\systemprofile\AppData\Local.IdentityService\AzureServiceAuth\tokenprovider.json'.
Previously, this message had stated the below message (which I understand to be the more recent location for this file), until I attempted to run under Visual Studio 2019 for comparison, at which point, it changed to the above message.
"Could not find a part of the path C:\Users[AppPoolName]\AppData\Local.IdentityService\AzureServiceAuth\tokenProvider.json".
At first, I noticed the path didn't exist from .IdentityService onward, and so followed the suggestion on this MSFT forum post to restore the AppAuthentification extension from VS2019 into VS2022's configuration to restore the C:\Users\<AppPoolName>\AppData\Local\.IdentityService\AzureServiceAuth\tokenprovider.json file and providers the TokenProviders as a path to C:\Program Files (x86)\Microsoft Visual Studio\<version>\Enterprise\Common7\IDE\Extensions\<random dir name>\TokenService\Microsoft.Asal.TokenService.exe. On the next build, I noticed .IdentityService had been created, but not the proceeding directory or file.
I then tried logging out and into Visual Studio a number of times, but this did not seem to create the remaining missing directory and file. Creating the directory and file manually of course resolve the System.IO.DirectoryNotFoundException, but the error message then informs me that the file schema is incorrect. I'm unable to find an example with the correct schema and values.
In terms of client configuration options, I've been explicitly limiting the modes of authentication flow to just ManagedIdentity and VisualStudioCredential for simplicity after noticing other methods (e.g. AzureCLI and Azure PowerShell Module` also failed, despite being logged in to them).
_client = new SecretClient(new Uri(options.KeyVaultUri), new DefaultAzureCredential(
new DefaultAzureCredentialOptions
{
ExcludeManagedIdentityCredential = false,
ExcludeVisualStudioCredential = false,
ExcludeInteractiveBrowserCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeAzureCliCredential = true,
ExcludeEnvironmentCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeSharedTokenCacheCredential = true,
ManagedIdentityClientId = options.ManagementIdentityClientId
}
));
I've also tried the suggestions on Azure SDK GitHub Issue #4590 of settings setProfileEnvironment and loadUserProfile to true in case it's an IIS permissions issue, but this made no difference - the same errors continue.
Finally, the only other reference I've found to the tokenProvider.json file is in Microsoft's documentation for App Authentication, but the re-authenticate button doesn't exist in the Tools > Options > Azure Service Authentication window as suggested.
"If you run into problems using Visual Studio, such as errors that
involve the token provider file, carefully review the preceding steps.
You may need to reauthenticate your developer token. To do so, select
Tools > Options, and then select Azure Service Authentication. Look
for a Re-authenticate link under the selected account. Select it to
authenticate."
As I'm able to locate C:\Program Files (x86)\Microsoft Visual Studio\<version>\Enterprise\Common7\IDE\Extensions\<random dir name>\TokenService\Microsoft.Asal.TokenService.exe and its related configuration file, I suspect it's the missing tokenProvider.json file that's the issue, but I'm not aware of what is responsible for creating that, nor what it should contain.
Any insight or pointers would be appreciated.
Notable packages and their versions in use:
Azure.Identity # v1.8.0
Azure.Security.KeyVault.Secrets # v4.4.0
Edit (1)
As one might expect, I'm able to configure an alternative flow to work by granting an RBAC record upon the Key Vault for an Azure AD Application Registration and then using the ClientSecretCredential flow in place of the DefaultAzureCredentials flow (as below). But this doesn't solve the problem in the best way so I'd be interested if anyone can spot where I'm going wrong with the DefaultAzureCredentials flow, if at all.
_client = new SecretClient(new Uri(options.KeyVaultUri),
new ClientSecretCredential(options.TenantId, options.ClientId, options.Secret)
);

Having found this question on SO, I found that tokenProvider.json existed in the same directory under C:\Users\<local user account> (via %LOCALAPPDATA%.IdentityService\AzureServiceAuth\) and was able to analyse it for reference and duplicate it to the IIS.
As suggested by Johnny5, it seems VisualStudioCredentials executes as the signed-in domain user, but IIS is running as the ApplicationPoolIdentity, hence it doesn't access the file under the domain user location and doesn't create one as it's not signed into Visual Studio. After a little research on how to alter this, I was able to set the IIS Application Pool identity as my domain user which matched the signed-in Visual Studio account.
To do this, follow these steps
Open IIS Manager > Go to Application Pools
Right-click the relevant pool, then click Advanced Settings...
Click the 3-dot button next to the Identity settings (likely ApplicationPoolIdentity)
Select Custom Account and enter your credentials (including any domain prefix - e.g. DOMAIN\MyUser)
If you aren't sure of your domain, open a command prompt and enter echo %USERDOMAIN% to find it.
I then set the SecretClient authentication flow back to utilise DefaultAzureCredential like so, re-tested locally, and success - secrets retrieved.
_client = new SecretClient(new Uri(options.KeyVaultUri), new DefaultAzureCredential(
new DefaultAzureCredentialOptions()
{
ExcludeEnvironmentCredential = true,
ExcludeVisualStudioCodeCredential = true,
ExcludeAzureCliCredential = true,
ExcludeAzurePowerShellCredential = true,
ExcludeSharedTokenCacheCredential = true,
ExcludeInteractiveBrowserCredential = true,
ExcludeManagedIdentityCredential = false,
ExcludeVisualStudioCredential = false,
ManagedIdentityClientId = options.ManagementIdentityClientId,
}));

Windows DPAPI in AWS AMI fails with Access is denied

We are using an AWS EC2 Windows AMI to do our builds from a Jenkins job.
Our libraries use the Windows Cryptography API: Next Generation (NG) (DPAPI) to protect sensitive data from C# and C++ components.
Our builds succeed without any issue, but our unit test trying to use this API in the AMI instance keeps on failing.
On the C# side we get the following exception (almost the same on C++ side):
Access is denied.
Source: System.Security
HRESULT: -2147024891
Stack:
at System.Security.Cryptography.ProtectedData.Protect(Byte[] userData, Byte[] optionalEntropy, DataProtectionScope scope)
We could get the Unit Tests passing with PsExec, but the output is lost and the step that normally takes about 20 minutes now takes more than 4 hours.
From some reading up it seems like the WinRM is the cause of the issue, the PsExec seems to verify that.
Are there any other options instead of PsExec we can try to allow the unit tests to use the DPAPI inside the AMI?
PS: We did try to sync the master keys with CryptProtectData(CRYP TPROTECT_CRED_SYNC) but that did not work.

Pass DataProtectionScope.LocalMachine to ProtectedData.Protect() for the scope. Seems like you are passing DataProtectionScope.CurrentUser which requires the same user for Protect and Unprotect. LocalMachine allows any admin user to UnProtect().

How to authenticate with a private repo on DockerHub through the Docker.Dotnet client?

I am currently writing a microservice in .NET Standard that writes a Dockerfile, builds the Docker image associated with that Dockerfile (which implicitly causes a pull image), and pushes the image to a private Docker registry. Those Docker operations are all performed using the Docker.Dotnet library that MSFT maintains. I believe that this is mostly just a wrapper around calls to the Docker Remote API. The execution context of this microservice is in a K8s cluster hosted on AWS, or internally on bare-metal, depending on the deployment.
Previously our Docker registry has just been a private registry hosted on Artifactory internally, but we are migrating to a private DockerHub registry/repository/. With this migration have come some authentication problems.
We authenticate all of the pull and push operations with an AuthConfig that consists of the username and password for the account associated with the registry. The AuthConfig is either added to a Parameters object and then passed to the call:
imageBuildParameters.AuthConfigs = new Dictionary<string,
AuthConfig>() { { DockerEnvVariables.DockerRegistry, authConfig } };
…
using (var responseStream = _dockerClient.Images.BuildImageFromDockerfileAsync(tarball, imageBuildParameters).GetAwaiter().GetResult())
Or it’s (strangely, to me) both passed in a parameter and separately to the call:
ImagePushParameters imagePushParameters = new ImagePushParameters() { ImageID = image.Id, RegistryAuth = authConfig, Tag = "latest" };
_dockerClient.Images.PushImageAsync(RepoImage(image.Id), imagePushParameters, authConfig, this).GetAwaiter().GetResult();
We are currently getting auth errors for any of the various Docker registries I’ve tried for DockerHub, as such (where I’ve redacted the organzation/namespace and image:
{“message”:“pull access denied for registry-1.docker.io//, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied”},“error”:“pull access denied for registry-1.docker.io//, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied”
The list of DockerHub urls that I’ve tried is following, all with either the error above or a different “Invalid reference format” error:
Hub.docker.io
Hub.docker.io/v1/
Docker.io
Docker.io/v1/
Index.docker.io
Index.docker.io/v1/
Registry.hub.docker.com
registry-1.docker.io
Strangely enough, if I run it locally on my Windows system, the bolded URL’s actually work. However, they all fail when deployed to the cluster (different method of interacting with the Docker socket/npipe?).
Does anyone know the correct URL I need to set to properly authenticate and interact with DockerHub? Or if my current implementation and usage of the Docker.Dotnet library is incorrect in some way? Again, it works just fine with a private Docker registry on Artifactory, and also with DockerHub when run on my local Windows system. Any help would be greatly appreciated. I can provide any more information that is necessary, be it code or configuration.

Deploying Container Instance with MSI and containers deployed to it cannot read keyvault secrets

I followed this guide from Microsoft:
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-managed-identity
All that works fine, but when I want to deploy with a ASP.NET Core 2.1 inside a container, where in my code I try to read the KeyVault secrets it does not work.
It will work without deploying it inside a Docker container though.
But the goal is to deploy the project into a docker container, register it into Azure Container Registry and then create a Container Instance with containers that can read keyvault secrets.
The error I get is typically what I would get when trying to read secrets from Keyvaults inside a running .NET core Docker container:
AzureServiceTokenProviderException: Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/. Exception Message: Tried the following 3 methods to get an access token, but none of them worked. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/. Exception Message: Tried to get token using Managed Service Identity. Unable to connect to the Managed Service Identity (MSI) endpoint. Please check that you are running on an Azure resource that has MSI setup. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/. Exception Message: Tried to get token using Visual Studio. Access token could not be acquired. Environment variable LOCALAPPDATA not set. Parameters: Connectionstring: [No connection string specified], Resource: https://vault.azure.net, Authority: https://login.windows.net/. Exception Message: Tried to get token using Azure CLI. Access token could not be acquired. /bin/bash: az: No such file or directory.
Any guide to get Docker containers read from Azure Keyvault secrets?
The source code for reading a keyvault secrets:
public static async Task<string> GetSecret(string baseUrl, string keyName)
{
AzureServiceTokenProvider azureServiceTokenProvider = new AzureServiceTokenProvider();
using (var keyVaultClient = new KeyVaultClient(new KeyVaultClient.AuthenticationCallback(azureServiceTokenProvider.KeyVaultTokenCallback)))
{
var secret = await keyVaultClient.GetSecretAsync(baseUrl, keyName).ConfigureAwait(false);
return secret
}
}

First of all, you should know managed identities is just the feather of Azure Services. Also, even if for Azure Services, not all the service are supported for the feather. You can know which Azure service support the managed identities here.
And you can take a look that how does the MSI work for Azure Service here. So it seems that you cannot use the MSI for the Docker container which does not belong to Azure inside the Azure Container Instance. The error also shows that:
Exception Message: Tried to get token using Managed Service Identity.
Unable to connect to the Managed Service Identity (MSI) endpoint.
Please check that you are running on an Azure resource that has MSI
setup.
But I suggest you can try to use the Service Principal to read the key stored in KeyVault or access other Azure Service.

Seems that I was not aware using AppAuthentication NuGet 1.1.0-preview produced the error.
Use 1.0.3 and everything works fine inside azure container instance with a container image running :)

Unable to start an NServiceBus Windows Service

Problem Description
I have a Windows service which is hosting an NServiceBus endpoint in NServiceBus.Host.exe.
The binaries are deployed to c:\inetpub\bus\services\myService folder on the server.
A DSC script makes sure the Windows service is created/exists on the server, with the "path to executable" property of the service set to "c:\inetpub\bus\services\myService´NServiceBus.Host.exe" -service NServicebus.Production.
Note! The -service switch is added when installing the service by using the built-in NServiceBus.Host.exe /install parameter, which is why I added it to the Windows service executable path in the DSC script.
Now, when I try to start the service manually on the server, it yields the following error message
Windows could not start the <service name> service on the Local Computer.
Error 1053: The service did not respond to the start or control request in a timely fashion.
Debugging Steps
I have looked through the event log and the following two error messages sticks out:
NServiceBust.Host.exe error:
Application: NServiceBus.Host.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info:
Topshelf.Exceptions.ConfigurationException
at Topshelf.Internal.Actions.RunAsServiceAction.Do(Topshelf.Configuration.IRunConfiguration)
at Topshelf.Runner.Host(Topshelf.Configuration.IRunConfiguration, System.String[])
at NServiceBus.Host.Program.Main(System.String[])
Local Activation permission error:
The application-specific permission settings do not grant Local
Activation permission for the COM Server application with CLSID
{D63B10C5-BB46-4990-A94F-E40B9D520160}
and APPID
{9CA88EE3-ACB7-47C8-AFC4-AB702511C276}
to the user <my_service_account_user> SID (<service_account_SID>) from
address LocalHost (Using LRPC) running in the application container
Unavailable SID (Unavailable). This security permission can be modified
using the Component Services administrative tool.`
Note! The error above only occurs once, i.e. the first time I try to start the service. It does not appear again in the event log for any subsequent attempts of starting the service.
What I have done so far:
Tried the suggestions in a closely related post here on SO, none of which were working.
Tried to install the service by using using the NServiceBus.Host.exe /install parameter. In this case, the service name is created with its name on the following format: MyService.EndpointConfig_v1.0.0.0. Using this approach, the service starts successfully without any error message
Stopping the service and then try to start the service created by the DSC script (with a different name) => success
Removing the service created by NServiceBus and then trying to start the DSC-created service again => failure
Tried granting the service account used for logon when running the service various privileges (neither of which yielded any success), among others:
Membership in the Administrators group
Membership in the Performance Log Users group
Full DCOM permissions via "Launch and Activation Permissions" in dcomcnfg
Tried running c:\inetpub\bus\services\myService´NServiceBus.Host.exe NServicebus.Production from the CLI => success
Code
My Init() method for the service looks like this:
namespace MyService
{
public class EndpointConfig : IConfigureThisEndpoint, AsA_Server, IWantCustomLogging, IWantCustomInitialization
{
public void Init()
{
Directory.SetCurrentDirectory(System.AppDomain.CurrentDomain.BaseDirectory);
SetLoggingLibrary.Log4Net(() => XmlConfigurator.Configure(File.OpenRead(#"log4net.config")));
GlobalContext.Properties["Hostname"] = Dns.GetHostName();
GlobalContext.Properties["Service"] = typeof(EndpointConfig).Namespace;
var container = new WindsorContainer(new XmlInterpreter());
Configure.With()
.CastleWindsorBuilder(container)
.XmlSerializer()
.MsmqTransport()
.IsTransactional(true)
.PurgeOnStartup(false)
.IsolationLevel(System.Transactions.IsolationLevel.RepeatableRead);
var connectionString = ConfigurationManager.ConnectionStrings["<some_conn_string>"].ConnectionString;
container.Register(
Component.For<IDatabaseProvider>()
.ImplementedBy<DatabaseProvider>()
.DependsOn(Property.ForKey("connectionString").Eq(connectionString)));
}
}
}
Theory
When installing the service using /install I assume that NServiceBus.Host.exe does some black magic under the hood - e.g. grants some necessary permissions - to make the service able to start.
Note! On the server, the latest version of NServiceBus is installed (v6.x). However, in my solution/service project version 2.x is used (please, do not ask if I can upgrade - unfortunately, that is not an option).
Appreciate any help I can get, as I am running out of ideas.
EDIT 1
I was asked why I can't just use the /install parameter of NServiceBus and be happy with that. The answer to that is that I could (and, actually, I currently am).
The reason I have still posted this question is split:
I wish to understand why one of two seemingly equivalent approaches fails
I am not completely happy with using the /install parameter. The reason? It boils down to a "chicken or the egg" problem. I use Powershell DSC to provision servers in Azure and I believe that ensuring that Windows Services exists on the server is the responsibility of DSC. However, the first time a server is provisioned the services cannot exist unless I script their creation with DSC, and point the executable path to where the service binaries will be deployed whenever that happens. The other alternative is to skip service creation in DSC, and run the NServiceBus.Host.exe /install as a part of the service/application deployment script instead. Obviously, deployment cannot happen until after a server has been provisioned. Thus, it requires the Windows Service part of the DSC script being stripped down to e.g. merely ensuring the service exist - a verification which will fail until a first time deployment of the application has been performed.