We Keep Coding

DisableAutomaticKeyGeneration in DataProtection does not work for current user

I have a console application named Keycreator that is used to create key using DataProtection.I am using ProtectKeysWithDpapiNG() extension for key configuration. Here's a snippet for the same:
var securityIdentifiers = identities.Select(identity => $"SID={new NTAccount(identity).Translate(typeof(SecurityIdentifier))}");
dataProtectionBuilder.ProtectKeysWithDpapiNG(string.Join(" OR ", securityIdentifiers), flags: DpapiNGProtectionDescriptorFlags.None);
The key generation works as expected.
There is another console app named DataProtector that takes in the key as one of the parameter that is generated using the keycreator console to encrypt/decrypt the file.
Working scenario: When I pass in identities to generate key that includes the current user ,the data protection exe works as expected ie .the encrypt/decrypt of the file being passed works with no issue.
Problem Area: When passing in identities excluding the current user, the data protection creates a new key that is unencrypted even though the encrypted key is already available.
There is an extension to disable Automatic Key Generation
[https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.dataprotection.dataprotectionbuilderextensions.disableautomatickeygeneration?view=aspnetcore-6.0][1]
but even with the extension dataprotection application creates an unencrypted key file.
The Point that i am trying to convey is ,this unencrypted key creation issue exists only when the key that is generated in the keycreator console does not include the current user as one of the identity.
Here is the error that is thrown:
The key ring does not contain a valid default protection key. The data protection system cannot create a new key because auto-generation of keys is disabled. For more information go to http://aka.ms/dataprotectionwarning
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
Any input is appreciated.

Cannot find the symmetric key 'keyFieldProtection', because it does not exist or you do not have permission.'

I am trying to use a stored procedure that has encrypted data, I have written the same program in Python with success. However when I use c# connecting to same db, it throws this error. Cannot find the symmetric key 'keyFieldProtection', because it does not exist or you do not have permission.'

You need to grant permissions to the keys. If you are unable to grant permissions, need to switch mode to windows authentication.
Else if you are opening master key, then refer to the following:
https://learn.microsoft.com/en-us/sql/t-sql/statements/open-master-key-transact-sql?redirectedfrom=MSDN&view=sql-server-2017
As per above;
“If the database master key was encrypted with the service master key, it will be automatically opened when it is needed for decryption or encryption. In this case, it is not necessary to use the OPEN MASTER KEY statement.”
Its hard to tell without looking at your code

Deploy CLR UDF in SQL Server 2017: procedure

I am trying to deploy a CLR assembly in a SQL Server 2017 DB, following the procedure described at https://sqlquantumleap.com/2017/08/09/sqlclr-vs-sql-server-2017-part-2-clr-strict-security-solution-1/. Basically, for testing purposes I'm just including an assembly with a couple of Regex-based functions:
public class TextUdf
{
[SqlFunction(Name = "RegexIsMatch", IsDeterministic = true, IsPrecise = true)]
public static SqlBoolean RegexIsMatch(SqlString text, SqlString pattern,
SqlInt32 options)
{
if (text.IsNull) return SqlBoolean.Null;
if (pattern.IsNull) pattern = "";
return Regex.IsMatch((string)text,
(string)pattern,
options.IsNull? RegexOptions.None : (RegexOptions)options.Value,
new TimeSpan(0, 0, 10))
? SqlBoolean.True
: SqlBoolean.False;
}
[SqlFunction(Name = "RegexReplace", IsDeterministic = true, IsPrecise = true)]
public static SqlString RegexReplace(SqlString text, SqlString pattern,
SqlString replacement, SqlInt32 options)
{
if (text.IsNull || pattern.IsNull) return text;
return Regex.Replace((string)text, (string)pattern,
(string)replacement,
options.IsNull ? RegexOptions.None : (RegexOptions)options.Value);
}
}
I have created a full repro solution at https://github.com/Myrmex/sqlclr. I can follow the whole procedure described there (readme) up to the point where I have to assign the PFX certificate to the CLR assembly to be deployed. At this point, I get this error:
MSB3325: Cannot import the following key file: pfx. The key file may be password protected. To correct this, try to import the certificate again or manually install the certificate to the Strong Name CSP with the following key container name: ...
Following the error message guidance, I then found that I could solve this by installing the PFX with sn, so that I can enter the password manually when prompted (see Cannot import the keyfile 'blah.pfx' - error 'The keyfile may be password protected').
Once done this, I could compile my CLR assembly with the UDF functions. Now, when I try to install it in a test database (just an empty database created for this purpose), via CREATE ASSEMBLY [SqlServerUdf] FROM 0x...binary stuff..., I get this error:
CREATE or ALTER ASSEMBLY for assembly 'SqlServerUdf' with the SAFE or EXTERNAL_ACCESS option failed because the 'clr strict security' option of sp_configure is set to 1. Microsoft recommends that you sign the assembly with a certificate or asymmetric key that has a corresponding login with UNSAFE ASSEMBLY permission. Alternatively, you can trust the assembly using sp_add_trusted_assembly.
which just defeats the purpose of the long procedure I had to follow in order to let SQL Server accept my CLR without dropping strict security.
Clearly I'm missing something, but I'm not sure about many details of the tricky procedure so it would be a hard guesswork. Could anyone suggest we what's wrong with the procedure, so that we can have a quick and dirty step-by-step reference on how to insert a CLR assembly in SQL Server? This simple task seems to have become very hard with the latest version...

Based on what I see in the GitHub repository, there seems to be some steps that you skipped:
You did not set a password on the original SQL2017_KeyAsm project. That's why it's still an .snk file (i.e. SQL2017_KeyAsm.snk) instead of a .pfx file (i.e. SQL2017_KeyAsm.pfx). I am guessing that you unchecked the "Protect my key file with a password" checkbox (I think it is checked by default), because you should not have an .snk at all. Also, in your SQL2017_KeyAsm.sqlproj file, you currently have:
<AssemblyOriginatorKeyFile>SQL2017_KeyAsm.snk</AssemblyOriginatorKeyFile>
When you should have the following:
<AssemblyOriginatorKeyFile>SQL2017_KeyAsm.pfx</AssemblyOriginatorKeyFile>
Although, your instructions in the readme are correct, as you have stated there: "enter in a password". Still, this probably explains the password error you got, since there is no password protecting the private key. Either that, or the password error is due to using the wrong pfx file (see below). I guess try fixing the below items first as it might be possible to get away with a password-less SNK, I've just never tried it.
At the end of the One-Time Procedures section, in the last two steps just after the paragraph starting with, "Finally, the following steps...", step 2 is using the wrong .pfx file. I am guessing that this is a direct result of not having a .pfx file in the previous step (due to not setting a password), so you grabbed the only .pfx file that was there. This is why you are getting the error when attempting to load the SqlServerUdf assembly: you signed the assembly with the certificate, but that certificate was only used as a mechanism to load the KeyAsm assembly into [master] so that the Asymmetric Key could be extracted from it. After that the certificate is dropped. The Asymmetric Key is the public key of the .snk file that was created when you told Visual Studio that you wanted the SQL2017_KeyAsm project signed. And so that .snk file (or .pfx if you protect it with a password) is what you need to select when you are telling Visual Studio to sign the SqlServerUdf project. Doing this will use the same private key to sign both projects / assemblies.
So, first thing to do is change the private key used to sign the SqlServerUdf assembly. It should not be the certificate .pfx file. It needs to be the same key used to sign the KeyAsm file. In your case, this would be SQL2017_KeyAsm.snk.
Try making that one change and see if everything works. If not, then go back and add a password to the snk file (via Visual Studio). You shouldn't need to re-generate the Certificate or its .pfx file because the public key should be staying the same, and that's all that gets loaded into [master] anyway. But, do let me know if this still doesn't work. In the mean time, I will update that post to be more explicit about the password, and about which pfx file to use when.

Creating asymmetric keys in SQL Server 2014 for unsafe assemblies

I have 2 DLLs signed in Visual Studio (VS) 2015. A former employee created an asymmetric key and login and I ran one of the assemblies in UNSAFE mode correctly.
I get the following error with the second one:
Msg 10327, Level 14, State 1, Line 27
CREATE ASSEMBLY for assembly 'TableFile' failed because assembly 'TableFile' is not authorized for PERMISSION_SET = UNSAFE. The assembly is authorized when either of the following is true: the database owner (DBO) has UNSAFE ASSEMBLY permission and the database has the TRUSTWORTHY database property on; or the assembly is signed with a certificate or an asymmetric key that has a corresponding login with UNSAFE ASSEMBLY permission.
I can't ask the former employee so how do I find out how to get this to run? I tried this too:
USE master;
GO
CREATE ASYMMETRIC KEY AProjectKey FROM EXECUTABLE FILE = 'E:\sqldlls\TableFile.dll'
CREATE LOGIN AProjectLogin FROM ASYMMETRIC KEY AProjectKey ;
GRANT UNSAFE ASSEMBLY TO AProjectLogin ;
GO
This gives the following errors:
Msg 15396, Level 16, State 1, Line 9
An asymmetric key with name 'AProjectKey' already exists or this asymmetric key already has been added to the database.
Msg 15151, Level 16, State 1, Line 10
Cannot find the asymmetric key 'AProjectKey', because it does not exist or you do not have permission.
Msg 15151, Level 16, State 1, Line 11
Cannot find the login 'AProjectLogin', because it does not exist or you do not have permission.
How do I get both these assemblies running in unsafe mode? Thanks in advance.

Regarding those 3 error messages:
The first could be due to the Asymmetric Key already existing, but under a different name. Keys and Certificates need to be unique in terms of their public key, not just the name (though that obviously needs to be unique as well). Each Key and Certificate has a hash of the public key which is referred to as the "thumbprint". The thumbprints of existing Keys / Certificates are checked when creating a new one and will prevent the creation, even of a differently named Key / Certificate, if the thumbprint already exists. That is what the error message means by "or this asymmetric key already has been added to the database".
OR:
It could mean that a different Asymmetric Key (i.e. different "thumbprint") exists in master, but with the name AProjectKey
Because you couldn't create the same key under a different name, no Asymmetric Key exists with that new name, hence you can't create a Login from it (since, again, the CREATE ASYMMETRIC KEY statement failed).
Because you couldn't create the Login, it doesn't exist to be granted any permissions.
Error # 2 helps narrow down the issue. If Error # 1 was caused by an existing Asymmetric Key having the same name but different "thumbprint", then you either would have been able to create the Login (if it did not already exist), or you would have gotten an error stating that the Login (i.e. "server principal") already exists. But the error is that an Asymmetric Key by that name cannot be found. This should mean that the Asymmetric Key itself does already exist, but under a different name. You can see what Asymmetric Keys have been created by executing the following:
SELECT * FROM sys.asymmetric_keys;
But that doesn't tell you which one came from that Assembly (or an Assembly signed with the same Strong Name Key used to sign this one). For that you need to know the "thumbprint", and for that you need to open a Command Prompt (preferably a "Developer Command Prompt" which Visual Studio sets up when installed, as it has the correct path set up upon opening it). Then, run the following:
CD /D E:\sqldlls\
sn -T TableFile.dll
You should see:
Public key token is XXXXXXXXXXXXXXXX
Copy and paste that XXXXX "token" (i.e. thumbprint) into the following query:
SELECT ak.[name], ak.[sid]
FROM sys.asymmetric_keys ak
WHERE ak.[thumbprint] = 0x{XXXXXXXXXXXXXXXX}; -- remove the { and }
Assuming that you get a row returned, we need to see if the Login exists. Simply trying to create a Login from the Asymmetric Key won't get us the Login's name if it does exist since you can only have 1 Login created per Key / Certificate, and the error message only reports back the name you are trying to create as the one that already exists, even though it can be a different name for the same Key. So, take the SID from the returned row and paste it in the following query:
SELECT sp.*
FROM sys.server_principals sp
WHERE sp.[sid] = 0x{SID_from_sys_asymmetric_keys}; -- remove the { and }
If no row is returned then you should create a Login from that Asymmetric Key.
At this point a Login should exist, so grant it the UNSAFE ASSEMBLY permission.
Now try creating the Assembly again.

Import certificate with private key programmatically

I'm trying to use the HttpListener class in a C# application to have a mini webserver serve content over SSL. In order to do this I need to use the httpcfg tool. I have a .pfx file with my public and private key pair. If I import this key pair manually using mmc into the local machine store, everything works fine. However, if I import this key pair programmatically using the X509Store class, I am not able to connect to my mini webserver. Note that in both methods the cert is getting imported to the MY store in LocalMachine. Oddly, I am able to view the certificate in mmc once I programmatically import it and when I view it, the UI indicates that a private key is also available for this certificate.
Digging a little deeper, I notice that when I manually import the key pair, I can see a new file appear in C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys, but one does not appear when I import programmatically. On a related note, when I delete a manually imported certificate, it does not remove the corresponding private key file from the previously mentioned directory.
Ultimately, my question is this: When I programmatically add the certificate to the store, where is the private key being stored and why isn't it accessible to the HttpListener class (HttpApi)?
Note that this question is slightly related but I don't think permissioning is the problem since this is all being done as the same Windows user:
How to set read permission on the private key file of X.509 certificate from .NET

Ok, I figured it out. It had to do with the key storage parameters for the certificate object. For anyone else that runs into this problem, make sure you construct your X509Certificate2 objects that you are adding to the store using the X509KeyStorageFlags.PersistKeySet and X509KeyStorageFlags.MachineKeySet flags. This will force the private key to persist in the machine key set location which is required by HttpApi (HttpListener wraps this).

Is this a 2 way SSL? If it is then did you send over a SSL Certificate Request file generated on your machine? This Certificate Request file will be used to create the SSL and they together form a public private key pair.
Also did you try assigning the cert permission for the user account that is being used to run the web app? You can do this by using the Microsoft WSE 3.0 tool.

Not exactly the answer to your question, but here for reference of others going down this path:
Here is a link to a MS chat that gives sample C# code to do what httpcfg does, thus eliminating the need for the tool on deployment.