We Keep Coding

Best practice for storing passwords for automation

I know that the "how do I securely store a password?" questions have been asked many times here; and I know the answer is always "Never do this! Store hashes, never store passwords! Use bcrypt!"
But what about times when you cannot use a hash for authentication? For example: automated processes. Say I wrote an automation application that needs to log into an SFTP site or some sort of external service that requires a username/password. I can't use a hash to authenticate with an external service, so what am I supposed to do?
I tagged this question with C# and ASP.NET as these are the two areas that this question applies to specifically for my situation.
Edit: Due to discussion raised in the comments, let me clarify the threat I'm trying to address: I want to prevent an attacker from being able to read the plain-text password used to access the external service. Meaning that if they somehow gained non-admin access to our network or database, even with the database dump they would not be able to read the passwords in plain text.

There are different options, both using encryption for password using a key, and protecting the key storage using HSM module.
option (1): Using Database with HSM module
You can store passwords encrypted in database and benefit from a feature in SQL 2016 "Always Encrypted (Database Engine)".
Always Encrypted allows clients to encrypt sensitive data inside client applications and never reveal the encryption keys to the Database Engine (SQL Database or SQL Server).
You can use Hardware Security Modules (HSM) with Always Encrypted.
The hardware security module (HSM) is a physical device that safeguards digital keys and performs cryptographic operations. These modules traditionally come in the form of a plug-in card or an external device that attaches directly to a computer or to the network.
When you get an HSM, you also get a software libraries implementing common APIs, such as Microsoft Crypto API and Cryptography API. These API are called Cryptographic Service Provider (CSP) and Cryptography API: Next Generation CNG providers.
Your applications can communicate with an HSM using those APIs.
For more securing the HSM module, you can:
- Tie the HSM to your Database Server.
- Tie the HSM to your admin login to Operating System Server.
for more details:
Always Encrypted (Database Engine)
Using Hardware Security Modules with Always Encrypted
Also, Oracle database and other engine can provide encryption with HSM
Securing Stored Data Using Transparent Data Encryption
Option (2): Store password in files in Protected storage using HSM module:
Encrypting files that contain passwords. This may be done by the operating system, an application, or a specialized utility such as password management software that is specifically designed to protect the confidentiality of passwords.
Using OS access control features to restrict access to files that contain passwords. For example, a host could be configured to permit only administrators and certain processes running with administrator-level privileges to access a password file, thus preventing users and user-level processes from accessing passwords.
As you are not using hashing, I exclude this option, but it's a mechanism for storing one-way cryptographic hashes for passwords instead of storing the passwords themselves.

AES encryption and store the key in an external encryption key storage module, if you happen to have access to one of those.
Otherwise, you could try scattering parts of the encryption key on different servers or something but it shouldn't be the first choice.

In my opinion, this shouldn't be solved with a HSM, as that also needs a password that you would need to save. Better is to always type in the password only when restarting the system (a server seldom restarts, so that's ok). The RAM is more or less safe (see here). See for discussion of small projects that have a maybe physical server that have a crontab here.

Establishing a secure connection to a secured database in SQL Server

I have just created a database secured with a symmetric key.
The question is, How can I access the database with a secure conncection from a standalone C# application created in .net 3.5 and SQL Server 2008 (as sql database is a part of application)
Just putting the the key inside the .exe file would be an unsecure way to access the data,or am I missing something?
Please provide specific guide as I am (almost) new to sql.
Thanks.
Update:I mean, I have encrypted a table column with sensitive data to avoid data abuse.
And I want to write a c# code which automatically decrypt ,perform read or write actions on the data column and encrypt again.
The problem is, if I write the direct password inside the code,the whole implementation would be useless.

I have just created a database secured with a symmetric key
What exactly did you do? There is no such thing as you describe. You may had created a symmetric key, but that is something completely different from 'securing' a database.
How can I access the database with a secure connection
Use SSL. See Encrypting Connections to SQL Server for step-by-step instructions.
If you need more specific answer about SQL Server security and/or cryptography you will need to describe what are you trying to achieve. What are you trying to protect, and against what specific threat. You should also go over the Encryption Hierachy. As a general rule though, one can never embed a secret inside an application. Data encrypted by a key hierarchy derived form a password is decrypted by asking the user for the password.

Securely storing (encrypting) data in an ASP.Net application

I have an asp.net application, accessed by people over the internet using web browsers. It runs on a web server and it talks to a backend database.
Some of the users would like to use the application to store some private data. The requirements of this are:
1) Only the user who stored the data should be able to see it.
2) The developers/dbas should not be able to see the data.
3) If the web server and database server were compromised a hacker must not be able to decrypt the data.
So, it's obvious I'm going to have to encrypt this data. If I encrypt it there will be a key somewhere and probably a salt/IV. The question is where do I store the data which is used to perform the decryption? If I store it in the database or the web server then a developer, dba or hacker can access it and decrypt the data.
I think my ideal solution to this would be for the private key to be on the clients machine, that way they are entirely responsible for it. But I'm not sure of how to go about this with an asp.net web application.
I believe I can create a certificate which also stores a private key (PFX). The client companies could use group policy to deploy the certificate to their domain. But it is possible that the ASP.Net application can request the web browser to send the private key to it so that it can perform the decryption? Decrypting on the client would be best but other than creating something in javascript I don't see how this is possible.
Any advice welcome.

Store the key in the mind of the user. Use any password/passphrase based key derivation algorithm you like. The standard is PBKDF2. The optimum choice of algorithm will depend on precisely what your security requirements and threat mode is. Ideally, that decision and the implementation should at least be reviewed by a security expert.

Is it possible that you deploy a ClickOnce application as a part of your solution? The ClickOnce could easily access the cert store on a local machine thus allowing you to perform client-side encryption.

How secure is ProtectedData.Protect (DPAPI)?

Suppose someone gets access all of my hard disk, I guess the weak spot would be my windows password. Without knowing/being able to retrieve that, the data should be pretty much safe, shouldn't it?
I'm asking specifically because of the EFS entry in wikipedia which states that
In Windows 2000, the local administrator is the default Data Recovery Agent, capable of decrypting all files encrypted with EFS by any local user.
and EFS happens to use DPAPI. So does the same apply to my own data protected using this:
ProtectedData.Protect(plain, null, DataProtectionScope.CurrentUser);
And if that is indeed the case, how could I prevent it?
[Edit] N.B. I'm trying to store credentials for a winforms app so that the user does not have to enter their password every time they login. In other words, if someone is able to login as that user (i.e. know the user password), then they might as well be able read the encrypted data.
Which - not coming from a windows background - now makes me wonder - can't the local admin login as any local user anyway? In that case I shouldn't be concerned about the admin being able to retrieve passwords anyway...
[Edit2] As google reveals, it looks like an Administrator cannot just login as any user without resetting/changing their password first. So my question still seems relevant...

EFS uses DPAPI, not the other way around. And Administrator can't read your key just like that.
Before forgetting about DPAPI, I would consider the alternatives. If you encrypt the file yourself,
You must select a strong algorithm and implement it well.
You will need a key. Where will it be ?
You will store the key in a file somewhere on your drive.
That key is sensitive, obviously, you will want to encrypt it
Goto 1
DPAPI does 1 to 3 well. 4 and 5 are moot. If a Windows password is not enough to protect data, ask yourself why it is enough to CRUD that data in the first place.
For better security, you can consider not saving the data but a (salted) hash of it, if possible. It makes your data write only, though. For example, if you want to verify a customer license number :
Save a salted hash value of it
Run the same hash on the salted license number you want to verify,
Compare the two. It they match, the license is valid.
If you must read back encrypted data and a locally encrypted key is not enough, consider encrypting your application key (step 2 above) with a private key stored on a smart card.
Either way, remember that things happens. You always need a backup key somewhere.

See this article on DPAPI Security. Basically, it is as secure as your Windows password -- if your password is reset by an administrator, the decryption key will be lost. The major attack vectors you'll need to look at are:
Password disclosure: "shoulder surfing", sticky notes, etc.
Capture of the computer's accounts database and the use of a password cracker
Online attack by "drive-by download", removable media AutoPlay, etc.
Capture of a password reset disk, if you've made one
Physical installation of a key-logging device or other "bug"

DPAPI can be used both with and without optional entropy. There are only two ways DPAPI blobs without optional entropy can be compromised:
A domain admin can directly retrieve anyone's history of DPAPI master keys at any time. Nothing else is required. These can be used to decrypt all blobs. Local administrators cannot do this.
The user's Windows credentials are compromised.
If you use optional entropy then the data cannot be decrypted by anyone who doesnt know the value. The entropy may be derived from a password required to launch the application etc. Without the value, the data is lost forever.
EFS works differently. The user's key is protected using DPAPI for his profile, but the decryption key for the file itself is additionally directly encrypted with the administrator's public key as well. Therefore a domain admin can access the files.

Are EncryptByCert and DecryptByCert a secure way of encrypting?

I would like to use TDE, but I cannot use it, so I have chosen to use the EncryptByCert and DecryptByCert functions. However, I was also considering encrypting/decrypting data in c# as shown here.
My question is are EncryptByCert and DecryptByCert unsecure because the certificate is also stored in the database? How do people get around this problem?
Is using the c# built-in encryption a better idea?
Thanks in advance for any help. :)

There is an Encryption Hierarchy one has to deploy:
(source: microsoft.com)
The root key used can be the service master key (actually the root is the service password, but this is transparent to the service start), in which case the applications can access the data just by simply connecting to the database and decrypting it. This protects the data if the database file is lost accidentally, but does not protect the data against compromised access to the server: since the decryption key is held by the server itself (the service master key) then anyone that has access to the data can see the data, because anyone can decrypt it. The data is protected through normal access rights and permissions, but the cryptography does not add any protection against authorized users.
The other option is to rely on a password at the root o encryption hierarchy, in which case the application must ask the user for the password to access data. For instance, if is a web site , the user would have to fill in a data decryption password in order to access a specific page. This truly protects the data cryptographically against users that gain access to the data through ordinary data access permissions and don't know the password(s).
SQL Server 2008 also offers the possibility to leverage a hardware TPM to store the decryption key (ie. an employee badge) but this is Enterprise Edition functionality only.
Ultimately, asking if the SQL Cryptographic API is secure or insecure is a non-question. The API itself of course is secure. Encrypting with the data with a symmetric key and then encrypting the symmetric key with a certificate and storing the private key of the certificate along with the data in the database is secure, as long as the private key is properly protected. The security or insecurity of a cryptographic deployment it always ultimately driven by the key management (bare some faulty implementation, but lets assume the implementation is perfect) and key management is 1% application design and 99% human process.
Ultimately, you need to do a serious threat modeling (sorry, but "The threat is that the data is private information and should not be seen plain text." is not threat modeling). Follow a methodology like STRIDE, or other similar methodologies. This is a very hard domain, and is deceptively simple to read about cryptographic APIs and believe that you know how to protect data:
The world is full of bad security
systems designed by people who read
Applied Cryptography