We Keep Coding

Enterprise Data

We are developing a huge financial, budget and expense management solution and one of the requirements is the user data and postings collected by our app stored in sql server MUST be encrypted by user supplied key.
We are using SQL server 2012,EF 6 and .NET 4.5.
What we have tried:
We created class library with two functions in c# that does the encryption and decryption.The assemblies are complied to sql assembly and that works fine using a single encryption key.
The challenge:
The database contain data from different users who supply different key.the question is
How do we store user supplied keys such that it's secure...e.g If a user lost/forgot the key used to encrypt their data..the app can recover it.. while the DBA who supports this database should NOT have access to the keys..?
if we have 1M users..that means million keys..the tables have relational references so it become tricky to encrypt each row differently per user...What's the industry standard in this scenario..?

First off, I would like to preface this answer by stating that I don't pretend to know the industry standard in this scenario -- I don't. That being said, here's what I would do.
In cryptography, there's an algorithm known as Shamir's Secret Sharing. In summary, it would let you split the key into multiple parts:
User chooses their private key, and splits it into 4 parts (n = 4), where any subset of 2 parts (k = 2) is sufficient to reconstruct their secret. You can vary n and k to suit your needs, where n would be the number of recovery options provided and k is the number that must be correct.
User then encrypts each part of the private key with their recovery options and sends the encrypted parts to the server to store.
When user requests file, server sends encrypted file to client who can then decrypt it with their key.
In the event the user forgets their key, they can request their encrypted key parts from the server, provide recovery answers in an attempt to decrypt at least k of them, and (hopefully) get their data back.
Notes:
Server doesn't store answers to the recovery options. This means it won't be able to decrypt the files without the user's help (unless you were to send the raw splits to the server as well, but that's a potential security risk). In essence, you could help the user get back to their key, but all bets are off if they can't remember any of their recovery options (e.g. amnesia, Alzheimer's, untimely death).
If the user were to change their secret key, every file would need to be decrypted and re-encrypted using the new key. This could be a potentially expensive task.
The sum of recovery options needed to remake the key must not be easy for an attacker to guess. For example, if I have 4 recovery options of which I must provide 2, and my choices are phone number, best friend's first name, and some others, then this would not be secure. There aren't very many possible choices, which would make that example combination very easy to brute force.

Securely storing and searching by social security number

So I'm working on a supplemental web-based system required by an HR department to store and search records of former personnel. I fought the requirement, but in the end it was handed down that the system has to both enable searching by full SSN, and retrieval of full SSN. My protestations aside, taking some steps to protect this data will actually be a huge improvement over what they are doing with it right now (you don't want to know).
I have been doing a lot of research, and I think I have come up with a reasonable plan -- but like all things crypto/security related there's an awful lot of complexity, and it's very easy to make a mistake. My rough plan is as follows:
On first time run of the application, generate a large random salt, and a 128bit AES key using RijndaelManaged
Write out both of these into a plaintext file for emergency recovery. This file will be stored offline in a secure physcial location. The application will check for the presence of the file, and scream warnings if it is still sitting there.
Store the salt and key securely somewhere. This is the part I don't have a great answer for. I was planning on using DPAPI -- but I don't know how secure it really is at the end of the day. Would I be better off just leaving it in plaintext and restricting filesystem access to the directory its stored in ?
When writing a record to the database, hash the SSN along with the large salt value above to generate a field that is searchable (but not recoverable without obtaining the salt and brute forcing all possible SSNs), and AES encrypt the raw SSN value with a new IV (stored alongside) to generate a field that is retrievable (with the key/iv) but not searchable (because encrypting the same SSN twice should yield different output).
When searching, just hash the search value with the same salt and look it up in the DB
When retrieving, decrypt the value from the DB using the AES key/iv
Other than needing a way to store the keys in a relatively secure way (number 3 above) it seems solid enough.
Things that won't work for us:
"Don't do any of this" Is not an option. This needs to be done, and if we don't do it they'll a) get mad at us and b) just pass all the numbers around in a plaintext document over email.
This will be internal to our network only, so we have that layer of protection at least on top of whatever is implemented here. And access to the application itself will be controlled by active directory.
Thank you for reading, and for any advice.
Update #1:
I realized from the comments that it makes no sense to keep a private IV for the SSN retrieval field. I updated the plan to properly generate a new IV for each record and store it alongside the encrypted value.
Update #2:
I'm removing the hardware stuff from my list of stuff we can't do. I did a bit of research, and it seems like that stuff is more accessible than I thought. Does making use of one of those USB security token things add meaningful security for key storage?

I've had to solve a similar problem recently and have decided to use an HMAC for the hashing. This would provide more security than a simple hash, especially as you can't salt the value (otherwise it wouldn't be searchable).
Then as you say, use AES with a random salt for the reversible encryption.
It maybe that you don't need to encrypt this data but I had no choice and this seemed like a reasonable solution.
My question on IT Security https://security.stackexchange.com/questions/39017/least-insecure-way-to-encrypt-a-field-in-the-database-so-that-it-can-still-be-in

With respect to key storage there are two methods you can use if you choose to store your AES key in the web.config. First method is to use DPAPI as you mentioned. This will encrypt your web.config application setting for that box. The other method you can use is via RSA key (check out this MSDN tutorial), this will encrypt your web.config just like DPAPI however you can use the RSA key on multiple boxes, so if the application is clustered then RSA key is better (just more complicated to setup).
As far as generating the key before you run your application not on the machine running the app this way there's no chance you're going to leave the text file in the directory. You should generate the key as follows.
Generate a random value using RngCryptoServiceProvider
Generate a random salt value using RngCryptoServiceProvider
Hash the two values with PBKDF2 (Rfc2898DeriveBytes)
The reason you use the key derivation method is it protects you in case RngCryptoServiceProvider was found to be insecure for some reason which happens with random number generators.
Use AES 256 instead of AES 128, reason is these algorithms are extremely fast anyway so get the higher security it's almost free. Also make sure you're using the algorithm in CBC or CTR mode (CTR is available in the BouncyCastle library).
Now this will not give your key absolute protection if someone were able to put up a aspx file in your directory. Because that file will become part of your application it would have access to your decrypted values including your key. The reason I'm mentioning this is your network and server security will have to be top notch, so I would highly recommend you work hand-in-hand with your network security team to ensure that nobody has access to that box except the parties in the HR department that need access (Firewall not Active directory). Do NOT make this application publically accessible from the internet in any way shape or form.
You also cannot trust your HR department, someone could become a victim of a social engineering attack and end up giving away their login thus destroying your security model. So in addition to working with your network team you should integrate a two factor authentication mechanism to get into the system, highly recommend going with an actual RSA key or something similar rather than implementing TOTP. This way even if someone from the dept gives away their password because they thought they were winning a free ipad, the attacker would still need a physical device to get into the application.
Log Everything, any time someone sees a SSN make sure to log it somewhere that will be part of a permanent record that's archived on a regular basis. This will allow you to mitigate quickly. I would also put limits on how many records a person can see in a particular time frame, this way you know if someone is mining data from within your application.
Create a SQL user specifically to access this table, do not let any other user have access to the table. This will ensure that only with a particular user id and password can you view the table data.
Before deploying to a production environment you should hire a penetration testing team to test the application and see what they can get, this will go a long way to harden the application from potential attackers, and they can give you great advice on how to harden the security of the application.

Create a new salt and IV for each record. If you need to dump the data into a report for some reason (hopefully without my SSN in it), you would be able to use the method you describe with the unique salt and IV. If you only need to search on an SSN, you could actually hash it instead of using a reversible encryption (more secure).

I think I read somewhere once that hashing a limited set of inputs gets you absolutely nothing. A quick google turned up this SO post with similar warnings:
Hashing SSNs and other limited-domain information
I must admit that I am also no security expert, but given that the possible number of inputs is much smaller than 10^9 which any decent hacker should be able to breeze through in a matter of hours, hashing a SSN seems like you are adding a small layer of annoyance rather than an actual security/difficulty barrier.
Rather than doing it this way, could you do something else? For example, SSN's only have value to an attacker if they can associate a name to a number (since anyone can enumerate out all numbers easily enough). In that case, could you encrypt the user id that the SSN links to in such a way that would be impractical to attack? I am assuming your employees table has some sort of ID, but maybe instead of that do a hash on their email or some sort of guid? That way, even if they do get your SSN data, they would not be able to tell which employee's it is until they managed to brute force that link.
Then again, that approach is also flawed since your company may not have that many employees total. At that point it would be a relatively simple matter of guessing and checking against a company directory to attain everything. No matter how you slice it, this security flaw is going to exist if SSN's must be stored with other identifying data.

Encryption of Data that should be stored in a Database. And understanding the concept of the "key" used,

I'm new to C# and ASP.NET and I have to do a project now. It deals with confidential data of a firm's employees so it needs to be encrypted. I am not sure if I will be able to get through with my own encryption algorithm. If I use any existing algorithms, they said that I should find a foolproof way to store the key.
To be honest, I don't really understand the term "key" in encryption. I would like someone to brief about it and help me with how I should move forward with this project.

http://en.wikipedia.org/wiki/Key_%28cryptography%29
dunno, but maybe start there?

IMHO:
as already advised, don't cobble up your "own", use existing algorithms in the framework that have been tested extensively. Whatever weaknesses they may have will (likely) still be better than what you can cobble up on your own.
understand what needs to be encrypted which pretty much means at some point will need to be decrypted vs. data that needs to be hashed (one-way - e.g. passwords).
decide if you want this to happen on the application side or perhaps, if resources are available to you like SQL server (to store data), on the database side (discuss this with your DBA). You can do both encryption and hashing in SQL server alone.
on the application side, you can think about storing keys in your web.config and subsequently encrypting that section - just like the option to do so for your db connection strings (encrypting the connection section of web.config). This way even your keys aren't in plain text.

The first rule of cryptography - never use your own algorithm, unless you are a Ph.D. and several other Ph.D's are helping you, even then, use only after public auditing.
What they mean about storing the key is that it shouldn't be exposed anywhere - if an attacker can get the key, they can decrypt all data in the database. Currently, there are no known ways to do this. You can store the key in a file outside the website's root folder - this way either the server itself must be compromised, your app must be compromised (e.g. by making it display the "../../key.txt" file, thus descending below the webroot) or your app must be tricked into encrypting/decrypting the data transparently for the attacker (e.g. by having a bug that allows authentication bypass, thus allowing them to use your app to talk to the database).
For the last part of the question, use #Haxx's answer :)

How to hide a database ID from HTML/Javascript

Obviously depending on the type/context of data returned to a web front-end (in my case the setup is HTML/Javascript, .NET Csharp back-end and JSON as the data transport), if I have to return an ID say of a message that is an auto-generated primary key (Int64), what is the best way to "hide" this real ID?
For most things of course, I can understand it doesn't make too much difference, however an application I am working on means if a user "guesses" an ID in the URL to pull back another record, it could prove to be a security issue..
There seems to be lots of ideas/commentary about methods, but nothing has quite clicked.
I was thinking of having an auto-generated primary INT, but also a secondary alternate GUID too. It would be the GUID returned to any front-end process, and of course the auto-generated primary ID would still be used in the backend..
The thinking of course is the GUID would be far more difficult to guess/obtain another one to access a record?
Any ideas or best practices people use?
Thanks in advance,
David.

Regarding security you have several aspects:
Session hijacking
Accessing/Modifying/Creating/Deleting records the user is not authorized to
Non-Authenticated access
Cross-Site* attacks
Man-in-the-middle attacks
etc.
The measures to deal with these depend on your architecture and security needs.
Since you don't say much about your arhcitecture and security needs it is really hard to give any specific advice...
Some points regarding "ID shouldn't be guessable":
"Correct" solution
The problem goes away in the moment you implement authentication + autherization properly
because properly implemented these two make sure that only authenticated users can access
anything at all AND that every user can only access things he is allowed to. Even if an authenticated user knows the correct ID of something he is not allowed to access this would be secure because he would prevented from accessing it.
"weak solution"
create a ConcurrentDictionary as a thread-safe in-memory-cache and put the real IDs plus the "temporary IDs" (for example upon first record access freshly generated GUIDs) in there. You can combine that temporary ID with some salt and/or encryption and/or hash of some connection-specific aspects (like client IP, time etc.). Then on every access you check with the ConcurrentDictionary and act accordingly... one positive effect: after app restart (for example app pool recycling) the same record gets a different ID because this is only an in-memory-cache... though this is hardly usable in a web-farming scenario

I am working on means if a user "guesses" an ID in the URL to pull back another record, it could prove to be a security issue.."
If this is the case then you really need to step back and review the approach to security. If a user can access records which they don't have authorisation to view you do not provide appropriate security of your Object References - https://www.owasp.org/index.php/Top_10_2010-A4-Insecure_Direct_Object_References
The GUID approach will attempt to provide security by obscurity see Is using a GUID security though obscurity? as to whether or not it does you will have to make your own mind up based on your circumstances.

Ofcourse technically, pulling back another record by quessing another ID is a bad thing- only when that other ID shouldnt be visible for the user who's pulling it back. But then you have a security problem anyways and you should focus on that rather then find a way to obfuscate the ID
Anyways, if you want to mess up the url, i recommend you looking into Rijndael. We use it alot here to pass around tokens. Basically, this encryption technique allows you to both encrypt and decrypt. Therefore you can encrypt the ID, send it to the client, the client posts it back and you can simply decrypt again. No need for an extra database record. Even more secure is to encrypt/decrypt the record ID salted with something like an IP for the current client, therefore even URL fishing will be a reduced problem.
See: http://msdn.microsoft.com/en-us/library/system.security.cryptography.rijndael.aspx

I would like to say that, the URL are meant to be public, it is not kind of confidential data. There's no need to hide the url from users. If a url can be seen by one user and should not be accessable to another user, you should check the privilege of the user from the server side instead of hiding that url.

All of the other answers (3) failed to cover the possibility of this being a non-cookied, non-authenticated, non-sessioned, non-logged-in user.
For example, a confirmation page after a order, etc...
In that case, your authentication is based on a secret in the URL. You use a secret that for all practical purposes is unguessable, and very unique per record. Then you assume that if the user has that secret, then they have access to said record, etc...
The real chalenge is to find a good way to make a secret UUID. Many developers will take the SHA1() of rand() + time() + uuid() + remote_ip() or something like that (which is typically sufficient), but I'm sure there is plenty of documentation out there on this.
Yes, in a situation where you have a non-authenticated user accessing a specific piece of data or performing an action (such as password reset), you need to have a second identifier (eg, varchar 40) on your records with a unique key (as you had outlined). Fill it with very random data, and if they have that secret, then let them in.
Take care.

Assistance with URL structure for accept/decline links

I am in the process of creating an app in which a customer can add email addresses to an event. This means that each email address is sent 2 urls via email when added to the list, 1 url to accept and the other to decline. The url is made up of a number of query parmatters, id's etc.
The issue I have is that I want to prevent the scenario in which someone could "guess" another persons url - as such guest the combination of parametters etc. While this is very unlikely, I still want to prevent such.
I have seen several scenarios to help prevent this, ie. add a hash value, encrypt the url etc. However I am looking for the most secure and best practise approach to this and would like any possible feedback.
As an aside I am coding in C# but I dont believe the solution to this is language specific.
Thanks in advance.

I agree this is not language specific. I had a situation very similar to this within the last few years. It needed to be extremely secure due to children and parents receiving the communications. The fastest solution was something like the following:
First store the information that you would use in the URL as parameters somewhere in a database. This should be relatively quick and simple.
Create two GUIDs.
Associate the first GUID with the data in the database that you would have used for processing an "acceptance".
Associate the second GUID for a "decline" record in the database.
Create the two URL's with only the GUID's as parameters.
If the Acceptance URL is clicked, use the database data associated with it to process the acceptance.
If the Decline is clicked, delete the data out of the database, or archive it, or whatever.
After a timeframe, is no URL is clicked, delete or archive the data associated with those GUID's so that they can no longer be used.
GUID's are extremely hard to guess, and the likelihood of guessing one that is actually usable would be so unlikely it is nearly impossible.

I'm guessing you are saving these email addresses somewhere. So it's quite easy to make a secure identifier for each entry you have. Whether that is a hash or some encryption technique, doesn't really matter. But I guess a hash is easier to implement and actually meant for this job.
So you hash for example the emailaddress, the PK value of the record, with the timestamp of when it was added, and some really impossible to guess salt. Just concatenate the various fields together and hash them.
In the end, you send nothing but the hashed key to the server. So when you send those two links, they could look as follows:
http://www.url.com/newsletter/acceptsubscription.aspx?id=x1r15ff2svosdf4r2s0f1
http://www.url.com/newsletter/cancelsubscription.aspx?id=x1r15ff2svosdf4r2s0f1
When the user clicks such a link, your server looks in the database for the record which contains the supplied key. Easy to implement, and really safe if done right. No way in hell someone can guess another persons key. Just bear in mind the standard things when doing something with hashing. Such as:
Do not forget to add salt.
Pick a really slow, and really secure, hashing algorithm.
Just make sure that no one can figure out their own hash, from information they can possess.
If you are really scared of people doing bad things, make sure to stop bruteforcing by adding throttle control to the website. Only allow X number of requests per minute for example. Or some form of banning on an IP-address.
I'm not an expert at these things, so there might be room for improvement. However I think this should point you in the right direction.
edit: I have to add; the solution provided by Tim C is also good. GUID's are indeed very useful for situations like these, and work effectively the same as my hashed solution above.