We Keep Coding

storing ntfs file metadata for retrieval via webserver and linking with database

I am looking for a few pointers to reaching am adequate solution to a problem/feature I need to implement/rectify in my asp.net mvc application.
My application is a LAN only interface that is run over a webserver. In this application there is a page that displays a bunch of files/folders.
I need to be able to store a set of attributes\properties about these files, and those props\attrs need to be independent of their location on the fileserver. This is my main issue, as I could easily link them to the db with the path as the primary key, but alas then as soon as the file moves their link to the db would be lost.
The types of files that need to be displayed unfortunately could be anything. .txt, .exe, media etc etc. So that provides a limiting option also from using something like the tagsharp lib.
One approach i was considering was simply storing a key somehow in the file itself, or with an ADS ( i have no experience in doing this, but am presently trying to research its potentiality).
Does anybody have any experience with this issue, and can recommend a simple approach. I am hoping i do not need to implement an ADS approach as what ive been reading so far is a little bit over my head and im not sure C# will handle the streams adequately for my needs.

Opinion based. Proposal anyway: what about an additional file which is found by a naming convention?
MyDocument.doxc
MyDomument.docx.properties
MyMovie.mp4
MyMovie.mp4.properties
When moving / renaming files, make sure you move / rename the properties file the same.

First of all thanks for taking a moment to reply.
I had considered the possibility of using a separate file. The problem is that the users of the filesystem ( which may or may not include users of the lan application ) need to be able to move/copy files independent of db application.
Therefore if a user moves a file in windows explorer, I need it to automatically move those additional properties with it. Unfortunately I cant rely on users to move those additional files on their own volition, and I cant ask users to only use the application to move files ( if i were to generate code for the program to do this ).

General file attribute to identify file without its path or even name

I am working on an application that keeps track and uses files stored on the file system. Users are allowed to open, create, delete and move files in the file system. Meanwhile my application is not constantly running so I can't keep track of all changes real-time. Afterwards my application has to find out what file is whom (i.c. as identified in my application).
The most preferred solution for the users is that the application solves every change itself. Each user-interaction is less preferred.
One of my ideas was to use an attribute of a file and assign a key-value to it so when it has been identified once, it can always been recognized afterwards. But I don't know if there is such an attribute. This article didn't give much hope: There is in Windows file systems a pre computed hash for each file?.
Does somebody know if there is such an attribute I can use? And how can I used it in C#?
Is there anyone who is running up against this problem? And how did you solve it?
I'd like to hear good suggestions.
regards, Jaap

If your files don't leave NTFS, this is easily achievable by alternative data streams, where you can store your data along with files. This is more-or-less good article about ADS: http://www.flexhex.com/docs/articles/alternate-streams.phtml
There is another suitable method - it's very efficient, but also very complicated to use, it requires quite good knowledge about NTFS internals - USN Change Journal; see http://msdn.microsoft.com/en-us/library/windows/desktop/aa363798.aspx. With USN Change Journal, you can "get" very efficiently all files that were changed (even all change events) within specified time period.
Nevertheless, if your files leave NTFS realm, e.g. if it's copied to FAT32, contents of ADS is lost.

Relying on a File attribute is "dangerous" in that some user could alter the attribute while your program isn't running. This could lead you to believe that a certain file is (or isn't) tracked by the program while it really isn't.
I would suggest to keep track of the files in a database, XML, or some other file. When your application starts you read the file/db and check for new/deleted/editted files.
You could store a Hash of the files to find out if a file has been moved/editted. Keeping track of files that are moved AND editted is going to be pretty difficult. (I have no clue how you could achieve it)
PS: Have you considered making your application a Windows service? Having the file-management running in the background no matter if the GUI part of your application is running or not?

Responding to interactions with files in C#?

I want to write a program that will encrypt an entire folder and it's sub-folders, I have no problem doing this but I would like to make the entire encryption process rather transparent by letting a user double click it and have it open as if it weren't encrypted, say if it were a picture or a word document and it'd open in it's respective application.
How can a running program of mine become notified about the opening of a target file, stop the file from opening, do what it needs to do (decrypt), followed by running the resulting decrypted file.
How can I watch a file and do this in C#? Can I watch for other interactions like the user copying a watched file (since it won't be in a watched folder, it should be decrypted i.e. it's dragged to a USB device), or for deleting a watched file (say if I want to shred a file before deletion)?
P.S. The FileSystemWatcher doesn't quite meet my needs. EDIT: What I mean is that FileSystemWatcher will tell me when a file is being opened, deleted and all those events, but it won't let me step in real quick, decrypt the file, and hand it back to the process that normally opens that file.

You can rename files, add them your own extension, like thepicture.jpg.encrypted. Set your program as a default program for this extension and handle opening them

It's impossible in C#. the bare minimum would need you to use user-mode hooks on NtCreateFile, NtOpenFile, etc. You can't achieve that in C#. That wouldn't even work properly due to kernel-mode code which may try to access your files. The proper way of doing this would be to write a I/O minifilter (in C of course).
EDIT: If you're really desperate, try EasyHook - it allows you to hook functions from C#. I haven't tried it though, and it does seem risky hooking vital functions like NtCreateFile. Plus you need a fair bit of Native API knowledge.

Are you using Windows? If so, why not use the built-in BitLocker?
See this link:
BitLocker drive encryption
If you are thinking about a competitive application to BitLocker, add a comment, as I can point you in that direction as well.

Instead of trying to reinvent the wheel, use NTFS file encryption. You can encrypt single files or entire folders or drives. Plus it's completely transparent to the user and does exactly what you asks (e.g. automatically decrypt when copying to a UBS drive, etc). Just use System.IO.File.Encrypt(string) - there couldn't be anything easier.

You can't do this from usermode.
Unfortunately the only way to do this is to write a minifilter driver. Minifilter drivers allow you to intercept IO requests to files, you can then encrypt/decrypt the files you care about on the fly.
It sounds simple, but encryption minifilter drivers are very, very, difficult to get right. You will have to end up shadowing file objects which is a real challenge. Check with www.osr.com, they have a ton of information on doing exactly what you want to do.
If you choose to go this route I would recommend getting a copy of VMWare Workstation and download VirtualKD. It will let you debug at near fire-wire speeds into a VM. I would start with x64 Win7 and get remote shares working first.

How do I programmatically get a list of locked files under a folder structure?

Following up from this question: How can I unlock a file that is locked by a process in .NET, how do I programmatically get a list of files that are locked in a particular folder and its subfolders?
I'm using Windows 2003, .NET 3.5, C# 3.0.
Update: some background... basically we're archiving closed websites on a shared server. After deleting the site from IIS we move the folder structure for the site to an archive where it's held for 6 weeks. Even though the IIS site and FTP has been stopped and deleted we still encounter locked files during the move. So it'd be nice to know upfront what's still got a lock on it.

As answered in another question :How to check for file lock?
The only way would be trying them all and add them to the list if it throws an exception.
As mentioned in the above question, it would not be a reliable list to use, but only give you an overview of a specific point in time.

The only reliable way to detect locked files in real time would be to write a device driver (like filemon), but I don't think you can do that in C#.

How to prevent the uploading of Warez. (asp.net, C#)

My company created an application that can send large attachements from one mail recipient to another (because most mailboxes are very limited).
But we were wondering how we can prevent the uploading of warez?
For now all extentions are allowed, but we could restrict the extentions to zip and images.
But if you zip warez you can still upload these.
Are there any tools, methods or something like it to prevent the uploading of warez through our system?
Some more info:
This project is semi-public. It will mostly be used for the communication between customer and company. Therefore an email address of our company is always required (either within the receivers as that of the senders, but you all know how easy it is to manipulate this).

Define what "warez" is first.
I'm pretty sure you're going to have problems with that.
You can probably implement heuristics that figure out that you're sending applications and just ban that, but there's no way you're going to figure out that one application is a pirated copy and another isn't and allow the legal one while ban the pirated one.
If you control the server, and is afraid that people will upload pirated copies of applications onto your server and use it to spread it with, then I'm pretty sure your only option is to check with a lawyer what you're obligated to do.
I think it boils down to that you need a system where copyright owners can inform you of pirated copies being present and that you have a system to remove said content within a time frame. I think that's all that is required.

EDIT
If as you said in your edit, that this is for customers to send stuff to you, then I'd be very careful about the allowed email addresses. Is there anything to stop somebody putting in Distribution Email addresses. e.g. If some naughty person sent a large file to All#YourCompany.com, will it be distributed or will it be blocked
ORIGINAL
If this is an open/public system, then its going to be abused. There are ways to unpack zip files on the fly to check their contents, and even to check the file mimetype headers to perform more restrictions, but it doesn't change the fact that someone might want to legitimately send an AVI file of a presentation, while someone else whats to upload a pirated movie.
If this is for internal use in your company, I'd suggest restricting access in someway (tie the system into your Company LDAP/ADSI system and make users login to the system.
Also putting some file size restrictions in place might be necessary as theres nothing to stop some script kiddie just sending 1Gb Junk Text files around, just to be a nuisance and eating up your bandwidth

You can always just rename, for example, a .rar extension to .jpg and let the downloader know to rename the file to open the "Warez".
There is no way to block it other than to take random samplings, test it your self, and then manually delete whatever it is you don't want.

Short answer: No, you can't.
You could look for filenames from a list, but that will fail (e.g. you might ban "MS Word", but then if someone uplads an innocent "MS Word.doc" you fail. Or if the bad guy renames his exe to "MS W0rd" you fail.
You could look for recognised sequences in the file - that fails as soon as they apply even simple encryption or compression.
You can create user accounts and ban users who misbehave, but this fails because you have to spend a lot of effort policing it and in any case users can just creat multiple accounts using web mail addresses.
My suggestion would be to make this someone else's problem. Get users to upload files to someone elses system (Microsoft SkyDrive, Amazon S3 etc) and then they can worry about the legal side.

What if someone sends a password-protected RAR archive to ensure security of some documents? You can't possibly look inside of it - and you shouldn't - not your business.
For example, we had a couple of times some access right issues with our network. And I needed to install some third-party components on my developer machine. As I was unable to access our repository at the time, I just got the installation package sent to me per email. Now, how can an outsider possibly decide whether a file "SomeCoolComponent.msi" is a warez copy downloaded from the intenet or a 100% legal copy which I have the rights to use?
We once had at university our email account suddenly block all password-protected archives as attachments. You guess what? I didn't stop encrypting them. I stopped using that account.

No - you cannot prevent this with a tool or framework.
You can prevent this by banning / blacklisting users who violate the policy.
Trying to do everything in code isn't always the best idea - sometime a simple "break the rules and you get banned" policy is best.

Assuming you can build decompression support in, you could use this heuristic method to determine whether a given uploaded archive is warez (derived from real world warez distribution methods):
get the filenames contained in the archive
if the archive contains (an .nfo OR a .diz file) and (an exe OR an msi or an archive containing one of the above) block the upload
if the archive contains a series of zip/rars/00X files block the upload
if the file is an exe, check whether it's a SFX and if it's a zip, rar or 7z SFX check the embedded archive
Otherwise just accept the upload, making sure you clearly state in your TOS that the company is not responsible for user uploads.