I am looking for a programmatic way to copy files from one network drive to another. I have created a program that does a simple copy but this is not enough as the line is not reliable and the files ends up being corrupted. Is there a technology that syncs the folders and does a sum-check or something like that.
I need it to be a c# app as I need to know when the copy is completed so I can process the files on the other side. I am using windows services to co-ordinate this.
I'm just looking for someone to point me in the right direction with tutorials if necessary.
What you are looking for is Robocopy (aka Robust Copy), it is built in to windows Vista and newer and it has features to retry on network failure.
One down side is there is no "Verify" functionality built in, but if you need to you can use a 3rd party file hashing program and put the copy in a script that verifies the hashes after the copy completes.
Here is a query that copies all files and sub-folders in a restartable mode, if a copy fails it will re-try 1,000,000 times by default waiting 30 seconds between tries. (you can change that with /r:<N> and /w:<N> where <N> is a number)
robocopy C:\SourceFolder \\DestComputer\DestFolder /zb /e
The only time I have ever done anything like this has been during the deploy process to distributed servers. In that instance I zipped the files, moved and unzipped them using PSEXEC.
If I were you, I would begin by figuring out how to zip the files (To prevent the corruption issues you have described) and then worry about how to move them.
Related
We have an app we wrote deployed onto our terminal servers at work, and keeping it up-to-date is a bit of a pain.
What update mechanisms do people use for app on terminal servers? At the moment we manually copy the new exe + dependencies on witch is just rubbish.
I'm a bit concerned about files being locked by users when trying to update, i cant really just kill the process in case someone is in the middle of doing something. We would like to be able to handle the odd fat client update as well.
Ideally we'd plug something into teamcity/octopus but are open to suggestions
Create a script that copies the exe file to a user specific temp folder, then launches the copied exe. To make the script more efficient, you can have it check the dates of the files. If they are different, then you copy the file over the old one located in the temp folder, else you just launch it.
There are several threads on SO that describe how to check which application creates a file with tools like Sysinternals process monitor. Is something like this possible programmatically from .net?
Background: My program has to remote-control a proprietary third party application using its automation interface, and one of the functions I need from this application has a bug where it creates a bunch of temporary files in %TEMP% that are called tmpXXXX.tmp (the same as .net's Path.GetTempFileName() does) but does not delete them. This causes the C drive to become full over time, eventually failing the application. I already filed a bug to the manufacturer, but we need a temporary workaround for the time being, so I thought of putting a FileSystemWatcher on %TEMP% that watches tmp*.tmp, collects these files, and after the operation on the third-party application finishes, deletes them. But this is risky as another application might also write files with the same file name pattern to %TEMP% so I only want to delete those created by NastyBuggyThirdPartyApplication.exe.
Is this anyhow possible?
This kind of things is possible, but maybe a bit tricky.
To know who created the file, look at the user that owns it. Therefore you might need to create a specific user, and that application will run under this specific user. In order to do that, you need to create a small application that will start your buggy app by impersonating another user, so anything done within the app will be under this user so as file creating...
I don't know how to monitor and get triggered when a file is created, but nothing can prevent you from setting a timer that wakes up every five or ten minutes, then checks if any file in the directory is owned by the application user and closed, so it deletes it.
Maybe if they react fast for this bug fixing, you won't need your app very long time. So another solution, if possible might just to change the Temp folder into another drive, which has lots of space...
One solution is that you use a FileWatcher to automatically delete all the files but before deleting you should check if the file is not currently locked or used by other process, for example the Sysinternal Suite has a tool called handle.exe that can do this. Use it from the command line:
handle.exe -a
You can invoke this from a c# program (there might be some performance issues though)
So what you would do is when a file is created you verify if it is in use or locked (for example u can use the code provided in Is there a way to check if a file is in use?) and then delete it.
Most of the time when an app is using a temp file it will lock it to prevent just what you fear, that you might delete files from other processes.
As far as I can tell there is no sure way to identify which process created a specific file.
We have a python application that checks a directory(C:\sample\folder) every 5 seconds, there's also this external application(.net app) that puts file into that same directory (C:\sample\folder).
Will there be any conflict when the two application access the same folder at the same time (accidentally)?
Conflicts like :
the external app wont be able to place a file because the python app is currently walking through that same directory?
It should be fine for the external app to create and write to a file. If the Python app is reading a file, the .NET app may not be able to write to it while Python is reading it, without both processes opening the file in a shareable way, however.
Likewise if the Python app is going to start reading the newly-created file, it may either find that it can't do so until the .NET app has finished writing to it, or it may read incomplete data. Again, changes would quite possibly be required to both processes to allow reading at all.
It's worth thoroughly testing all the poosibilities you're concerned about, possibly involving the creation of a "fake" external app which writes to a file very slowly, but opening it in the same way that the real one does.
I have a project that uses the .net FileSystemWatcher to watch a Samba network share for video files. When it sees a file, it adds it to an encode queue. When files are dequeued, they are moved to a local directory where the process then encodes the file to several different formats and spits them out to an output directory.
The problem arises because the video files are so big, that it often takes several minutes for them to copy completely into the network directory, so when a file is dequeued, it may or may not have completely finished being copied to the network share. When the file is being copied from a windows machine, I am able to work around it because trying to move a file that is still being copied throws an IOException. I simply catch the exception and retry every few seconds until it is done copying.
When a file is dropped into the Samba share from a computer running OS X however, that IOException is not thrown. Instead, a partial file is copied to the working directory which then fails to encode because it is not a valid video file.
So my question is, is there any way to make the FileSystemWatcher wait for files to be completely written before firing its "Created" event (based on this question I think the answer to that question is "no")? Alternatively, is there a way to get files copied from OS X to behave similarly to those in windows? Or do I need to find another solution for watching the Samba share? Thanks for any help.
Option 3. Your best bet is to have a process that watches the incoming share for files. When it sees a file, note its size and/or modification date.
Then, after some amount of time (like, 1 or 2 seconds), look again. Note any files that were seen before and compare their new sizes/mod dates to the one you saw last time.
Any file that has not changed for some "sufficiently long" period of time (1s? 5s?) is considered "done".
Once you have a "done" file, MOVE/rename that file to another directory. It is from THIS directory that your loading process can run. It "knows" that only files that are complete are in this directory.
By having this two stage process, you are able to later possibly add other rules for acceptance of a file, since all of those rules must pass before the file gets moved to its proper staging area (you can check format, check size, etc.) beyond a simple rule of just file existence.
Your later process can rely on file existence, both as a start mechanism and a restart mechanism. When the process restarts after failure or shut down, it can assume that any files in the second staging are either new or incomplete and take appropriate action based on its own internal state. When the processing is done it can choose to either delete the file, or move it to a "finished" area for archiving or what not.
I want to write a program that will encrypt an entire folder and it's sub-folders, I have no problem doing this but I would like to make the entire encryption process rather transparent by letting a user double click it and have it open as if it weren't encrypted, say if it were a picture or a word document and it'd open in it's respective application.
How can a running program of mine become notified about the opening of a target file, stop the file from opening, do what it needs to do (decrypt), followed by running the resulting decrypted file.
How can I watch a file and do this in C#? Can I watch for other interactions like the user copying a watched file (since it won't be in a watched folder, it should be decrypted i.e. it's dragged to a USB device), or for deleting a watched file (say if I want to shred a file before deletion)?
P.S. The FileSystemWatcher doesn't quite meet my needs. EDIT: What I mean is that FileSystemWatcher will tell me when a file is being opened, deleted and all those events, but it won't let me step in real quick, decrypt the file, and hand it back to the process that normally opens that file.
You can rename files, add them your own extension, like thepicture.jpg.encrypted. Set your program as a default program for this extension and handle opening them
It's impossible in C#. the bare minimum would need you to use user-mode hooks on NtCreateFile, NtOpenFile, etc. You can't achieve that in C#. That wouldn't even work properly due to kernel-mode code which may try to access your files. The proper way of doing this would be to write a I/O minifilter (in C of course).
EDIT: If you're really desperate, try EasyHook - it allows you to hook functions from C#. I haven't tried it though, and it does seem risky hooking vital functions like NtCreateFile. Plus you need a fair bit of Native API knowledge.
Are you using Windows? If so, why not use the built-in BitLocker?
See this link:
BitLocker drive encryption
If you are thinking about a competitive application to BitLocker, add a comment, as I can point you in that direction as well.
Instead of trying to reinvent the wheel, use NTFS file encryption. You can encrypt single files or entire folders or drives. Plus it's completely transparent to the user and does exactly what you asks (e.g. automatically decrypt when copying to a UBS drive, etc). Just use System.IO.File.Encrypt(string) - there couldn't be anything easier.
You can't do this from usermode.
Unfortunately the only way to do this is to write a minifilter driver. Minifilter drivers allow you to intercept IO requests to files, you can then encrypt/decrypt the files you care about on the fly.
It sounds simple, but encryption minifilter drivers are very, very, difficult to get right. You will have to end up shadowing file objects which is a real challenge. Check with www.osr.com, they have a ton of information on doing exactly what you want to do.
If you choose to go this route I would recommend getting a copy of VMWare Workstation and download VirtualKD. It will let you debug at near fire-wire speeds into a VM. I would start with x64 Win7 and get remote shares working first.