I have an application which has 10,000+ files in a directory. Once a day a few more files are added to the directory. In theory these files don't change. The problem is that it takes lots of time to check to make sure these files aren't in the database. I looked at FileSystemWatcher class but my app doesn't always run 24/7. It is also possible that the file timestamp may be older than the newest file. Therefore I can not count on file timestamps.
Having a false positive on a new file is better than a false negative.
I am looking for suggestions on a way to speed this up.
This is a Window 7+ and .NET app.
Related
I have made a game and wish to distribute it online. I have spent years playing around with many Installers (InstallWise, InstallShield, etc, etc, etc).
They are very complex, require time and in most cases, a decent amount of money. So I want to write my own Installer, that will install my game for the user.
My game is comprised of:
DLL files (these will go inside the Game's folder, inside Program Files.
The application file itself (a single .exe file).
I will also need to create a shortcut on the Desktop (if the user allows) which will launch the .exe in Program Files folder. I can already do this.
I know how to copy and write files to folders. What I am asking is, how do I "pack" the files into my installer file, so that I can give a user a single file to download, which will then "unpack" the game's files into the appropriate location?
I have asked this question 2 years ago on SO and was met with hostility; the person claimed that this is not possible - but incase they haven't noticed, 90% of installers are just a single file, which unpacks its contents into a directory/several directories. So I know it is possible.
The only way I can think of that I can get this to work is by going over each file that needs to be packed, and reading the bytes into the app and storing it into an embedded file. And when the app is run, it will look for embedded files/bytes and write those bytes to new files in the specified locations. Please correct me if I am wrong.
Any help will be greatly appreciated.
You will need to either save the bytes in your installer, which means that you will need a builder for the actual installer which will use CodeDom.
Or you can download the files from a server, which seems faster in this case.
You choose.
To do so i would use a Self-Extract Zip. this is an exe that will unpack itself with all necessary files then you set the after extract command to call and exe of yours which copy everything where you want and then create yourself link on desktop and such. not very difficult.
In the Self-Extract file you can also specify that the content is extracted in the temp folder of the computer allowing you to find stuff using environment special directory
I am looking for a programmatic way to copy files from one network drive to another. I have created a program that does a simple copy but this is not enough as the line is not reliable and the files ends up being corrupted. Is there a technology that syncs the folders and does a sum-check or something like that.
I need it to be a c# app as I need to know when the copy is completed so I can process the files on the other side. I am using windows services to co-ordinate this.
I'm just looking for someone to point me in the right direction with tutorials if necessary.
What you are looking for is Robocopy (aka Robust Copy), it is built in to windows Vista and newer and it has features to retry on network failure.
One down side is there is no "Verify" functionality built in, but if you need to you can use a 3rd party file hashing program and put the copy in a script that verifies the hashes after the copy completes.
Here is a query that copies all files and sub-folders in a restartable mode, if a copy fails it will re-try 1,000,000 times by default waiting 30 seconds between tries. (you can change that with /r:<N> and /w:<N> where <N> is a number)
robocopy C:\SourceFolder \\DestComputer\DestFolder /zb /e
The only time I have ever done anything like this has been during the deploy process to distributed servers. In that instance I zipped the files, moved and unzipped them using PSEXEC.
If I were you, I would begin by figuring out how to zip the files (To prevent the corruption issues you have described) and then worry about how to move them.
I download GB's of stuff every day. And I get all OCD and organize files and folders so many times during the day and it's driving me nuts.
So I plan on writing an app that detects when a file has finished downloading (to the Windows Downloads folder), and then places it in its relevant categorized folder.
E.g.:
I download an app. When the app detects that the file has finished downloading, it places it into Sub-folder Applciations. Or, when I finish downloading a Document, the document is then placed inside the Documents sub-folder of the Downloads folder.
The problem I have here is that I don't want to do this unless there is a definitive way to tell whether a file has finished downloading.
Things I have thought of:
I have thought about implementing FileSystemWatcher on the Downloads folder, and when a new file is created there, it gets added to a list. And when FileSystemWatcher detects that the file size has changed, or has been modified, it will start a timer; the purpose of this timer is to determine after x amount of seconds whether the download is complete. It does this by assuming (wrongly) that if a file's size has not increased in a specified period of time, the download is complete.
That's all I can think of. Any ideas on how this kind of thing can be accomplished?
File is blocked when it is accessed. Not every file. But you could check whether the file is open by another application. If the file is not open - this should tell you, that it has downloaded completely.
I have a project that uses the .net FileSystemWatcher to watch a Samba network share for video files. When it sees a file, it adds it to an encode queue. When files are dequeued, they are moved to a local directory where the process then encodes the file to several different formats and spits them out to an output directory.
The problem arises because the video files are so big, that it often takes several minutes for them to copy completely into the network directory, so when a file is dequeued, it may or may not have completely finished being copied to the network share. When the file is being copied from a windows machine, I am able to work around it because trying to move a file that is still being copied throws an IOException. I simply catch the exception and retry every few seconds until it is done copying.
When a file is dropped into the Samba share from a computer running OS X however, that IOException is not thrown. Instead, a partial file is copied to the working directory which then fails to encode because it is not a valid video file.
So my question is, is there any way to make the FileSystemWatcher wait for files to be completely written before firing its "Created" event (based on this question I think the answer to that question is "no")? Alternatively, is there a way to get files copied from OS X to behave similarly to those in windows? Or do I need to find another solution for watching the Samba share? Thanks for any help.
Option 3. Your best bet is to have a process that watches the incoming share for files. When it sees a file, note its size and/or modification date.
Then, after some amount of time (like, 1 or 2 seconds), look again. Note any files that were seen before and compare their new sizes/mod dates to the one you saw last time.
Any file that has not changed for some "sufficiently long" period of time (1s? 5s?) is considered "done".
Once you have a "done" file, MOVE/rename that file to another directory. It is from THIS directory that your loading process can run. It "knows" that only files that are complete are in this directory.
By having this two stage process, you are able to later possibly add other rules for acceptance of a file, since all of those rules must pass before the file gets moved to its proper staging area (you can check format, check size, etc.) beyond a simple rule of just file existence.
Your later process can rely on file existence, both as a start mechanism and a restart mechanism. When the process restarts after failure or shut down, it can assume that any files in the second staging are either new or incomplete and take appropriate action based on its own internal state. When the processing is done it can choose to either delete the file, or move it to a "finished" area for archiving or what not.
I have an app that has search functionality. The search algorithm is compiled to a separate dll. In the C# code for the search algorithm, I am using strings held in a settings file to point to the directory where the search index resides. But once the search code is compiled, the settings file is somehow incorporated in the dll. I want to have multiple versions of this code running on my server with each pointing to a different location for the index. And I want the operator to change a file to have each version point to something else as they find necessary. Both config files and settings files end up getting incorporated in the dll. How do I then accomplish this? What is the right industry standard way of doing this?
It's strange that the settings file is compiled... are you sure about that? Setting, config and resx files should be copied to the output directory, it's even a property you can modify on solution explorer. Then you should get it's values by doing
System.Configuration.ConfigurationManager.AppSettings.Get("YourKey")
But I think this won't know about user changes until app is restarted. If you want settings to be dynamic you should either store them on a database, or on a file that you open, read and close every time you need it.
Hope this helped!