I need to create a listener in C# that will watch a shared folder (UNC path) and copy the files with a specific extension (*.json) to a target folder when they arrive. The files can be delayed for about half a minute.
The folder is never empty.
Problems:
The files will arrive in a new sub folder, FileSystemWatcher can't be used since it can't listen to sub folders in a shared folder.
The files needs to be copied and left in the folder, so we need to assure that the same file isn't copied more than once.
Files that are edited/updated needs to be copied again and overwritten in the target folder.
Other files will be in the folder and new files will arrive that we need to ignore (doesn't have the right extension).
I thought about polling the folder , but I didn't come up with a good implementation.
I'm pretty sure I can't use the FilesystemWatcher object, but maybe someone can find a smart solution for using it.
One solution to your problem could be you can check the location constantly for a while and examine changes by yourself.
it is not a complete solution, but an idea to consider.
public async Task FindMyFile(string filePath)
{
int retries = 0;
this.Founded = false;
while (!this.Founded)
{
if (System.IO.File.Exists(filePath))
this.Founded = true;
else if (retries < this.maxTries)
{
Console.WriteLine($"File {filePath} not found. Going to wait for 15 minutes");
await Task.Delay(new TimeSpan(0, 15, 0));
++retries;
}
else
{
Console.WriteLine($"File {filePath} not found, retries exceeded.");
break;
}
}
}
Related
My code is searchcing inside a loop if a *txt file has been created.
If file will not be created after x time then i will throw an exception.
Here is my code:
var AnswerFile = #"C:\myFile.txt";
for (int i = 0; i <= 30; i++)
{
if (File.Exists(AnswerFile))
break;
await Task.Delay(100);
}
if (File.Exists(AnswerFile))
{
}
else
{
}
After the loop i check my file if has been created or not. Loop will expire in 3 seconds, 100ms * 30times.
My code is working, i am just looking for the performance and quality of my code. Is there any better approach than mine? Example should i use FileInfo class instead this?
var fi1 = new FileInfo(AnswerFile);
if(fi1.Exists)
{
}
Or should i use filewatcher Class?
You should perhaps use a FileSystemWatcher for this and decouple the process of creating the file from the process of reacting to its presence. If the file must be generated in a certain time because it has some expiry time then you could make the expiry datetime part of the file name so that if it appears after that time you know it's expired. A note of caution with the FileSystemWatcher - it can sometimes miss something (the fine manual says that events can be missed if large numbers are generated in a short time)
In the past I've used this for watching for files being uploaded via ftp. As soon as the notification of file created appears I put the file into a list and check it periodically to see if it is still growing - you can either look at the filesystem watcher lastwritetime event for this or directly check the size of the file now vs some time ago etc - in either approach it's probably easiest to use a dictionary to track the file and the previous size/most recent lastwritedate event.
After a minute of no growth I consider the file uploaded completely and I process it. It might be wise for you to implement a similar delay if using a file system watcher and the files are arriving by some slow generating method
Why you don't retrieve a list of files name, then search in the list? You can use Directory.GetFiles to get the files list inside a directory then search in this list.
This would be more fixable for you since you will create the list once, and reuse it across the application, instead of calling File.Exists for each file.
Example :
var path = #"C:\folder\"; // set the folder path, which contains all answers files
var ext = "*.txt"; // set the file extension.
// GET filename list (bare name) and make them all lowercase.
var files = Directory.GetFiles(path, ext).Select(x=> x.Substring(path.Length, (x.Length - path.Length) - ext.Length + 1 ).Trim().ToLower()).ToList();
// Search for this filename
var search = "myFile";
// Check
if(files.Contains(search.ToLower()))
{
Console.WriteLine($"File : {search} is already existed.");
}
else
{
Console.WriteLine($"File : {search} is not found.");
}
My app receives messages from another system to indicate that a file has been posted in a folder and it then checks the folder to copy that file to another location. This works perfectly, but the file gets posted in the folder 20 minutes after the app gets the message. I'd like some help/ ideas on how I can create a delay without stopping the polling for each file.
Eg:
10:00am: Message received that 20180405_file1.pdf created
10:02am: Message received that 20180405_file2.pdf created
10:20am: check folder for 20180405_file1.pdf
10:22am: check folder for 20180405_ file2.pdf
I tried using System.Thread.Timer, and Timespan, but I don’t want the app to wait until first file is read and then process second file. I want it to be continuously processing other messages and read the files
Any help would be great
MSDN:
FileSystemWatcher listens to the file system change notifications and
raises events when a directory, or file in a directory, changes.
Check msdn for more detail.
What you need:
FileSystemWatcher fileWatcher;
private void watch()
{
fileWatcher = new FileSystemWatcher();
fileWatcher.Path = path;
fileWatcher.Created += new FileSystemEventHandler(OnCreated);
fileWatcher.EnableRaisingEvents = true;
}
private void OnCreated(object source, FileSystemEventArgs e)
{
WaitForFile(e.FullPath);
//Copy files to another directory.
}
private void WaitForFile(string fullPath)
{
while (true)
{
try
{
using (StreamReader stream = new StreamReader(fullPath))
{
break;
}
}
catch
{
Thread.Sleep(1000);
}
}
}
OnCreated will hit once I file is created but wait till it is written. There is no need for any timer as FileSystemWatcher will monitor new file creation.
You can use this only there is a particular folder where all your pdf files are generating after you get the message. If it is a common folder where there are other files too, you need to manage a list of pdf files name same you got from the message notification and then copied them based on condition in OnCreated.
There are similar questions regarding this which were asked earlier. But I couldn't find a proper solution yet.
I have an application which uses a user defined dll library(Cplus_Function_Library.dll). Imagine that the application is launched already to customers. And if there's a new version of the dll available the application will automatically download it and replaces the old one. There's no problem there.
Now I want to create new dll libraries(a lot) and upload it in the sameplace where the Cplus_Function_Library.dll new version exsists(ex: http path/FTP server). And then I can add them by reference in the Cplus_Function_Library.dll. It's also clear. But my question is how can I download all the dll files in this path without giving the file names one by one inside my updater function? Because when I launch the app these files are not known.(updater function is included inside the application.)Is there an easy way to download all the dll files from a specified path without much hassle?
My current update function can be seen below.
Uri pathToNewVerNo = new Uri("//....../New_version.txt"); //Path to the new version number
Uri pathToCurrentVerNo = new Uri("...../Current_version.txt"); //Path to the current version number
Uri pathToDownload = new Uri(".....new_library.dll");
StreamReader readNewVer; //To read the new version number
StreamReader readCurVer; //To read the current version number
StreamWriter writeToCurVer;
WebClient verNew = new WebClient(); //will be used to download the New_version .txt file
WebClient verCur = new WebClient(); //will be used to download the Current_version .txt file
WebClient update = new WebClient(); //will be used to download the new dll file
verNew.DownloadFile(pathToNewVerNo, "New_version.txt"); //Download the New_version .txt file
readCurVer = new StreamReader("Current_version.txt"); //open Current_version.txt file to read
current_Version = readCurVer.ReadLine(); //assign the value to a string
readCurVer.Close(); //close the file
readNewVer = new StreamReader("New_version.txt"); //open New_version.txt file to read
new_Version = readNewVer.ReadLine(); //assign the value to a string
readNewVer.Close(); //close the file
current_ver_doub = Convert.ToDouble(current_Version); //convert the string value to a double
new_ver_doub = Convert.ToDouble(new_Version);
if (new_ver_doub > current_ver_doub) //check if the new version number is greater than the current version number
{
obj.SBO_Application.StatusBar.SetText("Please wait update in process", BoMessageTime.bmt_Medium, BoStatusBarMessageType.smt_Warning);
writeToCurVer = new StreamWriter("Current_version.txt"); //open the current_version.txt to write
writeToCurVer.Write(new_Version); //update with new version number
writeToCurVer.Close(); //close the file
update.DownloadFile(pathToDownload, "new_library.dll"); //download the new .dll file
//*************There will be a roll back functionality added in the future in case if the updated dll file is currupted.*****************
File.Replace("new_library.dll", "Cplus_Function_Library.dll", "Cplus_Function_Library.dll.bac", false); //make a back up file of the old .dll file and replace it
obj.SBO_Application.MessageBox("Update Successful. Please restart the AddOn", 1, "Ok");
try
{
foreach (Process proc in Process.GetProcessesByName("cplus_Global"))
{
proc.Kill();
proc.WaitForExit();
}
}
catch (Exception ex)
{
obj.SBO_Application.MessageBox(ex.Message, 1, "Ok");
}
}
else
{
// SBO_Application.MessageBox("No Update Available", 1, "Ok");
}
}//End of updater_cplus function */
First of all you need to ask yourself whether you absolutely want to reinvent the wheel, and are ready to tackle all problems you will encounter while doing that, such as the problem you now ran into. There are plenty of existing solutions to include installers and updaters with your software.
Anyway, to answer your question:
how can I download all the dll files in this path without giving the file names one by one inside my updater function?
You casually mention it should work over HTTP and FTP. The former has no formal "directory listing" command, so you'll have to invent that yourself. The latter does, but requires the directory to only contain relevant files, or you need to create a whitelist and/or blacklist to include or exclude certain files.
The easiest solution would be to define your version file format so as to include the list of files to download. Then you fetch the version file, interpret it and request the files mentioned in it.
Acording to CodeCaster's idea I put all my dll files inside one folder and the new_version text file in another folder. This way I made sure different file formats won't get mixed up.
The next step was to read the new_version.txt from the ftp server and compare the value with the current version. If it's greater than the later I took all the file names of dll files to a list. Then easily you can download the files one by one to your desired location.
I have three folders in my BIN directory.
1) Backlog
2) InProgress
3) Completed
There are TXT files in Backlog folder. I have used below code to copy files from "Backlog" to "InProgress" during some operation.
System.IO.File.Copy(source,target,true);
File succesfully copied to destination folder. now i want to delete the original file from "Backlog". so I have used below code.
System.IO.File.Delete(source);
It thorws exception, "Process can not access the file source..."
I think its issue with Garbage collection and I need to destroy this using "USING" statement. But i tried below, it says syntax for USING is wrong :(
using(System.IO.File.Copy(source,target,true)){}
I have tried below MOVE, but I still get same error.
using(System.IO.File.Move(source,target))
Note: MOVE is success during moving file from FOLDER 1 to FOLDER 2,
MOVE is FAILED during moving file from FOLDER 2 to FOLDER 3,
So I feel, after first MOVE, we have to reset something??
FULL CODE
string filepathBackLog = AppSettings["Backlog"];
string filepathInProgress = AppSettings["InProgress"];
string filepathCompleted = AppSettings["Completed"];
if(!System.IO.File.Exists(filepathBackLog ))
return;
System.IO.File.Move(filepathBackLog ,filepathInProgress ); //Success, I can see it in folder.
System.IO.File.Move(filepathInProgress ,filepathCompleted );//Exception, I think we need to give some time to execute first MOVE.
Kindly guide me how can I resolve this?
Why don't you use File.Move? It will take care of deleting, too.
I have a program that runs as a Windows Service which is processing files in a specific folder.
Since it's a service, it constantly monitors a folder for new files that have been added. Part of the program's job is to perform comparisons of files in the target folder and flag non-matching files.
What I would like to do is to detect a running copy operation and when it is completed, so that a file is not getting prematurely flagged if it's matching file has not been copied over to the target folder yet.
What I was thinking of doing was using the FileSystemWatcher to watch the target folder and see if a copy operation is occurring. If there is, I put my program's main thread to sleep until the copy operation has completed, then proceed to perform the operation on the folder like normal.
I just wanted to get some insight on this approach and see if it is valid. If anyone else has any other unique approaches to this problem, it would be greatly appreciated.
UPDATE:
I apologize for the confusion, when I say target directory, I mean the source folder containing all the files I want to process. A part of the function of my program is to copy the directory structure of the source directory to a destination directory and copy all valid files to that destination directory, preserving the directory structure of the original source directory, i.e. a user may copy folders containing files to the source directory. I want to prevent errors by ensuring that if a new set of folders containing more subfolders and files is copied to the source directory for processing, my program will not start operating on the target directory until the copy process has completed.
Yup, use a FileSystemWatcher but instead of watching for the created event, watch for the changed event. After every trigger, try to open the file. Something like this:
var watcher = new FileSystemWatcher(path, filter);
watcher.Changed += (sender, e) => {
FileStream file = null;
try {
Thread.Sleep(100); // hack for timing issues
file = File.Open(
e.FullPath,
FileMode.Open,
FileAccess.Read,
FileShare.Read
);
}
catch(IOException) {
// we couldn't open the file
// this is probably because the copy operation is not done
// just swallow the exception
return;
}
// now we have a handle to the file
};
This is about the best that you can do, unfortunately. There is no clean way to know that the file is ready for you to use.
What you are looking for is a typical producer/consumer scenario. What you need to do is outlined in 'Producer/consumer queue' section on this page. This will allow you to use multi threading (maybe span a backgroundworker) to copy files so you don't block the main service thread from listening to system events & you can perform more meaningful tasks there - like checking for new files & updating the queue. So on main thread do check for new files on background threads perform the actual coping task. From personal experience (have implemented this tasks) there is not too much performance gain from this approach unless you are running on multiple CPU machine but the process is very clean & smooth + the code is logically separated nicely.
In short, what you have to do is have an object like the following:
public class File
{
public string FullPath {get; internal set;}
public bool CopyInProgress {get; set;} // property to make sure
// .. other properties if desired
}
Then following the tutorial posted above issue a lock on the File object & the queue to update it & copy it. Using this approach you can use this type approaches instead of constantly monitoring for file copy completion.
The important point to realize here is that your service has only one instance of File object per actual physical file - just make sure you (1)lock your queue when adding & removing & (2) lock the actual File object when initializing an update.
EDIT: Above where I say "there is not too much performance gain from this approach unless" I refere to if you do this approach in a single thread compare to #Jason's suggesting this approach must be noticeably faster due to #Jason's solution performing very expensive IO operations which will fail on most cases. This I haven't tested but I'm quite sure as my approach does not require IO operations open(once only), stream(once only) & close file(once only). #Jason approach suggests multiple open,open,open,open operations which will all fail except the last one.
One approach is to attempt to open the file and see if you get an error. The file will be locked if it is being copied. This will open the file in shared mode so it will conflict with an already open write lock on the file:
using(System.IO.File.Open("file", FileMode.Open,FileAccess.Read, FileShare.Read)) {}
Another is to check the file size. It would change over time if the file is being copied to.
It is also possible to get a list of all applications that has opened a certain file, but I don't know the API for this.
I know this is an old question, but here's an answer I spun up after searching for an answer to just this problem. This had to be tweaked a lot to remove some of the proprietary-ness from what I was working on, so this may not compile directly, but it'll give you an idea. This is working great for me:
void BlockingFileCopySync(FileInfo original, FileInfo copyPath)
{
bool ready = false;
FileSystemWatcher watcher = new FileSystemWatcher();
watcher.NotifyFilter = NotifyFilters.LastWrite;
watcher.Path = copyPath.Directory.FullName;
watcher.Filter = "*" + copyPath.Extension;
watcher.EnableRaisingEvents = true;
bool fileReady = false;
bool firsttime = true;
DateTime previousLastWriteTime = new DateTime();
// modify this as you think you need to...
int waitTimeMs = 100;
watcher.Changed += (sender, e) =>
{
// Get the time the file was modified
// Check it again in 100 ms
// When it has gone a while without modification, it's done.
while (!fileReady)
{
// We need to initialize for the "first time",
// ie. when the file was just created.
// (Really, this could probably be initialized off the
// time of the copy now that I'm thinking of it.)
if (firsttime)
{
previousLastWriteTime = System.IO.File.GetLastWriteTime(copyPath.FullName);
firsttime = false;
System.Threading.Thread.Sleep(waitTimeMs);
continue;
}
DateTime currentLastWriteTime = System.IO.File.GetLastWriteTime(copyPath.FullName);
bool fileModified = (currentLastWriteTime != previousLastWriteTime);
if (fileModified)
{
previousLastWriteTime = currentLastWriteTime;
System.Threading.Thread.Sleep(waitTimeMs);
continue;
}
else
{
fileReady = true;
break;
}
}
};
System.IO.File.Copy(original.FullName, copyPath.FullName, true);
// This guy here chills out until the filesystemwatcher
// tells him the file isn't being writen to anymore.
while (!fileReady)
{
System.Threading.Thread.Sleep(waitTimeMs);
}
}