I am trying to implement file based autoincrement identity value (at int value stored in TXT file) and I am trying to come up with the best way to handle concurrency issues. This identity will be used for unique ID for my content. When saving new content this file gets opened, the value gets read, incremented, new content is saved and the incremented value is written back to the file (whether we store the next available ID or the last issued one doesn't really matter). While this is being done another process might come along and try to save new content. The previous process opens the file with FileShare.None so no other process will be able to read the file until it is released by the first process. While the odds of this happening are minimal it could still happen.
Now when this does happen we have two options:
wait for the file to become available -
Emulate waiting on File.Open in C# when file is locked
we are talking about miliseconds here, so I guess this wouldn't be an issue as long as something strange happens and file never becomes available, then this solution would result in an infinite loop, so not an ideal solution
implement some sort of a queue and run all operations on files within a queue. My user experience requirements are such that at the time of saving/modifying files user should never be informed about exceptions or that something went wrong - he would get informed about them through a very friendly user interface later when operations would fail on the queue too.
At the moment of writing this, the solution should work within ASP.NET MVC application (both synchronously and async thru AJAX) but, if possible, it should use the concepts that could also work in Silverlight or Windows Forms or WPF application.
With regards to those two options which one do you think is better and for the second option what are possible technologies to implement this?
The ReaderWriterLockSlim class seems like a good solution for synchronizing access to the shared resource.
Related
I am using Windows Workflow Foundation 4and as part of more complicated scenario I have those two states:
The idea is that the users are allowed to upload files which is part of the whole workflow. There may be several reason which my lead to the fact that an uploaded file can not be processed immediately. One of the reasons may be that currently there is a file uploaded from a certain user which is being processed so every other file uploaded during the processing should be in state WaitingProcessing. However when I enter the WaitingProcessing state I need to check this. In order to do that I have to implement something like this:
where generally the HasFileToProcess is a function which will call a stored procedure from the database to check if currently there is a file for this user which is in state Processing.
Almost all parts of the tasks are clear, one thing that I don't know how to do is how to call e function inside the condition field. In fact I have almost zero experience with Windows Workflow at all so I'm not even sure that this is the way to go so as a sub-question I would appreciate if someone knows and show me if there is better way to implement this kind of logic.
I'm currently working on a C# project of an application we'd like to develop. We're brainstorming over the question of sharing the data between users. We'd like to be able to specify a folder where all the files of the application are going to be saved and we'd like to be able to save them on a shared folder (server, different PC or Mac, Nas, etc.).
The deployment would be like so :
Installation on the first PC, we choose a network drive, share, whatever and create all the files for the application in this location.
On the second PC we install the application and we choose the same location (on the network), the application doesn't create anything, it sees that it's already existing and it uses these files as the application's data
Same thing on the other clients
The application's files are going to be documents (most likely XML formatted documents) and when opening the application we want to show all the existing documents. The thing is, we don't only want to have the list of documents and be able to edit their content, we also would like to be able to edit the document's property, so in a way we'd like a file (Sqlite, XML, whatever) representing the list of all the documents and their attributes. Same thing for a list of addresses.
I know all that looks exactly like a client / server with database solution, but this solution is out of the question. I was first looking at SQLite for my data files, but I know concurrency can be a real problem and file lock doesn't work well. The thing is, I would have the same problem with simple XML files (refreshing the content when several users are working, accessing locked files).
So I guess my final question is : Is it feasable? Is there an alternative I didn't see which would allow us to do that more easily?
EDIT :
OK I'm not responding to every post or comment, because I'm currently testing concurrency with SQLite. What I did, and please correct me if the way I test this is wrong, is launch X BackgroundWorker which are all going to insert record in a sample database (which is recreated everytime I start the application). I tried launching 100 iterations of INSERT in the database via these backgroundWorkers.
Of course concurrency is working with one application running, it's simply waiting for the last BackgroundWorker to do it's job and then writing the next record. I also tried inserting at (almost) the same time, meaning I put a loop in every BackgroundWorker waiting for a modulo 5 timestamp (every 5 seconds, every BackgroundWorker runs). Again, it's waiting for the previous insert query to end before doing the next and everything's working fine. I even tried it with 500 BackgroundWorkers and it worked fine.
I then tried launching my app several times and running them simultaneously. When doing this I did have some issue. With two instances of my app it was still working fine, but when trying this with 4-5 instances, it got really buggy and I got two types of error : 1. database is locked 2. disk I/O failure. But mostyle locked databases.
What I did was pretty intensive, in the scenario of my application, it will never ever come to 5 processes trying to simultaneously insert 500 hunded rows at the same time (maybe I'll get a concurrency of two or three connections). But what really bugged me and what makes me think my testing method is not really a good one, is that I got these errors trying to work on a database on a shared network, on a NAS AND on my own HDD. Everytime it worked for maybe 30-40 queries then throwing me "database is locked" error.
Am I testing it wrong? Maybe I shouldn't be trying so hard to make this work, but I'm still not convinced that SQLite is not a good alternative to what I'm trying to do, since the concurrency is going to be really small.
With your optimistic/pessimistic locking, you are ultimately trying to build a database. Also, you WILL have issues with consistency while trying to keep multiple files in sync with each other. Think about if you update the "metadata" file, and the write fails half-way through because of a network blip. File corruption will ensue, and you will be left trying to reconstruct things from backups.
I would suggest a couple of likely solutions:
1) Host the content yourselves, and let them be pure clients (cloud based deployments are ideal for this). Most network/firewall issues can be circumvented by using HTTP as your transport (web services).
2) Have one of the workstations be the "server", which keeps it data files on the NFS. This will give you transactional integrity, incremental backups, etc. There are lots of good embedded database managements systems to help you manage this complexity. MS SQL Server even has some great options for this.
You right, Sqlite uses file locks on database file, so storing all data files in database would bring write-starvation problem for editing your documents.
May be it's better choice to implement simple optimistic/pessimistic locking by yourself on particular-file level? For example, in case of using pessimistic lock you just don't allow anyone to edit particular file, if somebody already in process of editing it. In this case you will hold lock just on one file, but not on the entire database. If possibility of conflict(editing particular file at the same time) is pretty low, it is better to go with optimistic locking.
Simple optimistic locking implementation:
When user get file for reading - it's OK, no problem here. If user get file for editing, you could calculate hash for this file(or get timestamp of last updated time of the file), and then, when user tries to save edited file, compare current(at the moment of saving) hash/timestamp to make sure that file has not been changed by somebody else. If file has not been changed then it's ok to save it. IF file has been changed, then current user is out of luck, you need to inform him about it. This optimistic scenario is nice when possibility of this "out of luck" is pretty low. Otherwise it's better to stick with pessimistic locking, when you do not allow user even to start file editing if somebody else is doing it.
hello guys I have been given the task to create a web-app that would allow an end user to load a file to one of our servers. the server would then manipulate the file and then give it back to the end user.The server has IIS7 and .net framework 4
Here is my problem.
I have everything figured out except that I'm having a hard time getting rid of the file from the server once i manipulated. Is there a way that i can put a timer say after 30 mins to have the file removed from the server in my code? or is there another solution to doing this?
I know the timer suggestion perhaps does not make sense however i cant think of another way to do it so i'm looking for an opinion or another method.
A timer is a good method to schedule something in the future. You could even reset the timer if the user requests the file again. Just give the timer a delegate that deletes the file when the timer fires.
Are you able to set up a scheduled task on the server?
This sort of thing is perfect for a simple console app that simply deletes files that have a modified date/time that is older than, say, now.AddMinutes(-10).
The task can run every 10 minutes or so if you like too.
Sometimes best to keep this sort of thing away from your website. Let your site serve your users and create something else to serve your server.. :)
Update
If its a heavy-traffic site you could simply delete all old files the next time someone uploads a file.
So:
User Selects file to upload, clicks Upload -> you get file -> you
delete old files (regardless of who they belong to) -> you manipulate
file -> file will be deleted by next users upload...
Why not delete it after manipulating? Or whatever the last step in the process is? That would seem to be the best and easiest way.
Depending on volume, it's probably not a great idea to do a single task for each file - rather you should batch them into a queue and have a single thread process the queue.
For instance, you can spin up a background thread in global.asax (perhaps using a Timer) to handle cleanup tasks by comparing file times or something.
Or, as step 1 of the process you could clean any old files. Not exactly the same thing, but may be close enough.
Or, you could abuse the Cache remove callback as a timer.
If you can ensure the app stays running all the time you can skip scheduled tasks and use Quartz.NET. In this case, even if it shuts down using quartz would not be that bad -- unless there is something else to this having a few old files hanging around while the app is idle wouldn't hurt.
Insofar as handling this, I would store in an appropriate manner (eg -- your database) an list of the files with a marker for the job being complete as well as deleted. Your quartz task could then grab the files that are marked done but not marked deleted and clean those up. Bonus points for using transactions around the file deletes and updates and logging so you know what happened while the world was sleeping.
In order to give my application an autosave functionality, I'm looking at the best implementation that would optimise the 3 followings requirements:
safety: in order to reduce the risk of data corruption
user friendly: the user is not computer expert so the solution must be intuitive and friendly
quick to develop: I don't want to spend weeks over this implementation never
I have three solutions witch doesn't fit the 3 criteria and I'm looking for an alternative:
creating a simple shadow file so when the application crashes or the PC shutdown unexpectedly the application try to restore it
working the same way than above but storing several version of the file at different time in a temp folder
implement a true roll back system allowing the extend the undo/redo functionnality even the the application is restarted by keeping trace of the modification in a temp folder.
Does someone have anything to suggest?
For autosave, I'd simply have a background running thread that would run your Save() method silently (no popups) to a temp location (AppData system folder). You should probably keep a separate file for each session, so that you can always offer to return to a previous crashed session. On normal exit, you should delete the file to indicate the session has completed successfully.
I'd even keep 2 files for every session an alternate saving to each, so that if a crash happens during an autosave, it won't corrupt the previous autosave.
I have a text file and multiple threads/processes will write to it (it's a log file).
The file gets corrupted sometimes because of concurrent writings.
I want to use a file writing mode from all of threads which is sequential at file-system level itself.
I know it's possible to use locks (mutex for multiple processes) and synchronize writing to this file but I prefer to open the file in the correct mode and leave the task to System.IO.
Is it possible ? what's the best practice for this scenario ?
Your best bet is just to use locks/mutexex. It's a simple approach, it works and you can easily understand it and reason about it.
When it comes to synchronization it often pays to start with the simplest solution that could work and only try to refine if you hit problems.
To my knowledge, Windows doesn't have what you're looking for. There is no file handle object that does automatic synchronization by blocking all other users while one is writing to the file.
If your logging involves the three steps, open file, write, close file, then you can have your threads try to open the file in exclusive mode (FileShare.None), catch the exception if unable to open, and then try again until success. I've found that tedious at best.
In my programs that log from multiple threads, I created a TextWriter descendant that is essentially a queue. Threads call the Write or WriteLine methods on that object, which formats the output and places it into a queue (using a BlockingCollection). A separate logging thread services that queue--pulling things from it and writing them to the log file. This has a few benefits:
Threads don't have to wait on each other in order to log
Only one thread is writing to the file
It's trivial to rotate logs (i.e. start a new log file every hour, etc.)
There's zero chance of an error because I forgot to do the locking on some thread
Doing this across processes would be a lot more difficult. I've never even considered trying to share a log file across processes. Were I to need that, I would create a separate application (a logging service). That application would do the actual writes, with the other applications passing the strings to be written. Again, that ensures that I can't screw things up, and my code remains simple (i.e. no explicit locking code in the clients).
you might be able to use File.Open() with a FileShare value set to None, and make each thread wait if it can't get access to the file.