I have a logging application that works great, but I want to apply the ability to maintain the size of the log file - stop it from getting too large.
Ideally, I want to check the size of the file periodically, and if it's over the configured amount (5MB or something) delete from the beginning till it reaches some size, like 4MB.
From reading other questions I'm still unclear if I can update/delete a file without reading it's entire contents. My ideal situation would be:
if(filesize > 5MB)
{
while(filesize > 4MB)
Delete_First_X_Many_Lines(file);
}
Thankyou in advance for any pointers and direction.
I would do this:
Lock the log file (prevent writes).
Copy the end of the log file you want to keep to a new file.
Copy the new file on top of the old log file.
Unlock the log file.
Related
I have to change the specific line of the text file in asp.net.
Can I change/Replace the text in a particular line only??
I have used the replace function in text file but it is replacing text in entire file.
I want to replace only one line specified by me.
Waiting for the reply..
Thanks in advance..
File systems don't generally allow you to edit within a file other than directly overwriting byte-by-byte. If your text file uses the same number of bytes for every line, then you can very efficiently replace a line of text - but that's a relatively rare case these days.
It's more likely that you'll need to take one of these options:
Load the whole file into memory using File.ReadAllLines, change the relevant line, and then write it out again using File.WriteAllLines. This is inefficient in terms of memory, but really simple to code. If your file is small, it's a good option.
Open the input file and a new output file. Read a line of text at a time from the input, and either copying it to the output or writing a different line instead. Then close both files, delete the input file and rename the output file. This only requires a single line of text in memory at a time, but it's considerably more fiddly.
The second option has another benefit - you can shuffle the files around (using lots of rename steps) so that at no point do you ever have the possibility of losing the input file unless the output file is known to be complete and in the right place. That's even more complicated though.
I want to save some plain text periodically to a text file, and it will be really better if I can minimize the chance of corrupting the file in case the app gets terminated or the system restarts. What are the way to ensure that the plain text file is always good.
Edit
I will run the program from USB drive, so want to make sure the file is still perfect if I eject the drive without closing the App.
You could take a look at using transactions on the NTFS file system, assuming that the USB stick is formatted NTFS and your OS is Vista or better.
Do not use option FileMode.Create, which overwrites existing files. Instead, use option FileMode.Append when creating the file stream, so that any text will be appended to the file without modifying last data.
However, try not to keep the files opened for long period, just open them and read or write and then Dispose them.
Use File.AppendAllText to open, append and close in one go.
I have a binary data file that is written to from a live data stream, so it keeps on growing as stream comes. In the meanwhile, I need to open it at the same time in read-only mode to display data on my application (time series chart). Opening the whole file takes a few minutes as it is pretty large (a few 100' MBytes).
What I would like to do is, rather than re-opening/reading the whole file every x seconds, read only the last data that was added to the file and append it to the data that was already read.
I would suggest using FileSystemWatcher to be notified of changes to the file. From there, cache information such as the size of the file between events and add some logic to only respond to full lines, etc. You can use the Seek() method of the FileStream class to jump to a particular point in the file and read only from there. I hope it helps.
If you control the writing of this file, I would split it in several files of a predefined size.
When the writer determines that the current file is larger than, say, 50MB, close it and immediately create a new file to write data to. The process writing this data should always know the current file to write received data to.
The reader thread/process would read all these files in order, jumping to the next file when the current file was read completely.
You can probably use a FileSystemWatcher to monitor for changes in the file, like the example given here: Reading changes in a file in real-time using .NET.
But I'd suggest that you evaluate another solution, including a queue, like RabbitMQ, or Redis - any queue that has Subscriber-Publisher model. Then you'll just push the live data into the queue, and will have 2 different listeners(subscribers) - one to save in the file, and the other to process the last-appended data. This way you can achieve more flexibility with distributing load of the application.
I need to parse a large CSV file in real-time, while it's being modified (appended) by a different process. By large I mean ~20 GB at this point, and slowly growing. The application only needs to detect and report certain anomalies in the data stream, for which it only needs to store small state info (O(1) space).
I was thinking about polling the file's attributes (size) every couple of seconds, opening a read-only stream, seeking to the previous position, and then continuing to parse where I first stopped. But since this is a text (CSV) file, I obviously need to keep track of new-line characters when continuing somehow, to ensure I always parse an entire line.
If I am not mistaken, this shouldn't be such a problem to implement, but I wanted to know if there is a common way/library which solves some of these problems already?
Note: I don't need a CSV parser. I need info about a library which simplifies reading lines from a file which is being modified on the fly.
I did not test it, but I think you can use a FileSystemWatcher to detect when a different process modified your file. In the Changed event, you will be able to seek to a position you saved before, and read the additional content.
There is a small problem here:
Reading and parsing CSV requires a TextReader
Positioning doesn't work (well) with TextReaders.
First thought: Keep it open. If both the producer and the analyzer operate in non-exclusive mode It should be possible to ReadLine-until-null, pause, ReadLine-until-null, etc.
it should be 7-bit ASCII, just some Guids and numbers
That makes it feasible to track the file Position (pos += line.Length+2). Do make sure you open it with Encoding.ASCII. You can then re-open it as a plain binary Stream, Seek to the last position and only then attach a StreamReader to that stream.
Why don't you just spin off a separate process / thread each time you start parsing - that way, you move the concurrent (on-the-fly) part away from the data source and towards your data sink - so now you just have to figure out how to collect the results from all your threads...
This will mean doing a reread of the whole file for each thread you spin up, though...
You could run a diff program on the two versions and pick up from there, depending on how well-formed the csv data source is: Does it modify records already written? Or does it just append new records? If so, you can just split off the new stuff (last-position to current-eof) into a new file, and process those at leisure in a background thread:
polling thread remembers last file size
when file gets bigger: seek from last position to end, save to temp file
background thread processes any temp files still left, in order of creation/modification
Suppose I have a program running that periodically adds information to a .CSV file. Is there a way to write to the file while it is already open in Excel? Obviously the changes wouldn't be noticed until the file was re-opened in Excel, but as it stands right now, I'm catching IOException and just starting a new .csv file if the current one is already open.
Excel seems to open the file in exclusive mode, so the only way I can think of would be to write your changes to a temporary file, and then use FileSystemWatcher to see when the file is closed, and overwrite the file.
Not a very good idea, as you could lose data. The whole reason excel locks the file is so that you don't accidentally overwrite changes made in excel.
It sounds like the file is locked. I doubt you will be able to write to that file if it is open in another process.
As a former (and sort of current) VB Programmer, I can tell you Jared is correct - there is no way to do this directly. You can try to copy the file first, make your edits, then attempt to save the file back to its original location until the locked file becomes free. You should be able to copy that file, even while locked.
What about using Excel's object model and automating the addition of the data into the open spreadsheet? You'd probably need to prompt the user somehow to let them know what was happening.