We Keep Coding

How to monitor a text file and retrieve just the added lines? [duplicate]

I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?

More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.

You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.

To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.

This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}

You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.

private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}

If you are just looking for a tool to do this then check out free version of Bare tail

File is created once, then stops

I am trying to read from a folder, and delete a specified file from inside.
I am able to do this one time without any issues.
However after doing it the first time, I am no longer able to Create the new file. The old one is still deleted, but the new file is no longer being created as it should. My question is; From the code provided why would the task work once, but not after that? It's being deleted after the first time but won't re-create.
EDIT:
The issue was in permissions.
I changed the folder security settings to allow reading/writing and I am now able to do it as I wanted.
However, I need it to automatically set the security settings if possible as other users may not know how to do so.
if (Directory.Exists(path1))
{
if (File.Exists(path1 + "\\launchinfo.txt"))
{
File.Delete(path1 + "\\launchinfo.txt");
using (FileStream fs = File.Create(path1 + "\\launchinfo.txt"))
{
Byte[] info = new UTF8Encoding(true).GetBytes("[Connection]\n" + Form1.ipaddress + "\nport=0000\nclient_port=0\n[Details]\n" + Form1.playername);
fs.Write(info, 0, info.Length);
}
}
else
{
using (FileStream fs = File.Create(path1 + "\\launchinfo.txt"))
{
Byte[] info = new UTF8Encoding(true).GetBytes("[Connection]\n" + Form1.ipaddress + "\nport=0000\nclient_port=0\n[Details]\n" + Form1.playername);
fs.Write(info, 0, info.Length);
}
}
}

You've not posted any possible exception you may be running into - if you do have one, please post it.
That being said, it's possible that you're encountering File in use by another process when trying to delete the file - especially if you're calling your function moments after creating a file.
A method to get around this is to check if a process is using the file before you try to delete it.
string fullPath = Path.Combine(path1, "launchinfo.txt");
if (Directory.Exists(path1))
{
if (File.Exists(fullPath))
{
// Call a method to check if the file is in use.
if (IsFileLocked(new FileInfo(fullPath)){
// do something else because you can't delete the file
} else {
File.Delete(fullPath);
}
}
using (FileStream fs = File.Create(fullPath))
{
Byte[] info = new UTF8Encoding(true).GetBytes("[Connection]\n" + Form1.ipaddress + "\nport=0000\nclient_port=0\n[Details]\n" + Form1.playername);
fs.Write(info, 0, info.Length);
}
}
A method to check if the file is in use by another process
protected virtual bool IsFileLocked(FileInfo file)
{
FileStream stream = null;
try
{
stream = file.Open(FileMode.Open, FileAccess.Read, FileShare.None);
}
catch (IOException)
{
return true;
}
finally
{
if (stream != null)
stream.Close();
}
//file is not locked
return false;
}

FileStream and ObjectDisposedException in .NET CF

public byte[] GetFile(string filename)
{
FileStream aStream = File.Open(filename, FileMode.Open, FileAccess.Read);
BinaryReader binReader = new BinaryReader(aStream);
binReader.BaseStream.Position = 0;
byte[] binFile = binReader.ReadBytes(Convert.ToInt32(binReader.BaseStream.Length));
binReader.Close();
return binFile;
}
I run this method for a number of filepaths, problem is whenever a file cannot be accessed with File.Open (because it is used by another process) I get:
'aStream.Position' threw an exception of type 'System.ObjectDisposedException'
on the following line:
binReader.BaseStream.Position = 0;
And seldom I get
{System.IO.IOException: The process can not access the file '\folder\file.txt' because it is being used by another process.}
This is the exception I want. So why is the object disposed most of the times?
Note: I first had the FileStream line in a using statement but removed it because I thought that might have disposed the object. But the problem remains.
Edit: Using Compact Framework, which doesn't have ReadAllBytes.

Maybe that part of the time your FileStream is throwing the IOException whenever your file is in use, and at other times, perhaps, you are getting the ObjectDisposedException because your array is not initialized.
Obviously, I can not test this theory.
See if you can copy-n-paste this one with good results:
public byte[] GetFile(string filename)
{
byte[] binFile = null;
try
{
using (var aStream = File.Open(filename, FileMode.Open, FileAccess.Read))
{
BinaryReader binReader = new BinaryReader(aStream);
binFile = new byte[binReader.BaseStream.Length];
binReader.BaseStream.Position = 0; // <= this step should not be necessary
binFile = binReader.ReadBytes(binReader.BaseStream.Length);
binReader.Close();
}
} catch (IOException err) {
// file is being used by another process.
} catch (ObjectDisposedException err) {
// I am guessing you would never see this because your binFile is not disposed
}
return binFile;
}
Be sure to check for null return variables!
EDIT:
I wrote (what I think is) a simpler version. I tested it, and it seems to work OK. I also prefer Read() overload to ReadBytes(), because I know how much data was pulled in.
First, is the test function that calls the method for every image in my Pictures folder:
public void Test() {
DirectoryInfo dir = new DirectoryInfo(Environment.GetFolderPath(Environment.SpecialFolder.Personal));
foreach (var subDir in dir.GetDirectories()) {
if (-1 < subDir.Name.ToLower().IndexOf("pictures")) {
foreach (var file in subDir.GetFiles()) {
byte[] data = GetFile(file.FullName);
if (data != null) {
Console.WriteLine(data.Length);
}
}
}
}
}
public byte[] GetFile(string filename) {
byte[] result = null;
try {
if (File.Exists(filename)) {
int len = 0;
FileInfo file = new FileInfo(filename);
byte[] data = new byte[file.Length];
using (BinaryReader br = new BinaryReader(file.Open(FileMode.Open, FileAccess.Read))) {
len = br.Read(data, 0, data.Length);
br.Close();
}
if (0 < len) {
if (len == data.Length) {
return data;
} else {
// this section of code was never triggered in my tests;
// however, it is good to keep it as a backup.
byte[] dat2 = new byte[len];
Array.Copy(data, dat2, len);
return dat2;
}
}
}
} catch (IOException err) {
// file is being used by another process.
} catch (ObjectDisposedException err) {
// I am guessing you would never see this because your binFile is not disposed
}
return result;
}
I don't see any reason why these would not work - unless you are having an int overflow.

Just use this:
byte[] contents = File.ReadAllBytes(filename);

Why don't you simply use
public byte[] GetFile(string filename)
{
try { return File.ReadAllBytes(filename); }
catch { return null; }
}
Just for fun, you could even define an extension method
public static class Extensions
{
public static byte[] GetFile(this string filename)
{
try { return File.ReadAllBytes(filename); }
catch { return null; }
}
}
so you could do byte[] myfile = filename.GetFile();.
Remember you must check that return is not null before proceed:
if (myfile != null)
{
// Do what you need
}

How to join 2 or more .WAV files together programmatically?

I need the ability to join 2 or more .wav files together in to one .wav file. I must do this programmatically, using C# (3rd-party products are not an option). I know of the System.Media.SoundPlayer class, but I am not looking to play the the .wav, but only to create it.

Here's a basic WAV concatenation function built using NAudio. This will ensure that only the data chunks are concatenated (unlike the code example in this CodeProject article linked in another answer). It will also protect you against concatenating WAV files that do not share the same format.
public static void Concatenate(string outputFile, IEnumerable<string> sourceFiles)
{
byte[] buffer = new byte[1024];
WaveFileWriter waveFileWriter = null;
try
{
foreach (string sourceFile in sourceFiles)
{
using (WaveFileReader reader = new WaveFileReader(sourceFile))
{
if (waveFileWriter == null)
{
// first time in create new Writer
waveFileWriter = new WaveFileWriter(outputFile, reader.WaveFormat);
}
else
{
if (!reader.WaveFormat.Equals(waveFileWriter.WaveFormat))
{
throw new InvalidOperationException("Can't concatenate WAV Files that don't share the same format");
}
}
int read;
while ((read = reader.Read(buffer, 0, buffer.Length)) > 0)
{
waveFileWriter.WriteData(buffer, 0, read);
}
}
}
}
finally
{
if (waveFileWriter != null)
{
waveFileWriter.Dispose();
}
}
}

One comment on Mark's answer:
The == operator does not seem to work for me when comparing wave formats. It's safer to do this:
if (!reader.WaveFormat.Equals(waveFileWriter.WaveFormat))
Alternatively, you could wrap the reader in a WaveFormatConversionStream and get rid of the format check altogether (not sure if it will work on all scenarios but I was able to succesfully test it).

Check out this codeproject example, seems to be exactly what you need with a good explanation of how to do it too:
Concatenating Wave Files Using C# 2005
It seems to comprise essentially of extracting and merging the sound data from all the wav files into one chunk of data with a new file header on top
EDIT: I have no experience of using this, nor am I an expert. I just came across this article and thought it may be useful. See Mark Heath's answer for a better solution

Use from How to join .Wav files together
private void JoinWav()
{
string[] files = new string[] { "1990764-ENG-CONSEC-RESPONSE7.WAV","1990764-ND_A.WAV", "1990764-SIGHT-SP.WAV",
"1990764-SP-CONSEC-RESPONSE6.WAV","1990764-VOCABWORD-004-12-SP.WAV","bi-consec-1-successful.wav",
"bi-transition-instruct.wav","nd_B.wav","sightreceived_B.wav","teststamp_A.wav" };
AudioCompressionManager.Join("res.wav", files);
}

If you need get only byte array, to insert in database or somebody else. You may use memory stream:
public static byte[] Concatenate(IEnumerable<byte[]> sourceData)
{
var buffer = new byte[1024 * 4];
WaveFileWriter waveFileWriter = null;
using (var output = new MemoryStream())
{
try
{
foreach (var binaryData in sourceData)
{
using (var audioStream = new MemoryStream(binaryData))
{
using (WaveFileReader reader = new WaveFileReader(audioStream))
{
if (waveFileWriter == null)
waveFileWriter = new WaveFileWriter(output, reader.WaveFormat);
else
AssertWaveFormat(reader, waveFileWriter);
WaveStreamWrite(reader, waveFileWriter, buffer);
}
}
}
waveFileWriter.Flush();
return output.ToArray();
}
finally
{
waveFileWriter?.Dispose();
}
}
}
private static void AssertWaveFormat(WaveFileReader reader, WaveFileWriter writer)
{
if (!reader.WaveFormat.Equals(writer.WaveFormat))
{
throw new InvalidOperationException("Can't concatenate WAV Files that don't share the same format");
}
}
private static void WaveStreamWrite(WaveFileReader reader, WaveFileWriter writer, byte[] buffer)
{
int read;
while ((read = reader.Read(buffer, 0, buffer.Length)) > 0)
{
writer.Write(buffer, 0, read);
}
}

C#: Prepending to beginning of a file

What is the best way to add text to the beginning of a file using C#?
I couldn't find a straightforward way to do this, but came up with a couple work-arounds.
Open up new file, write the text that I wanted to add, append the text from the old file to the end of the new file.
Since the text I want to add should be less than 200 characters, I was thinking that I could add white space characters to the beginning of the file, and then overwrite the white space with the text I wanted to add.
Has anyone else come across this problem, and if so, what did you do?

This works for me, but for small files. Probably it's not a very good solution otherwise.
string currentContent = String.Empty;
if (File.Exists(filePath))
{
currentContent = File.ReadAllText(filePath);
}
File.WriteAllText(filePath, newContent + currentContent );

Adding to the beginning of a file (prepending as opposed to appending) is generally not a supported operation. Your #1 options is fine. If you can't write a temp file, you can pull the entire file into memory, preprend your data to the byte array and then overwrite it back out (this is only really feasible if your files are small and you don't have to have a bunch in memory at once because prepending the array is not necessarily easy without a copy either).

Yeah, basically you can use something like this:
public static void PrependString(string value, FileStream file)
{
var buffer = new byte[file.Length];
while (file.Read(buffer, 0, buffer.Length) != 0)
{
}
if(!file.CanWrite)
throw new ArgumentException("The specified file cannot be written.", "file");
file.Position = 0;
var data = Encoding.Unicode.GetBytes(value);
file.SetLength(buffer.Length + data.Length);
file.Write(data, 0, data.Length);
file.Write(buffer, 0, buffer.Length);
}
public static void Prepend(this FileStream file, string value)
{
PrependString(value, file);
}
Then
using(var file = File.Open("yourtext.txt", FileMode.Open, FileAccess.ReadWrite))
{
file.Prepend("Text you want to write.");
}
Not really effective though in case of huge files.

using two streams, you can do it in place, but keep in mind that this will still loop over the whole file on every addition
using System;
using System.IO;
using System.Text;
namespace FilePrepender
{
public class FilePrepender
{
private string file=null;
public FilePrepender(string filePath)
{
file = filePath;
}
public void prependline(string line)
{
prepend(line + Environment.NewLine);
}
private void shiftSection(byte[] chunk,FileStream readStream,FileStream writeStream)
{
long initialOffsetRead = readStream.Position;
long initialOffsetWrite= writeStream.Position;
int offset = 0;
int remaining = chunk.Length;
do//ensure that the entire chunk length gets read and shifted
{
int read = readStream.Read(chunk, offset, remaining);
offset += read;
remaining -= read;
} while (remaining > 0);
writeStream.Write(chunk, 0, chunk.Length);
writeStream.Seek(initialOffsetWrite, SeekOrigin.Begin);
readStream.Seek(initialOffsetRead, SeekOrigin.Begin);
}
public void prepend(string text)
{
byte[] bytes = Encoding.Default.GetBytes(text);
byte[] chunk = new byte[bytes.Length];
using (FileStream readStream = File.Open(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using(FileStream writeStream = File.Open(file, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
{
readStream.Seek(0, SeekOrigin.End);//seek chunk.Length past the end of the file
writeStream.Seek(chunk.Length, SeekOrigin.End);//which lets the loop run without special cases
long size = readStream.Position;
//while there's a whole chunks worth above the read head, shift the file contents down from the end
while(readStream.Position - chunk.Length >= 0)
{
readStream.Seek(-chunk.Length, SeekOrigin.Current);
writeStream.Seek(-chunk.Length, SeekOrigin.Current);
shiftSection(chunk, readStream, writeStream);
}
//clean up the remaining shift for the bytes that don't fit in size%chunk.Length
readStream.Seek(0, SeekOrigin.Begin);
writeStream.Seek(Math.Min(size, chunk.Length), SeekOrigin.Begin);
shiftSection(chunk, readStream, writeStream);
//finally, write the text you want to prepend
writeStream.Seek(0,SeekOrigin.Begin);
writeStream.Write(bytes, 0, bytes.Length);
}
}
}
}
}

I think the best way is to create a temp file. Add your text then read the contents of the original file adding it to the temp file. Then you can overwrite the original with the temp file.

prepend:
private const string tempDirPath = #"c:\temp\log.log", tempDirNewPath = #"c:\temp\log.new";
StringBuilder sb = new StringBuilder();
...
File.WriteAllText(tempDirNewPath, sb.ToString());
File.AppendAllText(tempDirNewPath, File.ReadAllText(tempDirPath));
File.Delete(tempDirPath);
File.Move(tempDirNewPath, tempDirPath);
using (FileStream fs = File.OpenWrite(tempDirPath))
{ //truncate to a reasonable length
if (16384 < fs.Length) fs.SetLength(16384);
fs.Close();
}

// The file we'll prepend to
string filePath = path + "\\log.log";
// A temp file we'll write to
string tempFilePath = path + "\\temp.log";
// 1) Write your prepended contents to a temp file.
using (var writer = new StreamWriter(tempFilePath, false))
{
// Write whatever you want to prepend
writer.WriteLine("Hi");
}
// 2) Use stream lib methods to append the original contents to the Temp
// file.
using (var oldFile = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Read, FileShare.Read))
{
using (var tempFile = new FileStream(tempFilePath, FileMode.Append, FileAccess.Write, FileShare.Read))
{
oldFile.CopyTo(tempFile);
}
}
// 3) Finally, dump the Temp file back to the original, keeping all its
// original permissions etc.
File.Replace(tempFilePath, filePath, null);
Even if what you're writing is small, the Temp file gets the entire original file appended to it before the .Replace(), so it does need to be on disk.
Note that this code is not Thread-safe; if more than one thread accesses this code you can lose writes in the file swapping going on here. That said, it's also pretty expensive, so you'd want to gate access to it anyway - pass writes via multiple Providers to a buffer, which periodically empties out via this prepend method on a single Consumer thread.

You should be able to do this without opening a new file. Use the following File method:
public static FileStream Open(
string path,
FileMode mode,
FileAccess access
)
Making sure to specify FileAccess.ReadWrite.
Using the FileStream returned from File.Open, read all of the existing data into memory. Then reset the pointer to the beginning of the file, write your new data, then write the existing data.
(If the file is big and/or you're suspicious of using too much memory, you can do this without having to read the whole file into memory, but implementing that is left as an exercise to the reader.)

The following algorithm may solve the problem pretty easily, it's most efficient for any size of file, including very big text files:
string outPutFile = #"C:\Output.txt";
string result = "Some new string" + DateTime.Now.ToString() + Environment.NewLine;
StringBuilder currentContent = new StringBuilder();
List<string> rawList = File.ReadAllLines(outPutFile).ToList();
foreach (var item in rawList) {
currentContent.Append(item + Environment.NewLine);
}
File.WriteAllText(outPutFile, result + currentContent.ToString());

Use this class:
public static class File2
{
private static readonly Encoding _defaultEncoding = new UTF8Encoding(false, true); // encoding used in File.ReadAll*()
private static object _bufferSizeLock = new Object();
private static int _bufferSize = 1024 * 1024; // 1mb
public static int BufferSize
{
get
{
lock (_bufferSizeLock)
{
return _bufferSize;
}
}
set
{
lock (_bufferSizeLock)
{
_bufferSize = value;
}
}
}
public static void PrependAllLines(string path, IEnumerable<string> contents)
{
PrependAllLines(path, contents, _defaultEncoding);
}
public static void PrependAllLines(string path, IEnumerable<string> contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllLines(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
public static void PrependAllText(string path, string contents)
{
PrependAllText(path, contents, _defaultEncoding);
}
public static void PrependAllText(string path, string contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllText(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
private static void AppendToTemp(string path, string temp, Encoding encoding)
{
var bufferSize = BufferSize;
char[] buffer = new char[bufferSize];
using (var writer = new StreamWriter(temp, true, encoding))
{
using (var reader = new StreamReader(path, encoding))
{
int bytesRead;
while ((bytesRead = reader.ReadBlock(buffer,0,bufferSize)) != 0)
{
writer.Write(buffer,0,bytesRead);
}
}
}
}
}

Put the file's contents in a string. Append new data you want to add to the top of the file to that string -- string = newdata + string. Then move the seek position of the file to 0 and write the string into the file.