Since this can't be solved so easily how can I implement 1 thread that writes strings to a file / buffer line by line using Console.WriteLine() and another thread that reads those strings from the same file / buffer also line by line ? I guess I need to:
redirect Console to file / buffer
read file / buffer thread save, when a line is written it must be read by the other thread
make that asynchronous (no ReadToEnd(), it must be live)
try Memory-Mapped files it will allow you to read write to one shared file from multiple threads. As to redirecting console try:
Console.SetIn
Console.SetOut
I would like to do that with a buffer though.
Solution with file:
class Program
{
private static bool terminated = false;
private static void listen()
{
StreamReader file = new StreamReader(new FileStream("C:/test.txt", FileMode.Open, FileAccess.Read, FileShare.ReadWrite));
while (!terminated || !file.EndOfStream)
if (!file.EndOfStream)
{
string text = file.ReadLine();
MessageBox.Show(text); // display it
}
}
static void Main(string[] args)
{
StreamWriter sw = new StreamWriter(new FileStream("C:/test.txt", FileMode.Create, FileAccess.Write, FileShare.Read));
sw.AutoFlush = true;
Console.SetOut(sw);
new Thread(new ThreadStart(listen)).Start();
for (int i = 0; i < 10; i++)
{
Thread.Sleep(250);
Console.Out.WriteLine("hello world - " + i);
}
terminated = true;
}
}
Works line by line and doesn't miss one.
Related
I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?
More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.
You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.
To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.
This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}
You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.
private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}
If you are just looking for a tool to do this then check out free version of Bare tail
I am trying to replace CRLF with LF (see reason # Read binary stdout data from adb shell?)
A brief summary of the above referenced thread is that basically when piping a screenshot from an Android device via the Android debug bridge, it looks like the line feeds line endings in the data stream are being replaced with carriage return line feeds, therefore I am receiving a corrupted file at the other end of the pipe. What has worked for others is undoing the replacement via code, as below, but it doesn't seem to be working from me.
My code is still spitting a corrupted file... any ideas why?
++ I know the code isn't as clean and efficient as it can be, will fix up after so please hold the comments related to my coding skill, or lack thereof.
Thanks
static void Main(string[] args)
{
// adb shell screencap -p > <path>
string path = #"<filepath>\screen.png";
var fs = new FileStream(path, FileMode.Open);
var fsw = new FileStream(path.Replace(".png", "_fixed.png"), FileMode.Create);
var buffer = new byte[fs.Length];
fs.Read(buffer, 0, buffer.Length);
fs.Flush();
fs.Close();
var switched = Repair(buffer);
fsw.Write(switched, 0, switched.Length);
fsw.Flush();
fsw.Close();
Console.WriteLine(buffer.Length);
Console.WriteLine(switched.Length);
Console.Read();
}
static byte[] Repair(byte[] enc)
{
var bstr = new MemoryStream();
for (int i = 0; i < enc.Length; i++)
{
if (enc.Length > i + 1 && enc[i] == 0x0d && enc[i + 1] == 0x0a)
{
bstr.WriteByte(0x0a);
i++;
}
else bstr.WriteByte(enc[i]);
}
bstr.Flush();
bstr.Close();
return bstr.ToArray();
}
I have multiple servers created.Each one has to send some data to its own client. I am using TCP/IP protocol. To prevent any data loss due to client getting disconnected, I am using a text file as a buffer. So in the program , there is a thread for each server which keeps on checking if the client is connected or not. If it is connected then it reads from the buffer and sends it to client. whenever some new data has to be send to client , I am first checking if client is connected.If client isn't connected then I am writing data to the same buffer text file. The problem I am facing is that I am unable to write to the file while thread is reading from it.
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
namespace WindowsFormsApplication1
{
public class TcpIp
{
public int machinePort;
public static int port1 = 1024;
public static int count = 0;
public string bufferName;
FileStream buffer;
Socket client;
public IPAddress localIp;
public TcpListener sender;
StreamReader reader ;
FileStream iStream;
//this.get
public TcpIp(string id)
{
this.machinePort = port1 + count;
while (!isAvailable(this.machinePort))
{
count++;
this.machinePort = port1 + count;
}
this.bufferName = WindowsFormsApplication1.Program.path + "machine_" + id + ".txt";
buffer = new FileStream(this.bufferName, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite);
localIp = IPAddress.Parse(WindowsFormsApplication1.Program.ip);
sender = new TcpListener(localIp, this.machinePort);
// this.oStream = new FileStream(this.bufferName, FileMode.Append, FileAccess.Write, FileShare.ReadWrite);
//this.iStream = new FileStream(this.bufferName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
// reader = new StreamReader(this.iStream);
}
bool isAvailable(int port)
{
bool isAvailable = true;
IPGlobalProperties ipGlobalProperties = IPGlobalProperties.GetIPGlobalProperties();
TcpConnectionInformation[] tcpConnInfoArray = ipGlobalProperties.GetActiveTcpConnections();
foreach (TcpConnectionInformation tcpi in tcpConnInfoArray)
{
if (tcpi.LocalEndPoint.Port == port)
{
isAvailable = false;
break;
}
}
return isAvailable;
}
public void createServer()
{
this.sender.Start();
string line;
reader = new StreamReader(buffer);
//client = sender.AcceptSocket();
while (true)
{
line = reader.ReadLine();
if (!connected())
{
client = sender.AcceptSocket();
}
while (reader.EndOfStream && line != null)
{
byte[] bytes = System.Text.Encoding.UTF8.GetBytes(line);
client.Send(bytes, 0, bytes.Length, SocketFlags.None);
line = reader.ReadLine();
}
// iStream.Flush();
Thread.Sleep(3000);
//reader = new StreamReader(iStream);
}
}
public void writeToClient(string data)
{
if (connected())
{
//send data to client
byte[] bytes = System.Text.Encoding.UTF8.GetBytes(data);
//System.Buffer.BlockCopy(data.ToCharArray(), 0, bytes, 0, bytes.Length);
this.client.Send(bytes, 0, bytes.Length, SocketFlags.None);
}
else
{
//write to file
while (true)
{
try
{
StreamWriter sw = File.AppendText(this.bufferName);
sw.WriteLine(data);
sw.Close();
break;
}
catch (Exception ex)
{
Console.WriteLine("WaitForFile {0} failed to get an exclusive lock: "+ex.Message );
// Wait for the lock to be released
System.Threading.Thread.Sleep(500);
}
}
}
}
bool connected()
{
if (client == null)
return false;
else
return client.Connected;
}
}
}
Any enlightenment would be appreciated. Thank you :)
The actual problem is that you mix up access to the file.
You open a stream by buffer = new FileStream(this.bufferName, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite); in order to create a StreamReader from the opened stream for later reading by reader = new StreamReader(buffer);.
OTOH, you want get a StreamWriter for writing to the file by StreamWriter sw = File.AppendText(this.bufferName);. This tries to open the file again which fails because of mismatching file sharing mode.
So you need to access the file for writing and reading via the very same "handle" (here FileStream). Furthermore, don't forget to serialize access by some locking mechanism in order to make it thread-safe. Otherwise, you'll get corrupted data. You'll probably need to maintain the read/write pointer (Stream.Position).
You can't read and write from the same text file at the same time.
If you really want to use text files, why not using 2 of them:
One to read from and one to write into.
Once the read file is empty -> the read file is your new write file and vice versa.
That sounds logic, it's not possible for 2 threads to access a file as the same time, because the file is in use.
Imagine that it's even possible, you will have some very strange behaviour.
Why are you not using a lock() to make sure that only a single thread can access the file at a given time?
And with async programming, you don't need to wait until the lock is released before continueing your program.
You will have to create Threads like this
Thread thread = new Thread(yourMethod());
When you have multiple threads then what you need is ReaderWriterLockSlim. Here is the link.
It lets you to read a file by multiple threads but write the file with one Thread. I think this will solve your problem.
I want to read a CSV file which can be at a size of hundreds of GBs and even TB. I got a limitation that I can only read the file in chunks of 32MB. My solution to the problem, not only does it work kinda slow, but it can also break a line in the middle of it.
I wanted to ask if you know of a better solution:
const int MAX_BUFFER = 33554432; //32MB
byte[] buffer = new byte[MAX_BUFFER];
int bytesRead;
using (FileStream fs = File.Open(filePath, FileMode.Open, FileAccess.Read))
using (BufferedStream bs = new BufferedStream(fs))
{
string line;
bool stop = false;
while ((bytesRead = bs.Read(buffer, 0, MAX_BUFFER)) != 0) //reading only 32mb chunks at a time
{
var stream = new StreamReader(new MemoryStream(buffer));
while ((line = stream.ReadLine()) != null)
{
//process line
}
}
}
Please do not respond with a solution which reads the file line by line (for example File.ReadLines is NOT an acceptable solution). Why? Because I'm just searching for another solution...
The problem with your solution is that you recreate the streams in each iteration. Try this version:
const int MAX_BUFFER = 33554432; //32MB
byte[] buffer = new byte[MAX_BUFFER];
int bytesRead;
StringBuilder currentLine = new StringBuilder();
using (FileStream fs = File.Open(filePath, FileMode.Open, FileAccess.Read))
using (BufferedStream bs = new BufferedStream(fs))
{
string line;
bool stop = false;
var memoryStream = new MemoryStream(buffer);
var stream = new StreamReader(memoryStream);
while ((bytesRead = bs.Read(buffer, 0, MAX_BUFFER)) != 0)
{
memoryStream.Seek(0, SeekOrigin.Begin);
while (!stream.EndOfStream)
{
line = ReadLineWithAccumulation(stream, currentLine);
if (line != null)
{
//process line
}
}
}
}
private string ReadLineWithAccumulation(StreamReader stream, StringBuilder currentLine)
{
while (stream.Read(buffer, 0, 1) > 0)
{
if (charBuffer [0].Equals('\n'))
{
string result = currentLine.ToString();
currentLine.Clear();
if (result.Last() == '\r') //remove if newlines are single character
{
result = result.Substring(0, result.Length - 1);
}
return result;
}
else
{
currentLine.Append(charBuffer [0]);
}
}
return null; //line not complete yet
}
private char[] charBuffer = new char[1];
NOTE: This needs some tweaking if newlines are two characters long and you need newline characters to be contained in the result. The worst case would be newline pair "\r\n" split across two blocks. However since you were using ReadLine I assumed that you don't need this.
Also, the problem is that in case your whole data contains only one line, this will end up in an attempt to read the whole data into memory anyway.
which can be at a size of hundreds of GBs and even TB
For a large file processing the most suitable class recommended is MemoryMappedFile Class
Some advantages:
It is ideal to access a data file on disk without performing file I/O operations and from buffering the file’s content. This works great when you deal with large data files.
You can use memory mapped files to allow multiple processes running on the same machine to share data with each other.
so try it and you will note the difference as swapping between memory and harddisk is a time consuming operation
I want to read file continuously like GNU tail with "-f" param. I need it to live-read log file.
What is the right way to do it?
More natural approach of using FileSystemWatcher:
var wh = new AutoResetEvent(false);
var fsw = new FileSystemWatcher(".");
fsw.Filter = "file-to-read";
fsw.EnableRaisingEvents = true;
fsw.Changed += (s,e) => wh.Set();
var fs = new FileStream("file-to-read", FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
using (var sr = new StreamReader(fs))
{
var s = "";
while (true)
{
s = sr.ReadLine();
if (s != null)
Console.WriteLine(s);
else
wh.WaitOne(1000);
}
}
wh.Close();
Here the main reading cycle stops to wait for incoming data and FileSystemWatcher is used just to awake the main reading cycle.
You want to open a FileStream in binary mode. Periodically, seek to the end of the file minus 1024 bytes (or whatever), then read to the end and output. That's how tail -f works.
Answers to your questions:
Binary because it's difficult to randomly access the file if you're reading it as text. You have to do the binary-to-text conversion yourself, but it's not difficult. (See below)
1024 bytes because it's a nice convenient number, and should handle 10 or 15 lines of text. Usually.
Here's an example of opening the file, reading the last 1024 bytes, and converting it to text:
static void ReadTail(string filename)
{
using (FileStream fs = File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
// Seek 1024 bytes from the end of the file
fs.Seek(-1024, SeekOrigin.End);
// read 1024 bytes
byte[] bytes = new byte[1024];
fs.Read(bytes, 0, 1024);
// Convert bytes to string
string s = Encoding.Default.GetString(bytes);
// or string s = Encoding.UTF8.GetString(bytes);
// and output to console
Console.WriteLine(s);
}
}
Note that you must open with FileShare.ReadWrite, since you're trying to read a file that's currently open for writing by another process.
Also note that I used Encoding.Default, which in US/English and for most Western European languages will be an 8-bit character encoding. If the file is written in some other encoding (like UTF-8 or other Unicode encoding), It's possible that the bytes won't convert correctly to characters. You'll have to handle that by determining the encoding if you think this will be a problem. Search Stack overflow for info about determining a file's text encoding.
If you want to do this periodically (every 15 seconds, for example), you can set up a timer that calls the ReadTail method as often as you want. You could optimize things a bit by opening the file only once at the start of the program. That's up to you.
To continuously monitor the tail of the file, you just need to remember the length of the file before.
public static void MonitorTailOfFile(string filePath)
{
var initialFileSize = new FileInfo(filePath).Length;
var lastReadLength = initialFileSize - 1024;
if (lastReadLength < 0) lastReadLength = 0;
while (true)
{
try
{
var fileSize = new FileInfo(filePath).Length;
if (fileSize > lastReadLength)
{
using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
fs.Seek(lastReadLength, SeekOrigin.Begin);
var buffer = new byte[1024];
while (true)
{
var bytesRead = fs.Read(buffer, 0, buffer.Length);
lastReadLength += bytesRead;
if (bytesRead == 0)
break;
var text = ASCIIEncoding.ASCII.GetString(buffer, 0, bytesRead);
Console.Write(text);
}
}
}
}
catch { }
Thread.Sleep(1000);
}
}
I had to use ASCIIEncoding, because this code isn't smart enough to cater for variable character lengths of UTF8 on buffer boundaries.
Note: You can change the Thread.Sleep part to be different timings, and you can also link it with a filewatcher and blocking pattern - Monitor.Enter/Wait/Pulse. For me the timer is enough, and at most it only checks the file length every second, if the file hasn't changed.
This is my solution
static IEnumerable<string> TailFrom(string file)
{
using (var reader = File.OpenText(file))
{
while (true)
{
string line = reader.ReadLine();
if (reader.BaseStream.Length < reader.BaseStream.Position)
reader.BaseStream.Seek(0, SeekOrigin.Begin);
if (line != null) yield return line;
else Thread.Sleep(500);
}
}
}
so, in your code you can do
foreach (string line in TailFrom(file))
{
Console.WriteLine($"line read= {line}");
}
You could use the FileSystemWatcher class which can send notifications for different events happening on the file system like file changed.
private void button1_Click(object sender, EventArgs e)
{
if (folderBrowserDialog.ShowDialog() == DialogResult.OK)
{
path = folderBrowserDialog.SelectedPath;
fileSystemWatcher.Path = path;
string[] str = Directory.GetFiles(path);
string line;
fs = new FileStream(str[0], FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
tr = new StreamReader(fs);
while ((line = tr.ReadLine()) != null)
{
listBox.Items.Add(line);
}
}
}
private void fileSystemWatcher_Changed(object sender, FileSystemEventArgs e)
{
string line;
line = tr.ReadLine();
listBox.Items.Add(line);
}
If you are just looking for a tool to do this then check out free version of Bare tail