Assume we have given an API function f(Stream s) to put binary data contained in a stream into a database. I want to put a file into the database using f but I want to compress the data in advance. Hence I thought I could do the following:
var fileStream= File.OpenRead(path);
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
f(dstream);
But it seems DeflateStream only writes into the stream fileStream but does not read from it when compressing. In all examples I found, the CopyTo method of the stream was used to compress or decompress. But this would mean that I have to keep a copy of the compressed data in memory before passing it to f for instance like this:
var memoryStream = new MemoryStream();
using(var fileStream= File.OpenRead(path))
using(var dstream = new DeflateStream(memoryStream, CompressionLevel.Optimal)) {
fileStream.CopyTo(dstream);
memoryStream.Seek(0, SeekOrigin.Begin);
f(memoryStream);
}
Is there any way to avoid using the MemoryStream?
Update
For the sake of the persistency of some commentators I add a complete example:
using System;
using System.IO;
using System.IO.Compression;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
dummyDatabaseClient.f(dstream);
}
}
The read operation in the dummy implementation of f throws an exception: InvalidOperationException: Reading from the compression stream is not supported. Concluding the discussion in the comments, I assume that the desired behaviour is not possible with DeflateStream but there are alternatives in third party libraries.
You can use SharpCompress for this. Its DeflateStream allows you to read the compressed data on the fly, which is exactly what you want.
Here's a complete example based on Sir Rufo's:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
using System.Linq;
public class Program
{
public static void Main()
{
var dataBuffer = Enumerable.Range(1, 50000).Select(e => (byte)(e % 256)).ToArray();
using (var dataStream = new MemoryStream(dataBuffer))
{
// Note: this refers to SharpCompress.Compressors.Deflate.DeflateStream
using (var deflateStream = new DeflateStream(dataStream, CompressionMode.Compress))
{
ConsumeStream(deflateStream);
}
}
}
public static void ConsumeStream(Stream stream)
{
// Let's just prove we can reinflate to the original data...
byte[] data;
using (var decompressed = new MemoryStream())
{
using (var decompressor = new DeflateStream(stream, CompressionMode.Decompress))
{
decompressor.CopyTo(decompressed);
}
data = decompressed.ToArray();
}
Console.WriteLine("Reinflated size: " + data.Length);
int errors = 0;
for (int i = 0; i < data.Length; i++)
{
if (data[i] != (i + 1) % 256)
{
errors++;
}
}
Console.WriteLine("Total errors: " + errors);
}
}
Or using your sample code:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(
fileStream, CompressionMode.Compress, CompressionLevel.BestCompression))
dummyDatabaseClient.f(dstream);
}
}
This now doesn't throw an exception, and will serve the compressed data.
The DeflateStream is just a wrapper and needs a stream for the compressed data. So you have to use two streams.
Is there any way to avoid using the MemoryStream?
Yes.
You need a stream to store temporary data without consuming (too much) memory. Instead using MemoryStream you can use a temporary file for that.
For the lazy people (like me in first place) let's create a class that will behave mostly like a MemoryStream
public class TempFileStream : FileStream
{
public TempFileStream() : base(
path: Path.GetTempFileName(),
mode: FileMode.Open,
access: FileAccess.ReadWrite,
share: FileShare.None,
bufferSize: 4096,
options: FileOptions.DeleteOnClose | FileOptions.Asynchronous | FileOptions.Encrypted | FileOptions.RandomAccess)
{
}
}
The important part here is FileOptions.DeleteOnClose which will remove the temporary file when you dispose the stream.
And then use it
using (var compressedStream = new TempFileStream())
{
using (var deflateStream = new DeflateStream(
stream: compressedStream,
compressionLevel: CompressionLevel.Optimal,
leaveOpen: true))
using (var fileStream = File.OpenRead(path))
{
fileStream.CopyTo(deflateStream);
}
f(compressedStream);
}
I have a method which will convert any file encoding to Windows(1252).
But it is not converting when the file is created in UNIX environment (with UTF-8 without BOM Encoding) and obviously file does not have any BOM. It happens only when this scenario occures.
This is my code.
public static void ConvertFileEncoding(string srcFile, Encoding srcEncoding, string tempFile)
{
using (var reader = new StreamReader(srcFile,srcEncoding,false))
using (var writer = new StreamWriter(System.IO.File.Open(tempFile, FileMode.Create), Encoding.GetEncoding(1252)))
{
char[] buf = new char[1024];
while (true)
{
int count = reader.Read(buf, 0, buf.Length);
if (count == 0)
{
break;
}
writer.Write(buf, 0, count);
}
}
System.IO.File.Copy(tempFile, srcFile, true); // Source file is replaced with Temp file
DeleteTempFile(tempFile);
// TO DO -- Log Sucess Details
}
Can anyone suggest better way of doing this.
I think in your case you can simply change...
using (var reader = new StreamReader(srcFile,srcEncoding,false))
{
...
}
TO...
using (var reader = new StreamReader(srcFile))
{
...
}
It's not required to define the encoding on StreamReader, it works very well without.
Another point is that there are unhandled exceptions in your code; and useless overwrites in some cases.
I created a similar function some time ago. I modified it a bit to share it here as example:
namespace Example
{
using System;
using System.IO;
using System.Text;
using System.Security.Cryptography;
public static class TextEx
{
public static void ChangeEncoding(string srcFile, Encoding encoding)
{
if (srcFile == null)
throw new ArgumentNullException(nameof(srcFile));
if (encoding == null)
throw new ArgumentNullException(nameof(encoding));
if (!File.Exists(srcFile))
throw new FileNotFoundException();
// get the current encoding of the source file and cancel if it equals
// with the specified encoding
if (encoding.Equals(GetEncoding(srcFile)))
return;
// creates the destination file on the same location to be sure we
// have rights to write there
var newFile = string.Concat(srcFile, ".new");
File.Create(newFile).Close();
// read the source file and write to the destination file
using (var sr = new StreamReader(srcFile))
{
var ca = new char[4096];
using (var sw = new StreamWriter(newFile, true, encoding))
{
int i;
while ((i = sr.Read(ca, 0, ca.Length)) > 0)
sw.Write(ca, 0, i);
}
}
// generate the hashcode of the source file for comparison
var srcHash = GetFileHash(srcFile);
// generate the hashcode of the finished destination file and compare
// it with the hashcode of the source file
var newHash = GetFileHash(newFile);
if (srcHash.Equals(newHash))
{
// delete the destination file if there are no changes inside
File.Delete(newFile);
return;
}
// overwrite the file if it is different
File.Delete(srcFile);
File.Move(newFile, srcFile);
}
public static Encoding GetEncoding(string file)
{
if (string.IsNullOrEmpty(file))
throw new ArgumentNullException(nameof(file));
if (!File.Exists(file))
throw new FileNotFoundException();
Encoding encoding;
using (var sr = new StreamReader(file, true))
{
sr.Peek();
encoding = sr.CurrentEncoding;
}
return encoding;
}
public static string GetFileHash(string file)
{
if (string.IsNullOrEmpty(file))
throw new ArgumentNullException(nameof(file));
if (!File.Exists(file))
throw new FileNotFoundException();
string hash;
using (var md5 = MD5.Create())
using (var fs = File.OpenRead(file))
{
var bytes = md5.ComputeHash(fs);
hash = BitConverter.ToString(bytes);
}
return hash;
}
}
}
Please note that there is an unhandled exception if the source file is locked by another process. But I think the solution that I have created is too overkill to share it here.
USAGE:
Example.TextEx.ChangeEncoding("C:\\example.txt", Encoding.GetEncoding(1252));
A friend of mine gave me the challenge to decompress an assembly that was packed with Fody.Costura. The assembly has a dll dependency that was embedded as a resource. I tried to extract this .zip resource with dotPeek and decompress it with this code here
public static void Decompress(string path)
{
using (var stream = File.OpenRead(path))
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress))
{
compressStream.Seek(0, SeekOrigin.Begin);
var fs = File.Create(path + ".decompressed");
compressStream.CopyTo(fs);
fs.Close();
}
}
This works when it comes to extracting the .zip but the result is quite unuseful
Is there a suitable solution to decompress this packed dll?
This is my simple C# Console App code (Framework 4), which I`m using simply by "drag and drop" (Costura Compressed) files over compiled (ConsoleApp1) executable.
using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text.RegularExpressions;
namespace ConsoleApp1
{
class Program
{
static void CopyTo(Stream source, Stream destination) {
int count;
var array = new byte[81920];
while ((count = source.Read(array, 0, array.Length)) != 0) {
destination.Write(array, 0, count);
}
}
static Stream LoadStream(string fullname) {
FileStream stream = default(FileStream);
if (fullname.EndsWith(".zip")) {
using (stream = new FileStream(fullname, FileMode.Open)) {
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress)) {
var memStream = new MemoryStream();
CopyTo(compressStream, memStream);
memStream.Position = 0;
return memStream;
}
}
}
return stream;
}
static void Main(string[] args) {
Stream stream; Stream file;
string RefilePath = #"^.+[^\\]+\\"; string fullname; string newFile;
for (int i = 0; i < args.Count(); i++) {
fullname = args[i];
newFile = Regex.Replace(fullname, "\\.zip$", string.Empty);
Console.Write("{0} -> {1}\r\n",
Regex.Replace(fullname, RefilePath, string.Empty),
Regex.Replace(newFile, RefilePath, string.Empty));
try
{
stream = LoadStream(fullname);
using (file = File.Create(newFile)) {
CopyTo(stream, file);
}
}
catch (Exception ex) {
Console.Write("{0}", ex.ToString());
}
}
}
}
}
Based On Cameron MacFarland Answer
The code that Costura uses to decompress those resources is here.
https://github.com/Fody/Costura/blob/master/src/Costura.Template/Common.cs
static void CopyTo(Stream source, Stream destination)
{
var array = new byte[81920];
int count;
while ((count = source.Read(array, 0, array.Length)) != 0)
{
destination.Write(array, 0, count);
}
}
static Stream LoadStream(string fullname)
{
var executingAssembly = Assembly.GetExecutingAssembly();
if (fullname.EndsWith(".zip"))
{
using (var stream = executingAssembly.GetManifestResourceStream(fullname))
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress))
{
var memStream = new MemoryStream();
CopyTo(compressStream, memStream);
memStream.Position = 0;
return memStream;
}
}
return executingAssembly.GetManifestResourceStream(fullname);
}
To decompress those resources there is this project to.
I'm overwriting a file using C# in Windows Phone 7. When I do this a seemingly random character is added to the start of each line.
Why is this happening?
Code:
public static bool overwriteFile(string filename, string[] inputArray)
{
try
{
IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
FileStream stream = store.OpenFile(filename, FileMode.Create);
BinaryWriter writer = new BinaryWriter(stream);
foreach (string input in inputArray)
{
writer.Write(input + "\n");
}
writer.Close();
return true;
}
catch (IOException ex)
{
return false;
}
}
Lodaing Code:
public static Idea[] getFile(string filename)
{
try
{
IsolatedStorageFile store = IsolatedStorageFile.GetUserStoreForApplication();
string fileContents = null;
if (store.FileExists(filename)) // Check if file exists
{
IsolatedStorageFileStream save = new IsolatedStorageFileStream(filename, FileMode.Open, store);
StreamReader streamReader = new StreamReader(save);
fileContents = streamReader.ReadToEnd();
save.Close();
}
string[] lines = null;
if (fileContents != null)
{
lines = fileContents.Split('\n');
}
Idea[] ideaList = null;
if (lines != null)
{
ideaList = new Idea[lines.Length];
for (int i = 0; i < lines.Length; i++)
{
ideaList[i] = new Idea(lines[i].TrimEnd('\r'));
}
}
return ideaList;
}
catch (IOException ex)
{
return null;
}
}
The random character is a length prefix; see http://msdn.microsoft.com/en-us/library/yzxa6408.aspx.
You should be using some type of TextWriter to write strings to the file; NOT a BinaryWriter.
A StreamWriter might be the best and then you could use the WriteLine method.
Instead of using '\n', try using Environment.NewLine
You are using a BinaryWriter to write, and a TextReader to read. Change your write code to use a StreamWriter (which is a TextWriter) instead of a BinaryWriter. This will also get you the WriteLine method that Naveed recommends.
try changing this
writer.Write(input + "\n");
to
writer.WriteLine(input);
What is the best way to add text to the beginning of a file using C#?
I couldn't find a straightforward way to do this, but came up with a couple work-arounds.
Open up new file, write the text that I wanted to add, append the text from the old file to the end of the new file.
Since the text I want to add should be less than 200 characters, I was thinking that I could add white space characters to the beginning of the file, and then overwrite the white space with the text I wanted to add.
Has anyone else come across this problem, and if so, what did you do?
This works for me, but for small files. Probably it's not a very good solution otherwise.
string currentContent = String.Empty;
if (File.Exists(filePath))
{
currentContent = File.ReadAllText(filePath);
}
File.WriteAllText(filePath, newContent + currentContent );
Adding to the beginning of a file (prepending as opposed to appending) is generally not a supported operation. Your #1 options is fine. If you can't write a temp file, you can pull the entire file into memory, preprend your data to the byte array and then overwrite it back out (this is only really feasible if your files are small and you don't have to have a bunch in memory at once because prepending the array is not necessarily easy without a copy either).
Yeah, basically you can use something like this:
public static void PrependString(string value, FileStream file)
{
var buffer = new byte[file.Length];
while (file.Read(buffer, 0, buffer.Length) != 0)
{
}
if(!file.CanWrite)
throw new ArgumentException("The specified file cannot be written.", "file");
file.Position = 0;
var data = Encoding.Unicode.GetBytes(value);
file.SetLength(buffer.Length + data.Length);
file.Write(data, 0, data.Length);
file.Write(buffer, 0, buffer.Length);
}
public static void Prepend(this FileStream file, string value)
{
PrependString(value, file);
}
Then
using(var file = File.Open("yourtext.txt", FileMode.Open, FileAccess.ReadWrite))
{
file.Prepend("Text you want to write.");
}
Not really effective though in case of huge files.
using two streams, you can do it in place, but keep in mind that this will still loop over the whole file on every addition
using System;
using System.IO;
using System.Text;
namespace FilePrepender
{
public class FilePrepender
{
private string file=null;
public FilePrepender(string filePath)
{
file = filePath;
}
public void prependline(string line)
{
prepend(line + Environment.NewLine);
}
private void shiftSection(byte[] chunk,FileStream readStream,FileStream writeStream)
{
long initialOffsetRead = readStream.Position;
long initialOffsetWrite= writeStream.Position;
int offset = 0;
int remaining = chunk.Length;
do//ensure that the entire chunk length gets read and shifted
{
int read = readStream.Read(chunk, offset, remaining);
offset += read;
remaining -= read;
} while (remaining > 0);
writeStream.Write(chunk, 0, chunk.Length);
writeStream.Seek(initialOffsetWrite, SeekOrigin.Begin);
readStream.Seek(initialOffsetRead, SeekOrigin.Begin);
}
public void prepend(string text)
{
byte[] bytes = Encoding.Default.GetBytes(text);
byte[] chunk = new byte[bytes.Length];
using (FileStream readStream = File.Open(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using(FileStream writeStream = File.Open(file, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
{
readStream.Seek(0, SeekOrigin.End);//seek chunk.Length past the end of the file
writeStream.Seek(chunk.Length, SeekOrigin.End);//which lets the loop run without special cases
long size = readStream.Position;
//while there's a whole chunks worth above the read head, shift the file contents down from the end
while(readStream.Position - chunk.Length >= 0)
{
readStream.Seek(-chunk.Length, SeekOrigin.Current);
writeStream.Seek(-chunk.Length, SeekOrigin.Current);
shiftSection(chunk, readStream, writeStream);
}
//clean up the remaining shift for the bytes that don't fit in size%chunk.Length
readStream.Seek(0, SeekOrigin.Begin);
writeStream.Seek(Math.Min(size, chunk.Length), SeekOrigin.Begin);
shiftSection(chunk, readStream, writeStream);
//finally, write the text you want to prepend
writeStream.Seek(0,SeekOrigin.Begin);
writeStream.Write(bytes, 0, bytes.Length);
}
}
}
}
}
I think the best way is to create a temp file. Add your text then read the contents of the original file adding it to the temp file. Then you can overwrite the original with the temp file.
prepend:
private const string tempDirPath = #"c:\temp\log.log", tempDirNewPath = #"c:\temp\log.new";
StringBuilder sb = new StringBuilder();
...
File.WriteAllText(tempDirNewPath, sb.ToString());
File.AppendAllText(tempDirNewPath, File.ReadAllText(tempDirPath));
File.Delete(tempDirPath);
File.Move(tempDirNewPath, tempDirPath);
using (FileStream fs = File.OpenWrite(tempDirPath))
{ //truncate to a reasonable length
if (16384 < fs.Length) fs.SetLength(16384);
fs.Close();
}
// The file we'll prepend to
string filePath = path + "\\log.log";
// A temp file we'll write to
string tempFilePath = path + "\\temp.log";
// 1) Write your prepended contents to a temp file.
using (var writer = new StreamWriter(tempFilePath, false))
{
// Write whatever you want to prepend
writer.WriteLine("Hi");
}
// 2) Use stream lib methods to append the original contents to the Temp
// file.
using (var oldFile = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Read, FileShare.Read))
{
using (var tempFile = new FileStream(tempFilePath, FileMode.Append, FileAccess.Write, FileShare.Read))
{
oldFile.CopyTo(tempFile);
}
}
// 3) Finally, dump the Temp file back to the original, keeping all its
// original permissions etc.
File.Replace(tempFilePath, filePath, null);
Even if what you're writing is small, the Temp file gets the entire original file appended to it before the .Replace(), so it does need to be on disk.
Note that this code is not Thread-safe; if more than one thread accesses this code you can lose writes in the file swapping going on here. That said, it's also pretty expensive, so you'd want to gate access to it anyway - pass writes via multiple Providers to a buffer, which periodically empties out via this prepend method on a single Consumer thread.
You should be able to do this without opening a new file. Use the following File method:
public static FileStream Open(
string path,
FileMode mode,
FileAccess access
)
Making sure to specify FileAccess.ReadWrite.
Using the FileStream returned from File.Open, read all of the existing data into memory. Then reset the pointer to the beginning of the file, write your new data, then write the existing data.
(If the file is big and/or you're suspicious of using too much memory, you can do this without having to read the whole file into memory, but implementing that is left as an exercise to the reader.)
The following algorithm may solve the problem pretty easily, it's most efficient for any size of file, including very big text files:
string outPutFile = #"C:\Output.txt";
string result = "Some new string" + DateTime.Now.ToString() + Environment.NewLine;
StringBuilder currentContent = new StringBuilder();
List<string> rawList = File.ReadAllLines(outPutFile).ToList();
foreach (var item in rawList) {
currentContent.Append(item + Environment.NewLine);
}
File.WriteAllText(outPutFile, result + currentContent.ToString());
Use this class:
public static class File2
{
private static readonly Encoding _defaultEncoding = new UTF8Encoding(false, true); // encoding used in File.ReadAll*()
private static object _bufferSizeLock = new Object();
private static int _bufferSize = 1024 * 1024; // 1mb
public static int BufferSize
{
get
{
lock (_bufferSizeLock)
{
return _bufferSize;
}
}
set
{
lock (_bufferSizeLock)
{
_bufferSize = value;
}
}
}
public static void PrependAllLines(string path, IEnumerable<string> contents)
{
PrependAllLines(path, contents, _defaultEncoding);
}
public static void PrependAllLines(string path, IEnumerable<string> contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllLines(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
public static void PrependAllText(string path, string contents)
{
PrependAllText(path, contents, _defaultEncoding);
}
public static void PrependAllText(string path, string contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllText(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
private static void AppendToTemp(string path, string temp, Encoding encoding)
{
var bufferSize = BufferSize;
char[] buffer = new char[bufferSize];
using (var writer = new StreamWriter(temp, true, encoding))
{
using (var reader = new StreamReader(path, encoding))
{
int bytesRead;
while ((bytesRead = reader.ReadBlock(buffer,0,bufferSize)) != 0)
{
writer.Write(buffer,0,bytesRead);
}
}
}
}
}
Put the file's contents in a string. Append new data you want to add to the top of the file to that string -- string = newdata + string. Then move the seek position of the file to 0 and write the string into the file.