I have a StreamReader object that I initialized with a stream, now I want to save this stream to disk (the stream may be a .gif or .jpg or .pdf).
Existing Code:
StreamReader sr = new StreamReader(myOtherObject.InputStream);
I need to save this to disk (I have the filename).
In the future I may want to store this to SQL Server.
I have the encoding type also, which I will need if I store it to SQL Server, correct?
As highlighted by Tilendor in Jon Skeet's answer, streams have a CopyTo method since .NET 4.
var fileStream = File.Create("C:\\Path\\To\\File");
myOtherObject.InputStream.Seek(0, SeekOrigin.Begin);
myOtherObject.InputStream.CopyTo(fileStream);
fileStream.Close();
Or with the using syntax:
using (var fileStream = File.Create("C:\\Path\\To\\File"))
{
myOtherObject.InputStream.Seek(0, SeekOrigin.Begin);
myOtherObject.InputStream.CopyTo(fileStream);
}
You have to call Seek if you're not already at the beginning or you won't copy the entire stream.
You must not use StreamReader for binary files (like gifs or jpgs). StreamReader is for text data. You will almost certainly lose data if you use it for arbitrary binary data. (If you use Encoding.GetEncoding(28591) you will probably be okay, but what's the point?)
Why do you need to use a StreamReader at all? Why not just keep the binary data as binary data and write it back to disk (or SQL) as binary data?
EDIT: As this seems to be something people want to see... if you do just want to copy one stream to another (e.g. to a file) use something like this:
/// <summary>
/// Copies the contents of input to output. Doesn't close either stream.
/// </summary>
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[8 * 1024];
int len;
while ( (len = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, len);
}
}
To use it to dump a stream to a file, for example:
using (Stream file = File.Create(filename))
{
CopyStream(input, file);
}
Note that Stream.CopyTo was introduced in .NET 4, serving basically the same purpose.
public void CopyStream(Stream stream, string destPath)
{
using (var fileStream = new FileStream(destPath, FileMode.Create, FileAccess.Write))
{
stream.CopyTo(fileStream);
}
}
private void SaveFileStream(String path, Stream stream)
{
var fileStream = new FileStream(path, FileMode.Create, FileAccess.Write);
stream.CopyTo(fileStream);
fileStream.Dispose();
}
I don't get all of the answers using CopyTo, where maybe the systems using the app might not have been upgraded to .NET 4.0+. I know some would like to force people to upgrade, but compatibility is also nice, too.
Another thing, I don't get using a stream to copy from another stream in the first place. Why not just do:
byte[] bytes = myOtherObject.InputStream.ToArray();
Once you have the bytes, you can easily write them to a file:
public static void WriteFile(string fileName, byte[] bytes)
{
string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!path.EndsWith(#"\")) path += #"\";
if (File.Exists(Path.Combine(path, fileName)))
File.Delete(Path.Combine(path, fileName));
using (FileStream fs = new FileStream(Path.Combine(path, fileName), FileMode.CreateNew, FileAccess.Write))
{
fs.Write(bytes, 0, (int)bytes.Length);
//fs.Close();
}
}
This code works as I've tested it with a .jpg file, though I admit I have only used it with small files (less than 1 MB). One stream, no copying between streams, no encoding needed, just write the bytes! No need to over-complicate things with StreamReader if you already have a stream you can convert to bytes directly with .ToArray()!
Only potential downsides I can see in doing it this way is if there's a large file you have, having it as a stream and using .CopyTo() or equivalent allows FileStream to stream it instead of using a byte array and reading the bytes one by one. It might be slower doing it this way, as a result. But it shouldn't choke since the .Write() method of the FileStream handles writing the bytes, and it's only doing it one byte at a time, so it won't clog memory, except that you will have to have enough memory to hold the stream as a byte[] object. In my situation where I used this, getting an OracleBlob, I had to go to a byte[], it was small enough, and besides, there was no streaming available to me, anyway, so I just sent my bytes to my function, above.
Another option, using a stream, would be to use it with Jon Skeet's CopyStream function that was in another post - this just uses FileStream to take the input stream and create the file from it directly. It does not use File.Create, like he did (which initially seemed to be problematic for me, but later found it was likely just a VS bug...).
/// <summary>
/// Copies the contents of input to output. Doesn't close either stream.
/// </summary>
public static void CopyStream(Stream input, Stream output)
{
byte[] buffer = new byte[8 * 1024];
int len;
while ( (len = input.Read(buffer, 0, buffer.Length)) > 0)
{
output.Write(buffer, 0, len);
}
}
public static void WriteFile(string fileName, Stream inputStream)
{
string path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
if (!path.EndsWith(#"\")) path += #"\";
if (File.Exists(Path.Combine(path, fileName)))
File.Delete(Path.Combine(path, fileName));
using (FileStream fs = new FileStream(Path.Combine(path, fileName), FileMode.CreateNew, FileAccess.Write)
{
CopyStream(inputStream, fs);
}
inputStream.Close();
inputStream.Flush();
}
Here's an example that uses proper usings and implementation of idisposable:
static void WriteToFile(string sourceFile, string destinationfile, bool append = true, int bufferSize = 4096)
{
using (var sourceFileStream = new FileStream(sourceFile, FileMode.OpenOrCreate))
{
using (var destinationFileStream = new FileStream(destinationfile, FileMode.OpenOrCreate))
{
while (sourceFileStream.Position < sourceFileStream.Length)
{
destinationFileStream.WriteByte((byte)sourceFileStream.ReadByte());
}
}
}
}
...and there's also this
public static void WriteToFile(Stream stream, string destinationFile, int bufferSize = 4096, FileMode mode = FileMode.OpenOrCreate, FileAccess access = FileAccess.ReadWrite, FileShare share = FileShare.ReadWrite)
{
using (var destinationFileStream = new FileStream(destinationFile, mode, access, share))
{
while (stream.Position < stream.Length)
{
destinationFileStream.WriteByte((byte)stream.ReadByte());
}
}
}
The key is understanding the proper use of using (which should be implemented on the instantiation of the object that implements idisposable as shown above), and having a good idea as to how the properties work for streams. Position is literally the index within the stream (which starts at 0) that is followed as each byte is read using the readbyte method. In this case I am essentially using it in place of a for loop variable and simply letting it follow through all the way up to the length which is LITERALLY the end of the entire stream (in bytes). Ignore in bytes because it is practically the same and you will have something simple and elegant like this that resolves everything cleanly.
Keep in mind, too, that the ReadByte method simply casts the byte to an int in the process and can simply be converted back.
I'm gonna add another implementation I recently wrote to create a dynamic buffer of sorts to ensure sequential data writes to prevent massive overload
private void StreamBuffer(Stream stream, int buffer)
{
using (var memoryStream = new MemoryStream())
{
stream.CopyTo(memoryStream);
var memoryBuffer = memoryStream.GetBuffer();
for (int i = 0; i < memoryBuffer.Length;)
{
var networkBuffer = new byte[buffer];
for (int j = 0; j < networkBuffer.Length && i < memoryBuffer.Length; j++)
{
networkBuffer[j] = memoryBuffer[i];
i++;
}
//Assuming destination file
destinationFileStream.Write(networkBuffer, 0, networkBuffer.Length);
}
}
}
The explanation is fairly simple: we know that we need to keep in mind the entire set of data we wish to write and also that we only want to write certain amounts, so we want the first loop with the last parameter empty (same as while). Next, we initialize a byte array buffer that is set to the size of what's passed, and with the second loop we compare j to the size of the buffer and the size of the original one, and if it's greater than the size of the original byte array, end the run.
Why not use a FileStream object?
public void SaveStreamToFile(string fileFullPath, Stream stream)
{
if (stream.Length == 0) return;
// Create a FileStream object to write a stream to a file
using (FileStream fileStream = System.IO.File.Create(fileFullPath, (int)stream.Length))
{
// Fill the bytes[] array with the stream data
byte[] bytesInStream = new byte[stream.Length];
stream.Read(bytesInStream, 0, (int)bytesInStream.Length);
// Use FileStream object to write to the specified file
fileStream.Write(bytesInStream, 0, bytesInStream.Length);
}
}
//If you don't have .Net 4.0 :)
public void SaveStreamToFile(Stream stream, string filename)
{
using(Stream destination = File.Create(filename))
Write(stream, destination);
}
//Typically I implement this Write method as a Stream extension method.
//The framework handles buffering.
public void Write(Stream from, Stream to)
{
for(int a = from.ReadByte(); a != -1; a = from.ReadByte())
to.WriteByte( (byte) a );
}
/*
Note, StreamReader is an IEnumerable<Char> while Stream is an IEnumbable<byte>.
The distinction is significant such as in multiple byte character encodings
like Unicode used in .Net where Char is one or more bytes (byte[n]). Also, the
resulting translation from IEnumerable<byte> to IEnumerable<Char> can loose bytes
or insert them (for example, "\n" vs. "\r\n") depending on the StreamReader instance
CurrentEncoding.
*/
Another option is to get the stream to a byte[] and use File.WriteAllBytes. This should do:
using (var stream = new MemoryStream())
{
input.CopyTo(stream);
File.WriteAllBytes(file, stream.ToArray());
}
Wrapping it in an extension method gives it better naming:
public void WriteTo(this Stream input, string file)
{
//your fav write method:
using (var stream = File.Create(file))
{
input.CopyTo(stream);
}
//or
using (var stream = new MemoryStream())
{
input.CopyTo(stream);
File.WriteAllBytes(file, stream.ToArray());
}
//whatever that fits.
}
public void testdownload(stream input)
{
byte[] buffer = new byte[16345];
using (FileStream fs = new FileStream(this.FullLocalFilePath,
FileMode.Create, FileAccess.Write, FileShare.None))
{
int read;
while ((read = input.Read(buffer, 0, buffer.Length)) > 0)
{
fs.Write(buffer, 0, read);
}
}
}
Related
I have a memory stream that I want to compress:
public static MemoryStream ZipChunk(MemoryStream unZippedChunk) {
MemoryStream zippedChunk = new MemoryStream();
ZipOutputStream zipOutputStream = new ZipOutputStream(zippedChunk);
zipOutputStream.SetLevel(3);
ZipEntry entry = new ZipEntry("name");
zipOutputStream.PutNextEntry(entry);
Utils.StreamCopy(unZippedChunk, zippedChunk, new byte[4096]);
zipOutputStream.CloseEntry();
zipOutputStream.IsStreamOwner = false;
zipOutputStream.Close();
zippedChunk.Close();
return zippedChunk;
}
public static void StreamCopy(Stream source, Stream destination, byte[] buffer, bool bFlush = true) {
bool flag = true;
while (flag) {
int num = source.Read(buffer, 0, buffer.Length);
if (num > 0) {
destination.Write(buffer, 0, num);
}
else {
if (bFlush) {
destination.Flush();
}
flag = false;
}
}
}
It's supposed to be quite simple. You provide it with a stream you want to compress. The methods compresses the stream and returns it. Great.
However, I don't get compressed stream back. What I get are streams that have about 20ish bytes added at the beginning and end, which seem to have something to do with the zip library. But the data in the middle is completely uncompressed (ranges of 256 bytes that have same value, etc). I tried upping the level to 9, but nothing changed.
Why aren't my streams compressing?
You yourself copy original stream right into output stream via:
Utils.StreamCopy(unZippedChunk, zippedChunk, new byte[4096]);
You should copy to zipOutputStream instead:
StreamCopy(unZippedChunk, zipOutputStream, new byte[4096]);
Side note: instead of using custom copy stream methods - use a default one:
unZippedChunk.CopyTo(zipOutputStream);
I need help converting a VERY LARGE binary file (ZIP file) to a Base64String and back again. The files are too large to be loaded into memory all at once (they throw OutOfMemoryExceptions) otherwise this would be a simple task. I do not want to process the contents of the ZIP file individually, I want to process the entire ZIP file.
The problem:
I can convert the entire ZIP file (test sizes vary from 1 MB to 800 MB at present) to Base64String, but when I convert it back, it is corrupted. The new ZIP file is the correct size, it is recognized as a ZIP file by Windows and WinRAR/7-Zip, etc., and I can even look inside the ZIP file and see the contents with the correct sizes/properties, but when I attempt to extract from the ZIP file, I get: "Error: 0x80004005" which is a general error code.
I am not sure where or why the corruption is happening. I have done some investigating, and I have noticed the following:
If you have a large text file, you can convert it to Base64String incrementally without issue. If calling Convert.ToBase64String on the entire file yielded: "abcdefghijklmnopqrstuvwx", then calling it on the file in two pieces would yield: "abcdefghijkl" and "mnopqrstuvwx".
Unfortunately, if the file is a binary then the result is different. While the entire file might yield: "abcdefghijklmnopqrstuvwx", trying to process this in two pieces would yield something like: "oiweh87yakgb" and "kyckshfguywp".
Is there a way to incrementally base 64 encode a binary file while avoiding this corruption?
My code:
private void ConvertLargeFile()
{
FileStream inputStream = new FileStream("C:\\Users\\test\\Desktop\\my.zip", FileMode.Open, FileAccess.Read);
byte[] buffer = new byte[MultipleOfThree];
int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
while(bytesRead > 0)
{
byte[] secondaryBuffer = new byte[buffer.Length];
int secondaryBufferBytesRead = bytesRead;
Array.Copy(buffer, secondaryBuffer, buffer.Length);
bool isFinalChunk = false;
Array.Clear(buffer, 0, buffer.Length);
bytesRead = inputStream.Read(buffer, 0, buffer.Length);
if(bytesRead == 0)
{
isFinalChunk = true;
buffer = new byte[secondaryBufferBytesRead];
Array.Copy(secondaryBuffer, buffer, buffer.length);
}
String base64String = Convert.ToBase64String(isFinalChunk ? buffer : secondaryBuffer);
File.AppendAllText("C:\\Users\\test\\Desktop\\Base64Zip", base64String);
}
inputStream.Dispose();
}
The decoding is more of the same. I use the size of the base64String variable above (which varies depending on the original buffer size that I test with), as the buffer size for decoding. Then, instead of Convert.ToBase64String(), I call Convert.FromBase64String() and write to a different file name/path.
EDIT:
In my haste to reduce the code (I refactored it into a new project, separate from other processing to eliminate code that isn't central to the issue) I introduced a bug. The base 64 conversion should be performed on the secondaryBuffer for all iterations save the last (Identified by isFinalChunk), when buffer should be used. I have corrected the code above.
EDIT #2:
Thank you all for your comments/feedback. After correcting the bug (see the above edit), I re-tested my code, and it is actually working now. I intend to test and implement #rene's solution as it appears to be the best, but I thought that I should let everyone know of my discovery as well.
Based on the code shown in the blog from Wiktor Zychla the following code works. This same solution is indicated in the remarks section of Convert.ToBase64String as pointed out by Ivan Stoev
// using System.Security.Cryptography
private void ConvertLargeFile()
{
//encode
var filein= #"C:\Users\test\Desktop\my.zip";
var fileout = #"C:\Users\test\Desktop\Base64Zip";
using (FileStream fs = File.Open(fileout, FileMode.Create))
using (var cs=new CryptoStream(fs, new ToBase64Transform(),
CryptoStreamMode.Write))
using(var fi =File.Open(filein, FileMode.Open))
{
fi.CopyTo(cs);
}
// the zip file is now stored in base64zip
// and decode
using (FileStream f64 = File.Open(fileout, FileMode.Open) )
using (var cs=new CryptoStream(f64, new FromBase64Transform(),
CryptoStreamMode.Read ) )
using(var fo =File.Open(filein +".orig", FileMode.Create))
{
cs.CopyTo(fo);
}
// the original file is in my.zip.orig
// use the commandlinetool
// fc my.zip my.zip.orig
// to verify that the start file and the encoded and decoded file
// are the same
}
The code uses standard classes found in System.Security.Cryptography namespace and uses a CryptoStream and the FromBase64Transform and its counterpart ToBase64Transform
You can avoid using a secondary buffer by passing offset and length to Convert.ToBase64String, like this:
private void ConvertLargeFile()
{
using (var inputStream = new FileStream("C:\\Users\\test\\Desktop\\my.zip", FileMode.Open, FileAccess.Read))
{
byte[] buffer = new byte[MultipleOfThree];
int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
while(bytesRead > 0)
{
String base64String = Convert.ToBase64String(buffer, 0, bytesRead);
File.AppendAllText("C:\\Users\\test\\Desktop\\Base64Zip", base64String);
bytesRead = inputStream.Read(buffer, 0, buffer.Length);
}
}
}
The above should work, but I think Rene's answer is actually the better solution.
Use this code:
public void ConvertLargeFile(string source , string destination)
{
using (FileStream inputStream = new FileStream(source, FileMode.Open, FileAccess.Read))
{
int buffer_size = 30000; //or any multiple of 3
byte[] buffer = new byte[buffer_size];
int bytesRead = inputStream.Read(buffer, 0, buffer.Length);
while (bytesRead > 0)
{
byte[] buffer2 = buffer;
if(bytesRead < buffer_size)
{
buffer2 = new byte[bytesRead];
Buffer.BlockCopy(buffer, 0, buffer2, 0, bytesRead);
}
string base64String = System.Convert.ToBase64String(buffer2);
File.AppendAllText(destination, base64String);
bytesRead = inputStream.Read(buffer, 0, buffer.Length);
}
}
}
I am using ICSharpCode.SharpZipLib v 0.86.0.518.
I have a a stream whose contents I would like to add as a file to a Zip, which must also be created in memory (as opposed to on on disk).
The resulting Zip opens for browsing, but when attempting to extract any contents, I get an error stating "The Archive is either in an unknown format or damaged".
In the code below, when asZip=false the text file is sent, and received as expected. When asZip=true, the file is sent, but suffers the corruption described above.
When I replace MemoryStream zipStream, with FileStream zipStream, the file on disk is OK.
Can anyone see what I've missed?
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net; // .NET 4.0
using System.Net.Http;
using System.Net.Http.Headers;
using System.Web.Mvc;
using ICSharpCode.SharpZipLib.Zip;//0.86.0.518
namespace Demo
{
public class DemoApiController : Controller
{
/// <summary>
/// demos the zipfile error
/// </summary>
/// <param name="withFiles">Creates the zip if set to <c>true</c> [with files].</param>
[HttpGet]
public void ZipErrorDemo(bool asZip)
{
const string fileContent = "Hello World!";
MemoryStream rawContentStream = new MemoryStream(System.Text.Encoding.ASCII.GetBytes(fileContent));
if (!asZip)
{
//This File is recieved as text, opens without erros and has correct content.
WriteStreamToDownload(rawContentStream, "text/plain", "HelloWorld.txt");
}
else
{
MemoryStream zipStream = new MemoryStream(1024 * 2048);//2MB
using (ZipOutputStream s = new ZipOutputStream(zipStream))
{
s.UseZip64 = ICSharpCode.SharpZipLib.Zip.UseZip64.Off; //No Zip64 for better compatability
s.SetLevel(0); //No compression
byte[] buffer = new byte[4096];
//Add the text file
ZipEntry csvEntry = new ZipEntry("HelloWorld.txt");
s.PutNextEntry(csvEntry);
int sourceBytes = 0;
do
{
sourceBytes = rawContentStream.Read(buffer, 0, buffer.Length);
s.Write(buffer, 0, sourceBytes);
} while (sourceBytes > 0);
s.CloseEntry();
s.IsStreamOwner = false;//Tells s.Close to not mess invoke zipStream.Close()
s.Flush();
s.Finish();
s.Close();
byte[] streamBuffer = zipStream.GetBuffer();//Before doing this things were worse.
MemoryStream newStream = new MemoryStream(streamBuffer);
//This File is recieved as a zip, opens to list contents, but atemtps at extraction result in an error.
WriteStreamToDownload(newStream, "application/zip", "HelloWorld.zip");
}
}
}
// Adapted from: http://stackoverflow.com/questions/5596747/download-stream-file-from-url-asp-net
// Accessed: 3/17/15. Works.
private static void WriteStreamToDownload(Stream stream, string contentType, string fileName)
{
// 100 kb
const int bytesToRead = 102400;
byte[] buffer = new byte[bytesToRead];
var contextResponse = System.Web.HttpContext.Current.Response;
try
{
contextResponse.ContentType = contentType;
contextResponse.AddHeader("Content-Disposition", "attachment; filename=\"" + Path.GetFileName(fileName) + "\"");
contextResponse.AddHeader("Content-Length", stream.Length.ToString());
int length;
do
{
if (contextResponse.IsClientConnected)
{
length = stream.Read(buffer, 0, bytesToRead);
contextResponse.OutputStream.Write(buffer, 0, length);
contextResponse.Flush();
buffer = new byte[bytesToRead];
}
else
{
length = -1;
}
} while (length > 0);
}
finally
{
if (stream != null)
{
stream.Close();
stream.Dispose();
}
}
}
}
}
At the moment, you are over-reading your stream. GetBuffer() returns the oversized backing buffer; you should usually limit yourself to the first zipStream.Length bytes of the buffer.
So the first thing to try is:
MemoryStream newStream = new MemoryStream(streamBuffer, 0
(int)zipStream.Length);
However, if that works, you can probably also achieve the same thing by simply sending zipStream, as long as you:
rewind the stream after writing
ensure that it doesn't get closed by the using
You might also be interested to hear that zip support is present inside the .NET framework itself, without requiring additional tools.
Your copying code, btw, is inefficient (especially when the buffer is constantly recreated) and could probably just use:
stream.CopyTo(contextResponse.OutputStream);
I need something similar to ReadToEnd or ReadAllBytes to read all of the contents of the MemoryMappedFile using the MappedViewAccessor if I don't know the size of it, how can I do it?
I have searched for it, I have seen this question, but it is not the thing I am looking for:
How can I quickly read bytes from a memory mapped file in .NET?
Edit:
There is a problem, the (int)stream.Length is not giving me the correct length, it rather gives the size of the internal buffer used! I need to refresh this question because it is very pressing.
Rather use the Stream:
public static Byte[] ReadMMFAllBytes(string fileName)
{
using (var mmf = MemoryMappedFile.OpenExisting(fileName))
{
using (var stream = mmf.CreateViewStream())
{
using (BinaryReader binReader = new BinaryReader(stream))
{
return binReader.ReadBytes((int)stream.Length);
}
}
}
}
This is difficult to answer since there are still many details of your application that you haven't specified, but I think both Guffa's and Amer's answers are still partially correct:
A MemoryMappedFile is more memory than file; it is a sequence of 4Kb pages in memory. So, stream.Length will in fact give you all of the bytes (there is no "internal buffer size"), but it might give you more bytes than you expect since the size will always be rounded up to a 4Kb boundary.
The "file" semantic comes from associating the MemoryMappedFile to a real filesystem file. Assuming that the process which creates the file always adjusts the file size, then you can get the precise size of the file via the fileSystem.
If all of the above would fit your application, then the following should work:
static byte[] ReadMemoryMappedFile(string fileName)
{
long length = new FileInfo(fileName).Length;
using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite))
{
using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, length, MemoryMappedFileAccess.Read, null, HandleInheritability.Inheritable, false))
{
using (var viewStream = mmf.CreateViewStream(0, length, MemoryMappedFileAccess.Read))
{
using (BinaryReader binReader = new BinaryReader(viewStream))
{
var result = binReader.ReadBytes((int)length);
return result;
}
}
}
}
}
To write the data, you can use this:
private static void WriteData(string fileName, byte[] data)
{
using (var stream = File.Open(fileName, FileMode.OpenOrCreate, FileAccess.ReadWrite, FileShare.ReadWrite))
{
using (var mmf = MemoryMappedFile.CreateFromFile(stream, null, data.Length, MemoryMappedFileAccess.ReadWrite, null, HandleInheritability.Inheritable, true))
{
using (var view = mmf.CreateViewAccessor())
{
view.WriteArray(0, data, 0, data.Length);
}
}
stream.SetLength(data.Length); // Make sure the file is the correct length, in case the data got smaller.
}
}
But, by the time you do all of the above you might do just as well to use the file directly and avoid the memory mapping. If mapping it to the filesystem isn't acceptable, then Guffa's answer of encoding the length (or an end marker) in the data itself is probably best.
You can't do that.
A view accessor is created with a minimum size of a system page, which means that it may be larger than the actual file. A view stream is just a stream form of an accessor, so it will also give the same behaviour.
"views are provided in units of system pages, and the size of the view
is rounded up to the next system page size"
http://msdn.microsoft.com/en-us/library/dd267577.aspx
The accessor will gladly read and write outside the actual file without throwing an exception. When reading, any bytes outside the file will just be zero. When writing, the bytes written outside the file are just ignored.
To read the file from a memory mapped file with the exact size of the original file, you have to already know that size.
Stream created by MemoryMappedFile has a length aligned to file system page size (usually 4096). You have to get the file size from somewhere else. If it is memory mapped file you could use that code:
byte[] ReadAllMemoryMappedFileBytes(string filePath)
{
var fileInfo = new FileInfo(filePath);
using (var file = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open))
using (var stream = file.CreateViewAccessor())
{
byte[] bytes = new byte[fileInfo.Length];
stream.ReadArray(0, bytes, 0, bytes.Length);
return bytes;
}
}
Use FileInfo class to get length as shown below
using System.Data;
using System.IO;
using System.IO.Compression;
using System.IO.MemoryMappedFiles;
// ...
public void WriteToMemoryMap(DataSet ds, string key, string fileName)
{
var bytes = CompressData(ds);
using (MemoryMappedFile objMf = MemoryMappedFile.CreateFromFile(fileName, FileMode.OpenOrCreate, key, bytes.Length))
{
using (MemoryMappedViewAccessor accessor = objMf.CreateViewAccessor())
{
accessor.WriteArray(0, bytes, 0, bytes.Length);
}
}
}
public DataSet ReadFromMemoryMap(string fileName)
{
var fi = new FileInfo(fileName);
var length = (int)fi.Length;
var newBytes = new byte[length];
using (MemoryMappedFile objMf = MemoryMappedFile.CreateFromFile(fileName, FileMode.Open))
{
using (MemoryMappedViewAccessor accessor = objMf.CreateViewAccessor())
{
accessor.ReadArray(0, newBytes, 0, length);
}
}
return DecompressData(newBytes);
}
public byte[] CompressData(DataSet ds)
{
try
{
byte[] data = null;
var memStream = new MemoryStream();
var zipStream = new GZipStream(memStream, CompressionMode.Compress);
ds.WriteXml(zipStream, XmlWriteMode.WriteSchema);
zipStream.Close();
data = memStream.ToArray();
memStream.Close();
return data;
}
catch (Exception)
{
return null;
}
}
public DataSet DecompressData(byte[] data)
{
try
{
var memStream = new MemoryStream(data);
var unzipStream = new GZipStream(memStream, CompressionMode.Decompress);
var objDataSet = new DataSet();
objDataSet.ReadXml(unzipStream, XmlReadMode.ReadSchema);
unzipStream.Close();
memStream.Close();
return objDataSet;
}
catch (Exception)
{
return null;
}
}
Just the #Amer Sawan solution translated to Vb.NET:
' Usage Example:
' Dim ReadBytes As Byte() = ReadMemoryMappedFile(Name:="My MemoryMappedFile Name") ' Read the byte-sequence from memory.
' Dim Message As String = System.Text.Encoding.ASCII.GetString(ReadBytes.ToArray) ' Convert the bytes to String.
' Message = Message.Trim({ControlChars.NullChar}) ' Remove null chars (leading zero-bytes)
' MessageBox.Show(Message, "", MessageBoxButtons.OK) ' Show the message. '
'
''' <summary>
''' Reads a byte-sequence from a <see cref="IO.MemoryMappedFiles.MemoryMappedFile"/> without knowing the exact size.
''' Note that the returned byte-length is rounded up to 4kb,
''' this means if the mapped memory-file was written with 1 byte-length, this method will return 4096 byte-length.
''' </summary>
''' <param name="Name">Indicates an existing <see cref="IO.MemoryMappedFiles.MemoryMappedFile"/> assigned name.</param>
''' <returns>System.Byte().</returns>
Private Function ReadMemoryMappedFile(ByVal Name As String) As Byte()
Try
Using MemoryFile As IO.MemoryMappedFiles.MemoryMappedFile =
IO.MemoryMappedFiles.MemoryMappedFile.OpenExisting(Name, IO.MemoryMappedFiles.MemoryMappedFileRights.ReadWrite)
Using Stream = MemoryFile.CreateViewStream()
Using Reader As New BinaryReader(Stream)
Return Reader.ReadBytes(CInt(Stream.Length))
End Using ' Reader
End Using ' Stream
End Using ' MemoryFile
Catch exNoFile As IO.FileNotFoundException
Throw
Return Nothing
Catch ex As Exception
Throw
End Try
End Function
I would like to have something from MemoryStream .ToArray() method to return all bytes, and the code below work for me:
using (MemoryMappedFile mmf = MemoryMappedFile.OpenExisting(MemoryMappedName))
{
using (MemoryMappedViewStream stream = mmf.CreateViewStream())
{
using (MemoryStream memStream = new MemoryStream())
{
stream.CopyTo(memStream);
return memStream.ToArray();
}
}
}
Cheers!
What is the best way to add text to the beginning of a file using C#?
I couldn't find a straightforward way to do this, but came up with a couple work-arounds.
Open up new file, write the text that I wanted to add, append the text from the old file to the end of the new file.
Since the text I want to add should be less than 200 characters, I was thinking that I could add white space characters to the beginning of the file, and then overwrite the white space with the text I wanted to add.
Has anyone else come across this problem, and if so, what did you do?
This works for me, but for small files. Probably it's not a very good solution otherwise.
string currentContent = String.Empty;
if (File.Exists(filePath))
{
currentContent = File.ReadAllText(filePath);
}
File.WriteAllText(filePath, newContent + currentContent );
Adding to the beginning of a file (prepending as opposed to appending) is generally not a supported operation. Your #1 options is fine. If you can't write a temp file, you can pull the entire file into memory, preprend your data to the byte array and then overwrite it back out (this is only really feasible if your files are small and you don't have to have a bunch in memory at once because prepending the array is not necessarily easy without a copy either).
Yeah, basically you can use something like this:
public static void PrependString(string value, FileStream file)
{
var buffer = new byte[file.Length];
while (file.Read(buffer, 0, buffer.Length) != 0)
{
}
if(!file.CanWrite)
throw new ArgumentException("The specified file cannot be written.", "file");
file.Position = 0;
var data = Encoding.Unicode.GetBytes(value);
file.SetLength(buffer.Length + data.Length);
file.Write(data, 0, data.Length);
file.Write(buffer, 0, buffer.Length);
}
public static void Prepend(this FileStream file, string value)
{
PrependString(value, file);
}
Then
using(var file = File.Open("yourtext.txt", FileMode.Open, FileAccess.ReadWrite))
{
file.Prepend("Text you want to write.");
}
Not really effective though in case of huge files.
using two streams, you can do it in place, but keep in mind that this will still loop over the whole file on every addition
using System;
using System.IO;
using System.Text;
namespace FilePrepender
{
public class FilePrepender
{
private string file=null;
public FilePrepender(string filePath)
{
file = filePath;
}
public void prependline(string line)
{
prepend(line + Environment.NewLine);
}
private void shiftSection(byte[] chunk,FileStream readStream,FileStream writeStream)
{
long initialOffsetRead = readStream.Position;
long initialOffsetWrite= writeStream.Position;
int offset = 0;
int remaining = chunk.Length;
do//ensure that the entire chunk length gets read and shifted
{
int read = readStream.Read(chunk, offset, remaining);
offset += read;
remaining -= read;
} while (remaining > 0);
writeStream.Write(chunk, 0, chunk.Length);
writeStream.Seek(initialOffsetWrite, SeekOrigin.Begin);
readStream.Seek(initialOffsetRead, SeekOrigin.Begin);
}
public void prepend(string text)
{
byte[] bytes = Encoding.Default.GetBytes(text);
byte[] chunk = new byte[bytes.Length];
using (FileStream readStream = File.Open(file, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
using(FileStream writeStream = File.Open(file, FileMode.OpenOrCreate, FileAccess.Write, FileShare.ReadWrite))
{
readStream.Seek(0, SeekOrigin.End);//seek chunk.Length past the end of the file
writeStream.Seek(chunk.Length, SeekOrigin.End);//which lets the loop run without special cases
long size = readStream.Position;
//while there's a whole chunks worth above the read head, shift the file contents down from the end
while(readStream.Position - chunk.Length >= 0)
{
readStream.Seek(-chunk.Length, SeekOrigin.Current);
writeStream.Seek(-chunk.Length, SeekOrigin.Current);
shiftSection(chunk, readStream, writeStream);
}
//clean up the remaining shift for the bytes that don't fit in size%chunk.Length
readStream.Seek(0, SeekOrigin.Begin);
writeStream.Seek(Math.Min(size, chunk.Length), SeekOrigin.Begin);
shiftSection(chunk, readStream, writeStream);
//finally, write the text you want to prepend
writeStream.Seek(0,SeekOrigin.Begin);
writeStream.Write(bytes, 0, bytes.Length);
}
}
}
}
}
I think the best way is to create a temp file. Add your text then read the contents of the original file adding it to the temp file. Then you can overwrite the original with the temp file.
prepend:
private const string tempDirPath = #"c:\temp\log.log", tempDirNewPath = #"c:\temp\log.new";
StringBuilder sb = new StringBuilder();
...
File.WriteAllText(tempDirNewPath, sb.ToString());
File.AppendAllText(tempDirNewPath, File.ReadAllText(tempDirPath));
File.Delete(tempDirPath);
File.Move(tempDirNewPath, tempDirPath);
using (FileStream fs = File.OpenWrite(tempDirPath))
{ //truncate to a reasonable length
if (16384 < fs.Length) fs.SetLength(16384);
fs.Close();
}
// The file we'll prepend to
string filePath = path + "\\log.log";
// A temp file we'll write to
string tempFilePath = path + "\\temp.log";
// 1) Write your prepended contents to a temp file.
using (var writer = new StreamWriter(tempFilePath, false))
{
// Write whatever you want to prepend
writer.WriteLine("Hi");
}
// 2) Use stream lib methods to append the original contents to the Temp
// file.
using (var oldFile = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Read, FileShare.Read))
{
using (var tempFile = new FileStream(tempFilePath, FileMode.Append, FileAccess.Write, FileShare.Read))
{
oldFile.CopyTo(tempFile);
}
}
// 3) Finally, dump the Temp file back to the original, keeping all its
// original permissions etc.
File.Replace(tempFilePath, filePath, null);
Even if what you're writing is small, the Temp file gets the entire original file appended to it before the .Replace(), so it does need to be on disk.
Note that this code is not Thread-safe; if more than one thread accesses this code you can lose writes in the file swapping going on here. That said, it's also pretty expensive, so you'd want to gate access to it anyway - pass writes via multiple Providers to a buffer, which periodically empties out via this prepend method on a single Consumer thread.
You should be able to do this without opening a new file. Use the following File method:
public static FileStream Open(
string path,
FileMode mode,
FileAccess access
)
Making sure to specify FileAccess.ReadWrite.
Using the FileStream returned from File.Open, read all of the existing data into memory. Then reset the pointer to the beginning of the file, write your new data, then write the existing data.
(If the file is big and/or you're suspicious of using too much memory, you can do this without having to read the whole file into memory, but implementing that is left as an exercise to the reader.)
The following algorithm may solve the problem pretty easily, it's most efficient for any size of file, including very big text files:
string outPutFile = #"C:\Output.txt";
string result = "Some new string" + DateTime.Now.ToString() + Environment.NewLine;
StringBuilder currentContent = new StringBuilder();
List<string> rawList = File.ReadAllLines(outPutFile).ToList();
foreach (var item in rawList) {
currentContent.Append(item + Environment.NewLine);
}
File.WriteAllText(outPutFile, result + currentContent.ToString());
Use this class:
public static class File2
{
private static readonly Encoding _defaultEncoding = new UTF8Encoding(false, true); // encoding used in File.ReadAll*()
private static object _bufferSizeLock = new Object();
private static int _bufferSize = 1024 * 1024; // 1mb
public static int BufferSize
{
get
{
lock (_bufferSizeLock)
{
return _bufferSize;
}
}
set
{
lock (_bufferSizeLock)
{
_bufferSize = value;
}
}
}
public static void PrependAllLines(string path, IEnumerable<string> contents)
{
PrependAllLines(path, contents, _defaultEncoding);
}
public static void PrependAllLines(string path, IEnumerable<string> contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllLines(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
public static void PrependAllText(string path, string contents)
{
PrependAllText(path, contents, _defaultEncoding);
}
public static void PrependAllText(string path, string contents, Encoding encoding)
{
var temp = Path.GetTempFileName();
File.WriteAllText(temp, contents, encoding);
AppendToTemp(path, temp, encoding);
File.Replace(temp, path, null);
}
private static void AppendToTemp(string path, string temp, Encoding encoding)
{
var bufferSize = BufferSize;
char[] buffer = new char[bufferSize];
using (var writer = new StreamWriter(temp, true, encoding))
{
using (var reader = new StreamReader(path, encoding))
{
int bytesRead;
while ((bytesRead = reader.ReadBlock(buffer,0,bufferSize)) != 0)
{
writer.Write(buffer,0,bytesRead);
}
}
}
}
}
Put the file's contents in a string. Append new data you want to add to the top of the file to that string -- string = newdata + string. Then move the seek position of the file to 0 and write the string into the file.