We Keep Coding

How can I return a collection of large files without putting them in memory?

I need to write an API that retrieves a collection of file streams and then returns the files. These files can get large (~1GB). I need this to be as quick as possible, but this service can't consume too much memory.
In the past when I've had to do something like this, the files weren't large, so I just created a ZIP in memory and returned that. I'm not able to do that this time due to the memory constraint. From what I can tell, multipart responses don't exist, so I can't do that either. What options do I have? Is there some way I can stream zip back as a response?
public async Task GetFiles(string someId)
{
List<Stream> streamList = GetStreams(someId);
using (ZipArchive archive = new ZipArchive(responseStream /* ?? */, ZipArchiveMode.Create, true))
{
...
}
}

You could try using a gzipsteam, which avoids loading the files in memory.
https://msdn.microsoft.com/en-us/library/system.io.compression.gzipstream(v=vs.110).aspx
Here is the example from the page:
using System;
using System.IO;
using System.IO.Compression;
namespace zip
{
public class Program
{
private static string directoryPath = #"c:\temp";
public static void Main()
{
DirectoryInfo directorySelected = new DirectoryInfo(directoryPath);
Compress(directorySelected);
foreach (FileInfo fileToDecompress in directorySelected.GetFiles("*.gz"))
{
Decompress(fileToDecompress);
}
}
public static void Compress(DirectoryInfo directorySelected)
{
foreach (FileInfo fileToCompress in directorySelected.GetFiles())
{
using (FileStream originalFileStream = fileToCompress.OpenRead())
{
if ((File.GetAttributes(fileToCompress.FullName) &
FileAttributes.Hidden) != FileAttributes.Hidden & fileToCompress.Extension != ".gz")
{
using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz"))
{
using (GZipStream compressionStream = new GZipStream(compressedFileStream,
CompressionMode.Compress))
{
originalFileStream.CopyTo(compressionStream);
}
}
FileInfo info = new FileInfo(directoryPath + "\\" + fileToCompress.Name + ".gz");
Console.WriteLine("Compressed {0} from {1} to {2} bytes.",
fileToCompress.Name, fileToCompress.Length.ToString(), info.Length.ToString());
}
}
}
}
public static void Decompress(FileInfo fileToDecompress)
{
using (FileStream originalFileStream = fileToDecompress.OpenRead())
{
string currentFileName = fileToDecompress.FullName;
string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);
using (FileStream decompressedFileStream = File.Create(newFileName))
{
using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
{
decompressionStream.CopyTo(decompressedFileStream);
Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
}
}
}
}
}
}

Crypted file is not working

I started to learn about cryptography and I wanted to make my own file crypter.
After you run the program you must select file to encrypt and then stub. For now it is now encrypting but it is binding two files. After selecting this two files it will create file first.exe. Splitter string is for program to know where stub code ends and when original file code starts. It is made like this
stub
splitter (some string)
original file
When you run first.exe then it will create file cos.exe which should be the original file. But when i run it, it doesnt work. It shows "This app is not going to work on your computer" and it says that i need to find version for my OS.
Here is the code for crypter:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication2
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
string stubpath, filepath;
private void Form1_Load(object sender, EventArgs e)
{
if (openFileDialog1.ShowDialog(this) == DialogResult.OK)
{
filepath = openFileDialog1.InitialDirectory + openFileDialog1.FileName;
}
if (openFileDialog1.ShowDialog(this) == DialogResult.OK)
{
stubpath = openFileDialog1.InitialDirectory + openFileDialog1.FileName;
}
byte[] originalfile = File.ReadAllBytes(filepath);
byte[] array = Encoding.ASCII.GetBytes("password");
byte[] originalstubfile = File.ReadAllBytes(stubpath);
byte[] arr1 = Combine(array, originalfile);
ByteArrayToFile("first.exe", Combine(originalstubfile,arr1));
}
public bool ByteArrayToFile(string fileName, byte[] byteArray)
{
try
{
using (var fs = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
fs.Write(byteArray, 0, byteArray.Length);
return true;
}
}
catch (Exception ex)
{
return false;
}
}
public static byte[] Combine(byte[] first, byte[] second)
{
byte[] ret = new byte[first.Length + second.Length];
Buffer.BlockCopy(first, 0, ret, 0, first.Length);
Buffer.BlockCopy(second, 0, ret, first.Length, second.Length);
return ret;
}
}
}
And here is the code for stub:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.IO;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.Windows.Forms;
namespace WindowsFormsApplication3
{
public partial class Form1 : Form
{
public Form1()
{
InitializeComponent();
}
private void Form1_Load(object sender, EventArgs e)
{
string exePath = Application.ExecutablePath;
byte[] originalfile = File.ReadAllBytes(exePath);
var str = System.Text.Encoding.Default.GetString(originalfile);
string[] tokens = str.Split(new[] { "password" }, StringSplitOptions.None);
//0 stub
//1 original file
byte[] toBytes = Encoding.ASCII.GetBytes(tokens[1]);
File.WriteAllBytes(System.AppDomain.CurrentDomain.BaseDirectory+ "cos.exe", toBytes);
}
}
}
EDIT:
Here is the error(Polish Language)
Also I found that when I delete splitter string and I dont use stub but I use 2 files like ex. putty.exe then it works, so I think that there is a problem in the stub.

Exception, named pipe + serialization

In the below example, NamedPipeServer is started or registered with name “File Transfer”.
The NamedPipeServer reads the content of a source file and creates an instance for “TransferFile” class setting the attributes such as source file name and file content (stream).
Server then serializes the "TransferFile" object and writes to the stream.
NamedPipeClient connects to the NamedPipeServer based whose server name is “File Transfer”.
and reads the stream. After reading the stream, Client deserializes to get the “TransferFile” object.
from the “FileTransfer” object, client creates the target file in specific directory with name from “FileName” attribute and content(byte[]) from “FileContent”.
Exception, when i use formatter.Deserialize(pipeClient).
TransferFile.cs
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
namespace FileTransfer
{
//file transfer class (serializable)
[Serializable]
public class TransferFile
{
public string FileName;
public Stream FileContent;
}
}
Named Pipe Server
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.IO.Pipes;
using System.Threading;
using System.Runtime.Serialization;
using FileTransfer;
namespace ServerNamedPipe
{
class Program
{
static void Main()
{
//creating object for file transfer
using (NamedPipeServerStream pipeServer =
new NamedPipeServerStream("File Transfer", PipeDirection.Out))
{
Console.WriteLine("File Transfer Named Pipe Stream is ready...");
Console.Write("Waiting for client connection...");//waiting for any client connections
pipeServer.WaitForConnection();
Console.WriteLine("Client connected.");
try
{
string strFile = #"c:\Test\1\Srinivas.txt";
//creating FileTransfer Object ans setting the file name
TransferFile objTransferFile = new TransferFile() { FileName = new FileInfo(strFile).Name };
objTransferFile.FileContent = new MemoryStream();
//opening the source file to read bytes
using (FileStream fs = File.Open(strFile, FileMode.Open, FileAccess.Read))
{
byte[] byteBuffer = new byte[1024];
int numBytes = fs.Read(byteBuffer, 0, 1024);
//writing the bytes to file transfer content stream
objTransferFile.FileContent.Write(byteBuffer, 0, 1024);
//below code is to write the bytes directly to the pipe stream
//pipeServer.Write(byteBuffer, 0, 1024);
//Reading each Kbyte and writing to the file content
while (numBytes > 0)
{
numBytes = fs.Read(byteBuffer, 0, numBytes);
objTransferFile.FileContent.Write(byteBuffer, 0, 1024);
//below code is to write the bytes to pipe stream directly
//pipeServer.Write(byteBuffer, 0, numBytes);
}
//setting the file content (stream) position to begining
objTransferFile.FileContent.Seek(0, SeekOrigin.Begin);
//serializing the file transfer object to a stream
IFormatter formatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
try
{
//serialzing and writing to pipe stream
formatter.Serialize(pipeServer, objTransferFile);
}
catch (Exception exp)
{
throw exp;
}
}
}
// Catch the IOException that is raised if the pipe is
// broken or disconnected.
catch (IOException e)
{
Console.WriteLine("ERROR: {0}", e.Message);
}
}
}
}
}
Named Pipe Client
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.IO.Pipes;
using System.Runtime.Serialization;
namespace ClientNamedPipe
{
class Program
{
static void Main(string[] args)
{
//connecting to the known pipe stream server which runs in localhost
using (NamedPipeClientStream pipeClient =
new NamedPipeClientStream(".", "File Transfer", PipeDirection.In))
{
// Connect to the pipe or wait until the pipe is available.
Console.Write("Attempting to connect to File Transfer pipe...");
//time out can also be specified
pipeClient.Connect();
Console.WriteLine("Connected to File Transfer pipe.");
IFormatter formatter = new System.Runtime.Serialization.Formatters.Binary.BinaryFormatter();
//deserializing the pipe stream recieved from server
FileTransfer.TransferFile objTransferFile = (FileTransfer.TransferFile)formatter.Deserialize(pipeClient);
//creating the target file with name same as specified in source which comes using
//file transfer object
byte[] byteBuffer = new byte[1024];
using (FileStream fs = new FileStream(#"c:\Test\2\" + objTransferFile.FileName, FileMode.Create, FileAccess.Write))
{
//writing each Kbyte to the target file
int numBytes = objTransferFile.FileContent.Read(byteBuffer, 0, 1024);
fs.Write(byteBuffer, 0, 1024);
while (numBytes > 0)
{
numBytes = objTransferFile.FileContent.Read(byteBuffer, 0, numBytes);
fs.Write(byteBuffer, 0, numBytes);
}
}
Console.WriteLine("File, Received from server: {0}", objTransferFile.FileName);
}
Console.Write("Press Enter to continue...");
Console.ReadLine();
}
}
}

You don't say what exception you got, specifically. But just looking at the code, the most obvious problem is that you have a Stream field in your TransferFile object. You can't serialize a Stream object.
One option would be to use a byte[] object instead in the TransferFile class. You can initialize it from your file using File.ReadAllBytes().
Note that serializing the whole file is not very efficient. If you want to make the best use of your pipe bandwidth, you should skip the whole serialization-based scheme, and instead use BinaryWriter.Write(string) to send the file name, and then just write the actual contents of the file to the stream.
For example:
Server:
static void Main()
{
//creating object for file transfer
using (NamedPipeServerStream pipeServer =
new NamedPipeServerStream("File Transfer", PipeDirection.Out))
{
Console.WriteLine("File Transfer Named Pipe Stream is ready...");
Console.Write("Waiting for client connection...");//waiting for any client connections
pipeServer.WaitForConnection();
Console.WriteLine("Client connected.");
try
{
string strFile = #"c:\Test\1\Srinivas.txt";
using (BinaryWriter writer = new BinaryWriter(pipeServer, Encoding.UTF8, true))
{
writer.Write(strFile);
}
//opening the source file to read bytes
using (FileStream fs = File.Open(strFile, FileMode.Open, FileAccess.Read))
{
fs.CopyTo(pipeServer);
}
}
// Catch the IOException that is raised if the pipe is
// broken or disconnected.
catch (IOException e)
{
Console.WriteLine("ERROR: {0}", e.Message);
}
}
}
Client:
static void Main(string[] args)
{
//connecting to the known pipe stream server which runs in localhost
using (NamedPipeClientStream pipeClient =
new NamedPipeClientStream(".", "File Transfer", PipeDirection.In))
{
// Connect to the pipe or wait until the pipe is available.
Console.Write("Attempting to connect to File Transfer pipe...");
//time out can also be specified
pipeClient.Connect();
Console.WriteLine("Connected to File Transfer pipe.");
using (BinaryReader reader = new BinaryReader(pipeClient, Encoding.UTF8, true))
{
string fileName = reader.ReadString();
}
//creating the target file with name same as specified in source which comes using
//file transfer object
using (FileStream fs = new FileStream(#"c:\Test\2\" + fileName, FileMode.Create, FileAccess.Write))
{
pipeClient.CopyTo(fs);
}
Console.WriteLine("File, Received from server: {0}", fileName);
}
Console.Write("Press Enter to continue...");
Console.ReadLine();
}

Deadlock with MemoryMappedFile streaming

I have a small problem performing MemoryMappedFile streaming.
I have 2 projects, one for sending bytes and another for reading bytes. Between these two processes should be a 2 second sleep timer.
I've implemented all of this, but when the software attempts to do the read it appears to encounter a deadlock. The code for both processes is below.
Can anyone help me find the issue?
namespace ProcesComunication
{
class Program
{
static void Main(string[] args)
{
MemoryMappedFile mmf = MemoryMappedFile.CreateNew("AAB", 1024);
MemoryMappedViewStream mStream = mmf.CreateViewStream();
BinaryWriter bw = new BinaryWriter(mStream);
Mutex mx = new Mutex(true, "sync");
while (true)
{
Thread.Sleep(2000);
Console.WriteLine("TEST");
bw.Write(DateTime.Now.ToString());
mx.ReleaseMutex();
}
bw.Close();
mStream.Close();
}
}
}
namespace ProcesRead
{
class Program
{
static void Main(string[] args)
{
MemoryMappedFile mmf = MemoryMappedFile.OpenExisting("AAB");
MemoryMappedViewStream mStream = mmf.CreateViewStream();
BinaryReader br = new BinaryReader(mStream);
Mutex mx = Mutex.OpenExisting("sync");
while (true)
{
mx.WaitOne();
Console.Write(br.ReadString());
mx.ReleaseMutex();
}
br.Close();
mStream.Close();
}
}
}

I tried and found simple solution, below is a code:
Thanks to all contributors for answers.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Threading;
using System.IO.MemoryMappedFiles;
namespace ProcesComunication
{
class Program
{
static void Main(string[] args)
{
MemoryMappedFile mmf = MemoryMappedFile.CreateNew("AAB", 1024);
MemoryMappedViewStream mStream = mmf.CreateViewStream();
BinaryWriter bw = new BinaryWriter(mStream);
Mutex mx = new Mutex(true, "sync");
while (true)
{
mx.WaitOne();
Thread.Sleep(2000);
var random = new Random();
var nextValue = random.Next().ToString();
Console.WriteLine(nextValue);
bw.Write(nextValue);
mx.ReleaseMutex();
}
bw.Close();
mStream.Close();
}
}
}
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Threading;
using System.IO.MemoryMappedFiles;
namespace ProcesRead
{
class Program
{
static void Main(string[] args)
{
MemoryMappedFile mmf = MemoryMappedFile.OpenExisting("AAB");
MemoryMappedViewStream mStream = mmf.CreateViewStream();
BinaryReader br = new BinaryReader(mStream);
Mutex emx = Mutex.OpenExisting("sync");
while (true)
{
Console.WriteLine(br.ReadString());
emx.WaitOne(2000);
}
br.Close();
mStream.Close();
}
}
}

There is no need to use a synchronization object (Mutex). MemoryMappedFile is thread safety between the processes. Don't use mutex. And control reader to it have data for reading.

Save modified WordprocessingDocument to new file

I'm attempting to open a Word document, change some text and then save the changes to a new document. I can get the first bit done using the code below but I can't figure out how to save the changes to a NEW document (specifying the path and file name).
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Diagnostics;
using DocumentFormat.OpenXml.Packaging;
using System.IO;
namespace WordTest
{
class Program
{
static void Main(string[] args)
{
string template = #"c:\data\hello.docx";
string documentText;
using (WordprocessingDocument wordDoc = WordprocessingDocument.Open(template, true))
{
using (StreamReader reader = new StreamReader(wordDoc.MainDocumentPart.GetStream()))
{
documentText = reader.ReadToEnd();
}
documentText = documentText.Replace("##Name##", "Paul");
documentText = documentText.Replace("##Make##", "Samsung");
using (StreamWriter writer = new StreamWriter(wordDoc.MainDocumentPart.GetStream(FileMode.Create)))
{
writer.Write(documentText);
}
}
}
}
}
I'm a complete beginner at this, so forgive the basic question!

If you use a MemoryStream you can save the changes to a new file like this:
byte[] byteArray = File.ReadAllBytes("c:\\data\\hello.docx");
using (MemoryStream stream = new MemoryStream())
{
stream.Write(byteArray, 0, (int)byteArray.Length);
using (WordprocessingDocument wordDoc = WordprocessingDocument.Open(stream, true))
{
// Do work here
}
// Save the file with the new name
File.WriteAllBytes("C:\\data\\newFileName.docx", stream.ToArray());
}

In Open XML SDK 2.5:
File.Copy(originalFilePath, modifiedFilePath);
using (var wordprocessingDocument = WordprocessingDocument.Open(modifiedFilePath, isEditable: true))
{
// Do changes here...
}
wordprocessingDocument.AutoSave is true by default so Close and Dispose will save changes.
wordprocessingDocument.Close is not needed explicitly because the using block will call it.
This approach doesn't require entire file content to be loaded into memory like in accepted answer. It isn't a problem for small files, but in my case I have to process more docx files with embedded xlsx and pdf content at the same time so the memory usage would be quite high.

Simply copy the source file to the destination and make changes from there.
File.copy(source,destination);
using (WordprocessingDocument wordDoc = WordprocessingDocument.Open(destination, true))
{
\\Make changes to the document and save it.
wordDoc.MainDocumentPart.Document.Save();
wordDoc.Close();
}
Hope this works.

This approach allows you to buffer the "template" file without batching the whole thing into a byte[], perhaps allowing it to be less resource intensive.
var templatePath = #"c:\data\hello.docx";
var documentPath = #"c:\data\newFilename.docx";
using (var template = File.OpenRead(templatePath))
using (var documentStream = File.Open(documentPath, FileMode.OpenOrCreate))
{
template.CopyTo(documentStream);
using (var document = WordprocessingDocument.Open(documentStream, true))
{
//do your work here
document.MainDocumentPart.Document.Save();
}
}

For me this worked fine:
// To search and replace content in a document part.
public static void SearchAndReplace(string document)
{
using (WordprocessingDocument wordDoc = WordprocessingDocument.Open(document, true))
{
string docText = null;
using (StreamReader sr = new StreamReader(wordDoc.MainDocumentPart.GetStream()))
{
docText = sr.ReadToEnd();
}
Regex regexText = new Regex("Hello world!");
docText = regexText.Replace(docText, "Hi Everyone!");
using (StreamWriter sw = new StreamWriter(wordDoc.MainDocumentPart.GetStream(FileMode.Create)))
{
sw.Write(docText);
}
}
}