We Keep Coding

How to create a .tar.gz file from text string

I have a dlm file and I want to create a .tar.gz file from the content in dlm file. When I am trying to create the file, it is created but when I manually unzip that it is failed. My code is below for creating .tar.gz file, targetFileName is like C:\Folder\xxx.tar.gz:
using (StreamWriter write = new StreamWriter(targetFileName, false, Encoding.Default))
{
write.Write(text.ToString());
write.Close();
}
In the above code text is content from dlm file. Is there anything that I am missing? please help.

try use SharpZipLib from Nuget
using System;
using System.IO;
using System.Text;
using ICSharpCode.SharpZipLib.GZip;
using ICSharpCode.SharpZipLib.Tar;
add method:
private static void CreateTarGZ(string tgzFilename, string innerFilename, string text)
{
var uncompressed = Encoding.UTF8.GetBytes(text);
using (Stream outStream = File.Create(tgzFilename))
{
using (GZipOutputStream gzoStream = new GZipOutputStream(outStream))
{
gzoStream.SetLevel(9);
using (TarOutputStream taroStream = new TarOutputStream(gzoStream, Encoding.UTF8))
{
taroStream.IsStreamOwner = false;
TarEntry entry = TarEntry.CreateTarEntry(innerFilename);
entry.Size = uncompressed.Length;
taroStream.PutNextEntry(entry);
taroStream.Write(uncompressed, 0, uncompressed.Length);
taroStream.CloseEntry();
taroStream.Close();
}
}
}
}
then call:
CreateTarGZ("test.tar.gz", "FileName.txt", "my text");
CreateTarGZ("c:\\temp\\test.tar.gz", "foo-folder\\FileName.txt", "my text");

This is a quick example to create a .tar.gz and .gz file that will include the file that you might be creating using the stream.
Note that I'm using SharpZipLib which you can find in Nuget Package Manager for you project. Then make sure to add reference in your code:
Making tar.gz
using ICSharpCode.SharpZipLib.GZip;
using ICSharpCode.SharpZipLib.Tar;
using System.IO;
using System.Text;
static void Main(string[] args)
{
string text = ".Net is Awesome";
string filename = "D:\\text.txt";
string tarfilename = "D:\\text.tar.gz";
using (StreamWriter write = new StreamWriter(filename, false, Encoding.Default))
{
//Writing a text file
write.Write(text.ToString());
write.Close();
//Creating a tar.gz Stream
Stream TarFileStream = File.Create(tarfilename);
Stream GZStream = new GZipOutputStream(TarFileStream);
TarArchive tarArchive = TarArchive.CreateOutputTarArchive(GZStream);
tarArchive.RootPath = "D:/"; //Setting the Root Path for the archive
//Creating a file entry for the tar archive
TarEntry tarEntry = TarEntry.CreateEntryFromFile(filename);
//Writing the entry in the archive.
tarArchive.WriteEntry(tarEntry, false); //set false to only add the concerned file in the archive.
tarArchive.Close();
}
}
Making only .gz
You can create a method to make it more reusable like:
private static void MakeGz(string targetFile)
{
string TargetGz = targetFile + ".gz";
using (Stream GzStream = new GZipOutputStream(File.Create(TargetGz)))
{
using (FileStream fs = File.OpenRead(targetFile))
{
byte[] FileBuffer = new byte[fs.Length];
fs.Read(FileBuffer, 0, (int)fs.Length);
GzStream.Write(FileBuffer, 0, FileBuffer.Length);
fs.Close();
GzStream.Close();
}
}
}
Then you can call this method whenever you are creating a file to make an archive for the same at the same time like:
MakeGz(filename);

Avoid copying compressed data when using DeflateStream

Assume we have given an API function f(Stream s) to put binary data contained in a stream into a database. I want to put a file into the database using f but I want to compress the data in advance. Hence I thought I could do the following:
var fileStream= File.OpenRead(path);
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
f(dstream);
But it seems DeflateStream only writes into the stream fileStream but does not read from it when compressing. In all examples I found, the CopyTo method of the stream was used to compress or decompress. But this would mean that I have to keep a copy of the compressed data in memory before passing it to f for instance like this:
var memoryStream = new MemoryStream();
using(var fileStream= File.OpenRead(path))
using(var dstream = new DeflateStream(memoryStream, CompressionLevel.Optimal)) {
fileStream.CopyTo(dstream);
memoryStream.Seek(0, SeekOrigin.Begin);
f(memoryStream);
}
Is there any way to avoid using the MemoryStream?
Update
For the sake of the persistency of some commentators I add a complete example:
using System;
using System.IO;
using System.IO.Compression;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
dummyDatabaseClient.f(dstream);
}
}
The read operation in the dummy implementation of f throws an exception: InvalidOperationException: Reading from the compression stream is not supported. Concluding the discussion in the comments, I assume that the desired behaviour is not possible with DeflateStream but there are alternatives in third party libraries.

You can use SharpCompress for this. Its DeflateStream allows you to read the compressed data on the fly, which is exactly what you want.
Here's a complete example based on Sir Rufo's:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
using System.Linq;
public class Program
{
public static void Main()
{
var dataBuffer = Enumerable.Range(1, 50000).Select(e => (byte)(e % 256)).ToArray();
using (var dataStream = new MemoryStream(dataBuffer))
{
// Note: this refers to SharpCompress.Compressors.Deflate.DeflateStream
using (var deflateStream = new DeflateStream(dataStream, CompressionMode.Compress))
{
ConsumeStream(deflateStream);
}
}
}
public static void ConsumeStream(Stream stream)
{
// Let's just prove we can reinflate to the original data...
byte[] data;
using (var decompressed = new MemoryStream())
{
using (var decompressor = new DeflateStream(stream, CompressionMode.Decompress))
{
decompressor.CopyTo(decompressed);
}
data = decompressed.ToArray();
}
Console.WriteLine("Reinflated size: " + data.Length);
int errors = 0;
for (int i = 0; i < data.Length; i++)
{
if (data[i] != (i + 1) % 256)
{
errors++;
}
}
Console.WriteLine("Total errors: " + errors);
}
}
Or using your sample code:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(
fileStream, CompressionMode.Compress, CompressionLevel.BestCompression))
dummyDatabaseClient.f(dstream);
}
}
This now doesn't throw an exception, and will serve the compressed data.

The DeflateStream is just a wrapper and needs a stream for the compressed data. So you have to use two streams.
Is there any way to avoid using the MemoryStream?
Yes.
You need a stream to store temporary data without consuming (too much) memory. Instead using MemoryStream you can use a temporary file for that.
For the lazy people (like me in first place) let's create a class that will behave mostly like a MemoryStream
public class TempFileStream : FileStream
{
public TempFileStream() : base(
path: Path.GetTempFileName(),
mode: FileMode.Open,
access: FileAccess.ReadWrite,
share: FileShare.None,
bufferSize: 4096,
options: FileOptions.DeleteOnClose | FileOptions.Asynchronous | FileOptions.Encrypted | FileOptions.RandomAccess)
{
}
}
The important part here is FileOptions.DeleteOnClose which will remove the temporary file when you dispose the stream.
And then use it
using (var compressedStream = new TempFileStream())
{
using (var deflateStream = new DeflateStream(
stream: compressedStream,
compressionLevel: CompressionLevel.Optimal,
leaveOpen: true))
using (var fileStream = File.OpenRead(path))
{
fileStream.CopyTo(deflateStream);
}
f(compressedStream);
}

How to decompress a Fody.Costura packed assembly

A friend of mine gave me the challenge to decompress an assembly that was packed with Fody.Costura. The assembly has a dll dependency that was embedded as a resource. I tried to extract this .zip resource with dotPeek and decompress it with this code here
public static void Decompress(string path)
{
using (var stream = File.OpenRead(path))
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress))
{
compressStream.Seek(0, SeekOrigin.Begin);
var fs = File.Create(path + ".decompressed");
compressStream.CopyTo(fs);
fs.Close();
}
}
This works when it comes to extracting the .zip but the result is quite unuseful
Is there a suitable solution to decompress this packed dll?

This is my simple C# Console App code (Framework 4), which I`m using simply by "drag and drop" (Costura Compressed) files over compiled (ConsoleApp1) executable.
using System;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text.RegularExpressions;
namespace ConsoleApp1
{
class Program
{
static void CopyTo(Stream source, Stream destination) {
int count;
var array = new byte[81920];
while ((count = source.Read(array, 0, array.Length)) != 0) {
destination.Write(array, 0, count);
}
}
static Stream LoadStream(string fullname) {
FileStream stream = default(FileStream);
if (fullname.EndsWith(".zip")) {
using (stream = new FileStream(fullname, FileMode.Open)) {
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress)) {
var memStream = new MemoryStream();
CopyTo(compressStream, memStream);
memStream.Position = 0;
return memStream;
}
}
}
return stream;
}
static void Main(string[] args) {
Stream stream; Stream file;
string RefilePath = #"^.+[^\\]+\\"; string fullname; string newFile;
for (int i = 0; i < args.Count(); i++) {
fullname = args[i];
newFile = Regex.Replace(fullname, "\\.zip$", string.Empty);
Console.Write("{0} -> {1}\r\n",
Regex.Replace(fullname, RefilePath, string.Empty),
Regex.Replace(newFile, RefilePath, string.Empty));
try
{
stream = LoadStream(fullname);
using (file = File.Create(newFile)) {
CopyTo(stream, file);
}
}
catch (Exception ex) {
Console.Write("{0}", ex.ToString());
}
}
}
}
}
Based On Cameron MacFarland Answer

The code that Costura uses to decompress those resources is here.
https://github.com/Fody/Costura/blob/master/src/Costura.Template/Common.cs
static void CopyTo(Stream source, Stream destination)
{
var array = new byte[81920];
int count;
while ((count = source.Read(array, 0, array.Length)) != 0)
{
destination.Write(array, 0, count);
}
}
static Stream LoadStream(string fullname)
{
var executingAssembly = Assembly.GetExecutingAssembly();
if (fullname.EndsWith(".zip"))
{
using (var stream = executingAssembly.GetManifestResourceStream(fullname))
using (var compressStream = new DeflateStream(stream, CompressionMode.Decompress))
{
var memStream = new MemoryStream();
CopyTo(compressStream, memStream);
memStream.Position = 0;
return memStream;
}
}
return executingAssembly.GetManifestResourceStream(fullname);
}

To decompress those resources there is this project to.

Creating a Zip in memory with SharpZipLib resulting in a corrupt archive

I am using ICSharpCode.SharpZipLib v 0.86.0.518.
I have a a stream whose contents I would like to add as a file to a Zip, which must also be created in memory (as opposed to on on disk).
The resulting Zip opens for browsing, but when attempting to extract any contents, I get an error stating "The Archive is either in an unknown format or damaged".
In the code below, when asZip=false the text file is sent, and received as expected. When asZip=true, the file is sent, but suffers the corruption described above.
When I replace MemoryStream zipStream, with FileStream zipStream, the file on disk is OK.
Can anyone see what I've missed?
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net; // .NET 4.0
using System.Net.Http;
using System.Net.Http.Headers;
using System.Web.Mvc;
using ICSharpCode.SharpZipLib.Zip;//0.86.0.518
namespace Demo
{
public class DemoApiController : Controller
{
/// <summary>
/// demos the zipfile error
/// </summary>
/// <param name="withFiles">Creates the zip if set to <c>true</c> [with files].</param>
[HttpGet]
public void ZipErrorDemo(bool asZip)
{
const string fileContent = "Hello World!";
MemoryStream rawContentStream = new MemoryStream(System.Text.Encoding.ASCII.GetBytes(fileContent));
if (!asZip)
{
//This File is recieved as text, opens without erros and has correct content.
WriteStreamToDownload(rawContentStream, "text/plain", "HelloWorld.txt");
}
else
{
MemoryStream zipStream = new MemoryStream(1024 * 2048);//2MB
using (ZipOutputStream s = new ZipOutputStream(zipStream))
{
s.UseZip64 = ICSharpCode.SharpZipLib.Zip.UseZip64.Off; //No Zip64 for better compatability
s.SetLevel(0); //No compression
byte[] buffer = new byte[4096];
//Add the text file
ZipEntry csvEntry = new ZipEntry("HelloWorld.txt");
s.PutNextEntry(csvEntry);
int sourceBytes = 0;
do
{
sourceBytes = rawContentStream.Read(buffer, 0, buffer.Length);
s.Write(buffer, 0, sourceBytes);
} while (sourceBytes > 0);
s.CloseEntry();
s.IsStreamOwner = false;//Tells s.Close to not mess invoke zipStream.Close()
s.Flush();
s.Finish();
s.Close();
byte[] streamBuffer = zipStream.GetBuffer();//Before doing this things were worse.
MemoryStream newStream = new MemoryStream(streamBuffer);
//This File is recieved as a zip, opens to list contents, but atemtps at extraction result in an error.
WriteStreamToDownload(newStream, "application/zip", "HelloWorld.zip");
}
}
}
// Adapted from: http://stackoverflow.com/questions/5596747/download-stream-file-from-url-asp-net
// Accessed: 3/17/15. Works.
private static void WriteStreamToDownload(Stream stream, string contentType, string fileName)
{
// 100 kb
const int bytesToRead = 102400;
byte[] buffer = new byte[bytesToRead];
var contextResponse = System.Web.HttpContext.Current.Response;
try
{
contextResponse.ContentType = contentType;
contextResponse.AddHeader("Content-Disposition", "attachment; filename=\"" + Path.GetFileName(fileName) + "\"");
contextResponse.AddHeader("Content-Length", stream.Length.ToString());
int length;
do
{
if (contextResponse.IsClientConnected)
{
length = stream.Read(buffer, 0, bytesToRead);
contextResponse.OutputStream.Write(buffer, 0, length);
contextResponse.Flush();
buffer = new byte[bytesToRead];
}
else
{
length = -1;
}
} while (length > 0);
}
finally
{
if (stream != null)
{
stream.Close();
stream.Dispose();
}
}
}
}
}

At the moment, you are over-reading your stream. GetBuffer() returns the oversized backing buffer; you should usually limit yourself to the first zipStream.Length bytes of the buffer.
So the first thing to try is:
MemoryStream newStream = new MemoryStream(streamBuffer, 0
(int)zipStream.Length);
However, if that works, you can probably also achieve the same thing by simply sending zipStream, as long as you:
rewind the stream after writing
ensure that it doesn't get closed by the using
You might also be interested to hear that zip support is present inside the .NET framework itself, without requiring additional tools.
Your copying code, btw, is inefficient (especially when the buffer is constantly recreated) and could probably just use:
stream.CopyTo(contextResponse.OutputStream);

Copy between two streams in .net 2.0

I have been using the following code to Compress data in .Net 4.0:
public static byte[] CompressData(byte[] data_toCompress)
{
using (MemoryStream outFile = new MemoryStream())
{
using (MemoryStream inFile = new MemoryStream(data_toCompress))
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
inFile.CopyTo(Compress);
}
return outFile.ToArray();
}
}
However, in .Net 2.0 Stream.CopyTo method is not available. So, I tried making a replacement:
public static byte[] CompressData(byte[] data_toCompress)
{
using (MemoryStream outFile = new MemoryStream())
{
using (MemoryStream inFile = new MemoryStream(data_toCompress))
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
//inFile.CopyTo(Compress);
Compress.Write(inFile.GetBuffer(), (int)inFile.Position, (int)(inFile.Length - inFile.Position));
}
return outFile.ToArray();
}
}
The compression fails, though, when using the above attempt - I get an error saying:
MemoryStream's internal buffer cannot be accessed.
Could anyone offer any help on this issue? I'm really not sure what else to do here.
Thank you,
Evan

This is the code straight out of .Net 4.0 Stream.CopyTo method (bufferSize is 4096):
byte[] buffer = new byte[bufferSize];
int count;
while ((count = this.Read(buffer, 0, buffer.Length)) != 0)
destination.Write(buffer, 0, count);

Since you have access to the array already, why don't you do this:
using (MemoryStream outFile = new MemoryStream())
{
using (GZipStream Compress = new GZipStream(outFile, CompressionMode.Compress))
{
Compress.Write(data_toCompress, 0, data_toCompress.Length);
}
return outFile.ToArray();
}
Most likely in the sample code you are using inFile.GetBuffer() will throw an exception since you do not use the right constructor - not all MemoryStream instances allow you access to the internal buffer - you have to look for this in the documentation:
Initializes a new instance of the MemoryStream class based on the
specified region of a byte array, with the CanWrite property set as
specified, and the ability to call GetBuffer set as specified.
This should work - but is not needed anyway in the suggested solution:
using (MemoryStream inFile = new MemoryStream(data_toCompress,
0,
data_toCompress.Length,
false,
true))

Why are you constructing a memory stream with an array and then trying to pull the array back out of the memory stream?
You could just do Compress.Write(data_toCompress, 0, data_toCompress.Length);
If you need to replace the functionality of CopyTo, you can create a buffer array of some length, read data from the source stream and write that data to the destination stream.

You can try
infile.WriteTo(Compress);

try to replace the line:
Compress.Write(inFile.GetBuffer(), (int)inFile.Position, (int)(inFile.Length - inFile.Position));
with:
Compress.Write(data_toCompress, 0, data_toCompress.Length);
you can get rid of this line completely:
using (MemoryStream inFile = new MemoryStream(data_toCompress))
Edit: find an example here: Why does gzip/deflate compressing a small file result in many trailing zeroes?

You should manually read and write between these 2 streams:
private static void CopyStream(Stream from, Stream to)
{
int bufSize = 1024, count;
byte[] buffer = new byte[bufSize];
count = from.Read(buffer, 0, bufSize);
while (count > 0)
{
to.Write(buffer, 0, count);
count = from.Read(buffer, 0, bufSize);
}
}

The open-source NuGet package Stream.CopyTo implements Stream.CopyTo for all versions of the .NET Framework.
Available on GitHub and via NuGet (Install-Package Stream.CopyTo)