We Keep Coding

How to create a .tar.gz file from text string

I have a dlm file and I want to create a .tar.gz file from the content in dlm file. When I am trying to create the file, it is created but when I manually unzip that it is failed. My code is below for creating .tar.gz file, targetFileName is like C:\Folder\xxx.tar.gz:
using (StreamWriter write = new StreamWriter(targetFileName, false, Encoding.Default))
{
write.Write(text.ToString());
write.Close();
}
In the above code text is content from dlm file. Is there anything that I am missing? please help.

try use SharpZipLib from Nuget
using System;
using System.IO;
using System.Text;
using ICSharpCode.SharpZipLib.GZip;
using ICSharpCode.SharpZipLib.Tar;
add method:
private static void CreateTarGZ(string tgzFilename, string innerFilename, string text)
{
var uncompressed = Encoding.UTF8.GetBytes(text);
using (Stream outStream = File.Create(tgzFilename))
{
using (GZipOutputStream gzoStream = new GZipOutputStream(outStream))
{
gzoStream.SetLevel(9);
using (TarOutputStream taroStream = new TarOutputStream(gzoStream, Encoding.UTF8))
{
taroStream.IsStreamOwner = false;
TarEntry entry = TarEntry.CreateTarEntry(innerFilename);
entry.Size = uncompressed.Length;
taroStream.PutNextEntry(entry);
taroStream.Write(uncompressed, 0, uncompressed.Length);
taroStream.CloseEntry();
taroStream.Close();
}
}
}
}
then call:
CreateTarGZ("test.tar.gz", "FileName.txt", "my text");
CreateTarGZ("c:\\temp\\test.tar.gz", "foo-folder\\FileName.txt", "my text");

This is a quick example to create a .tar.gz and .gz file that will include the file that you might be creating using the stream.
Note that I'm using SharpZipLib which you can find in Nuget Package Manager for you project. Then make sure to add reference in your code:
Making tar.gz
using ICSharpCode.SharpZipLib.GZip;
using ICSharpCode.SharpZipLib.Tar;
using System.IO;
using System.Text;
static void Main(string[] args)
{
string text = ".Net is Awesome";
string filename = "D:\\text.txt";
string tarfilename = "D:\\text.tar.gz";
using (StreamWriter write = new StreamWriter(filename, false, Encoding.Default))
{
//Writing a text file
write.Write(text.ToString());
write.Close();
//Creating a tar.gz Stream
Stream TarFileStream = File.Create(tarfilename);
Stream GZStream = new GZipOutputStream(TarFileStream);
TarArchive tarArchive = TarArchive.CreateOutputTarArchive(GZStream);
tarArchive.RootPath = "D:/"; //Setting the Root Path for the archive
//Creating a file entry for the tar archive
TarEntry tarEntry = TarEntry.CreateEntryFromFile(filename);
//Writing the entry in the archive.
tarArchive.WriteEntry(tarEntry, false); //set false to only add the concerned file in the archive.
tarArchive.Close();
}
}
Making only .gz
You can create a method to make it more reusable like:
private static void MakeGz(string targetFile)
{
string TargetGz = targetFile + ".gz";
using (Stream GzStream = new GZipOutputStream(File.Create(TargetGz)))
{
using (FileStream fs = File.OpenRead(targetFile))
{
byte[] FileBuffer = new byte[fs.Length];
fs.Read(FileBuffer, 0, (int)fs.Length);
GzStream.Write(FileBuffer, 0, FileBuffer.Length);
fs.Close();
GzStream.Close();
}
}
}
Then you can call this method whenever you are creating a file to make an archive for the same at the same time like:
MakeGz(filename);

Avoid copying compressed data when using DeflateStream

Assume we have given an API function f(Stream s) to put binary data contained in a stream into a database. I want to put a file into the database using f but I want to compress the data in advance. Hence I thought I could do the following:
var fileStream= File.OpenRead(path);
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
f(dstream);
But it seems DeflateStream only writes into the stream fileStream but does not read from it when compressing. In all examples I found, the CopyTo method of the stream was used to compress or decompress. But this would mean that I have to keep a copy of the compressed data in memory before passing it to f for instance like this:
var memoryStream = new MemoryStream();
using(var fileStream= File.OpenRead(path))
using(var dstream = new DeflateStream(memoryStream, CompressionLevel.Optimal)) {
fileStream.CopyTo(dstream);
memoryStream.Seek(0, SeekOrigin.Begin);
f(memoryStream);
}
Is there any way to avoid using the MemoryStream?
Update
For the sake of the persistency of some commentators I add a complete example:
using System;
using System.IO;
using System.IO.Compression;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(fileStream, CompressionLevel.Optimal))
dummyDatabaseClient.f(dstream);
}
}
The read operation in the dummy implementation of f throws an exception: InvalidOperationException: Reading from the compression stream is not supported. Concluding the discussion in the comments, I assume that the desired behaviour is not possible with DeflateStream but there are alternatives in third party libraries.

You can use SharpCompress for this. Its DeflateStream allows you to read the compressed data on the fly, which is exactly what you want.
Here's a complete example based on Sir Rufo's:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
using System.Linq;
public class Program
{
public static void Main()
{
var dataBuffer = Enumerable.Range(1, 50000).Select(e => (byte)(e % 256)).ToArray();
using (var dataStream = new MemoryStream(dataBuffer))
{
// Note: this refers to SharpCompress.Compressors.Deflate.DeflateStream
using (var deflateStream = new DeflateStream(dataStream, CompressionMode.Compress))
{
ConsumeStream(deflateStream);
}
}
}
public static void ConsumeStream(Stream stream)
{
// Let's just prove we can reinflate to the original data...
byte[] data;
using (var decompressed = new MemoryStream())
{
using (var decompressor = new DeflateStream(stream, CompressionMode.Decompress))
{
decompressor.CopyTo(decompressed);
}
data = decompressed.ToArray();
}
Console.WriteLine("Reinflated size: " + data.Length);
int errors = 0;
for (int i = 0; i < data.Length; i++)
{
if (data[i] != (i + 1) % 256)
{
errors++;
}
}
Console.WriteLine("Total errors: " + errors);
}
}
Or using your sample code:
using System;
using System.IO;
using SharpCompress.Compressors;
using SharpCompress.Compressors.Deflate;
public class ThisWouldBeTheDatabaseClient {
public void f(Stream s) {
// some implementation I don't have access to
// The only thing I know is that it reads data from the stream in some way.
var buffer = new byte[10];
s.Read(buffer,0,10);
}
}
public class Program {
public static void Main() {
var dummyDatabaseClient = new ThisWouldBeTheDatabaseClient();
var dataBuffer = new byte[1000];
var fileStream= new MemoryStream( dataBuffer ); // would be "File.OpenRead(path)" in real case
using(var dstream = new DeflateStream(
fileStream, CompressionMode.Compress, CompressionLevel.BestCompression))
dummyDatabaseClient.f(dstream);
}
}
This now doesn't throw an exception, and will serve the compressed data.

The DeflateStream is just a wrapper and needs a stream for the compressed data. So you have to use two streams.
Is there any way to avoid using the MemoryStream?
Yes.
You need a stream to store temporary data without consuming (too much) memory. Instead using MemoryStream you can use a temporary file for that.
For the lazy people (like me in first place) let's create a class that will behave mostly like a MemoryStream
public class TempFileStream : FileStream
{
public TempFileStream() : base(
path: Path.GetTempFileName(),
mode: FileMode.Open,
access: FileAccess.ReadWrite,
share: FileShare.None,
bufferSize: 4096,
options: FileOptions.DeleteOnClose | FileOptions.Asynchronous | FileOptions.Encrypted | FileOptions.RandomAccess)
{
}
}
The important part here is FileOptions.DeleteOnClose which will remove the temporary file when you dispose the stream.
And then use it
using (var compressedStream = new TempFileStream())
{
using (var deflateStream = new DeflateStream(
stream: compressedStream,
compressionLevel: CompressionLevel.Optimal,
leaveOpen: true))
using (var fileStream = File.OpenRead(path))
{
fileStream.CopyTo(deflateStream);
}
f(compressedStream);
}

Can we unzip file in FTP server using C#

Can I extract the ZIP file in FTP and place this extracted file on the same location using C#?

It's not possible.
There's no API in the FTP protocol to un-ZIP a file on a server.
Though, it's not uncommon that one, in addition to an FTP access, have also an SSH access. If that's the case, you can connect with the SSH and execute the unzip shell command (or similar) on the server to decompress the files.
See C# send a simple SSH command.
If you need, you can then download the extracted files using the FTP protocol (Though if you have the SSH access, you will also have an SFTP access. Then, use the SFTP instead of the FTP.).
Some (very few) FTP servers offer an API to execute an arbitrary shell (or other) commands using the SITE EXEC command (or similar). But that's really very rare. You can use this API the same way as the SSH above.
If you want to download and unzip the file locally, you can do it in-memory, without storing the ZIP file to physical (temporary) file. For an example, see How to import data from a ZIP file stored on FTP server to database in C#.

Download via FTP to MemoryStream, then you can unzip, example shows how to get stream, just change to MemoryStream and unzip. Example doesn't use MemoryStream but if you are familiar with streams it should be trivial to modify these two examples to work for you.
example from: https://learn.microsoft.com/en-us/dotnet/framework/network-programming/how-to-download-files-with-ftp
using System;
using System.IO;
using System.Net;
using System.Text;
namespace Examples.System.Net
{
public class WebRequestGetExample
{
public static void Main ()
{
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://www.contoso.com/test.htm");
request.Method = WebRequestMethods.Ftp.DownloadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential ("anonymous","janeDoe#contoso.com");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
StreamReader reader = new StreamReader(responseStream);
Console.WriteLine(reader.ReadToEnd());
Console.WriteLine("Download Complete, status {0}", response.StatusDescription);
reader.Close();
response.Close();
}
}
}
decompress stream, example from: https://learn.microsoft.com/en-us/dotnet/standard/io/how-to-compress-and-extract-files
using System;
using System.IO;
using System.IO.Compression;
namespace ConsoleApplication
{
class Program
{
static void Main(string[] args)
{
using (FileStream zipToOpen = new FileStream(#"c:\users\exampleuser\release.zip", FileMode.Open))
{
using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update))
{
ZipArchiveEntry readmeEntry = archive.CreateEntry("Readme.txt");
using (StreamWriter writer = new StreamWriter(readmeEntry.Open()))
{
writer.WriteLine("Information about this package.");
writer.WriteLine("========================");
}
}
}
}
}
}
here is a working example of downloading zip file from ftp, decompressing that zip file and then uploading the compressed files back to the same ftp directory
using System.IO;
using System.IO.Compression;
using System.Net;
using System.Text;
namespace ConsoleApp1
{
class Program
{
static void Main(string[] args)
{
string location = #"ftp://localhost";
byte[] buffer = null;
using (MemoryStream ms = new MemoryStream())
{
FtpWebRequest fwrDownload = (FtpWebRequest)WebRequest.Create($"{location}/test.zip");
fwrDownload.Method = WebRequestMethods.Ftp.DownloadFile;
fwrDownload.Credentials = new NetworkCredential("anonymous", "janeDoe#contoso.com");
using (FtpWebResponse response = (FtpWebResponse)fwrDownload.GetResponse())
using (Stream stream = response.GetResponseStream())
{
//zipped data stream
//https://stackoverflow.com/a/4924357
byte[] buf = new byte[1024];
int byteCount;
do
{
byteCount = stream.Read(buf, 0, buf.Length);
ms.Write(buf, 0, byteCount);
} while (byteCount > 0);
//ms.Seek(0, SeekOrigin.Begin);
buffer = ms.ToArray();
}
}
//include System.IO.Compression AND System.IO.Compression.FileSystem assemblies
using (MemoryStream ms = new MemoryStream(buffer))
using (ZipArchive archive = new ZipArchive(ms, ZipArchiveMode.Update))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
FtpWebRequest fwrUpload = (FtpWebRequest)WebRequest.Create($"{location}/{entry.FullName}");
fwrUpload.Method = WebRequestMethods.Ftp.UploadFile;
fwrUpload.Credentials = new NetworkCredential("anonymous", "janeDoe#contoso.com");
byte[] fileContents = null;
using (StreamReader sr = new StreamReader(entry.Open()))
{
fileContents = Encoding.UTF8.GetBytes(sr.ReadToEnd());
}
if (fileContents != null)
{
fwrUpload.ContentLength = fileContents.Length;
try
{
using (Stream requestStream = fwrUpload.GetRequestStream())
{
requestStream.Write(fileContents, 0, fileContents.Length);
}
}
catch(WebException e)
{
string status = ((FtpWebResponse)e.Response).StatusDescription;
}
}
}
}
}
}
}

If you're trying to unzip the files in place after they have been ftp uploaded, you will need to run a server side script with proper permissions that can be fired from within your c# application, or c# ssh as already described earlier.

Creating a Zip in memory with SharpZipLib resulting in a corrupt archive

I am using ICSharpCode.SharpZipLib v 0.86.0.518.
I have a a stream whose contents I would like to add as a file to a Zip, which must also be created in memory (as opposed to on on disk).
The resulting Zip opens for browsing, but when attempting to extract any contents, I get an error stating "The Archive is either in an unknown format or damaged".
In the code below, when asZip=false the text file is sent, and received as expected. When asZip=true, the file is sent, but suffers the corruption described above.
When I replace MemoryStream zipStream, with FileStream zipStream, the file on disk is OK.
Can anyone see what I've missed?
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net; // .NET 4.0
using System.Net.Http;
using System.Net.Http.Headers;
using System.Web.Mvc;
using ICSharpCode.SharpZipLib.Zip;//0.86.0.518
namespace Demo
{
public class DemoApiController : Controller
{
/// <summary>
/// demos the zipfile error
/// </summary>
/// <param name="withFiles">Creates the zip if set to <c>true</c> [with files].</param>
[HttpGet]
public void ZipErrorDemo(bool asZip)
{
const string fileContent = "Hello World!";
MemoryStream rawContentStream = new MemoryStream(System.Text.Encoding.ASCII.GetBytes(fileContent));
if (!asZip)
{
//This File is recieved as text, opens without erros and has correct content.
WriteStreamToDownload(rawContentStream, "text/plain", "HelloWorld.txt");
}
else
{
MemoryStream zipStream = new MemoryStream(1024 * 2048);//2MB
using (ZipOutputStream s = new ZipOutputStream(zipStream))
{
s.UseZip64 = ICSharpCode.SharpZipLib.Zip.UseZip64.Off; //No Zip64 for better compatability
s.SetLevel(0); //No compression
byte[] buffer = new byte[4096];
//Add the text file
ZipEntry csvEntry = new ZipEntry("HelloWorld.txt");
s.PutNextEntry(csvEntry);
int sourceBytes = 0;
do
{
sourceBytes = rawContentStream.Read(buffer, 0, buffer.Length);
s.Write(buffer, 0, sourceBytes);
} while (sourceBytes > 0);
s.CloseEntry();
s.IsStreamOwner = false;//Tells s.Close to not mess invoke zipStream.Close()
s.Flush();
s.Finish();
s.Close();
byte[] streamBuffer = zipStream.GetBuffer();//Before doing this things were worse.
MemoryStream newStream = new MemoryStream(streamBuffer);
//This File is recieved as a zip, opens to list contents, but atemtps at extraction result in an error.
WriteStreamToDownload(newStream, "application/zip", "HelloWorld.zip");
}
}
}
// Adapted from: http://stackoverflow.com/questions/5596747/download-stream-file-from-url-asp-net
// Accessed: 3/17/15. Works.
private static void WriteStreamToDownload(Stream stream, string contentType, string fileName)
{
// 100 kb
const int bytesToRead = 102400;
byte[] buffer = new byte[bytesToRead];
var contextResponse = System.Web.HttpContext.Current.Response;
try
{
contextResponse.ContentType = contentType;
contextResponse.AddHeader("Content-Disposition", "attachment; filename=\"" + Path.GetFileName(fileName) + "\"");
contextResponse.AddHeader("Content-Length", stream.Length.ToString());
int length;
do
{
if (contextResponse.IsClientConnected)
{
length = stream.Read(buffer, 0, bytesToRead);
contextResponse.OutputStream.Write(buffer, 0, length);
contextResponse.Flush();
buffer = new byte[bytesToRead];
}
else
{
length = -1;
}
} while (length > 0);
}
finally
{
if (stream != null)
{
stream.Close();
stream.Dispose();
}
}
}
}
}

At the moment, you are over-reading your stream. GetBuffer() returns the oversized backing buffer; you should usually limit yourself to the first zipStream.Length bytes of the buffer.
So the first thing to try is:
MemoryStream newStream = new MemoryStream(streamBuffer, 0
(int)zipStream.Length);
However, if that works, you can probably also achieve the same thing by simply sending zipStream, as long as you:
rewind the stream after writing
ensure that it doesn't get closed by the using
You might also be interested to hear that zip support is present inside the .NET framework itself, without requiring additional tools.
Your copying code, btw, is inefficient (especially when the buffer is constantly recreated) and could probably just use:
stream.CopyTo(contextResponse.OutputStream);

How to use ZipPackage to create a zip

I've seen the document on MSDN about the ZipPackage class.
The example there is not very useful, can anyone provide an example about this class?

Here an example, note that:
- ZipPackage seem to do not compress
- The generated zip has an undesired file "[Content_Types].xml"
- System.IO.Compression since .Net 4.5 seems to be a good alternative
You have, in Visual Studio, to add reference to "WindowsBase" (without prefix like "System.IO.")
using System;
using System.Linq;
using System.Text;
using System.IO.Packaging;
using System.IO;
namespace TestZip
{
public static class Program
{
public static void Main(string[] args)
{
byte[] data = Encoding.UTF8.GetBytes(String.Join("\n", new string[1000].Select(s => "Something to zip.").ToArray()));
byte[] zippedBytes;
using(MemoryStream zipStream = new MemoryStream())
{
using (Package package = Package.Open(zipStream, FileMode.Create))
{
PackagePart document = package.CreatePart(new Uri("/test.txt", UriKind.Relative), "");
using (MemoryStream dataStream = new MemoryStream(data))
{
document.GetStream().WriteAll(dataStream);
}
}
zippedBytes = zipStream.ToArray();
}
File.WriteAllBytes("test.zip", zippedBytes);
}
private static void WriteAll(this Stream target, Stream source)
{
const int bufSize = 0x1000;
byte[] buf = new byte[bufSize];
int bytesRead = 0;
while ((bytesRead = source.Read(buf, 0, bufSize)) > 0)
target.Write(buf, 0, bytesRead);
}
}
}

take a look at this code project -
C# Use Zip Archives without External Libraries.