So the question is pretty simple. I'm using Xamarin.Android and I have a zip file in the Assets folder named "MyZipFile.zip", which I want extracted to the following path: System.Environment.GetFolderPath(System.Environment.SpecialFolder.Personal);
It sounds simple enough, but I cannot figure out how to read the Asset into memory through the AssetManager and then unzip it at the targeted location.
Is there a simple way to do this?
The Android Java framework includes a Java.Util.Zip package, so without adding any additional app libraries, I directly use it instead of using C# framework code, thus no bloat that linking can not remove.
So basically you are creating an asset stream and feeding that to a ZipInputStream and iterating over each ZipEntry in that zip stream to either create directories or files to your destination path.
UnZipAssets
public void UnZipAssets(string assetName, string destPath)
{
byte[] buffer = new byte[1024];
int byteCount;
var destPathDir = new Java.IO.File(destPath);
destPathDir.Mkdirs();
using (var assetStream = Assets.Open(assetName, Android.Content.Res.Access.Streaming))
using (var zipStream = new ZipInputStream(assetStream))
{
ZipEntry zipEntry;
while ((zipEntry = zipStream.NextEntry) != null)
{
if (zipEntry.IsDirectory)
{
var zipDir = new Java.IO.File(Path.Combine(destPath, zipEntry.Name));
zipDir.Mkdirs();
continue;
}
// Note: This is deleting existing entries(!!!) for debug purposes only...
#if DEBUG
if (File.Exists(Path.Combine(destPath, zipEntry.Name)))
File.Delete(Path.Combine(destPath, zipEntry.Name));
#endif
using (var fileStream = new FileStream(Path.Combine(destPath, zipEntry.Name), FileMode.CreateNew))
{
while ((byteCount = zipStream.Read(buffer)) != -1)
{
fileStream.Write(buffer, 0, byteCount);
}
}
Log.Debug("UnZipAssets", zipEntry.Name);
zipEntry.Dispose();
}
}
}
Usage:
UnZipAssets("gameModLevels.zip", Path.Combine(Application.Context.CacheDir.AbsolutePath, "assets"));
Note: Even through the asset/zip steam is fast, depending upon number/size of the zip entries and the speed of the flash the entry is being written to, this should be done on a background thread as not to block UI thread and cause an ANR
Related
I am trying to create a Zip from a list of files in parallel and stream it to client.
I have a working code where I iterate over files sequentially, but I want it instead to be zipped in parallel (multiple files with >100mb each).
using ZipArchive zipArchive = new(Response.BodyWriter.AsStream(), ZipArchiveMode.Create, leaveOpen: false);
for (int i = 0; i < arrLocalFilesPath.Length; i++) // iterate over files
{
string strFilePath = arrLocalFilesPath[i]; // list of files path
string strFileName = Path.GetFileName(strFilePath);
ZipArchiveEntry zipEntry = zipArchive.CreateEntry(strFileName, CompressionLevel.Optimal);
using Stream zipStream = zipEntry.Open();
using FileStream fileStream = System.IO.File.Open(strFilePath, FileMode.Open, FileAccess.Read);
fileStream.CopyTo(zipStream);
}
return new EmptyResult();
Parallel.For and Parallel.ForEach do not work with ZipArchive
Since ZipArchive is not thread safe, I am trying to use DotNetZip to accomplish this task.
I looked at the docs and here's what I have so far using DotNetZip
using Stream streamResponseBody = Response.BodyWriter.AsStream();
Parallel.For(0, arrLocalFilesPath.Length, i =>
{
string strFilePath = arrLocalFilesPath[i]; // list of files path
string strFileName = Path.GetFileName(strFilePath);
string strCompressedOutputFile = strFilePath + ".compressed";
byte[] arrBuffer = new byte[8192]; //[4096];
int n = -1;
using FileStream input = System.IO.File.OpenRead(strFilePath);
using FileStream raw = new(strCompressedOutputFile, FileMode.Create, FileAccess.ReadWrite);
using Stream compressor = new ParallelDeflateOutputStream(raw);
while ((n = input.Read(arrBuffer, 0, arrBuffer.Length)) != 0)
{
compressor.Write(arrBuffer, 0, n);
}
input.CopyTo(streamResponseBody);
});
return new EmptyResult();
However, this doesn't zip files and send to client (it only creates local zip files on the server).
Using MemoryStream or creating a local zip file is out of the question and not what I am looking for.
The server should seamlessly stream read bytes of a file, zip it on the fly and send it to client as chunks (like in my ZipArchive), but with the added benefits of reading those files in parallel and creating a zip of them.
I know that parallelism is usually not optimal for I/O (sometimes a bit worse), but parallel zipping multiple big files should be faster for this case.
I also tried to use SharpZipLib without success.
Usage of any other libraries is fine as long as it read and stream files to client seamlessly without impacting memory.
Any help is appreciated.
If these files are on the same drive there won't be any speed up. The parallelization is used to compress/decompress data, but the disk IO operation cannot be done in parallel.
Assuming that files are not on the same drive and there is a chance to speed up this process...
Are you sure the Stream.CopyTo() is thread safe? Either check the docs or use single thread or set lock on it.
EDIT:
I've checked my old codes, where I was packing huge amount of data into a zip file using ZipArchive. I did it in parallel, but there was no IO read there.
You can use ZipArchive with Parallel.For but you need to use lock:
//create zip into stream
using (ZipArchive zipArchive = new ZipArchive(zipFS, ZipArchiveMode.Update, false))
{
//use parallel foreach instead of parallel, but not for IO read operation!
Parallel.ForEach(listOfFiles, filename =>
{
//create a file entry
ZipArchiveEntry zipFileEntry = zipArchive.CreateEntry(filename);
//prepare memory for the entry
MemoryStream ms = new MemoryStream();
/*fill the memory stream here - I did another packing with BZip2OutputStream, because the zip was packed without compression to speed up random decompression */
//only one thread can write to zip!
lock (zipFileEntry)
{
//open stream for writing
using (Stream zipEntryStream = zipFileEntry.Open())
{
ms.Position = 0; // rewind the stream
StreamUtils.Copy(ms, zipEntryStream, new byte[4096]); //from ICSharpCode.SharpZipLib.Core, copy memory stream data into zip entry with packing.
}
}
}
}
Anyway, if you need to read the files first, it's your performance bottleneck. You won't gain a lot (if anything) from parallel approach here.
With std::filesystem::resize_file in C++, it is possible to change the size of a file without opening the file.
Is there any similar function in C#, which allows changing the size of a file without opening it?
I think opening a file as a FileStream and saving it again with a new size will be slower.
Using FileStream.SetLength() will be about as fast as you can make it.
It ends up calling the Windows API to set the length of the file, the same as the std::filesystem::resize_file().
So you just need to do something like this, and it will be fast enough:
using (var file = File.Open(myFilePath, FileMode.Open))
{
file.SetLength(myRequiredFileSize);
}
The implementation of FileStream.SetLength() is:
private void SetLengthCore(long value)
{
Contract.Assert(value >= 0, "value >= 0");
long origPos = _pos;
if (_exposedHandle)
VerifyOSHandlePosition();
if (_pos != value)
SeekCore(value, SeekOrigin.Begin);
if (!Win32Native.SetEndOfFile(_handle)) {
int hr = Marshal.GetLastWin32Error();
if (hr==__Error.ERROR_INVALID_PARAMETER)
throw new ArgumentOutOfRangeException("value", Environment.GetResourceString("ArgumentOutOfRange_FileLengthTooBig"));
__Error.WinIOError(hr, String.Empty);
}
// Return file pointer to where it was before setting length
if (origPos != value) {
if (origPos < value)
SeekCore(origPos, SeekOrigin.Begin);
else
SeekCore(0, SeekOrigin.End);
}
}
(Note that SeekCore() just calls the the Windows API SetFilePointer() function.)
Doing this does NOT read the file into memory.
Also, the Windows API function SetEndOfFile() does not write to the extended region, so it is fast. The documentation states If the file is extended, the contents of the file between the old end of the file and the new end of the file are not defined. - this is as a result of data not being written to the extended region.
As test, I tried the following code:
using System;
using System.Diagnostics;
using System.IO;
namespace Demo
{
public class Program
{
public static void Main()
{
string filename = #"e:\tmp\test.bin";
File.WriteAllBytes(filename, new byte[0]); // Create empty file.
var sw = Stopwatch.StartNew();
using (var file = File.Open(filename, FileMode.Open))
{
file.SetLength(1024*1024*1024);
}
Console.WriteLine(sw.Elapsed);
}
}
}
My E:\ drive is a hard drive, not an SSD.
The output was: 00:00:00.0003574
So it took less than a hundreth of a second to extend the file to 1GB in size.
I'm a complete beginner to dealing with streams or anything like this, so if you see anything obviously wrong... that's why. I have some files stored on azure. I need to take the files, zip them up, and return the zip.
I've tested this with a 1GB file, and although it works, it ends up using 2.5GB of memory. Memory usage spikes between when the last line starts and completes. I'm not sure why this is loading everything in memory, so I'm not quite sure what i'm supposed to do to prevent that from happening. What's the correct way to do it? The only thing I can think of is to specify buffer sizes somewhere, but everywhere i've seen where it's possible has a small default.
FileStream zipToOpen = new FileStream("test.zip", FileMode.Create);
ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Update, true);
ZipArchiveEntry zipEntry = archive.CreateEntry("entryName", CompressionLevel.Optimal);
// ... some azure code
file = await dir.GetFileReference(fileName);
fileContents = await file.OpenReadAsync().ConfigureAwait(false);
await fileContents.CopyToAsync(zipEntry.Open()).ConfigureAwait(false);
Just create archive as
ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Create);
Your memory consumption will drop to minimal (In my test case it dropped from 900M to 36M)...
Seems like problem is related with ZipArchiveMode.Update
void Zip(IEnumerable<string> files, Stream inputStream)
{
using (var zip = new System.IO.Compression.ZipArchive(inputStream, System.IO.Compression.ZipArchiveMode.Create))
{
foreach (var file in files.Select(f => new FileInfo(f)))
{
var entry = zip.CreateEntry(file.Name, System.IO.Compression.CompressionLevel.Fastest);
using (var s = entry.Open())
{
using (var f = File.Open(file.FullName, FileMode.Open, FileAccess.Read, FileShare.Read))
{
f.CopyTo(s);
}
}
}
}
}
I am using .NET 4.5, and the ZipFile class works great if I am trying to zip up an entire directory with "CreateFromDirectory". However, I only want to zip up one file in the directory. I tried pointing to a specific file (folder\data.txt), but that doesn't work. I considered the ZipArchive class since it has a "CreateEntryFromFile" method, but it seems this only allows you to create an entry into an existing file.
Is there no way to simply zip up one file without creating an empty zipfile (which has its issues) and then using the ZipArchiveExtension's "CreateEntryFromFile" method?
**This is also assuming I am working on a company program which cannot use third-party add-ons at the moment.
example from:http://msdn.microsoft.com/en-us/library/ms404280%28v=vs.110%29.aspx
string startPath = #"c:\example\start";
string zipPath = #"c:\example\result.zip";
string extractPath = #"c:\example\extract";
ZipFile.CreateFromDirectory(startPath, zipPath);
ZipFile.ExtractToDirectory(zipPath, extractPath);
But if startPath were to be #"c:\example\start\myFile.txt;", it would throw an error that the directory is invalid.
Use the CreateEntryFromFile off a an archive and use a file or memory stream:
Using a filestream if you are fine creating the zip file and then adding to it:
using (FileStream fs = new FileStream(#"C:\Temp\output.zip",FileMode.Create))
using (ZipArchive arch = new ZipArchive(fs, ZipArchiveMode.Create))
{
arch.CreateEntryFromFile(#"C:\Temp\data.xml", "data.xml");
}
Or if you need to do everything in memory and write the file once it is done, use a memory stream:
using (MemoryStream ms = new MemoryStream())
using (ZipArchive arch = new ZipArchive(ms, ZipArchiveMode.Create))
{
arch.CreateEntryFromFile(#"C:\Temp\data.xml", "data.xml");
}
Then you can write the MemoryStream to a file.
using (FileStream file = new FileStream("file.bin", FileMode.Create, System.IO.FileAccess.Write)) {
byte[] bytes = new byte[ms.Length];
ms.Read(bytes, 0, (int)ms.Length);
file.Write(bytes, 0, bytes.Length);
ms.Close();
}
Using file (or any) stream:
using (var zip = ZipFile.Open("file.zip", ZipArchiveMode.Create))
{
var entry = zip.CreateEntry("file.txt");
entry.LastWriteTime = DateTimeOffset.Now;
using (var stream= File.OpenRead(#"c:\path\to\file.txt"))
using (var entryStream = entry.Open())
stream.CopyTo(entryStream);
}
or briefer:
// reference System.IO.Compression
using (var zip = ZipFile.Open("file.zip", ZipArchiveMode.Create))
zip.CreateEntryFromFile("file.txt", "file.txt");
make sure you add references to System.IO.Compression
Update
Also, check out the new dotnet API documentation for ZipFile and ZipArchive too. There are a few examples there. There is also a warning about referencing System.IO.Compression.FileSystem to use ZipFile.
To use the ZipFile class, you must reference the
System.IO.Compression.FileSystem assembly in your project.
The simplest way to get this working is to use a temporary folder.
FOR ZIPPING:
Create a temp folder
Move file to folder
Zip folder
Delete folder
FOR UNZIPPING:
Unzip archive
Move file from temp folder to your location
Delete temp folder
In .NET, there are quite a few ways to tackle the problem, for a single file. If you don't want to learn everything there, you can get an abstracted library, like SharpZipLib (long standing open source library), sevenzipsharp (requires 7zip libs underneath) or DotNetZip.
just use following code for compressing a file.
public void Compressfile()
{
string fileName = "Text.txt";
string sourcePath = #"C:\SMSDBBACKUP";
DirectoryInfo di = new DirectoryInfo(sourcePath);
foreach (FileInfo fi in di.GetFiles())
{
//for specific file
if (fi.ToString() == fileName)
{
Compress(fi);
}
}
}
public static void Compress(FileInfo fi)
{
// Get the stream of the source file.
using (FileStream inFile = fi.OpenRead())
{
// Prevent compressing hidden and
// already compressed files.
if ((File.GetAttributes(fi.FullName)
& FileAttributes.Hidden)
!= FileAttributes.Hidden & fi.Extension != ".gz")
{
// Create the compressed file.
using (FileStream outFile =
File.Create(fi.FullName + ".gz"))
{
using (GZipStream Compress =
new GZipStream(outFile,
CompressionMode.Compress))
{
// Copy the source file into
// the compression stream.
inFile.CopyTo(Compress);
Console.WriteLine("Compressed {0} from {1} to {2} bytes.",
fi.Name, fi.Length.ToString(), outFile.Length.ToString());
}
}
}
}
}
}
How can I list the contents of a zipped folder in C#? For example how to know how many items are contained within a zipped folder, and what is their name?
.NET 4.5 or newer finally has built-in capability to handle generic zip files with the System.IO.Compression.ZipArchive class (http://msdn.microsoft.com/en-us/library/system.io.compression.ziparchive%28v=vs.110%29.aspx) in assembly System.IO.Compression. No need for any 3rd party library.
string zipPath = #"c:\example\start.zip";
using (ZipArchive archive = ZipFile.OpenRead(zipPath))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
Console.WriteLine(entry.FullName);
}
}
DotNetZip - Zip file manipulation in .NET languages
DotNetZip is a small, easy-to-use class library for manipulating .zip files. It can enable .NET applications written in VB.NET, C#, any .NET language, to easily create, read, and update zip files.
sample code to read a zip:
using (var zip = ZipFile.Read(PathToZipFolder))
{
int totalEntries = zip.Entries.Count;
foreach (ZipEntry e in zip.Entries)
{
e.FileName ...
e.CompressedSize ...
e.LastModified...
}
}
If you are using .Net Framework 3.0 or later, check out the System.IO.Packaging Namespace. This will remove your dependancy on an external library.
Specifically check out the ZipPackage Class.
Check into SharpZipLib
ZipInputStream inStream = new ZipInputStream(File.OpenRead(fileName));
while (inStream.GetNextEntry())
{
ZipEntry entry = inStream.GetNextEntry();
//write out your entry's filename
}
Ick - that code using the J# runtime is hideous! And I don't agree that it is the best way - J# is out of support now. And it is a HUGE runtime, if all you want is ZIP support.
How about this - it uses DotNetZip (Free, MS-Public license)
using (ZipFile zip = ZipFile.Read(zipfile) )
{
bool header = true;
foreach (ZipEntry e in zip)
{
if (header)
{
System.Console.WriteLine("Zipfile: {0}", zip.Name);
if ((zip.Comment != null) && (zip.Comment != ""))
System.Console.WriteLine("Comment: {0}", zip.Comment);
System.Console.WriteLine("\n{1,-22} {2,9} {3,5} {4,9} {5,3} {6,8} {0}",
"Filename", "Modified", "Size", "Ratio", "Packed", "pw?", "CRC");
System.Console.WriteLine(new System.String('-', 80));
header = false;
}
System.Console.WriteLine("{1,-22} {2,9} {3,5:F0}% {4,9} {5,3} {6:X8} {0}",
e.FileName,
e.LastModified.ToString("yyyy-MM-dd HH:mm:ss"),
e.UncompressedSize,
e.CompressionRatio,
e.CompressedSize,
(e.UsesEncryption) ? "Y" : "N",
e.Crc32);
if ((e.Comment != null) && (e.Comment != ""))
System.Console.WriteLine(" Comment: {0}", e.Comment);
}
}
I'm relatively new here so maybe I'm not understanding what's going on. :-)
There are currently 4 answers on this thread where the two best answers have been voted down. (Pearcewg's and cxfx's) The article pointed to by pearcewg is important because it clarifies some licensing issues with SharpZipLib.
We recently evaluated several .Net compression libraries, and found that DotNetZip is currently the best aleternative.
Very short summary:
System.IO.Packaging is significantly slower than DotNetZip.
SharpZipLib is GPL - see article.
So for starters, I voted those two answers up.
Kim.
If you are like me and do not want to use an external component, here is some code I developed last night using .NET's ZipPackage class.
var zipFilePath = "c:\\myfile.zip";
var tempFolderPath = "c:\\unzipped";
using (Package package = ZipPackage.Open(zipFilePath, FileMode.Open, FileAccess.Read))
{
foreach (PackagePart part in package.GetParts())
{
var target = Path.GetFullPath(Path.Combine(tempFolderPath, part.Uri.OriginalString.TrimStart('/')));
var targetDir = target.Remove(target.LastIndexOf('\\'));
if (!Directory.Exists(targetDir))
Directory.CreateDirectory(targetDir);
using (Stream source = part.GetStream(FileMode.Open, FileAccess.Read))
{
source.CopyTo(File.OpenWrite(target));
}
}
}
Things to note:
The ZIP archive MUST have a [Content_Types].xml file in its root. This was a non-issue for my requirements as I will control the zipping of any ZIP files that get extracted through this code. For more information on the [Content_Types].xml file, please refer to: A New Standard For Packaging Your Data There is an example file below Figure 13 of the article.
This code uses the Stream.CopyTo method in .NET 4.0
The best way is to use the .NET built in J# zip functionality, as shown in MSDN: http://msdn.microsoft.com/en-us/magazine/cc164129.aspx. In this link there is a complete working example of an application reading and writing to zip files. For the concrete example of listing the contents of a zip file (in this case a Silverlight .xap application package), the code could look like this:
ZipFile package = new ZipFile(packagePath);
java.util.Enumeration entries = package.entries();
//We have to use Java enumerators because we
//use java.util.zip for reading the .zip files
while ( entries.hasMoreElements() )
{
ZipEntry entry = (ZipEntry) entries.nextElement();
if (!entry.isDirectory())
{
string name = entry.getName();
Console.WriteLine("File: " + name + ", size: " + entry.getSize() + ", compressed size: " + entry.getCompressedSize());
}
else
{
// Handle directories...
}
}
Aydsman had a right pointer, but there are problems. Specifically, you might find issues opening zip files, but is a valid solution if you intend to only create pacakges. ZipPackage implements the abstract Package class and allows manipulation of zip files. There is a sample of how to do it in MSDN: http://msdn.microsoft.com/en-us/library/ms771414.aspx. Roughly the code would look like this:
string packageRelationshipType = #"http://schemas.microsoft.com/opc/2006/sample/document";
string resourceRelationshipType = #"http://schemas.microsoft.com/opc/2006/sample/required-resource";
// Open the Package.
// ('using' statement insures that 'package' is
// closed and disposed when it goes out of scope.)
foreach (string packagePath in downloadedFiles)
{
Logger.Warning("Analyzing " + packagePath);
using (Package package = Package.Open(packagePath, FileMode.Open, FileAccess.Read))
{
Logger.OutPut("package opened");
PackagePart documentPart = null;
PackagePart resourcePart = null;
// Get the Package Relationships and look for
// the Document part based on the RelationshipType
Uri uriDocumentTarget = null;
foreach (PackageRelationship relationship in
package.GetRelationshipsByType(packageRelationshipType))
{
// Resolve the Relationship Target Uri
// so the Document Part can be retrieved.
uriDocumentTarget = PackUriHelper.ResolvePartUri(
new Uri("/", UriKind.Relative), relationship.TargetUri);
// Open the Document Part, write the contents to a file.
documentPart = package.GetPart(uriDocumentTarget);
//ExtractPart(documentPart, targetDirectory);
string stringPart = documentPart.Uri.ToString().TrimStart('/');
Logger.OutPut(" Got: " + stringPart);
}
// Get the Document part's Relationships,
// and look for required resources.
Uri uriResourceTarget = null;
foreach (PackageRelationship relationship in
documentPart.GetRelationshipsByType(
resourceRelationshipType))
{
// Resolve the Relationship Target Uri
// so the Resource Part can be retrieved.
uriResourceTarget = PackUriHelper.ResolvePartUri(
documentPart.Uri, relationship.TargetUri);
// Open the Resource Part and write the contents to a file.
resourcePart = package.GetPart(uriResourceTarget);
//ExtractPart(resourcePart, targetDirectory);
string stringPart = resourcePart.Uri.ToString().TrimStart('/');
Logger.OutPut(" Got: " + stringPart);
}
}
}
The best way seems to use J#, as shown in MSDN: http://msdn.microsoft.com/en-us/magazine/cc164129.aspx
There are pointers to more c# .zip libraries with different licenses, like SharpNetZip and DotNetZip in this article: how to read files from uncompressed zip in c#?. They might be unsuitable because of the license requirements.