We Keep Coding

How to unzip a file in memory to an Azure CloudBlob [duplicate]

I'm probably doing something obviously stupid here. Please point it out!
I have some C# code that is pulling down a bunch of .gz files from SFTP (using the SSH.NET Nuget package - works great!). Each gz contains only a single .CSV file inside of them. I want to keep these files in memory without hitting disk (yes, I know, server memory management concerns exist - that's fine as these files are fairly small), decompress them in memory to extract the CSV file inside, and then return a collection of CSV files in a custom DTO (FtpFile).
My problem is that while my MemoryStream from the SFTP connection has data in it, either it doesn't ever seem to be populated in my GZipStream or the copy from the GZipStream to my output MemoryStream is failing. I have tried with the more traditional looping over Read with my own buffer but it had the same results as this code.
Aside from connection details (it connects successfully, so no worries there), here's all of my code:
Logic:
public static List<FtpFile> Foo()
{
var connectionInfo = new ConnectionInfo("example.com",
"username",
new PasswordAuthenticationMethod("username", "password"));
using (var client = new SftpClient(connectionInfo))
{
client.Connect();
var searchResults = client.ListDirectory("/testdir")
.Where(obj => obj.IsRegularFile
&& obj.Name.ToLowerInvariant().StartsWith("test_")
&& obj.Name.ToLowerInvariant().EndsWith(".gz"))
.Take(2)
.ToList();
var fileResults = new List<FtpFile>();
foreach (var file in searchResults)
{
var ftpFile = new FtpFile { FileName = file.Name, FileSize = file.Length };
using (var fileStream = new MemoryStream())
{
client.DownloadFile(file.FullName, fileStream); // Success! All is good here, so far. :)
using (var gzStream = new GZipStream(fileStream, CompressionMode.Decompress))
{
using (var outputStream = new MemoryStream())
{
gzStream.CopyTo(outputStream);
byte[] outputBytes = outputStream.ToArray(); // No data. Sad panda. :'(
ftpFile.FileContents = Encoding.ASCII.GetString(outputBytes);
fileResults.Add(ftpFile);
}
}
}
}
return fileResults;
}
}
FtpFile (just a simple DTO I'm populating):
public class FtpFile
{
public string FileName { get; set; }
public long FileSize { get; set; }
public string FileContents { get; set; }
}
PSA If anybody comes and copies this code, be aware that this is NOT good code in that you could have some serious memory management problems with this code! It's best practice to instead stream it to disk, which is not being done in this code! My needs are very specific in that I have to have these files simultaneously in memory for what I'm building with them.

If you are inserting data into the stream, make sure to seek back to its origin before un-gzipping it.
The following should fix your troubles:
using (var fileStream = new MemoryStream())
{
client.DownloadFile(file.FullName, fileStream); // Success! All is good here, so far. :)
fileStream.Seek(0, SeekOrigin.Begin);
using (var gzStream = new GZipStream(fileStream, CompressionMode.Decompress))
{
using (var outputStream = new MemoryStream())
{
gzStream.CopyTo(outputStream);
byte[] outputBytes = outputStream.ToArray(); // No data. Sad panda. :'(
ftpFile.FileContents = Encoding.ASCII.GetString(outputBytes);
fileResults.Add(ftpFile);
}
}
}

ICSharpCode.SharpZipLib Compression method not supported PKZIP

I have a problem with a zip-file created with the old PKZIP® Command Line for Windows version 4 from the year 2000.
I am using ICSharpCode.SharpZipLib to extract the file.
Windows has no problem to open the file in the Explorer.
Here is the code:
private void Extract(string zipFile, string outputfolder)
{
try
{
_logger.InfoFormat("Extracting {0}", zipFile);
System.IO.Stream stream = new System.IO.FileStream(zipFile, System.IO.FileMode.Open);
ZipInputStream zipInputStream = new ZipInputStream(stream);
ZipEntry zipEntry = zipInputStream.GetNextEntry(); //Throws Compression error exception
while (zipEntry != null)
{
String entryFileName = zipEntry.Name;
_logger.InfoFormat("Entry-Filename: {0}", entryFileName);
byte[] buffer = new byte[4096];
String fullZipToPath = Path.Combine(outputfolder, entryFileName);
string directoryName = Path.GetDirectoryName(fullZipToPath);
if (directoryName.Length > 0)
{
Directory.CreateDirectory(directoryName);
}
using (FileStream streamWriter = File.Create(fullZipToPath))
{
StreamUtils.Copy(zipInputStream, streamWriter, buffer);
}
zipEntry = zipInputStream.GetNextEntry();
}
}
catch (Exception ex)
{
_logger.Error("Error during extraction",ex);
throw;
}
}
Any idea how to fix this problem?

I had this same issue when decompressing a zip-file made with 7-zip.
I changed it from Deflate64 to Deflate, and then it worked.

I cant copy files using FileStream

I am trying to read/write files using FileStream. Code is working but After copied files all I get an empty file. String data inside the file is not copied.
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
if(saveFileDialog1.ShowDialog()==DialogResult.OK)
{
FileStream streamR = new FileStream(openFileDialog1.FileName, FileMode.Open);
byte[] buffer = new byte[streamR.Length];
streamR.Read(buffer, 0, buffer.Length);
FileStream streamW = new FileStream(saveFileDialog1.FileName,FileMode.Create);
int read_byte = 0;
while ((read_byte = streamR.Read(buffer, 0, buffer.Length)) > 0)
{
streamW.Write(buffer, 0, read_byte);
}
}
}

When using streams, you should use the 'using' command:
if (openFileDialog1.ShowDialog() == DialogResult.OK)
{
if(saveFileDialog1.ShowDialog()==DialogResult.OK)
{
using (FileStream streamR = new FileStream(openFileDialog1.FileName, FileMode.Open))
{
using (FileStream streamW = new FileStream(saveFileDialog1.FileName,FileMode.Create))
{
byte[] buffer = new byte[streamR.Length];
int read_byte = 0;
while ((read_byte = streamR.Read(buffer, 0, buffer.Length)) > 0)
{
streamW.Write(buffer, 0, read_byte);
}
}
}
}
}
It will automatically flush, close and dispose the streams for you.
What actually stops your code from working, is the flush() and close() command.
However, it's still recommended to use the 'using' command.
A second way is to wrap everything in a try finally block and dispose the stream in the finally block:
using statement FileStream and / or StreamReader - Visual Studio 2012 Warnings
Anyway, I would suggest reading some more information about streams before continuing.
On the other hand ... if it's just for copying files, it would be simpler to use the Fil.Copy method.
Edit: Also ... loading the original file completely into a byte-array can cause some extra problems when your file is quite large.
The buffer is there to read chunks from the original file and process them.
I just corrected your code to make it work ... but it's far from perfect.

I would do something along these lines:
if (openFileDialog1.ShowDialog() == DialogResult.OK
&& saveFileDialog1.ShowDialog() == DialogResult.OK){
try {
if (File.Exists(saveFileDialog1.FileName)) {
File.Delete(saveFileDialog1.FileName);
}
File.Copy(openFileDialog1.FileName, saveFileDialog1.FileName);
} catch (Exception e){
//handle or throw e
}
}

Nesting Zip Files and Folders in Memory using DotNetZip Library

We have a page that users can download media and we construct a folder structure similar to the following and zip it up and send it back to the user in the response.
ZippedFolder.zip
- Folder A
- File 1
- File 2
- Folder B
- File 3
- File 4
The existing implementation that accomplishes this saves files and directories temporarily to file system and then deletes them at the end. We are trying to get away from doing this and would like to accomplish this entirely in memory.
I am able to successfully create a ZipFile with files in it, but the problem I am running into is creating Folder A and Folder B and adding files to those and then adding those two folders to the Zip File.
How can I do this without saving to the file system?
The code for just saving the file streams to the zip file and then setting the Output Stream on the response is the following.
public Stream CompressStreams(IList<Stream> Streams, IList<string> StreamNames, Stream OutputStream = null)
{
MemoryStream Response = null;
using (ZipFile ZippedFile = new ZipFile())
{
for (int i = 0, length = Streams.Count; i < length; i++)
{
ZippedFile.AddEntry(StreamNames[i], Streams[i]);
}
if (OutputStream != null)
{
ZippedFile.Save(OutputStream);
}
else
{
Response = new MemoryStream();
ZippedFile.Save(Response);
// Move the stream back to the beginning for reading
Response.Seek(0, SeekOrigin.Begin);
}
}
return Response;
}
EDIT We are using DotNetZip for the zipping/unzipping library.

Here's another way of doing it using System.IO.Compression.ZipArchive
public Stream CompressStreams(IList<Stream> Streams, IList<string> StreamNames, Stream OutputStream = null)
{
MemoryStream Response = new MemoryStream();
using (ZipArchive ZippedFile = new ZipArchive(Response, ZipArchiveMode.Create, true))
{
for (int i = 0, length = Streams.Count; i < length; i++)
using (var entry = ZippedFile.CreateEntry(StreamNames[i]).Open())
{
Streams[i].CopyTo(entry);
}
}
if (OutputStream != null)
{
Response.Seek(0, SeekOrigin.Begin);
Response.CopyTo(OutputStream);
}
return Response;
}
and a little test:
using (var write = new FileStream(#"C:\users\Public\Desktop\Testzip.zip", FileMode.OpenOrCreate, FileAccess.Write))
using (var read = new FileStream(#"C:\windows\System32\drivers\etc\hosts", FileMode.Open, FileAccess.Read))
{
CompressStreams(new List<Stream>() { read }, new List<string>() { #"A\One.txt" }, write);
}
re: your comment -- sorry, not sure if it creates something in the background, but you're not creating it yourself to do anything

How to save MP3 file from the internet on Windows Phone 7?

I'm trying to download and save an .mp3 file from the internet, but got stuck with stream from the external link:
private void saveSound()
{
IsolatedStorageFile iso = IsolatedStorageFile.GetUserStoreForApplication();
using (var fs = new IsolatedStorageFileStream("123.mp3", FileMode.Create, iso))
{
//Here should be this Stream from the Internet...
//Uri: "http://my-site.com/mega-popular-song.mp3"
StreamResourceInfo rs = new StreamResourceInfo(stream, "audio/mpeg");
int count = 0;
byte[] buffer = new byte[4096];
while (0 < (count = rs.Stream.Read(buffer, 0, buffer.Length)))
{
fs.Write(buffer, 0, count);
}
fs.Close();
}
}
What should this stream look like? What is the best way to download and save .mp3 files?

I'm sure this article gets you there. Like Bob mentioned, you'll have to use a WebClient. Basically this is the code that does the magic:
wc.OpenReadCompleted += ((s, args) =>
{
using (var store = IsolatedStorageFile.GetUserStoreForApplication())
{
if (store.FileExists(fileName))
store.DeleteFile(fileName);
using (var fs = new IsolatedStorageFileStream(fileName, FileMode.Create, store))
{
byte[] bytesInStream = new byte[args.Result.Length];
args.Result.Read(bytesInStream, 0, (int)bytesInStream.Length);
fs.Write(bytesInStream, 0, bytesInStream.Length);
fs.Flush();
}
}
});
But I would read the complete article to fully understand what happens. Hope this helps!