This is my code
I have an FTP with many zip files. and each zip file has a XML with the same name.
I want to parse these xml files.
What i did is this:
get a list of all zip files in the FTP and save the names in this variable directories.
Now I want to open each zip file, which its name is in the directories list. I did this.
foreach (string fileNameInFTP in directories)
{
}
Now to read the content of that zip file, I tried this:.
string fileName = FTPAddress + fileNameInFTP;
using (var file = File.OpenRead(fileName))
using (var zip = new ZipArchive(file, ZipArchiveMode.Read))
{
foreach (var entry in zip.Entries)
{
using (var stream = entry.Open())
{
// do whatever we want with stream
// ...
}
}
}
I got this exception The given path's format is not supported. on this line:
using (var file = File.OpenRead("ftp://" +FTPAddress +"/" + fileNameInFTP)) could u help please
You should use something like this instead of trying to use File.OpenRead for remote FTP file download.
http://msdn.microsoft.com/en-us/library/ms229711%28v=vs.110%29.aspx
// Get the object used to communicate with the server.
FtpWebRequest request = (FtpWebRequest)WebRequest.Create("ftp://www.contoso.com/test.htm");
request.Method = WebRequestMethods.Ftp.DownloadFile;
// This example assumes the FTP site uses anonymous logon.
request.Credentials = new NetworkCredential ("anonymous","janeDoe#contoso.com");
FtpWebResponse response = (FtpWebResponse)request.GetResponse();
Stream responseStream = response.GetResponseStream();
using (var zip = new ZipArchive(responseStream , ZipArchiveMode.Read))
{
//Loops through each file in the zip that has the ".xml" extension
foreach (var entry in zip.Entries.Where(x=> (Path.GetExtension(x.Name) ?? "").ToLower() ==".xml"))
{
using (var stream = entry.Open())
{
//Load xml file and do whatever you like with it.
var xmlDocument = XDocument.Load(stream);
}
}
}
Console.WriteLine("Download Complete, status {0}", response.StatusDescription);
response.Close();
you can't use File IO to open FTP stream, here is a sample of how we can open FTP using WebRequest in .NET:
private static void Main(string[] args)
{
var ftp = WebRequest.Create(#"ftp://ftp.microsoft.com/softlib/MSLFILES/aspwebwiz2k.zip");
//ftp.Credentials=new NetworkCredential("anonymous","anonymous");
var response=ftp.GetResponse();
var stream=response.GetResponseStream();
var ms = ToMemoryStream(stream);
var archive = new ZipArchive(ms, ZipArchiveMode.Read);
var entry=archive.GetEntry("file name here");
var doc=XDocument.Load(entry.Open());
}
public static MemoryStream ToMemoryStream( Stream stream)
{
var memoryStream = new MemoryStream();
var buffer = new byte[4096];
while (true)
{
var readCount = stream.Read(buffer, 0, buffer.Length);
if (readCount == 0)
break;
memoryStream.Write(buffer, 0, readCount);
}
memoryStream.Position = 0;
return memoryStream;
}
Related
I created a BizTalk Custom Pipeline Component, which zips all message parts into a zip stream. After some fails I created the following test method in a separate test project.
Basically I get a XML file which contains a filename and an UUID which I use to call a stored procedure and get the Base64 encoded content of the database entry.
The base64 content seems valid, because after decoding it and saving it to the file system the files can be read by the windows explorer without problems.
After saving the archiveStream to the file system I get the following error message from 7Zip when I try to extract the file:
"Unexpected end of data". if I try to just open the file with 7Zip there is no problem. I even can open the files from inside the 7Zip explorer.
If I try to read the file from C# with the following code I get the error message:
"End of Central Directory record could not be found."
Unzip code:
private static void ReadDat() {
var path = #"...\zip\0e00128b-0a6e-4b99-944d-68e9c20a51c2.zip";
var stream = System.IO.File.OpenRead(path);
// End of Central Directory record could not be found:
var zipArchive = new ZipArchive(stream, ZipArchiveMode.Read, false);
foreach(var zipEntry in zipArchive.Entries) {
var stream = zipEntry.Open();
Console.WriteLine(stream.Length);
}
}
Zip Code:
private static void StreamUuidList() {
var path = #"...\2017-08-05T132705.xml";
var xdoc = XDocument.Load(System.IO.File.OpenRead(path));
var files = xdoc.Root.Descendants().Where(d => d.Name.LocalName.Equals("NodeName"));
using (var archiveStream = new MemoryStream())
using (var archive = new ZipArchive(archiveStream, ZipArchiveMode.Create, true)) {
foreach (var file in files) {
var fileName = file.Elements().Where(e => e.Name.LocalName.Equals("FileName")).FirstOrDefault()?.Value ?? "";
var streamUuid = file.Elements().Where(e => e.Name.LocalName.Equals("StreamUUID")).FirstOrDefault()?.Value ?? "";
// validation here...
// get base64 content and convert content
var base64Content = GetStreamContent(streamUuid);
var data = Convert.FromBase64String(base64Content);
var dataStream = new MemoryStream(data);
dataStream.Seek(0, SeekOrigin.Begin);
// debug - save to file location
using (var fileStream = new FileStream($#"...\files\{fileName}", FileMode.Create)) {
dataStream.CopyTo(fileStream);
}
dataStream.Seek(0, SeekOrigin.Begin);
// create zip entry
var zipFile = archive.CreateEntry(fileName, GetCompressionLevelFromString("Optimal"));
using (var zipFileStream = zipFile.Open()) {
// copy data from mesage part stream into zip entry stream
dataStream.Seek(0, SeekOrigin.Begin);
dataStream.CopyTo(zipFileStream);
}
Console.WriteLine(fileName + ": " + streamUuid);
}
// debug - save to file location
archiveStream.Seek(0, SeekOrigin.Begin);
using (var fileStream = new FileStream($#"...\zip\{Guid.NewGuid()}.dat", FileMode.Create)) {
archiveStream.CopyTo(fileStream);
}
// debug end
}
}
I have some files inside in one .tar.gz archive. These files are on a linux server.How can I read from a specific file inside this archive if I know it's name?
For reading direct from the txt file, I used the following code:
Uri urlFile = new Uri("ftp://" + ServerName + "/%2f" + FilePath + "/" + fileName);
WebClient req = new WebClient() { Credentials=new NetworkCredential("user","psw")};
string result = req.DownloadString(urlFile);
It's possible to read this file without copying the archive on the local machine, something like the code above?
I found a solution. Maybe this can help you guys.
// archivePath="ftp://myLinuxServer.com/%2f/move/files/archive/20170225.tar.gz";
public static string ExtractFileFromArchive(string archivePath, string fileName)
{
string stringFromFile="File not found";
WebClient wc = new WebClient() { Credentials = cred, Proxy= webProxy }; //Create webClient with all necessary settings
using (Stream source = new GZipInputStream(wc.OpenRead(archivePath))) //wc.OpenRead() create one stream with archive tar.gz from our server
{
using (TarInputStream tarStr =new TarInputStream(source)) //TarInputStream is a stream from ICSharpCode.SharpZipLib.Tar library(need install SharpZipLib in nutgets)
{
TarEntry te;
while ((te = tarStr.GetNextEntry())!=null) // Go through all files from archive
{
if (te.Name == fileName)
{
using (Stream fs = new MemoryStream()) //Create a empty stream that we will be fill with file contents.
{
tarStr.CopyEntryContents(fs);
fs.Position = 0; //Move stream position to 0, in order to read from beginning
stringFromFile = new StreamReader(fs).ReadToEnd(); //Convert stream to string
}
break;
}
}
}
}
return stringFromFile;
}
I have an URL which contains a zip file. The files need to be unzipped from the URL. The URL is Opened and Read using webclient and then added to a Stream. It is then used in the ZipArchive object which will unzip the files and store them in the D:\ drive. When a file is around 400Mb I get the 'System.OutOfMemoryException'.
Stream has to be used since the webClient.OpenRead(Uri Address) returns a Stream. As well as the use ZipArchive(Stream stream).
How can I stop from getting this message?
string zipFileUrl = "https://www.dropbox.com/s/clersbjdcshpdy6/oversize_zip_test_0.zip?dl=0"
string output_path = #"D:\";
using (WebClient webClient = new WebClient())
{
using (Stream streamFile = webClient.OpenRead(zipFileUrl))
{
using (ZipArchive archive = new ZipArchive(streamFile))//ERROR HERE
{
var entries = archive.Entries;
//Loops thru each file in Zip and adds it to directory
foreach (var entry in entries)
{
if (entry.FullName != "/" && entry.Name != "")
{
string completeFileName = Path.Combine(output_path, entry.FullName);
string directory = Path.GetDirectoryName(completeFileName);
//If directory does not exist then we create it.
if (!Directory.Exists(directory))
{
Directory.CreateDirectory(directory);
}
//Extracts zip from URL to extract path, and overwrites if file exists.
entry.ExtractToFile(completeFileName, true);
}
}
}
}
I think here might be your problem, from the ZipArchive.Init method
private void Init(Stream stream, ZipArchiveMode mode, Boolean leaveOpen)
{
Stream extraTempStream = null;
try
{
_backingStream = null;
//check stream against mode
switch (mode)
{
case ZipArchiveMode.Create:
// (SNIP)
case ZipArchiveMode.Read:
if (!stream.CanRead)
throw new ArgumentException(SR.ReadModeCapabilities);
if (!stream.CanSeek)
{
_backingStream = stream;
extraTempStream = stream = new MemoryStream();
_backingStream.CopyTo(stream);
stream.Seek(0, SeekOrigin.Begin);
}
break;
case ZipArchiveMode.Update:
// (SNIP)
default:
// (SNIP)
}
// (SNIP)
}
if streamFile.CanSeek is false (which from a WebClient it will be) it copies the entire file in to memory then works on the file. This is what is using up all the memory.
Try to find a 3rd party library that handles Zip files and does not need a stream that supports seeking. If you can't, copy the file to disk first to the temp folder with a FileStream with the FileOptions.DeleteOnClose option passed in, then use that stream in your zip before you close the stream.
string zipFileUrl = "https://www.dropbox.com/s/clersbjdcshpdy6/oversize_zip_test_0.zip?dl=0";
string output_path = #"D:\";
using (var tempFileStream = new FileStream(Path.GetTempFileName(), FileMode.Create,
FileAccess.ReadWrite, FileShare.None,
4096, FileOptions.DeleteOnClose))
{
using (WebClient webClient = new WebClient())
{
using (Stream streamFile = webClient.OpenRead(zipFileUrl))
{
streamFile.CopyTo(tempFileStream);
}
}
tempFileStream.Position = 0;
using (ZipArchive archive = new ZipArchive(tempFileStream))
{
var entries = archive.Entries;
//Loops thru each file in Zip and adds it to directory
foreach (var entry in entries)
{
if (entry.FullName != "/" && entry.Name != "")
{
string completeFileName = Path.Combine(output_path, entry.FullName);
string directory = Path.GetDirectoryName(completeFileName);
//If directory does not exist then we create it.
if (!Directory.Exists(directory))
{
Directory.CreateDirectory(directory);
}
//Extracts zip from URL to extract path, and overwrites if file exists.
entry.ExtractToFile(completeFileName, true);
}
}
}
}
I am using SSH.NET library to download files. I want to save the downloaded file as a file in memory, rather than a file on disk but it is not happening.
This is my code which works fine:
using (var sftp = new SftpClient(sFTPServer, sFTPPassword, sFTPPassword))
{
sftp.Connect();
sftp.DownloadFile("AFile.txt", System.IO.File.Create("AFile.txt"));
sftp.Disconnect();
}
and this is the code which doesn't work fine as it gives 0 bytes stream.
using (var sftp = new SftpClient(sFTPServer, sFTPPassword, sFTPPassword))
{
sftp.Connect();
System.IO.MemoryStream mem = new System.IO.MemoryStream();
System.IO.TextReader textReader = new System.IO.StreamReader(mem);
sftp.DownloadFile("file.txt", mem);
System.IO.TextReader textReader = new System.IO.StreamReader(mem);
string s = textReader.ReadToEnd(); // it is empty
sftp.Disconnect();
}
You can try the following code, which opens the file on the server and reads it back into a stream:
using (var sftp = new SftpClient(sFTPServer, sFTPUsername, sFTPPassword))
{
sftp.Connect();
// Load remote file into a stream
using (var remoteFileStream = sftp.OpenRead("file.txt"))
{
var textReader = new System.IO.StreamReader(remoteFileStream);
string s = textReader.ReadToEnd();
}
}
For simple text files, it's even easier:
var contents = sftp.ReadAllText(fileSpec);
I had a similar issue with the ScpClient, I needed to reset the stream position to the beginning after downloading the file.
using (var sftp = new SftpClient(sFTPServer, sFTPPassword, sFTPPassword))
{
sftp.Connect();
System.IO.MemoryStream mem = new System.IO.MemoryStream();
System.IO.TextReader textReader = new System.IO.StreamReader(mem);
sftp.DownloadFile("file.txt", mem);
// Reset stream to the beginning
mem.Seek(0, SeekOrigin.Begin);
System.IO.TextReader textReader = new System.IO.StreamReader(mem);
string s = textReader.ReadToEnd();
sftp.Disconnect();
}
I am trying to download a zipped file from the server and trying to show the content of each files in zipped folder to the view.
I wrote a separate code where the file is on my laptop and I ran across each file and dislpayed the content such as
static void Main(string[] args)
{
string filePath = "C:\\ACL Data\\New folder\\files.zip";
var zip= new ZipInputStream(File.OpenRead(filePath));
var filestream=new FileStream(filePath,FileMode.Open,FileAccess.Read);
ZipFile zipfile = new ZipFile(filestream);
ZipEntry item;
while ((item = zip.GetNextEntry()) != null)
{
Console.WriteLine(item.Name);
using (StreamReader s = new StreamReader(zipfile.GetInputStream(item)))
{
Console.WriteLine(s.ReadToEnd());
}
}
Console.Read();
}
I am using sharplibzip library to implement this
This is the case when the zip file is located locally in the system. My next task scenario is what if the zipped file is located on the server. I am figuring out the way to implement it, below is the code what I assume should work
static void Main(string[] args)
{
string url = "https://test/code/304fd9c6-7e53-42a2-845a-624608bfd2ce.zip";
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse();
var zip = new ZipInputStream(webResponse.GetResponseStream());
ZipEntry item1;
//var zip= new ZipInputStream(File.OpenRead(filePath));
var filestream = new FileStream(filepath, FileMode.Open, FileAccess.Read);
ZipFile zipfile = new ZipFile(filestream);
ZipEntry item;
while ((item = zip.GetNextEntry()) != null)
{
Console.WriteLine(item.Name);
using (StreamReader s = new StreamReader(zipfile.GetInputStream(item)))
{
Console.WriteLine(s.ReadToEnd());
}
}
Console.Read();
}
I am stuck at this part: var filestream = new FileStream(filepath, FileMode.Open, FileAccess.Read);
This expect the first parameter to be path of the zip file. Since in the new scenario zip file is located remotely on the server. What should be the parameter in this case?
Your original code opens the stream twice on the following rows, which I think is causing some confusion:
var zip= new ZipInputStream(File.OpenRead(filePath));
var filestream=new FileStream(filePath,FileMode.Open,FileAccess.Read);
There is an overload to the ZipFile constructor that takes "any" Stream rather than specifically a FileStream, which you - unsurprisingly - can only create for files.
However, you cannot use the stream returned by GetResponseStream directly, because it's CanSeek property is false. This is because it's a NetworkStream, which can only be read once from beginning to end. SharpZipLib needs random access to read the file contents.
Depending on the size of the ZIP file, loading it in memory may be an option. If you expect large files, writing it to a temporary file may be better.
This should do the trick, without using both ZipInputStream and ZipFile, by enumerating through ZipFile instead:
string url = "https://test/code/304fd9c6-7e53-42a2-845a-624608bfd2ce.zip";
WebRequest webRequest = WebRequest.Create(url);
webRequest.Method = "GET";
WebResponse webResponse = webRequest.GetResponse();
using (var responseStream = webResponse.GetResponseStream())
using (var ms = new MemoryStream())
{
// Copy entire file into memory. Use a file if you expect a lot of data
responseStream.CopyTo(ms);
var zipFile = new ZipFile(ms);
foreach (ZipEntry item in zipFile)
{
Console.WriteLine(item.Name);
using (var s = new StreamReader(zipFile.GetInputStream(item)))
{
Console.WriteLine(s.ReadToEnd());
}
}
}
Console.Read();
PS: starting .NET 4.5, there is support for ZIP files built in. See the ZipArchive class.