We Keep Coding

Find a string in a zipped file without unzipping the file

Is there a way to search for a string within a file(s) within a zipped folder WITHOUT unzipping the files?
My situation is I have over 1 million files zipped by months of the year.
For example 2008_01, 2008_02, etc.
I need to extract/unzip only the files with specific serial numbers within the files.
The only thing I can find is unzipping the data to a temporary location to perform that search, but it takes me 45-60 minutes just to unzip the data manually. So I assume the code would take just as long to perform that task, plus I don't have that much available space.
Please Help.

Unfortunately, there isn't a way to do this. The zip format maintains an uncompressed manifest that shows file names and directory structure, but the contents of the files themselves are compressed, and therefore any string inside a file won't match your search until the file is decompressed.
This same limitation exists with just about any general-purpose file compression format (7zip, gzip, rar, etc.). You're essentially reclaiming disk space at the expense of CPU cycles.

Using some extension methods, you can scan through the Zip files. I don't think you can gain anything by trying to scan a single zip in parallel, but you could probably scan multiple zip files in parallel.
public static class ZipArchiveEntryExt {
public static IEnumerable<string> GetLines(this ZipArchiveEntry e) {
using (var stream = e.Open()) {
using (var sr = new StreamReader(stream)) {
string line;
while ((line = sr.ReadLine()) != null)
yield return line;
}
}
}
}
public static class ZipArchiveExt {
public static IEnumerable<string> FilesContain(this ZipArchive arch, string target) {
foreach (var entry in arch.Entries.Where(e => !e.FullName.EndsWith("/")))
if (entry.GetLines().Any(line => line.Contains(target)))
yield return entry.FullName;
}
public static void ExtractFilesContaining(this ZipArchive arch, string target, string extractPath) {
if (!extractPath.EndsWith(Path.DirectorySeparatorChar.ToString(), StringComparison.Ordinal))
extractPath += Path.DirectorySeparatorChar;
foreach (var entry in arch.Entries.Where(e => !e.FullName.EndsWith("/")))
if (entry.GetLines().Any(line => line.Contains(target)))
entry.ExtractToFile(Path.Combine(extractPath, entry.Name));
}
}
With these, you can search a zip file with:
var arch = ZipFile.OpenRead(zipPath);
var targetString = "Copyright";
var filesToExtract = arch.FilesContain(targetString);
You could also extract them to a particular path (assuming no filename conflicts) with:
var arch = ZipFile.OpenRead(zipPath);
var targetString = "Copyright";
arch.ExtractFilesContaining(targetString, #"C:\Temp");
You could modify ExtractFilesContaining to e.g. add the year-month to the file names to help avoid conflicts.

File.OpenRead() Directories , Sub Directories , Files, and then Write Folders and files to another directory

I am looking to read folders and files from a directory structure like so..
e.g
C:\RootFolder
SubFolder1
SubFolder2
File1
File2
SubFolder3
File3
Files....
Files....
I would like to read both, files and folders and write to another directory I cant use copy , because the directory I want to write to is remote and not local.
I read the files here.... Id love to be able to read folders and files and write both to another directory.
public static IEnumerable<FileInfo> GetFiles(string dir)
{
return Directory.EnumerateFiles(dir, "*", SearchOption.AllDirectories)
.Select(path =>
{
var stream = File.OpenRead(path);
return new FileInfo(Path.GetFileName(path), stream);
})
.DisposeEach(c => c.Content);
}
this function writes files to a remote sftp site.
public Task Write(IEnumerable<FileInfo> files)
{
return Task.Run(() =>
{
using (var sftp = new SftpClient(this.sshInfo))
{
sftp.Connect();
sftp.ChangeDirectory(this.remoteDirectory);
foreach (var file in files)
{
sftp.UploadFile(file.Content, file.RelativePath);
}
}
});
}
In this function I write the read files from the above function.
private async static Task SendBatch(Config config, Batch batch, IRemoteFileWriter writer)
{
var sendingDir = GetClientSendingDirectory(config, batch.ClientName);
Directory.CreateDirectory(Path.GetDirectoryName(sendingDir));
Directory.Move(batch.LocalDirectory, sendingDir);
Directory.CreateDirectory(batch.LocalDirectory);
//Use RemoteFileWriter...
var files = GetFiles(sendingDir);
await writer.Write(files).ContinueWith(t =>
{
if(t.IsCompleted)
{
var zipArchivePath = GetArchiveDirectory(config, batch);
ZipFile.CreateFromDirectory(
sendingDir,
zipArchivePath + " " +
DateTime.Now.ToString("yyyy-MM-dd hh mm ss.Zip")
);
}
});
}
Thank you!

You are getting UnauthorizedAccessException: Access to the path 'C:\temp' is denied. because you can't open a stream from a folder as it doesn't contain bytes.
From what I can understand you are looking to copy the files from one folder to another.
This answer seems to cover what you are doing. https://stackoverflow.com/a/3822913/3634581
Once you have copied the directories you can then create the zip file.
If you don't need to copy the files and just create the Zip I would recommend that since it will reduce the disk IO and speed up the process.
ZipArchive (https://msdn.microsoft.com/en-us/library/system.io.compression.ziparchive(v=vs.110).aspx) can be used to create a zip file straight to a stream.

I figured it out here is the solution
public Task Write(IEnumerable<FileInfo> files)
{
return Task.Run(() =>
{
using (var sftp = new SftpClient(this.sshInfo))
{
sftp.Connect();
sftp.ChangeDirectory(this.remoteDirectory);
foreach (var file in files)
{
var parts = Path.GetDirectoryName(file.RelativePath)
.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
sftp.ChangeDirectory(this.remoteDirectory);
foreach (var p in parts)
{
try
{
sftp.ChangeDirectory(p);
}
catch (SftpPathNotFoundException)
{
sftp.CreateDirectory(p);
sftp.ChangeDirectory(p);
}
}
sftp.UploadFile(file.Content, Path.GetFileName(file.RelativePath));
}
}
});
}
****Key point to the solution was
this
var parts = Path.GetDirectoryName(file.RelativePath)
.Split(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
We call *Path.GetDirectoryName
on the file itself to get the directory that correlates to the file.
We split the file directory to get the folder name and finally create the folder name we obtained from the split and upload the file to it.
I hope this helps others who may encounter such issue.

File is already use in some other process

Here I want to delete line in a textfiles containg specific string like "21309#003" where item1 is a filename but It shows runtime exception that item1 (file) is already use in some process.How I Solve this problem.I am new in .net C#.
private void button1_Click(object sender, EventArgs e)
{
var selectedItems = listBox1.SelectedItems.Cast<String>().ToList();
foreach (var item in selectedItems)
{
listBox1.Items.Remove(item);
}
foreach (var item1 in selectedItems)
{
listBox1.Items.Remove(item1);
string line = null;
//string line_to_delete = "the line i want to delete";
using (StreamReader reader = new StreamReader(item1))
//item1= "C:\\IMP2711\\textpresent.txt"
{
using (StreamWriter writer = new StreamWriter(item1))
{
while ((line = reader.ReadLine()) != null)
{
//if (String.Compare(line, #"*21349#003*") == 0)
//if (!line.Contains("21349#003") )
if (!line.StartsWith("21349#003"))
{**strong text**
writer.WriteLine(line);
}
}
}

You are reading and writing to the same file at the same time.
var item2 = item1;
If the file is not to big you can read the lines into memory and then write the lines you want to keep back to the file. We can even simplify your code a little bit.
File.WriteAllLines(item1,
File.ReadLines(item1).Where(l => !l.StartsWith("21349#003")).ToList());
Another option if the file is very large is to write to a temporary file. Delete the original and then rename the temporary.
var tmp = Path.GetTempFileName();
File.WriteAllLines(tmp, File.ReadLines(item1).Where(l => !l.StartsWith("21349#003")));
File.Delete(item1);
File.Move(tmp, item1);

If your file is small first read it to memory and then try to write on it, you have two stream on the same file, a file can share between multiple streams but you can not modify a file when it is open by another stream, if your file is huge and you can not moved to memory you can create a temp file and write to temp file when your reading finished replacing original file with temp file and removing temp file.

There's some process that's locking the file c:\imp2711\textpresent.txt. You have to find and kill it.
To find it out, please refer to this question: https://superuser.com/questions/117902/find-out-which-process-is-locking-a-file-or-folder-in-windows

c# zip file - Extract file last

Quick question: I need to extract zip file and have a certain file extract last.
More info: I know how to extract a zip file with c# (fw 4.5).
The problem I'm having now is that I have a zip file and inside it there is always a file name (for example) "myFlag.xml" and a few more files.
Since I need to support some old applications that listen to the folder I'm extracting to, I want to make sure that the XML file will always be extract the last.
Is there some thing like "exclude" for the zip function that can extract all but a certain file so I can do that and then extract only the file alone?
Thanks.

You could probably try a foreach loop on the ZipArchive, and exclude everything that doesn't match your parameters, then, after the loop is done, extract the last file.
Something like this:
private void TestUnzip_Foreach()
{
using (ZipArchive z = ZipFile.Open("zipfile.zip", ZipArchiveMode.Read))
{
string LastFile = "lastFileName.ext";
int curPos = 0;
int lastFilePosition = 0;
foreach (ZipArchiveEntry entry in z.Entries)
{
if (entry.Name != LastFile)
{
entry.ExtractToFile(#"C:\somewhere\" + entry.FullName);
}
else
{
lastFilePosition = curPos;
}
curPos++;
}
z.Entries[lastFilePosition].ExtractToFile(#"C:\somewhere_else\" + LastFile);
}
}

How to read multiple files from server into c#

I want to know how to read multiple (about 500-1000) text files which are located on a server.
So far, I've written code for a program that only reads a single text file.
Here's how I'm currently reading a single file.
public void button1_Click(object sender, EventArgs e)
{
// Reading/Inputing column values
OpenFileDialog ofd = new OpenFileDialog();
if (ofd.ShowDialog() == System.Windows.Forms.DialogResult.OK)
{
string[] fileLines = File.ReadAllLines(ofd.FileName);
I would like to get rid of the open file dialog box, and let the program automatically read the 500-1000 text files where are located in the sever.
I'm thinking something along the lines of
for (int i =0; i<numFiles; i++)
{
//just use string[] fileLines =File.ReadAllLines()
//how would i specify the path for multiple files?
}
Questions are then:
How would I approach this?
How exactly should I get the number of files?
(I'm guessing I'd have to read the server file which contains them.)

You can use Directory.GetFiles(string path) to get all files from a certain directory. You can then use a foreach loop to iterate through all the files in that directory and do your processing.

You can use recursion to loop through all directories. Using Directory.EnumerateFiles also allows you to use a foreach loop so you don't have to worry about the file count.
private static void ReadAllFilesStartingFromDirectory(string topLevelDirectory)
{
const string searchPattern = "*.txt";
var subDirectories = Directory.EnumerateDirectories(topLevelDirectory);
var filesInDirectory = Directory.EnumerateFiles(topLevelDirectory, searchPattern);
foreach (var subDirectory in subDirectories)
{
ReadAllFilesStartingFromDirectory(subDirectory);//recursion
}
IterateFiles(filesInDirectory, topLevelDirectory);
}
private static void IterateFiles(IEnumerable<string> files, string directory)
{
foreach (var file in files)
{
Console.WriteLine("{0}", Path.Combine(directory, file));//for verification
try
{
string[] lines = File.ReadAllLines(file);
foreach (var line in lines)
{
//Console.WriteLine(line);
}
}
catch (IOException ex)
{
//Handle File may be in use...
}
}
}
Also note Directory.EnumerateFiles provides overload that lets you provide a search pattern to narrow the scope of the files.

How you go about getting your files depends on if they are all located in the same directory of if you'll need to recursively search through a directory and all child directories. Directory.GetFiles is where you want to start. It has 3 overloads seen here. So you might try something like this:
string path = "\mypath\tosomehwere";
string searchPattern = "*.txt";
string[] MyFiles = Directory.GetFiles(path, searchPattern, SearchOption.AllDirectories);
Then just loop through the string array and proccess each file as you would normally.
foreach (string filePath in MyFiles)
{
MyFileProcessMethod(filePath)
}
Path.GetFileName(filePath) will return the individual text file name should you need it for your processing requirements.