I am trying to create a function that will retrieve all the uploaded files (which are now saved as byte in the database) and download it in a single zip file. I currently have 6000 files to download (and the number could grow).
The functionality is already working (from retrieval to download) if I limit the number of files being downloaded, otherwise, I get an OutOfMemoryException on the ForEach loop.
Here's a pseudo code: (files variable is a list of byte array and file name)
var files = getAllFilesFromDB();
foreach (var file in files)
{
var tempFilePath = Path.Combine(path, filename);
using (FileStream stream = new FileStream(tempfileName, FileMode.Create, FileAccess.ReadWrite))
{
stream.Write(file.byteArray, 0, file.byteArray.Length);
}
}
private readonly IEntityRepository<File> fileRepository;
IEnumerable<FileModel> getAllFilesFromDb()
{
return fileRepository.Select(f => new FileModel(){ fileData = f.byteArray, filename = f.fileName});
}
My question is, is there any other way to do this to avoid getting such errors?
To avoid this problem, you could avoid loading all the contents of all the files in one go. Most likely you will need to split your database call in to two database calls.
Retrieve a list of all the files without their contents but with some identifier - like the PK of the table.
A method which retrieves the contents of an individual file.
Then your (pseudo)code becomes
get list of all files
for each file
get the file contents
write the file to disk
Another possibility is to alter the way your query works currently, so that it uses deferred execution - this means it will not actually load all the files at once, but stream them one at a time from the database - but without seeing more code from your repository implementation, I cannot/ will not guess the right solution for you.
Related
I would like to know if it is possible to check the size of the zip file that is being created dynamically, because I am reading a directory and generate a 19 MB zip and I would like two zips instead to be created, one 10MB and the other 9MB. However, when I give a .Length in the zip file inside my loop it says the size is 0. When I finish adding the files it says that is 19MB. Would anyone know how to do this?
I am using only System.IO.Compression to this task.
here is some example to show how I am trying
String FilePath = "D:\Invoices";
string[] oFiles = Directory.GetFiles(FilePath,"*.pdf");
string name = DateTime.Now.Ticks.ToString()+".zip";
using(FileStream archive1 = File.Open(name,FileMode.Create))
{
using(var arch = new ZipArchive(archive1,ZipArchiveMode.Update))
{
for(int i =0; i<oFiles.Length;i++)
{
var fileinf = new FileInfo(oFiles[i]);
arch.CreateEntryFromFile(fileinf.FullName,fileinf.Name);
//here the zip file is always 0
Console.WriteLine(archive1.Length);
}
}
}
//here the zip file is updated
From the documentation:
When you set the mode to Update … The content of the entire archive is held in memory, and no data is written to the underlying file or stream until the archive is disposed.
If you want to be able to read the size of the file as you're adding things, you need to use ZipArchiveMode.Create.
Otherwise, you should use the ZipArchiveEntry.CompressedSize property to monitor how much you've added to the archive.
I have a CSV file (E.g. Directories.csv) which contains a huge list of directories. I am looping through the directories from the CSV using streamreader and performing some task. I am updating the completed directory list to a dictionary and stuck at this step now.
Ask: I want to capture the data through the loop on which directories are complete in the same CSV just in case the application crashes or server reboots, so that I don't have to re-iterate through the loop again which got completed. (Or) Delete the completed directories row from the CSV
I tried to check online for suggestions and asking to create temp file and move the copy of it. Can this be possible in case the server reboots or application crashes? Please suggest how can I take this forward.
My code:
Dictionary<string, string> directoryDictionary = new Dictionary<string, string>();
using (FileStream fileStreamDirectory = File.Open(outputdir + "\\Directories.csv", FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (BufferedStream bufferStreamDirectory = new BufferedStream(fileStreamDirectory))
using (StreamReader streamReaderDirectory = new StreamReader(bufferStreamDirectory))
{
while ((Directoryline = streamReaderDirectory.ReadLine()) != null)
{
#Doing the task here
directoryDictionary.Add(Directoryline, "Completed");
}
}
You can't really insert data into middle of a text file (unless it is fixed width format which in not the case of CSV).
Two options:
read to memory, update in-memory data, rewrite whole table back to the file (may need to keep previous version in case of write failures)
use database that satisfy you criteria and import CSV there to work with.
I want to store files in my SQL Server database by C# which I have done it without problem.
This is my code:
byte[] file;
using (var stream = new FileStream(letter.FilePath, FileMode.Open, FileAccess.Read))
{
using (var reader = new BinaryReader(stream))
{
file = reader.ReadBytes((int)stream.Length);
letter.ltr_Image = file;
}
}
LetterDB letterDB = new LetterDB();
id = letterDB.LetterActions(letter);
The insert SQL action in the LetterActions module. But I want to know, in order to reduce the size of the database (which increases daily) is there any solution for compressing the files and then store them in the database?
Yes , you can zip your files before storing them in the database, using the ZipFile class. Take a look here: https://msdn.microsoft.com/en-us/library/system.io.compression.zipfile(v=vs.110).aspx
Plenty of sample code out there too. See here:http://imar.spaanjaars.com/414/storing-uploaded-files-in-a-database-or-in-the-file-system-with-aspnet-20
You can compress file like this. Then insert compressed file stream into DB, but when you read it you need decompress it.
If you really need store file in DB, suggest you compress and decompress it by client.
And better way handle file is store them in disk, and only store file path in DB, when client need file use file path get file.
I'm looking for a way to store files in a zip file without a memory stream. My goal is to save a maximum of system memory, while direct disk IO is no problem.
I iterate over a database result set where I have collected some blobs. These are byte-arrays.
What I do it the following (System.IO.Compression):
using (var archive = ZipFile.Open("data.zip", ZipArchiveMode.Update))
{
foreach (var result in results)
{
string fileName = $"{result.Id}.bin";
using (var fileStream = new FileStream(fileName, FileMode.Create, FileAccess.Write))
{
// write the blob data from result.Value
fileStream.Write(result.Value, 0, result.Value.Length);
fileStream.Close();
}
archive.CreateEntryFromFile(fileName, fileName);
}
}
There are 2 problems with this implementation.
I have my *.bin files AND the one *.zip (only need the zip)
I don't know why, but this uses a lot of RAM (~100MB for 15x1.5MB bin files)
Is there a way to completely bybass the memory?
UPDATE:
What I'm trying to achieve is to generate one ZIP file that contains single binary files generated from database blobs. This should happen inside a ASP.NET Web API controller. A user can request the data, but instead of sending the whole data in the HTTP response, I generate the ZIP file in the time of the request, save it to a local file server and send a download link back to the user.
I think your >100 MBs coming from
the results object which should contain at least 15x1.5 MB of blob data
holding the resulting data.zip open inside the foreach Scope.
to minimize the RAM amount of the worker process:
create empty zip-file
do {
(single BLOB query from DB)
(write blob to new or overwrite File)
(open zip file for append)
(append file to zip)
(close and dispose **both** file handles / objects )
}
How can I read content of a text file inside a zip archive?
For example I have an archive qwe.zip, and insite it there's a file asd.txt, so how can I read contents of that file?
Is it possible to do without extracting the whole archive? Because it need to be done quick, when user clicks a item in a list, to show description of the archive (it needed for plugin system for another program). So extracting a whole archive isn't the best solution... because it might be few Mb, which will take at least few seconds or even more to extract... while only that single file need to be read.
You could use a library such as SharpZipLib or DotNetZip to unzip the file and fetch the contents of individual files contained inside. This operation could be performed in-memory and you don't need to store the files into a temporary folder.
Unzip to a temp-folder take the file and delete the temp-data
public static void Decompress(string outputDirectory, string zipFile)
{
try
{
if (!File.Exists(zipFile))
throw new FileNotFoundException("Zip file not found.", zipFile);
Package zipPackage = ZipPackage.Open(zipFile, FileMode.Open, FileAccess.Read);
foreach (PackagePart part in zipPackage.GetParts())
{
string targetFile = outputDirectory + "\\" + part.Uri.ToString().TrimStart('/');
using (Stream streamSource = part.GetStream(FileMode.Open, FileAccess.Read))
{
using (Stream streamDestination = File.OpenWrite(targetFile))
{
Byte[] arrBuffer = new byte[10000];
int iRead = streamSource.Read(arrBuffer, 0, arrBuffer.Length);
while (iRead > 0)
{
streamDestination.Write(arrBuffer, 0, iRead);
iRead = streamSource.Read(arrBuffer, 0, arrBuffer.Length);
}
}
}
}
}
catch (Exception)
{
throw;
}
}
Although late in the game and the question is already answered, in hope that this still might be useful for others who find this thread, I would like to add another solution.
Just today I encountered a similar problem when I wanted to check the contents of a ZIP file with C#. Other than NewProger I cannot use a third party library and need to stay within the out-of-the-box .NET classes.
You can use the System.IO.Packaging namespace and use the ZipPackage class. If it is not already included in the assembly, you need to add a reference to WindowsBase.dll.
It seems, however, that this class does not always work with every Zip file. Calling GetParts() may return an empty list although in the QuickWatch window you can find a property called _zipArchive that contains the correct contents.
If this is the case for you, you can use Reflection to get the contents of it.
On geissingert.com you can find a blog article ("Getting a list of files from a ZipPackage") that gives a coding example for this.
SharpZipLib or DotNetZip may still need to get/read the whole .zip file to unzip a file. Actually, there is still method could make you just extract special file from the .zip file without reading the entire .zip file but just reading small segment.
I needed to have insights into Excel files, I did it like so:
using (var zip = ZipFile.Open("ExcelWorkbookWithMacros.xlsm", ZipArchiveMode.Update))
{
var entry = zip.GetEntry("xl/_rels/workbook.xml.rels");
if (entry != null)
{
var tempFile = Path.GetTempFileName();
entry.ExtractToFile(tempFile, true);
var content = File.ReadAllText(tempFile);
[...]
}
}