We Keep Coding

Best practice for writing big files

I need to write a big file in my project.
What I learned:
I should NOT write the big file directly to the destination path,
because this may leave a incomplete file in case the app crash while writing it.
Instead, I should write to a temporary file and move (rename) it. (called atomic file operation)
My code snippet:
[NotNull]
public static async Task WriteAllTextAsync([NotNull] string path, [NotNull] string content)
{
string temporaryFilePath = null;
try {
temporaryFilePath = Path.GetTempFileName();
using (var stream = new StreamWriter(temporaryFilePath, true)) {
await stream.WriteAsync(content).ConfigureAwait(false);
}
File.Delete(path);
File.Move(temporaryFilePath, path);
}
finally {
if (temporaryFilePath != null) File.Delete(temporaryFilePath);
}
}
My Question:
The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Is there any other best practice for writing big files?
Is there any suggestion on my code?

The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Not that I'm aware of, but you can detect it - and if you use a more predictable filename, you can recover from that. It helps if you tweak the process somewhat to use three file names: the target, a "new" file and an "old" file. The process becomes:
Write to "new" file (e.g. foo.txt.new)
Rename the target file to the "old" file (e.g. foo.txt.old)
Rename the "new" file to the target file
Delete the "old" file
You then have three files, each of which may be present or absent. That can help you to detect the situation when you come to read the new file:
No files: Nothing's written data yet
Just target: All is well
Target and new: App crashed while writing new file
Target and old: App failed to delete old file
New and old: App failed after the first rename, but before the second
All three, or just old, or just new: Something very odd is going on! User may have interfered
Note: I was unaware of File.Replace before, but I suspect it's effectively just a simpler and possibly more efficient way of doing the code you're already doing. (That's great - use it!) The recovery process would still be the same though.

You can use File.Replace instead of deleting and moving files. In case of hard fault (electricity cut or something like this) you will always lost data, you have to count with that.

Xamarin Android throws IOException when using ZipFile.ExtractToDirectory method

I try to use the following code in my project without any success and it's driving me mad.
System.IO.Compression.ZipFile.ExtractToDirectory(filePath, appPath);
The parameters are:
filePath = "/storage/emulated/0/Flashback_Backup/memory_backup.zip"
appPath = "/storage/emulated/0/Flashback"
According to the documentation here IOException should be thrown if:
The directory specified by destinationDirectoryName already exists.
-or- The name of an entry in the archive is Empty, contains only white space, or contains at least one invalid character.
-or- Extracting an archive entry would create a file that is outside the directory specified by destinationDirectoryName. (For example,
this might happen if the entry name contains parent directory
accessors.)
-or- An archive entry to extract has the same name as an entry that has already been extracted from the same archive.
As far as I know none of it applies. The zip file is a totally valid one, which I compressed with the Directory.CreateDirectory method, and only contains a few uniquely named JSON files. I tried with and without existing "Flashback" folder too, but nothing seems to work.
If anyone have any ideas or solutions please tell me because I'm seriously lost at this. I can provide more info if needed.

Try to wrap extraction intro try-catch, it may give you a better understanding of what is going on.
try {
System.IO.Compression.ZipFile.ExtractToDirectory(filePath, appPath);
} catch (Exception ex) {
Console.Log(ex);
}
If there is an error, it will be one of your listed above.

Include source code for library in that library

I've a plugin based web app that allows an administrator to assign various small pieces of functionality (the actual functionality is unimportant here) to users.
This functionality is configurable and an administrator having an understanding of the plugins is important. The administrator is technical enough to be able to read the very simple source code for these plugins. (Mostly just arithmetic).
I'm aware there are a couple of questions already about accessing source code from within a DLL built from that source:
How to include source code in dll? for example.
I've played with getting the .cs files into a /Resources folder. However doing this with a pre-build event obviously don't include these files in the project. So VS never copies them and I'm unable to access them on the Resources object.
Is there a way to reference the source for a particular class... from the same assembly that I'm missing? Or alternatively a way to extract COMPLETE source from the pdb for that assembly? I'm quite happy to deploy detailed PDB files. There's no security risk for this portion of the solution.
As I've access to the source code, I don't want to go about decompiling it to display it. This seems wasteful and overly complicated.

The source code isn't included in the DLLs, and it isn't in the PDBs either (PDBs only contain a link between the addresses in the DLL and the corresponding lines of code in the sources, as well as other trivia like variable names).
A pre-build event is a possible solution - just make sure that it produces a single file that's included in the project. A simple zip archive should work well enough, and it's easy to decompress when you need to access the source. Text compresses very well, so it might make sense to compress it anyway. If you don't want to use zip, anything else will do fine as well - an XML file, for example. It might even give you the benefit of using something like Roslyn to provide syntax highlighting with all the necessary context.
Decompilation isn't necessarily a terrible approach. You're trading memory for CPU, basically. You'll lose comments, but that shouldn't be a problem if your code is very simple. Method arguments keep their names, but locals don't - you'd need the PDBs for that, which is a massive overkill. It actually does depend a lot on the kind of calculations you're doing. For most cases, it probably isn't the best solution, though.
A bit roundabout way of handling this would be a multi-file T4 template. Basically, you'd produce as many files as there are source code files, and have them be embedded resources. I'm not sure how simple this is, and I'm sure not going to include the code :D
Another (a bit weird) option is to use file links. Simply have the source code files in the project as usual, but also make a separate folder where the same files will be added using "Add as link". The content will be shared with the actual source code, but you can specify a different build action - namely, Embedded Resource. This requires a (tiny) bit of manual work when adding or moving files, but it's relatively simple. If needed, this could also be automated, though that sounds like an overkill.
The cleanest option I can think of is adding a new build action - Compile + Embed. This requires you to add a build target file to your project, but that's actually quite simple. The target file is just an XML file, and then you just manually edit your SLN/CSPROJ file to include that target in the build, and you're good to go. The tricky part is that you'll also need to force the Microsoft.CSharp.Core.target to use your Compile + Embed action to be used as both the source code and the embedded resource. This is of course easily done by manually changing that target file, but that's a terrible way of handling that. I'm not sure what the best way of doing that is, though. Maybe there's a way to redefine #(Compile) to mean #(Compile;MyCompileAndEmbed)? I know it's possible with the usual property groups, but I'm not sure if something like this can be done with the "lists".

Taking from #Luaan's suggestion of using a pre-build step to create a single Zipped folder I created a basic console app to package the source files into a zip file at a specific location.
static void Main(string[] args)
{
Console.WriteLine("Takes a folder of .cs files and flattens and compacts them into a .zip." +
"Arg 1 : Source Folder to be resursively searched" +
"Arg 2 : Destination zip file" +
"Arg 3 : Semicolon List of folders to ignore");
if (args[0] == null || args[1] == null)
{
Console.Write("Args 1 or 2 missing");
return;
};
string SourcePath = args[0];
string ZipDestination = args[1];
List<String> ignoreFolders = new List<string>();
if (args[2] != null)
{
ignoreFolders = args[2].Split(';').ToList();
}
var files = DirSearch(SourcePath, "*.cs", ignoreFolders);
Console.WriteLine($"{files.Count} files found to zip");
if (File.Exists(ZipDestination))
{
Console.WriteLine("Destination exists. Deleting zip file first");
File.Delete(ZipDestination);
}
int zippedCount = 0;
using (FileStream zipToOpen = new FileStream(ZipDestination, FileMode.OpenOrCreate))
{
using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Create))
{
foreach (var filePath in files)
{
Console.WriteLine($"Writing {Path.GetFileName(filePath)} to zip {Path.GetFileName(ZipDestination)}");
archive.CreateEntryFromFile(filePath, Path.GetFileName(filePath));
zippedCount++;
}
}
}
Console.WriteLine($"Zipped {zippedCount} files;");
}
static List<String> DirSearch(string sDir, string filePattern, List<String> excludeDirectories)
{
List<String> filePaths = new List<string>();
foreach (string d in Directory.GetDirectories(sDir))
{
if (excludeDirectories.Any(ed => ed.ToLower() == d.ToLower()))
{
continue;
}
foreach (string f in Directory.GetFiles(d, filePattern))
{
filePaths.Add(f);
}
filePaths.AddRange(DirSearch(d, filePattern, excludeDirectories));
}
return filePaths;
}
Takes 3 parameters for source dir, output zip file and a ";" separated list of paths to exclude. I've just built this as a binary. Committed it to source control for simplicity and included it in the pre-build for projects I want the source for.
No error checking really and I'm certain it will fail for missing args. But if anyone wants it. Here it is! Again Thanks to #Luaan for clarifying PDBs aren't all that useful!

How to backup a complete file?

What is the most efficient way to make a backup of a file when it's being opened into the program, so that when the user changes and saves it, there is always a way to go back?
Example:
private void open_click(object sender, EventArgs e)
{
ofd.DefaultExt = "";
if (ofd.ShowDialog() == System.Windows.Forms.DialogResult.OK)
{
fileIn = ofd.FileName;
fileOut = Path.GetTempFileName();
string encoded = File.ReadAllText(fileIn);
etc. etc. etc
}
The file that gets loaded into the program needs to get backed up as backup_01 and put in the same folder as the original file. When backup_01 exists, backup as backup_02, and so on).
Examples are more than welcome!

I would generally create a copy of the file itself, place it in a "backup" folder, and apply some naming scheme to it to indicate its age.
Eg: folder/originalFile.xyz ==> folder/backup/originalFile_2013-04-14-12-48.bak
Update/afterthought: I think the efficiency of this will depend on the OS performing the Copy-operation, but it should in general not be too bad. Unless you have good reason to do so, I would avoid trying to add extra logic to do it more efficiently.
Update in response to comment:
I won't provide a detailed implementation here, but I'll try to point you in the right direction: Check out System.IO.File, specifically the methods Copy and Exists. (This list of other common IO-taks may also be useful)
With these, you should be able to check if a file exists (eg. if you already have "backup_1.xyz" in your backup-folder), and based on that, generate a new name for your next backup file.
Create a loop that replaces 1 with increasing numbers until you find a "free" filename, and then copy the original file to a new file with that name.
Good luck! :)

UnauthorizedAccessException on newly created files

I have an application that is looking through some files for old data. In order to make sure we don't corrupt good projects, I'm copying the files to a temporary location. Some of the directories I'm checking are source-code directories, and they have .svn folders. We use Subversion to manage our code.
Once I've searched through all of the files, I want to delete the temp cache. Sounds easy, right?
For some reason, all of my .svn directories won't delete from the cache. They crash the app.
For reasons (too deep to go into here), I have to use the temp folder, so just "scan the original file" is out of the question for political reasons.
I can go into explorer and delete them. No problem. No warnings. Just deletes. But the code crashes with "Access to {file} is denied." I'm at my wits end with this one, so any help would be appreciated.
While I've simplified the function a LITTLE for sake of your sanity, the code REALLY is about this simple.
List<string> tmpCacheManifest = new List<string>();
string oldRootPath = "C:\\some\\known\\directory\\";
string tempPath = "C:\\temp\\cache\\";
foreach (string file in ListOfFilesToScan)
{
string newFile = file.Replace(oldRootPath, tempPath);
// This works just fine.
File.Copy(file, newFile);
tmpCacheManifest.add(newFile);
}
// ... do some stuff to the cache to verify what I need.
// Okay.. I'm done.. Delete the cache.
foreach (string file in tmpCacheManifest)
{
// CRASH!
File.Delete(file);
}
* Update *: The exception is UnauthorizedAccessException. The text is "Access to the path 'C:\temp\cache\some-sub-dirs\.svn\entries' is denied."
It happens under XP, XP-Pro and Windows 7.
* Update 2 * None of my validation even ATTEMPTS to look at subversion files. I do need them, however. That's part of the political crap. I have to show that EVERY file was copied... wheter it was scanned or not.
And I realize what the usual suspects are for File.Delete. I realize what UnauthorizedAccessException means. I don't have access. That's a no-brainer. But I just copied the file. How can I NOT have access to the file?
* Update 3 *
The answer was in the "read-only" flag. Here's the code I used to fix it:
foreach (string file in ListOfFilesToScan)
{
string newFile = file.Replace(oldRootPath, tempPath);
// This works just fine.
File.Copy(file, newFile);
//// NEW CODE ////
// Clear any "Read-Only" flags
FileInfo fi3 = new FileInfo(fn);
if ((fi3.Attributes & FileAttributes.ReadOnly) == FileAttributes.ReadOnly)
{
fi3.Attributes = (FileAttributes)(Convert.ToInt32(fi3.Attributes) - Convert.ToInt32(FileAttributes.ReadOnly));
}
tmpCacheManifest.add(newFile);
}
// ... do some stuff to the cache to verify what I need.

As far as I recall, Subversion marks the files in its .svn subdirectories as read-only.
Try resetting the read-only attribute before deleting the file. I don't really know any C#, but a quick Google suggests this might do the trick:
File.SetAttributes(file, FileAttributes.Normal);

The only problem I see would be in this part:
// ... do some stuff to the cache to verify what I need.
If you do open the file and forget to close it, you still have exclusive access to it, and thus can't delete it later on.

Sounds like you don't have access to delete the file...
system.io.file.delete
The above link says you get UnauthorizedAccessException when:
The caller does not have the required permission.
-or-
path is a directory.
-or-
path specified a read-only file.
It's one of those.

Sounds like a permissions issue. Tricky one though as you obviously have write access if the File.Copy already works....
Only thing I could think of is the file still has a handle opened somewhere (as others have suggested perhaps in your do some stuff to the cache part).

First of all: "Crash" means an exception, right? Which one? Can you catch it and show it?
Second thing: You are copying subversion repositories, although you don't care about the subversion metadata? That's what svn export is about (no .svn directory in the target).
The answer to the first question is what you really need to provide. Maybe something grabs the .svn and locks some files. TortoiseSVN maybe (to give you nice overlay icons..)?

If a folder contains read only files, Directory.Delete won't delete it and raise the exception you're getting. For future visitors of this page, I've found a simple solution which doesn't require us to recurse through all the files and changing their read-only attribute:
Process.Start("cmd.exe", "/c " + #"rmdir /s/q C:\Test\TestDirectoryContainingReadOnlyFiles");
(Change a bit to not to fire a cmd window momentarily, which is available all over the internet)

Not understanding what you want to do so much, but what about chmoding it to 777 or 775. :-/
Edit:
Noticed your on windows. You'd have to change the permissions. Don't know how windows does that :-/