I keep getting the error "Stream was not writable" whenever I try to execute the following code. I understand that there's still a reference to the stream in memory, but I don't know how to solve the problem. The two blocks of code are called in sequential order. I think the second one might be a function call or two deeper in the call stack, but I don't think this should matter, since I have "using" statements in the first block that should clean up the streams automatically. I'm sure this is a common task in C#, I just have no idea how to do it...
string s = "";
using (Stream manifestResourceStream =
Assembly.GetExecutingAssembly().GetManifestResourceStream("Datafile.txt"))
{
using (StreamReader sr = new StreamReader(manifestResourceStream))
{
s = sr.ReadToEnd();
}
}
...
string s2 = "some text";
using (Stream manifestResourceStream =
Assembly.GetExecutingAssembly().GetManifestResourceStream("Datafile.txt"))
{
using (StreamWriter sw = new StreamWriter(manifestResourceStream))
{
sw.Write(s2);
}
}
Any help will be very much appreciated. Thanks!
Andrew
Embedded resources are compiled into your assembly, you can't edit them.
As stated above, embedded resources are read only. My recommendation, should this be applicable, (say for example your embedded resource was a database file, XML, CSV etc.) would be to extract a blank resource to the same location as the program, and read/write to the extracted resource.
Example Pseudo Code:
if(!Exists(new PhysicalResource())) //Check to see if a physical resource exists.
{
PhysicalResource.Create(); //Extract embedded resource to disk.
}
PhysicalResource pr = new PhysicalResource(); //Create physical resource instance.
pr.Read(); //Read from physical resource.
pr.Write(); //Write to physical resource.
Hope this helps.
Additional:
Your embedded resource may be entirely blank, contain data structure and / or default values.
A bit late, but for descendants=)
About embedded .txt:
Yep, on runtime you couldnt edit embedded because its embedded. You could play a bit with disassembler, but only with outter assemblies, which you gonna load in current context.
There is a hack if you wanna to write to a resource some actual information, before programm starts, and to not keep the data in a separate file.
I used to worked a bit with winCE and compact .Net, where you couldnt allow to store strings at runtime with ResourceManager. I needed some dynamic information, in order to catch dllNotFoundException before it actually throws on start.
So I made embedded txt file, which I filled at the pre-build event.
like this:
cd $(ProjectDir)
dir ..\bin\Debug /a-d /b> assemblylist.txt
here i get files in debug folder
and the reading:
using (var f = new StreamReader(Assembly.GetExecutingAssembly().GetManifestResourceStream("Market_invent.assemblylist.txt")))
{
str = f.ReadToEnd();
}
So you could proceed all your actions in pre-build event run some exes.
Enjoy! Its very usefull to store some important information and helps avoid redundant actions.
Related
I have code that creates 3 files from 3 strings and zips it into the archive like:
private static async Task<byte[]> CreateArchive(CertificateCredentials certificate)
{
using (var ms = new MemoryStream())
{
using (var archive = new ZipArchive(ms, ZipArchiveMode.Create, false))
{
await Add(archive, "certificate.der", certificate.CertificateContent);
await Add(archive, "issuing_ca.der", certificate.IssuingCACertificateContent);
await Add(archive, "private_key.der", certificate.PrivateKeyContent);
}
return ms.ToArray();
async Task Add(ZipArchive zipArchive, string filename, string content)
{
ZipArchiveEntry zipEntry = zipArchive.CreateEntry(filename);
using (var originalFileStream = new MemoryStream(Convert.FromBase64String(content)))
using (Stream zipEntryStream = zipEntry.Open())
{
await originalFileStream.CopyToAsync(zipEntryStream);
}
}
}
}
In SonarQube report I got Critical Security Hotspot at this line
using (Stream zipEntryStream = zipEntry.Open())
with message:
Make sure that decompressing this archive file is safe
https://rules.sonarsource.com/csharp?search=Expanding%20archive%20files%20is%20security-sensitive
How can I fix that? It looks like secure for me.
Thank you in advance
Regarding security related rules you can find the actual documentation here. You can read in the "What to expect from security-related rules" that the chance of false positives is greater and that a human should have a look at it when an issue is raised. So, given the example, it is well possible that this issue can be identified as a false positive and no code change is needed. In the user guide you can find here how to handle reported issues. In the "Automatic Issue Assignment" section (Technical Review) you can read how to mark an issue as a false-positive (this requires Administer Issues permission on the project) using the SonarQube UI. This prevents future issue reporting on this code. When there are a lot of issues reported based on this rule you could decide to disable the rule or lower its priority. An other possiblity is to narrow the focus by ignore Issues in a few Blocks for example. It all depends on the type of project and the project/security requirements.
EDIT:
The rule warns for the following issues:
CVE-2018-1263: It is possible to construct a (external) zip with files that, when extracted, will be placed outside the extraction directory. When adding a file to a zip, the file will be identified inside the zip by a filename (including a path if needed). The CVE database does not mention an example yet.
CVE-2018-16131: It is possible to create a (extrenal) zip that, when extracted will consume all the memory available and this will crash the host ('Zip Bomb'). The CVE database is pointing to this issue where someone succeeds to exploit this issue.
SonarQube doesn't want to know how ZipArchive is implemented. It is quite possible that, when adding a new item to a zip, the original Zip if first extracted exposing the issues above. Your are creating and using a (in memory, but that is not very relevant) zip archive in your own code, not using any external provided zip file so both are of no issue here if you trust the .Net implementation of the methods used.
The SonarQube documentation is also pointing to this Java example containing a security compliant solution. It is quite possible that even then SonarQube will warn you.
I need to write a big file in my project.
What I learned:
I should NOT write the big file directly to the destination path,
because this may leave a incomplete file in case the app crash while writing it.
Instead, I should write to a temporary file and move (rename) it. (called atomic file operation)
My code snippet:
[NotNull]
public static async Task WriteAllTextAsync([NotNull] string path, [NotNull] string content)
{
string temporaryFilePath = null;
try {
temporaryFilePath = Path.GetTempFileName();
using (var stream = new StreamWriter(temporaryFilePath, true)) {
await stream.WriteAsync(content).ConfigureAwait(false);
}
File.Delete(path);
File.Move(temporaryFilePath, path);
}
finally {
if (temporaryFilePath != null) File.Delete(temporaryFilePath);
}
}
My Question:
The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Is there any other best practice for writing big files?
Is there any suggestion on my code?
The file will be missing if the app crashes between File.Delete and File.Move. Can I avoid this?
Not that I'm aware of, but you can detect it - and if you use a more predictable filename, you can recover from that. It helps if you tweak the process somewhat to use three file names: the target, a "new" file and an "old" file. The process becomes:
Write to "new" file (e.g. foo.txt.new)
Rename the target file to the "old" file (e.g. foo.txt.old)
Rename the "new" file to the target file
Delete the "old" file
You then have three files, each of which may be present or absent. That can help you to detect the situation when you come to read the new file:
No files: Nothing's written data yet
Just target: All is well
Target and new: App crashed while writing new file
Target and old: App failed to delete old file
New and old: App failed after the first rename, but before the second
All three, or just old, or just new: Something very odd is going on! User may have interfered
Note: I was unaware of File.Replace before, but I suspect it's effectively just a simpler and possibly more efficient way of doing the code you're already doing. (That's great - use it!) The recovery process would still be the same though.
You can use File.Replace instead of deleting and moving files. In case of hard fault (electricity cut or something like this) you will always lost data, you have to count with that.
I've a plugin based web app that allows an administrator to assign various small pieces of functionality (the actual functionality is unimportant here) to users.
This functionality is configurable and an administrator having an understanding of the plugins is important. The administrator is technical enough to be able to read the very simple source code for these plugins. (Mostly just arithmetic).
I'm aware there are a couple of questions already about accessing source code from within a DLL built from that source:
How to include source code in dll? for example.
I've played with getting the .cs files into a /Resources folder. However doing this with a pre-build event obviously don't include these files in the project. So VS never copies them and I'm unable to access them on the Resources object.
Is there a way to reference the source for a particular class... from the same assembly that I'm missing? Or alternatively a way to extract COMPLETE source from the pdb for that assembly? I'm quite happy to deploy detailed PDB files. There's no security risk for this portion of the solution.
As I've access to the source code, I don't want to go about decompiling it to display it. This seems wasteful and overly complicated.
The source code isn't included in the DLLs, and it isn't in the PDBs either (PDBs only contain a link between the addresses in the DLL and the corresponding lines of code in the sources, as well as other trivia like variable names).
A pre-build event is a possible solution - just make sure that it produces a single file that's included in the project. A simple zip archive should work well enough, and it's easy to decompress when you need to access the source. Text compresses very well, so it might make sense to compress it anyway. If you don't want to use zip, anything else will do fine as well - an XML file, for example. It might even give you the benefit of using something like Roslyn to provide syntax highlighting with all the necessary context.
Decompilation isn't necessarily a terrible approach. You're trading memory for CPU, basically. You'll lose comments, but that shouldn't be a problem if your code is very simple. Method arguments keep their names, but locals don't - you'd need the PDBs for that, which is a massive overkill. It actually does depend a lot on the kind of calculations you're doing. For most cases, it probably isn't the best solution, though.
A bit roundabout way of handling this would be a multi-file T4 template. Basically, you'd produce as many files as there are source code files, and have them be embedded resources. I'm not sure how simple this is, and I'm sure not going to include the code :D
Another (a bit weird) option is to use file links. Simply have the source code files in the project as usual, but also make a separate folder where the same files will be added using "Add as link". The content will be shared with the actual source code, but you can specify a different build action - namely, Embedded Resource. This requires a (tiny) bit of manual work when adding or moving files, but it's relatively simple. If needed, this could also be automated, though that sounds like an overkill.
The cleanest option I can think of is adding a new build action - Compile + Embed. This requires you to add a build target file to your project, but that's actually quite simple. The target file is just an XML file, and then you just manually edit your SLN/CSPROJ file to include that target in the build, and you're good to go. The tricky part is that you'll also need to force the Microsoft.CSharp.Core.target to use your Compile + Embed action to be used as both the source code and the embedded resource. This is of course easily done by manually changing that target file, but that's a terrible way of handling that. I'm not sure what the best way of doing that is, though. Maybe there's a way to redefine #(Compile) to mean #(Compile;MyCompileAndEmbed)? I know it's possible with the usual property groups, but I'm not sure if something like this can be done with the "lists".
Taking from #Luaan's suggestion of using a pre-build step to create a single Zipped folder I created a basic console app to package the source files into a zip file at a specific location.
static void Main(string[] args)
{
Console.WriteLine("Takes a folder of .cs files and flattens and compacts them into a .zip." +
"Arg 1 : Source Folder to be resursively searched" +
"Arg 2 : Destination zip file" +
"Arg 3 : Semicolon List of folders to ignore");
if (args[0] == null || args[1] == null)
{
Console.Write("Args 1 or 2 missing");
return;
};
string SourcePath = args[0];
string ZipDestination = args[1];
List<String> ignoreFolders = new List<string>();
if (args[2] != null)
{
ignoreFolders = args[2].Split(';').ToList();
}
var files = DirSearch(SourcePath, "*.cs", ignoreFolders);
Console.WriteLine($"{files.Count} files found to zip");
if (File.Exists(ZipDestination))
{
Console.WriteLine("Destination exists. Deleting zip file first");
File.Delete(ZipDestination);
}
int zippedCount = 0;
using (FileStream zipToOpen = new FileStream(ZipDestination, FileMode.OpenOrCreate))
{
using (ZipArchive archive = new ZipArchive(zipToOpen, ZipArchiveMode.Create))
{
foreach (var filePath in files)
{
Console.WriteLine($"Writing {Path.GetFileName(filePath)} to zip {Path.GetFileName(ZipDestination)}");
archive.CreateEntryFromFile(filePath, Path.GetFileName(filePath));
zippedCount++;
}
}
}
Console.WriteLine($"Zipped {zippedCount} files;");
}
static List<String> DirSearch(string sDir, string filePattern, List<String> excludeDirectories)
{
List<String> filePaths = new List<string>();
foreach (string d in Directory.GetDirectories(sDir))
{
if (excludeDirectories.Any(ed => ed.ToLower() == d.ToLower()))
{
continue;
}
foreach (string f in Directory.GetFiles(d, filePattern))
{
filePaths.Add(f);
}
filePaths.AddRange(DirSearch(d, filePattern, excludeDirectories));
}
return filePaths;
}
Takes 3 parameters for source dir, output zip file and a ";" separated list of paths to exclude. I've just built this as a binary. Committed it to source control for simplicity and included it in the pre-build for projects I want the source for.
No error checking really and I'm certain it will fail for missing args. But if anyone wants it. Here it is! Again Thanks to #Luaan for clarifying PDBs aren't all that useful!
It was clearly stated that File.Move is atomic operation here: Atomicity of File.Move.
But the following code snippet results in visibility of moving the same file multiple times.
Does anyone know what is wrong with this code?
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
namespace FileMoveTest
{
class Program
{
static void Main(string[] args)
{
string path = "test/" + Guid.NewGuid().ToString();
CreateFile(path, new string('a', 10 * 1024 * 1024));
var tasks = new List<Task>();
for (int i = 0; i < 10; i++)
{
var task = Task.Factory.StartNew(() =>
{
try
{
string newPath = path + "." + Guid.NewGuid();
File.Move(path, newPath);
// this line does NOT solve the issue
if (File.Exists(newPath))
Console.WriteLine(string.Format("Moved {0} -> {1}", path, newPath));
}
catch (Exception e)
{
Console.WriteLine(string.Format(" {0}: {1}", e.GetType(), e.Message));
}
});
tasks.Add(task);
}
Task.WaitAll(tasks.ToArray());
}
static void CreateFile(string path, string content)
{
string dir = Path.GetDirectoryName(path);
if (!Directory.Exists(dir))
{
Directory.CreateDirectory(dir);
}
using (FileStream f = new FileStream(path, FileMode.OpenOrCreate))
{
using (StreamWriter w = new StreamWriter(f))
{
w.Write(content);
}
}
}
}
}
The paradoxical output is below. Seems that file was moved multiple times onto different locations. On the disk only one of them is present. Any thoughts?
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.0018d317-ed7c-4732-92ac-3bb974d29017
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.3965dc15-7ef9-4f36-bdb7-94a5939b17db
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.fb66306a-5a13-4f26-ade2-acff3fb896be
Moved test/eb85560d-8c13-41c1-926a-6871be030742 -> test/eb85560d-8c13-41c1-926a-6871be030742.c6de8827-aa46-48c1-b036-ad4bf79eb8a9
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
System.IO.FileNotFoundException: Could not find file 'C:\file-move-test\test\eb85560d-8c13-41c1-926a-6871be030742'.
The resulting file is here: eb85560d-8c13-41c1-926a-6871be030742.fb66306a-5a13-4f26-ade2-acff3fb896be
UPDATE. I can confirm that checking File.Exists also does NOT solve the issue - it can report that single file was really moved into several different locations.
SOLUTION. The solution I end up with is following: Prior to operations with source file create special "lock" file, if it succeeded then we can be sure that only this thread got exclusive access to the file and we are safe to do anything we want. The below is right set of parameters to create suck "lock" file.
File.Open(lockPath, FileMode.CreateNew, FileAccess.Write);
Does anyone know what is wrong with this code?
I guess that depends on what you mean by "wrong".
The behavior you're seeing is not IMHO unexpected, at least if you're using NTFS (other file systems may or may not behave similarly).
The documentation for the underlying OS API (MoveFile() and MoveFileEx() functions) is not specific, but in general the APIs are thread-safe, in that they guarantee the file system will not be corrupted by concurrent operations (of course, your own data could be corrupted, but it will be done in a file-system-coherent way).
Most likely what is occurring is that as the move-file operation proceeds, it does so by first getting the actual file handle from the given directory link to it (in NTFS, all "file names" that you see are actually hard links to an underlying file object). Having obtained that file handle, the API then creates a new file name for the underlying file object (i.e. as a hard link), and then deletes the previous hard link.
Of course, as this progresses, there is a window during the time between a thread having obtained the underlying file handle but before the original hard link has been deleted. This allows some but not all of the other concurrent move operations to appear to succeed. I.e. eventually the original hard link doesn't exist and further attempts to move it won't succeed.
No doubt the above is an oversimplification. File system behaviors can be complex. In particular, your stated observation is that you only wind up with a single instance of the file when all is said and done. This suggests that the API does also somehow coordinate the various operations, such that only one of the newly-created hard links survives, probably by virtue of the API actually just renaming the associated hard link after retrieving the file object handle, as opposed to creating a new one and deleting the old one (implementation detail).
At the end of the day, what's "wrong" with the code is that it is intentionally attempting to perform concurrent operations on a single file. While the file system itself will ensure that it remains coherent, it's up to your own code to ensure that such operations are coordinated so that the results are predictable and reliable.
I have a file (an xml), that is accessed in my code, I would like it to be some how added to the executable, so my utility can access it at runtime, but still be all in one file. Is there a way to doing that? (C#)
Thanks.
Look at embedded resources (first result from a Google search, but looks good at first glance)
Actually this article has the advantage of actually telling you how to make something an embedded resource. Between the two of them you should be able to sort out your problem.
Add it as an embedded resource (set the build action for the file to be "Embedded Resource") and use Assembly.GetManifestResourceStream to access it.
Be aware that when accessing a resource stream the name is case sensitive.
In the properties windows, set the properties Build Action as Embedded Resource.
After that you can access your file like this:
Assembly assbl = Assembly.GetAssembly(this.GetType());
using(Stream s = assbl.GetManifestResourceStream("projectnamespace.embeddedfilename.xml"))
{
XmlDocument doc = new XmlDocument();
using (StreamReader reader = new StreamReader(s))
{
doc.LoadXml(reader.ReadToEnd());
reader.Close();
}
}
In GetManifestResourceStream, you need to specify the "path" of your file in your project.