There are number of possible solutions to do file backup application. I need to know which method would be rock-solid and professional way to perform copying of data files even though the file is being used or very large sized.
There is a known method called Volume shadow copy (VSS), however I've read that it is an overkill for a simple copying operation and instead the PInvoke BackupRead can be used.
.NET framework provides it's own methods:
File.Copy was (and possibly still is) problematic with large files and sharing the resources
FileStream seems to be suitable for backup purposes however I didn't locate comprehensive description and I am not sure if I'm correct.
Could you please enlighten me which method should be used (maybe I have overlooked some options) and why? If the VSS or PInvoke methods are preferred could you please also provide an example how to use it or some reference to comprehensive documentation (particularly I'm interested in the correct settings to create file handle, which would allow sharing the resources when the file is in use).
Thanks in advance.
Everything you're going to try in a live (i.e. currently running OS) volume will suffer from not being able to open some files. The reason is, applications and the OS itself opens files exclusively - that is, they open the files with ShareMode=0. You won't be able to read those files.
VSS negotiates with VSS-aware applications to release their open files for the duration, but relatively few applications outside Microsoft are VSS aware.
An alternative approach is to boot to another OS (on a USB stick or another on-disk volume) and do your work from there. For example, you could use the Microsoft Preinstallation environment (WinPE). You can, with some effort run a .Net 4.x application from there. From such an environment, you can get to pretty much any file on the target volume without sharing violations.
WinPE runs as local administrator. As such, you need to assert privileges, such as SE_BACKUP_NAME, SE_RESTORE_NAME, SE_SECURITY_NAME, SE_TAKE_OWNERSHIP_NAME, and you need to open the files with a flag called BACKUP_SEMANTCS...as described here.
The BackupRead/BackupWrite APIs are effective, if awkward. You can't use asynchronous file handles with these APIs...or at least MS claims you're in for "subtle errors" if you do. If those APIs are overkill, you can just use FileStreams.
There are bunches of little gotchas either way. For example, you should know when there are hardlinks in play or you'll be backing up redundant data...and when you restore, you don't want to break those links. There are APIs for getting all the hard links for a given file...NtQueryInformationFile, for example.
ReparsePoints (Junctions and SymLinks) require special handling, too...as they are low-level redirects to other file locations. You can run in circles following these reparse points if you're not careful, and even find yourself inadvertently backing up off-volume data.
Not easy to deal with all this stuff, but if thoroughness is an issue, you'll encounter them all before you're done.
Related
Overview
I'm currently working on a media streaming server using ASP.net Core REST Server. I'm currently using .net 5.0 and ASP.net Core MVC
What I need
I need to be able to dynamically down-res the original video file. from 1080p to 720p for example.
Also I need to be able to make the media file able to be transcoded to a different encoding based on client capabilities.
What I've Tried
I've been looking for a library that can manage this feat, but I can't seem to find one. I thought FFMpeg would be able to do this. I know this is possible because applications like plex and emby seem to manage this.
What I've Done
[HttpGet("/api/streaming/video")]
public IActionResult GetFile()
{
string path = "C:\Path\To\Video\FILE.mp4";
System.IO.FileStream stream = new(path, System.IO.FileMode.Open, System.IO.FileAccess.Read);
Microsoft.AspNetCore.Mvc.FileStreamResult file = File(stream, "video/mp4", true);
return file;
}
Framework Tried
Xabe.FFmpeg
FFMpegSharp
Given that you need this to work cross platform, as #Andy said the best solution is ffmpeg. You have two choices:
Invoke the ffmpeg command line utility from your web process; or
Compile the libav library suite (which underlies ffmpeg) to each native platform your code might be run on - Windows, Linux, etc. - make a native DLL wrapper, and use P/Invoke in your ASP.NET project to access it.
Command Line Utility
The command line utility is very easy to use and well documented. Documentation is here. Your basic approach would be to include ffmpeg.exe in your web project (make sure you have a version for each platform), and use Process.Start to invoke it, using the command line arguments to point it to your video file and configure the output. Once the output is finished, you can serve it by returning with File like in your example. There are also some open source .NET wrappers like this one that could save you some of the work. Unfortunately the command line utility doesn't offer much (any) control once started, or a programmatic way of determining progress. However, these issues should not be a problem if you follow my recommendation at the end.
Libav
If you do need or want total control, however - including frame by frame transcoding, progress reporting, etc. - then you would need to use libav. Before going further, note that you need to use at least some C/C++ to use libav. That means your server code is going to have to run with full trust, and you WILL be susceptible to the security risks of running native code. (Though the same would be true if you used ffmpeg.exe, but at least in that case you don't run the risk of introducing NEW security risks through your own code).
Also know that you can't just find nice clean, always up-to-date downloadable binaries for every platform (at least one reason for which is fear of patent lawsuits). Instead you have to build it yourself for every platform your code might run on. Find your platform(s) on here and then follow the instructions to the letter. If you make a single deviation no matter how small, you won't be able to build it, and you will pull your hair out figuring out why.
Once you have the builds, then your next major task is to expose the APIs you need to your C# code. The documentation for the libav APIs is not a model of clarity. They more or less assume you will look at the code for the ffmpeg command line utility to figure out how to use libav, and that's what you should do. But if you invest the time (days if not weeks) to construct the builds and learn the APIs, you will become a Master of Media. There is virtually nothing imaginable that you can't do using libav.
Now you're finally ready to integrate this into your app. Again you can take two approaches. If you're quite comfortable with C/C++, you should make a new C/C++ DLL project, link the libav DLLs to it, and do most of the heavy lifting there and just export couple of entrypoint functions that you can invoke from C#. Alternatively, you can P/Invoke directly to the libav DLLs, but you will need to do a ton of scaffolding of data structures and functions in C#. Unless you're extremely comfortable with marshalling, I would not attempt this.
In fact, I'm going to recommend going the command line utility route, because -
You Shouldn't Try to Transcode On The Fly Anyway
With all that out of the way, let's talk about your actual gameplan. You said you need to "dynamically" convert the video based on what the client wants/can receive. No one does this. What they do do is create multiple versions of the videos in advance and save each one on the server e.g., a 1080p, 720p, 480p, and maybe even 240p version. Then, the client application monitors the connection quality and, also considering the user's preference, directs the media player to the desired version. The server always serves the version requested and doesn't convert on the fly. Unless you're talking about streaming live events - and if so then that's beyond the scope of my expertise - this is what you should do.
So, what I would advise is use the ffmpeg utility to create different versions of the videos in advance - as part of an import or upload process for example. Track the videos in a database including what versions are available for each. Give the client a way to obtain this information. Then when it comes time to serve the videos, you just serve the one the client requests in a query parameter. Put the logic for determining the desired version on the client - either connection speed and/or user preference.
And Don't Forget to Support Content-Range Requests
Finally, you probably don't want to just use File to serve the media unless the users are just going to download the files for offline viewing. Assuming people are going to play videos in a browser, you need your API to accept content-range request headers. Here's a pretty good example of how to do that. Provided you implement it correctly, web browser media players will be able to play, seek, etc., transparently. If for whatever reason the format needs to change, just redirect the URL of the media player to the appropriate version, keep the position the same, and resume playing, and the user will barely notice a skip.
Hope at least some of this helps!
I have a large bespoke container file (~3TB) in size, and another application that needs to read from it.
However, the application doesn't understand the structure of the container, so I have to convert it first, which means creating another ~3TB file; I'm hoping to streamline this process.
What I'd like to do is to create a file/pipe/something on the file system, and when the other applications reads from it, my application simply returns the correct data from within the container.
I'm not sure if this can be done in C# and I don't really want to have to hook any OS components, so I was thinking that a named pipe might work, but I'm not sure, if anyone has any suggestions or ideas, I'd appreciate it.
If you don't control the consuming application and it expects to be reading from the file system, there may be a way of doing this but it's a fair bit of work.
Recent releases of Windows 10 have included the Windows Projected File System. Windows takes care of all of the file system interception and you just have to be able to answer questions like "what files are meant to be in this directory?" and the like. I believe it's now used for OneDrive and that's one of the intended uses - where the actual files may normally reside in cloud storage rather than locally.
You do have to make file content available as Windows demands it. The one thing to say though is that it's not an easy job direct from C#. If you're going to try binding to this API, it really helps if you understand a bit of C or C++ too.
Earlier this year I was looking to create a managed binding to this API to make consumption easier from .NET languages. It's not, however, currently in a releasable state. But the basics worked and proves that this is a viable approach.
Once .NET Core 3 is fully released I'll probably dust this off again and make it work well, but for now it's a WIP
I'm refactoring a WPF application and dealing with cleaning up storage of settings. I've re-written most to use the application's settings (Properties.Settings.Default) and this technically is working right now it seems to generate rather ugly paths in the %appdata% folder such as:
C:\Users\me\AppData\Local\Company_Name_With_Underscores\program.exe_Url_xs3vufrvyvfeg4xv01dvlt54k5g2xzfr\3.0.1.0\
These also then result in a new version number folder for each version that don't get cleaned up ever unless apparently I manually do so via file io functions.
Other programs don't seem to follow this format, including Microsoft programs, so I'm under the impression this seems like one of those 'technically the best way but not practical so nobody uses' solutions. Is this the case or am I missing something to make this more practical?
I ask mainly because I can foresee issues if we ever have to direct a customer to one of these folders to check or send us a file from there.
I'm using .NET 4.0 right now.
So, my application depends on a huge number of small files. The actual number is somewhere around 90,000. Now, I use a component that needs an access to these files, but the only way it accepts them is by the use of an URI.
So far I have simply added a directory containing all the files to my debug-folder while I have developed the application. However, now I have to consider the deployment. What are my options on including all these files with my deployment?
So far I have come up with a couple of different solutions, none of which I've managed to make work completely. First was to simply add all the files to the installer which would then copy them to their places. This would, in theory at least, work, but it'd make maintaining the installer (a standard MSI-installer generated with VS) an absolute hell.
The next option I came up with was to zip them into a single file and add this as a part of the installer and then unzip them by the use of a custom action. The standard libraries, however, do not seem to support complex zip-files, making this a rather hard option.
Finally, I realized that I could create a separate project and add all the files as resources in that project. What I don't know is how do the URIs pointing to resources stored in other assemblies work. Meaning, is it "standard" for everything to support the "application://,,,:Assembly"-format?
So, are these the only options I have, or are there some other ones as well? And what would be the best option to go about this?
I would use a single zip-like archive file, and not unzip that file on your hard disk but leave it as is. This is also the approach used by several well known applications that depend on lots of smaller files.
Windows supports using zip files as virtual folders (as of XP), users can see and edit their content with standard tools like Windows Explorer.
C# also has excellent support for zip files, if you're not happy with the built in tools I recommend one of the major Zip libraries out there - they're very easy to use.
In case you worry about performance, caching files in memory is a simple exercise. If your use case actually requires the files to exist on disk, also not an issue, just unzip them on first use - it's just a few lines of code.
In short, just use a zip archive and a good library and you won't run into any trouble.
In any case, I would not embed this huge amount of files in your application directly. Data files are to be separate.
You could include the files in a zip archive, and have the application itself unzip them on first launch as part of a final configuration, if it's not practical to do that from the installer. This isn't entirely atypical (e.g. it seems like most Microsoft apps do a post-install config on first run).
Depending on how the resources are used, you could could have a service that provides them on demand from a store of some kind and caches them, rather than dumping them somewhere. This may or may not make sense depending on what these resources are for, e.g. if they're UI elements a delay on first access might not be acceptable.
You could even serve them using http from a local or non-local server, or a SQL server if it's already using one, caching them as well, which would be great for maintenance, but may not work for the environment.
I wouldn't do anything that involves an embedded resource for each file individually, that would be hell to maintain.
Another option could be to create a self-extract zip/rar archive and extract it from the installer.
One of the options is to keep them in compound storage and access them right in the storage. The article on our site describes various types of storages and their advantages / specifics.
Hi I need your help if you are an expert in MOSS.
I have a system that needs to upload documents into a MOSS document library.
I decided that the easiest approach for a phase 1 system would simply be to map a network path to a MOSS Document library.
The whole thing seems too easy. After that its a straight copy using System.IO.
What I would like to know, is this method reliable enough to be used in a production system?
Speculation would be great, but if you have real experience with working with MOSS in this way, your answer would mean a lot.
Thanks.
So long as you do the proper error checking around the copy, its fine - if you bear in mind the standard caveats with SharePoint document libraries and file naming conventions.
SharePoint does not allow some characters in file names which NTFS and FAT do - these will cause an error when you try to copy them to the DL, regardless of how you do that, so you will need to sanitise your filenames beforehand.
The only downside to using a network path to the webdav interface of SharePoint is that if you stress it too much (a large copy of a lot of files), you can easily overwhelm it and it will cause the network share to become unavailable for a period of time. If you are talking about a few files every now and then, several an hour for example, it should be fine.
You are better off reading the files off the network path and then using the Object Model API or the web services to upload it.
You can use timer jobs which can be scheduled to run at a convenient time. The timer job can read it's configuration settings from an XML file.
This system would be easier to maintain, and troubleshoot, when compared to a straight copy using System.IO.