I have a large bespoke container file (~3TB) in size, and another application that needs to read from it.
However, the application doesn't understand the structure of the container, so I have to convert it first, which means creating another ~3TB file; I'm hoping to streamline this process.
What I'd like to do is to create a file/pipe/something on the file system, and when the other applications reads from it, my application simply returns the correct data from within the container.
I'm not sure if this can be done in C# and I don't really want to have to hook any OS components, so I was thinking that a named pipe might work, but I'm not sure, if anyone has any suggestions or ideas, I'd appreciate it.
If you don't control the consuming application and it expects to be reading from the file system, there may be a way of doing this but it's a fair bit of work.
Recent releases of Windows 10 have included the Windows Projected File System. Windows takes care of all of the file system interception and you just have to be able to answer questions like "what files are meant to be in this directory?" and the like. I believe it's now used for OneDrive and that's one of the intended uses - where the actual files may normally reside in cloud storage rather than locally.
You do have to make file content available as Windows demands it. The one thing to say though is that it's not an easy job direct from C#. If you're going to try binding to this API, it really helps if you understand a bit of C or C++ too.
Earlier this year I was looking to create a managed binding to this API to make consumption easier from .NET languages. It's not, however, currently in a releasable state. But the basics worked and proves that this is a viable approach.
Once .NET Core 3 is fully released I'll probably dust this off again and make it work well, but for now it's a WIP
Related
Overview
I'm currently working on a media streaming server using ASP.net Core REST Server. I'm currently using .net 5.0 and ASP.net Core MVC
What I need
I need to be able to dynamically down-res the original video file. from 1080p to 720p for example.
Also I need to be able to make the media file able to be transcoded to a different encoding based on client capabilities.
What I've Tried
I've been looking for a library that can manage this feat, but I can't seem to find one. I thought FFMpeg would be able to do this. I know this is possible because applications like plex and emby seem to manage this.
What I've Done
[HttpGet("/api/streaming/video")]
public IActionResult GetFile()
{
string path = "C:\Path\To\Video\FILE.mp4";
System.IO.FileStream stream = new(path, System.IO.FileMode.Open, System.IO.FileAccess.Read);
Microsoft.AspNetCore.Mvc.FileStreamResult file = File(stream, "video/mp4", true);
return file;
}
Framework Tried
Xabe.FFmpeg
FFMpegSharp
Given that you need this to work cross platform, as #Andy said the best solution is ffmpeg. You have two choices:
Invoke the ffmpeg command line utility from your web process; or
Compile the libav library suite (which underlies ffmpeg) to each native platform your code might be run on - Windows, Linux, etc. - make a native DLL wrapper, and use P/Invoke in your ASP.NET project to access it.
Command Line Utility
The command line utility is very easy to use and well documented. Documentation is here. Your basic approach would be to include ffmpeg.exe in your web project (make sure you have a version for each platform), and use Process.Start to invoke it, using the command line arguments to point it to your video file and configure the output. Once the output is finished, you can serve it by returning with File like in your example. There are also some open source .NET wrappers like this one that could save you some of the work. Unfortunately the command line utility doesn't offer much (any) control once started, or a programmatic way of determining progress. However, these issues should not be a problem if you follow my recommendation at the end.
Libav
If you do need or want total control, however - including frame by frame transcoding, progress reporting, etc. - then you would need to use libav. Before going further, note that you need to use at least some C/C++ to use libav. That means your server code is going to have to run with full trust, and you WILL be susceptible to the security risks of running native code. (Though the same would be true if you used ffmpeg.exe, but at least in that case you don't run the risk of introducing NEW security risks through your own code).
Also know that you can't just find nice clean, always up-to-date downloadable binaries for every platform (at least one reason for which is fear of patent lawsuits). Instead you have to build it yourself for every platform your code might run on. Find your platform(s) on here and then follow the instructions to the letter. If you make a single deviation no matter how small, you won't be able to build it, and you will pull your hair out figuring out why.
Once you have the builds, then your next major task is to expose the APIs you need to your C# code. The documentation for the libav APIs is not a model of clarity. They more or less assume you will look at the code for the ffmpeg command line utility to figure out how to use libav, and that's what you should do. But if you invest the time (days if not weeks) to construct the builds and learn the APIs, you will become a Master of Media. There is virtually nothing imaginable that you can't do using libav.
Now you're finally ready to integrate this into your app. Again you can take two approaches. If you're quite comfortable with C/C++, you should make a new C/C++ DLL project, link the libav DLLs to it, and do most of the heavy lifting there and just export couple of entrypoint functions that you can invoke from C#. Alternatively, you can P/Invoke directly to the libav DLLs, but you will need to do a ton of scaffolding of data structures and functions in C#. Unless you're extremely comfortable with marshalling, I would not attempt this.
In fact, I'm going to recommend going the command line utility route, because -
You Shouldn't Try to Transcode On The Fly Anyway
With all that out of the way, let's talk about your actual gameplan. You said you need to "dynamically" convert the video based on what the client wants/can receive. No one does this. What they do do is create multiple versions of the videos in advance and save each one on the server e.g., a 1080p, 720p, 480p, and maybe even 240p version. Then, the client application monitors the connection quality and, also considering the user's preference, directs the media player to the desired version. The server always serves the version requested and doesn't convert on the fly. Unless you're talking about streaming live events - and if so then that's beyond the scope of my expertise - this is what you should do.
So, what I would advise is use the ffmpeg utility to create different versions of the videos in advance - as part of an import or upload process for example. Track the videos in a database including what versions are available for each. Give the client a way to obtain this information. Then when it comes time to serve the videos, you just serve the one the client requests in a query parameter. Put the logic for determining the desired version on the client - either connection speed and/or user preference.
And Don't Forget to Support Content-Range Requests
Finally, you probably don't want to just use File to serve the media unless the users are just going to download the files for offline viewing. Assuming people are going to play videos in a browser, you need your API to accept content-range request headers. Here's a pretty good example of how to do that. Provided you implement it correctly, web browser media players will be able to play, seek, etc., transparently. If for whatever reason the format needs to change, just redirect the URL of the media player to the appropriate version, keep the position the same, and resume playing, and the user will barely notice a skip.
Hope at least some of this helps!
I'm refactoring a WPF application and dealing with cleaning up storage of settings. I've re-written most to use the application's settings (Properties.Settings.Default) and this technically is working right now it seems to generate rather ugly paths in the %appdata% folder such as:
C:\Users\me\AppData\Local\Company_Name_With_Underscores\program.exe_Url_xs3vufrvyvfeg4xv01dvlt54k5g2xzfr\3.0.1.0\
These also then result in a new version number folder for each version that don't get cleaned up ever unless apparently I manually do so via file io functions.
Other programs don't seem to follow this format, including Microsoft programs, so I'm under the impression this seems like one of those 'technically the best way but not practical so nobody uses' solutions. Is this the case or am I missing something to make this more practical?
I ask mainly because I can foresee issues if we ever have to direct a customer to one of these folders to check or send us a file from there.
I'm using .NET 4.0 right now.
There are number of possible solutions to do file backup application. I need to know which method would be rock-solid and professional way to perform copying of data files even though the file is being used or very large sized.
There is a known method called Volume shadow copy (VSS), however I've read that it is an overkill for a simple copying operation and instead the PInvoke BackupRead can be used.
.NET framework provides it's own methods:
File.Copy was (and possibly still is) problematic with large files and sharing the resources
FileStream seems to be suitable for backup purposes however I didn't locate comprehensive description and I am not sure if I'm correct.
Could you please enlighten me which method should be used (maybe I have overlooked some options) and why? If the VSS or PInvoke methods are preferred could you please also provide an example how to use it or some reference to comprehensive documentation (particularly I'm interested in the correct settings to create file handle, which would allow sharing the resources when the file is in use).
Thanks in advance.
Everything you're going to try in a live (i.e. currently running OS) volume will suffer from not being able to open some files. The reason is, applications and the OS itself opens files exclusively - that is, they open the files with ShareMode=0. You won't be able to read those files.
VSS negotiates with VSS-aware applications to release their open files for the duration, but relatively few applications outside Microsoft are VSS aware.
An alternative approach is to boot to another OS (on a USB stick or another on-disk volume) and do your work from there. For example, you could use the Microsoft Preinstallation environment (WinPE). You can, with some effort run a .Net 4.x application from there. From such an environment, you can get to pretty much any file on the target volume without sharing violations.
WinPE runs as local administrator. As such, you need to assert privileges, such as SE_BACKUP_NAME, SE_RESTORE_NAME, SE_SECURITY_NAME, SE_TAKE_OWNERSHIP_NAME, and you need to open the files with a flag called BACKUP_SEMANTCS...as described here.
The BackupRead/BackupWrite APIs are effective, if awkward. You can't use asynchronous file handles with these APIs...or at least MS claims you're in for "subtle errors" if you do. If those APIs are overkill, you can just use FileStreams.
There are bunches of little gotchas either way. For example, you should know when there are hardlinks in play or you'll be backing up redundant data...and when you restore, you don't want to break those links. There are APIs for getting all the hard links for a given file...NtQueryInformationFile, for example.
ReparsePoints (Junctions and SymLinks) require special handling, too...as they are low-level redirects to other file locations. You can run in circles following these reparse points if you're not careful, and even find yourself inadvertently backing up off-volume data.
Not easy to deal with all this stuff, but if thoroughness is an issue, you'll encounter them all before you're done.
The Windows Azure client libraries are very big (several MBs), and I have a fairly small project (on the order of a few hundred KBs) that uses only a few functions from them. Is there a way for me to link in those functions at build time, so that the resultant DLL doesn't get hugely bloated, and I don't have to link the functions in at runtime?
Something like this http://blogs.msdn.com/b/microsoft_press/archive/2010/02/03/jeffrey-richter-excerpt-2-from-clr-via-c-third-edition.aspx, but I get the impression that bundles in the whole DLL.
Thanks!
Edit: Because there are external constraints on the size of the final deliverable DLL, inflating it this much is an absolute last resort - the only other option I'm aware of is just to duplicate the code I use verbatim.
In a word: No.
Remember that even though you only use a few functions, there are likely many other function in the library that those functions use, that you don't even know about!
You can't do this, because you don't have access to all the dependencies. Remember also that those dependencies may even reside in another DLL, and you need to include that entire DLL for the same reason.
What you are looking for sounds very like .NET Native.
Unfortunately for you, its only preview and right now works only with Store Apps for devices. Statements like this
We will continue to evolve and improve native compilation for the range of .NET applications
can be found on the internet but nothing specific about web apps\Azure.
Until then, answer is No
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have the task of developing an application to pull data from remote REST services and generating Excel reports. This application will be used by a handful of users at the company (10-15). The data load can reach 10,000-200,000 records. I have been debating whether to use Python or C#...
The only reason I am considering Python is because I am familiar with it and it would be less of a risk.
Personally I want to try use C# since this would be a good opportunity to learn it. The application is not too complicated so the overhead of learning it won't be too much... I think.
Are there any issues with C# that I should be concerned about for this type of program? The users run Windows XP... would users not having .NET installed be a major concern?
Thanks in advance.
EDIT:
I guess I need to stress the fact that the end users should be able to run the application without installing additional libraries/frameworks.
Why not IronPython which merges the two worlds together?
If you want to learn C# and you don't have such a limited time constraint, now might be a good time to try it. Also, though I haven't done Excel work with either Python or C#, I would expect it to be easier to work with Excel files with a Microsoft product rather than Python. If you're just dumping data in CSV format to a file, though, either Python or C# will work fine.
If you want to try something even crazier, you can use IronPython such that you'll have .NET resources available but you can still write in Python.
I would normally say, use what you know.
However, here you may have some performance issues using an interpreted language. I, myself, would use C#. If you want to learn it, now is as good of a time as any.
However, if your own cost/benefit analysis reveals that you would be better off in python, use that.
.NET will be required for C# on windows, but you can use the .NET Client Profile to make the install pretty painless.
I realize this is an old post but I guess I have some pertinent information for anyone coming across this in more recent times, as I have.
The Python library, Openpyxl has worked well for me for manipulating and creating excel files in my desktop based automation scripts.
http://openpyxl.readthedocs.io/en/default/usage.html
It is accessing the file directly from my understanding, unlike the Interop technique in c#. So wherever it runs, you wouldn't need Excel installed. If you're trying to get something on a server it might be an issue getting a licensed copy of Excel or ensuring it exists if your user is running this locally.
Using C# you have the full features of visual studio to help code, such as the code completion, debugging tools and error highlighting. However you can get most of that with Python in Visual Studio, though I've found it more complicated trying to get the right version of python tied to the project and whatnot.
In C# you can easily compile an executable whereas I believe you'll need to mess with loading some Python libraries to get close to a similar local executable with dependencies comparable to the C# output.
If the users have to install a C# client then not having .NET installed would be an issue. You can package the installer so it downloads the .NET runtime, but depending on what features you use it could be quite a big download.
If your application is web based then all the C# code runs on the server and just delivers HTML to the browser so whether .NET is installed or not shouldn't matter.
Integration with Excel will almost certainly be simpler from C#. Given the requirements "Windows only" and "Integrates with Excel", it would seem a simple choice that C# is better suited to this individual problem.
And, no, users' not having .NET is not a concern compared with Python as the alternative. The Dot Net framework is a standard part of Microsoft Update. In contrast, Python will almost certainly not be there on the average end users' machine. In your case, with an internal app, that last point might not matter of course.
If you are doing any kind of HTTP work with C# make sure you take a look at the new HttpClient library that is in WCF REST Starter Preview2. Ignore the name of that download, the HttpClient part can be used independently of WCF.
It is a WAY better client than just using HttpWebRequest.
If you do a web based application not having installed .NET in the client machine wont be a problem .. but I suppose you dont want to try both C# and web development at the same time
It is easy with Python to pull data from a web server, parse the JSON or XML, and generate some kind of report. If you need really good Excel integration (for example, your output is supposed to be an Excel spreadsheet that includes numerous graphs) C# might be better, but Python is still possible; there is PyExcelerator for writing rich spreadsheet files, and there is Python support for COM if you want to use COM to talk to Excel.
With Python it is also very easy to make a stand-alone executable, for example with py2exe.
This sounds like a batch tool, one that won't have a GUI; I think I would write it in Python if I were you, unless I really wanted a fun project to help me learn C#. But anything with a GUI, C# is probably better. (I haven't done GUI development with Python, but I have seen comments here and elsewhere that the performance of Python GUIs tend to be slow.)
The answer probably has more to do with the circumstances than the question of which is the better language for the task at hand.
If you have no prior C# experience, it's probably not a good idea to jump into it now and learn it as you go along. Syntax differences are trivial to learn, mastery is defined by knowing the dirty details and how to use the language's strengths to get the job done.
If you have plenty of time and starting over a lot is NOT an issue, or you have experience with a language that is paradigmatically very similar to C# (say, Java or C++), you could probably afford to take the risk and adapt to the new language very quickly.
If you don't care about efficiency (e.g. it's a pet project, you don't care much about your job or efficiency simply isn't important), try C# just to learn something new (always a good thing) or stick to Python to make sure you won't have to adjust your thinking too much (always a source of vulnerabilities and misunderstandings).
If you definitely want to use C#, you should play around with it in your "free time" (not necessarily at home, but preferably at work when there's nothing else to do) and thus build a skillset you can use once you have to start with the actual project. This way you'll already be familiar with the paradigm (to use the term loosely) and encounter less problems where they count.