I am looking for an effective way to grab image data off video files. I am currently testing FilgraphManagerClass.GetCurrentImage() from the Interop.QuartzTypeLib library. This does what I need but is painfully slow. I need to process all frames of each video. What better options do I have?
Requirements
Must be frame accurate. <-- Very important!
Gives me access to the decoded pixel buffer (array of int or byte[]), ideally RGB24 or RGB32.
The buffer can be grabbed in realtime or faster. I do not need to display the video, I only need to analyze the pixels.
Handle mp4 files (h264/aac). I can rewrap or frame serve via AviSynth if needed but no retranscoding can be involved.
Any suggestions would be welcome.
Some code as requested:
FilgraphManagerClass graphClass = new FilgraphManagerClass();
graphClass.RenderFile(#"C:\tmp\tmp.avs");
int sz = (graphClass.Width * graphClass.Height + 10) * 4;
int[] buffer = new int[sz - 1];
I am then stepping through each frame. I have something like this in the loop:
graphClass.GetCurrentImage(ref sz, out buffer[0]);
//DoStuff(buffer);
graphClass.CurrentPosition += graphClass.AvgTimePerFrame;
IBasicVideo::GetCurrentImage method you are using is basically intended for snapshots, and works with legacy video rendering in legacy modes only. That is, (a) it is NOT time accurate, it can get you duplicate frames or, the opposite, lose frames; and (b) it assumes that you display video.
Instead you want to build a filter graph of the following kind: File Source -> ... -> Sample Grabber Filter -> Null Renderer. Sample Grabber, a standard component, can be provided with a callback so that it calls you with any frame data that comes through it.
Then you remove clock from the graph by calling SetReferenceClock(null) on the filter graph so that it run as fast as possible (as opposed to realtime). Then you Run the graph and all video frames are supplied to your callback.
To accomplish the task in C# you need to use DirectShow.NET library. It's Capture\DxSnap sample provides a brief example how to use Sample Grabber. They do it through BufferCB instead of SampleCB and it works well too. Other samples there are also using this approach.
You will find other code snippets very close to this task:
Seeking keyframes using DirectShowNet - use of Sample Grabber
BufferCB not being called by SampleGrabber - same task for audio part
How to access an audio stream using DirectShow.NET C#
Regarding MP4 files you should take into consideration the following:
Support for MPEG-4 is limited in Windows, and you might need third party components installed to make the files playable. If GraphEdit can read them, then you can too.
Windows Media Player might be using, and is likely to, a newer API and you should rather look at GraphEdit
Be sure to use Win32/x86 platform on your application to avoid running into scenario that your app is running in 64-bit domain, while support for MP4 only exists in 32-bit components/libraries installed
You could also look at creating an allocator-presenter using Windows Media Foundation. This will give you the decoded video frame as a GPU texture and you could also use CUDA or OpenCL to perform the processing required (if possible) which would help your processing speed immensely.
Related
I get a stream of frames, an initial SPS and PPS h264 data packet and then packets for the I and P frames.
Using c# .NET I want to convert into a series of JPEGs? Has anyone done this?
I have tried AForge.NET FFMpeg wrapper, but can only go from MP4 file to JPEGs?
Also looked at DirectShow.
I can't seem to find an examples that even come close to doing this?
Thanks
In Directshow, it sounds like you need to introduce a SampleGrabber filter into your filter graph. Insert this after the H264 decoder.
This is a pass through filter which can receive a callback containing each video frame. You can then obviously choose what you do with the frame, is save it to disk as a jpeg etc.
Rather than regurgitate MSDN, there is a great page explaining its usage here:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx
Update taking account of Roman's comments:
SampleGrabber is deprecated as its part of Directshow Editing Services. However, it is quite self contained. If it was removed from a later version of Windows it would be straight forward to replace with an alternative filter. I still use it in one of my consumer applications. Roman is correct though - it has a steep learning curve.
I'm developing a Windows 8 Store App, I have this problem,
I want the user to add the videos from a file picker and i managed it,
the problem is I want to display the videos images in a GridView,
like snap shot of the video in a certain position, i tried the media element and it's not working
also an image and it doesn't make sense.
There might not be a pure C# solution to this problem. Perhaps SharpDX would let you do that, but I haven't tried and I don't know if it was done before. If you look at these two threads:
http://social.msdn.microsoft.com/Forums/fil-PH/wingameswithdirectx/thread/05731d4f-5b7f-4ed1-8e28-94604655139e
http://social.msdn.microsoft.com/Forums/en/wingameswithdirectx/thread/ffacd05c-6e9e-43bf-b691-99127240730c
-- you should see that there is a TransferVideoFrame method you can use to copy a frame from a video stream to a DirectX texture. The Media engine native C++ video playback sample shows how you can use it natively. If you search for TransferVideoFrame+SharpDX - you can find this sample that uses the SharpDX version of this method in C#. After you transfer that frame to a texture - you can either copy the contents of the texture to a WriteableBitmap using the Map and Unmap methods like here. You can also save it to a file either using BitmapEncoder a here or directly from the DirectX texture to a file using WIC or say SharpDX Toolkit as in here. You will probably want to build a cache of thumbnails to avoid having to process the videos every time you display the list, so saving to a file is something you should do anyway.
Your own solution that you quoted elsewhere that should work for at least some videos:
var thumb = await
storageFile.GetThumbnailAsync(
Windows.Storage.FileProperties.ThumbnailMode.PicturesView,
1000,
Windows.Storage.FileProperties.ThumbnailOptions.UseCurrentScale);
I suggest using GetThumbnailAsync(), which works just as well. You don't even need to get the video properties or anything like that. If the StorageFile is loaded, this should work for a normal video of any format.
Is there a way I can capture a single frame from a video file (mpg, wmv, flv, etc) at a specific point in the video (e.g. 5 seconds or the 25th frame)? Then save it as an image?
[Edit]
Something link YouTube does. That can't be all done manually? ;)
I'd use DirectShow.NET, because it'll let you do a lot of the work in managed code, which is quite a bit more friendly than doing it in native code.
You'll have to construct a filter graph to render the file you want, and you'll also need a file reader for the format of the file (i.e. if it's an MP4 file, you'll need an MP4 demux), and you'll need a decoder for the format of video (i.e. if it's H264, you'll need an H264 decoder filter). I'd use Windows7 if possible, it has much better media support.
Your graph should look something like:
File Reader -> Video Decoder -> Sample Grabber -> Null Renderer
You'll construct your graph, and then call IMediaSeeking to seek to the approximate time of the sample you want. Then run the graph. The decompressed frames will come in through a Sample Grabber callback interface. You can check the timestamps and get the one that's closest to what you need.
From there, you can use .NET to save it as whatever image format you like (JPEG is probably best).
FFMPEG and .net is your best choice
I want to provide a choice between streaming 'channels', if you will, in a web application. These are Windows Media streams of live events that are being broadcast from a Windows Media Services 9 distribution network.
I want to provide a relatively recent thumbnail image of the stream (as a user, you would expect to see this), but although I've seen this done in Flash on CNN and countless other sites, I've never seen this done with Windows Media.
I already have a C# / DirectX library that can extract a thumbnail from a WMV file, but obviously the stream doesn't come from a file if it's a live source.
My assumptions so far are:
Will need to run some kind of service application/daemon that will receive a stream into a Windows Media Player object and somehow take thumbnails if WMPlayer supports it...
-or-
Configure the streams to archive to file, and use the existing class library to take a peek at the last frame available in the archive file being written to get the thumbnail.
I would much rather do #1 because it seems like the clean solution, but don't know if/how WMPlayer supports grabbing a frame.
Are there better ways of doing this?
Not sure if using media player is a good idea in such case. I would look into this:
http://sourceforge.net/projects/windowsmedianet/
I have a live 16-bit gray-scale video stream that is pushed through a ring-buffer in memory as a raw, uncompressed byte stream (2 bytes per pixel, 2^18 pixels/frame, 32 frames/sec). (This is coming from a scientific grade camera, via a PCI frame-grabber). I would like to do some simple processing on the video (clip dynamic range, colorize, add overlays) and then show it in a window, using C#.
I have this working using Windows Forms & GDI (for each frame, build a Bitmap object, write raw 32-bit RGB pixel values based on my post-processing steps, and then draw the frame using the Graphics class). But this uses a significant chunk of CPU that I'd like to use for other things. So I'm interested in using WPF for its GPU-accelerated video display. (I'd also like to start using WPF for its data binding & layout features.)
But I've never used WPF before, so I'm unsure how to approach this. Most of what I find online about video & WPF involves reading a compressed video file from disk (e.g. WMV), or getting a stream from a consumer-grade camera using a driver layer that Windows already understands. So it doesn't seem to apply here (but correct me if I'm wrong about this).
So, my questions:
Is there a straighforward, WPF-based way to play video from raw, uncompressed bytes in memory (even if just as 8-bit grayscale, or 24-bit RGB)?
Will I need to build DirectShow filters (or other DirectShow/Media Foundation-ish things) to get the post-processing working on the GPU?
Also, any general advice / suggestions for documentation, examples, blogs, etc that are appropriate to these tasks would be appreciated. Thanks!
Follow-up: After some experimentation, I found WriteableBitmap to be fast enough for my needs, and extremely easy to use correctly: Simply call WritePixels() and any Image controls bound to it will update themselves. InteropBitmap with memory-mapped sections is noticeably faster, but I had to write p/invokes to kernel32.dll to use it on .NET 3.5.
My VideoRendererElement, though very efficient, does use some hackery to make it work. You may also want to experiment with the WriteableBitmap in .NET 3.5 SP1.
Also the InteropBitmap is very fast too. Much more efficient than the WB as it's not double buffered. Though it can be subject to video tearing.
Some further Google-searching yielded this:
http://www.codeplex.com/VideoRendererElement
which I'm looking into now, but may be the right approach here. Of course further thoughts/suggestions are still very much welcome.