This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
bitmaps to avi file c# .Net
I am bit struck with an idea to convert the sequence image files into a single video file. I am using dotnet as a platform.How should i proceed. No clear idea...
And more to that need to add audio(mp3) speech while the image sequenceare displayed...
The general idea here is you want to pass your raw images through an encoder and encode the file that way. The encoder will take care of generating all your keyframes and intermediary (P and B) frames as well as generating any necessary decoding metadata that needs to be stored. On top of that running it through an encoding tool such as ffmpeg will also take care of saving the video file in a known container format and properly structuring your video headers. All of this is complicated and tedious to do by hand, not to mention error prone.
Whether you use ffmpeg or some other encoder it's up to you. I suggest using ffmpeg because it has the necessary functionality you need. If you want to do this all in code, ffmpeg is open source and you can wrap the pieces you need in a .net shell and call things that way. Though, keep in mind ffmpeg's licenses if you are developing a distributable application.
This should get you started: Making movies from image files using ffmpeg/mencoder
To add audio check this: https://stackoverflow.com/questions/1329333/how-can-i-add-audio-mp3-to-a-flv-just-video-with-ffmpeg
Now if you want to synchronize the audio and video (lets say the image sequence is people talking and the audio is their speech) you have a much more difficult problem on your hands. At this point you need to properly multiplex audio and video frames based on their durations. FFMpeg probably won't do that well since it will set each image in your video sequence to play at the same duration, which doesn't usually correlate properly with audio frames.
Related
I get a stream of frames, an initial SPS and PPS h264 data packet and then packets for the I and P frames.
Using c# .NET I want to convert into a series of JPEGs? Has anyone done this?
I have tried AForge.NET FFMpeg wrapper, but can only go from MP4 file to JPEGs?
Also looked at DirectShow.
I can't seem to find an examples that even come close to doing this?
Thanks
In Directshow, it sounds like you need to introduce a SampleGrabber filter into your filter graph. Insert this after the H264 decoder.
This is a pass through filter which can receive a callback containing each video frame. You can then obviously choose what you do with the frame, is save it to disk as a jpeg etc.
Rather than regurgitate MSDN, there is a great page explaining its usage here:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx
Update taking account of Roman's comments:
SampleGrabber is deprecated as its part of Directshow Editing Services. However, it is quite self contained. If it was removed from a later version of Windows it would be straight forward to replace with an alternative filter. I still use it in one of my consumer applications. Roman is correct though - it has a steep learning curve.
I've built a Windows Phone class to convert some WriteableBitmap into a AVI Full Frame (Uncompressed). Videos are really huge. Is there a simple codec implementation existing, like a codec that just zip images, or something that is making a next/previous XOR then zip it in jpeg?
Windows Phone doesn't allow any unsafe code and most DLL cannot be wrapped into a C# WP library. So that's why I'm coding something from scratch. Note that I'm more efficient at coding from scratch than in studying C++ existing sources (I'm not a C++ coder), so what I'm searching is infos about a compressed AVI format that can be achieved without writing 100000 lines. I've used AVI because the specs are simple.
[EDIT]
I've found something very interesting here on codeproject, from a 2004 article. It's a 100% C# source to convert frames to mpeg-1. Sadly that's i frame, and not p frame, so files are 3 times larger than an expected mpeg-1 average file size.
[EDIT]
To describe more my project, what I'll do is to apply some effects on a captured movie. This movie will then be uploaded on Youtube or some other websites. Thus, the user expect the exact resolution used on the phone, at least 25 frames/s, a decent quality, and a short upload time. So I can't stop with a Mpeg-1 I-Frames. I'll need to study about prediction in mpeg-1.
I take this to be a continuation/re-post of your previous question (but with a few more details). As I mentioned in the comments of that post, there is a whole universe of video codecs out there. One reason for the proliferation is that a lot of people like to re-invent wheels. However, a more salient reason is that there are a lot of different use cases for video.
You seem to be asking for a lot, yet there are a lot of variables you have not presented:
You express a need for a video encoder that will run purely in software on a Windows Phone device, which is necessarily a fairly low-powered machine; do you need it to run in real-time? I.e., do you expect a frame of video to be compressed almost immediately after to send in the uncompressed frame (within a few milliseconds)? Or can you let the device think about the compression for awhile?
How large are the video frames? Are you doing screen capture on a WP device, i.e., computer-generated data? Or are you reading raw frames from the camera and hoping to compress those?
Following from the previous point, what type of video data? Computer-generated data will look better with a certain class of codecs. Photo-quality images (from camera) implies a different family of codecs.
What bitrate are you aiming for? If you have 1 second of video, what's the max amount of bytes it should occupy (or so you hope)?
Who is the eventual consumer of the video? In the last post, you indicated you wanted to upload to YouTube. If that's the case, you're in luck, since YouTube -- backed by FFmpeg -- handles nearly every codec in the universe, so you would have a lot of options.
I don't know much about Windows Phone programming. However, any WP device is going to technically have hardware video encoding capabilities. I've done some cursory Googling to determine if you get any access to that at the application programming level but I can't find any evidence that it's possible (and this SO answer states that the functionality is not there).
I hope to impress upon you that writing a video encoder is a LOT of work (look at my username; I know from whence I speak). Generally, they require quite a lot of CPU horsepower (and, consequently, battery power, especially when implemented in pure software). However, you have already made some guesses about a codec that uses standard zlib. In fact, there are a few video codecs based on straight zlib, namely MSZH and ZLIB, collectively the Lossless Codec Libraries. That wiki page has a basic bitstream description (disclosure: I operate that wiki site). I'm confident the WP libraries include access to zlib encoding, so this might be a starting point, and YouTube should be able to digest the resulting files.
There is also a video codec that combines XOR and zlib as you guessed (Dosbox Capture Codec), but it's probably not appropriate for your application.
Do the libraries provide access to standard JPEG (i.e., can it encode JPEG files)? Another option (depending on the video type) would be successive frames of still JPEG images stuffed in the AVI file. This is known as Motion JPEG or MJPEG. However, it's roughly equivalent in bitrate to intra-only MPEG-1, which you expressed as being inadequate.
I hope I have given you some ideas and useful avenues to pursue on your path to a solution.
I am developing a windows phone 7 application and it does video recording. I would like to get the sound portion of the video file (MP4) and do some enhancements on the sound. I believe sound is saved as AAC frames in MP4. (Right?) How can I extract sound of a videa MP4 file?
Since this is a video file, it can be huge file. So uploading to cloud and processing there is not a good option. Since this WP7 application I cannot use unmaged dlls :( Is there a way to do in pure C#? Any open source tools/samples?
Thanks!
MP4 is a container format and realistically the sound portion isn't always AAC. It could be MP3 or any other number of different audio formats. You may be thinking of M4A, which I believe requires either AAC or ALAC.
On the subject of audio extraction, it should be possible to extract the audio from an MP4 using just managed code. You'll have to read up on the MP4 format (here, for example - this question is also worth reading) and then search through the file for the location of the audio and then either copy it to its own buffer or do your manipulations in chunks. Even then, you'll have to be able to recognize when it isn't an audio format that your app won't support.
It's possible that there already exists a .net library that can do all of this but I don't know of any. It's probably not very popular because managed code is definitely not the best angle to approach this from, but considering this is Windows Phone, it is, as you noted, your only avenue of approach.
Good luck!
Is there a way I can capture a single frame from a video file (mpg, wmv, flv, etc) at a specific point in the video (e.g. 5 seconds or the 25th frame)? Then save it as an image?
[Edit]
Something link YouTube does. That can't be all done manually? ;)
I'd use DirectShow.NET, because it'll let you do a lot of the work in managed code, which is quite a bit more friendly than doing it in native code.
You'll have to construct a filter graph to render the file you want, and you'll also need a file reader for the format of the file (i.e. if it's an MP4 file, you'll need an MP4 demux), and you'll need a decoder for the format of video (i.e. if it's H264, you'll need an H264 decoder filter). I'd use Windows7 if possible, it has much better media support.
Your graph should look something like:
File Reader -> Video Decoder -> Sample Grabber -> Null Renderer
You'll construct your graph, and then call IMediaSeeking to seek to the approximate time of the sample you want. Then run the graph. The decompressed frames will come in through a Sample Grabber callback interface. You can check the timestamps and get the one that's closest to what you need.
From there, you can use .NET to save it as whatever image format you like (JPEG is probably best).
FFMPEG and .net is your best choice
I'm looking to develop a Silverlight application which will take a stream of data (not an audio stream as such) from a web server.
The data stream would then be manipulated to give audio of a certain format (G.711 a-Law for example) which would then be converted into PCM so that additional effects can be applied (such as boosting the volume).
I'm OK up to this point. I've got my data, converted the G.711 into PCM but my problem is being able to output this PCM audio to the sound card.
I basing a solution on some C# code intended for a .Net application but in Silverlight there is a problem with trying to take a copy of a delegate (function pointer) which will be the topic of a separate question once I've produced a simple code sample.
So, the question is... How can I output the PCM audio that I have held in a data structure (currently an array) in my Silverlight to the user? (Please don't say write the byte values to a text box)
If it were a MP3 or WMA file I would play it using a MediaElement but I don't want to have to make it into a file as this would put a crimp on applying dynamic effects to the audio.
I've seen a few posts from people saying low level audio support is poor/non-existant in Silverlight so I'm open to any suggestions/ideas people may have.
The simple answer is that there is no support for PCM playback from Silverlight in version 2. So unless you want to write a fully managed PCM to MP3 converter you are stuck. Even then I'm not sure you could get the MediaElement to play from isolated storage.
Is there any chance you could use a web service to perform the conversion?
See also this question:
Where's the sound API in Silverlight? Or, how do I write a music app to run in the browser?
Update: Silverlight 3 supports your custom audio sources. However, it won't let you intercept samples to perform effects on WMA or MP3, presumably for DRM reasons, so you would still potentially need to write your own decoder.
Short answer is use a MediaElement + a MediaStreamSource
Check out these:
http://blogs.msdn.com/gillesk/archive/2009/03/23/playing-back-wave-files-in-silverlight.aspx
http://code.msdn.microsoft.com/wavmss/Release/ProjectReleases.aspx?ReleaseId=2417
Basically, write a decoder in managed code to convert G.711 a-Law to PCM, then do whatever modifications you want to the raw values, then pass those into a MediaStreamSource.
Looks like Silverlight 3 supports direct PCM output now, or will when released. I don't see anything in the docs about the raw AV pipeline yet.
Mark Heath's answer is correct - only certain formats are supported - mp3 and certain flavours of WMA (unfortunately not WMA lossless which would be 'closer' to PCM).
To play PCM data in Silverlight, you could do the following:
* Convert the PCM into mp3 data, and store it in memory.
* Play the mp3 data using the technique presented at ManagedMediaHelpers. The idea here involves a class called Mp3MediaStreamSource (derived from System.Windows.Media.MediaStreamSource) that provides mp3 chunks to a MediaElement to play. The chunks will need to be in a stream, but of course a memory stream will do.
I initially thought you might be able to provide PCM chunks via MediaStreamSource, but this does not work. It's a real shame as it would solve your problem (and the one I was facing - making a Speex audio file player) really easily!