We're building an app which requires really fast video streaming.
We've never done nothing like it so during the research we thought to ask the pros :P
Which codecs support fast encoding/decoding for real time video streaming (<150ms) with around 30fps and low bandwidth?
Edit:
If you can list some apis for either java,c# or c++ it would be nice
Question is: what's the task? You want to encode video on the fly?
Most codecs with compression (or even all) require complete video stream for analysis and encoding. Any video stream sent without any compression (RGB) will be to much too handle with low bandwidth. That's why any pseudo real-time digital sources send picture in some format like mpeg-2 or other and provide 20 fps (and only pretty good hardware gives 25-30 fps).
For your task the best way is to receive encoded media stream (web camera, some video file) sent over with some restrictions and preliminary checks made, and then schedule its conversion to some other standard format.
This way you have a binary stream stored first-hand and then processed normally.
Related
I've built a Windows Phone class to convert some WriteableBitmap into a AVI Full Frame (Uncompressed). Videos are really huge. Is there a simple codec implementation existing, like a codec that just zip images, or something that is making a next/previous XOR then zip it in jpeg?
Windows Phone doesn't allow any unsafe code and most DLL cannot be wrapped into a C# WP library. So that's why I'm coding something from scratch. Note that I'm more efficient at coding from scratch than in studying C++ existing sources (I'm not a C++ coder), so what I'm searching is infos about a compressed AVI format that can be achieved without writing 100000 lines. I've used AVI because the specs are simple.
[EDIT]
I've found something very interesting here on codeproject, from a 2004 article. It's a 100% C# source to convert frames to mpeg-1. Sadly that's i frame, and not p frame, so files are 3 times larger than an expected mpeg-1 average file size.
[EDIT]
To describe more my project, what I'll do is to apply some effects on a captured movie. This movie will then be uploaded on Youtube or some other websites. Thus, the user expect the exact resolution used on the phone, at least 25 frames/s, a decent quality, and a short upload time. So I can't stop with a Mpeg-1 I-Frames. I'll need to study about prediction in mpeg-1.
I take this to be a continuation/re-post of your previous question (but with a few more details). As I mentioned in the comments of that post, there is a whole universe of video codecs out there. One reason for the proliferation is that a lot of people like to re-invent wheels. However, a more salient reason is that there are a lot of different use cases for video.
You seem to be asking for a lot, yet there are a lot of variables you have not presented:
You express a need for a video encoder that will run purely in software on a Windows Phone device, which is necessarily a fairly low-powered machine; do you need it to run in real-time? I.e., do you expect a frame of video to be compressed almost immediately after to send in the uncompressed frame (within a few milliseconds)? Or can you let the device think about the compression for awhile?
How large are the video frames? Are you doing screen capture on a WP device, i.e., computer-generated data? Or are you reading raw frames from the camera and hoping to compress those?
Following from the previous point, what type of video data? Computer-generated data will look better with a certain class of codecs. Photo-quality images (from camera) implies a different family of codecs.
What bitrate are you aiming for? If you have 1 second of video, what's the max amount of bytes it should occupy (or so you hope)?
Who is the eventual consumer of the video? In the last post, you indicated you wanted to upload to YouTube. If that's the case, you're in luck, since YouTube -- backed by FFmpeg -- handles nearly every codec in the universe, so you would have a lot of options.
I don't know much about Windows Phone programming. However, any WP device is going to technically have hardware video encoding capabilities. I've done some cursory Googling to determine if you get any access to that at the application programming level but I can't find any evidence that it's possible (and this SO answer states that the functionality is not there).
I hope to impress upon you that writing a video encoder is a LOT of work (look at my username; I know from whence I speak). Generally, they require quite a lot of CPU horsepower (and, consequently, battery power, especially when implemented in pure software). However, you have already made some guesses about a codec that uses standard zlib. In fact, there are a few video codecs based on straight zlib, namely MSZH and ZLIB, collectively the Lossless Codec Libraries. That wiki page has a basic bitstream description (disclosure: I operate that wiki site). I'm confident the WP libraries include access to zlib encoding, so this might be a starting point, and YouTube should be able to digest the resulting files.
There is also a video codec that combines XOR and zlib as you guessed (Dosbox Capture Codec), but it's probably not appropriate for your application.
Do the libraries provide access to standard JPEG (i.e., can it encode JPEG files)? Another option (depending on the video type) would be successive frames of still JPEG images stuffed in the AVI file. This is known as Motion JPEG or MJPEG. However, it's roughly equivalent in bitrate to intra-only MPEG-1, which you expressed as being inadequate.
I hope I have given you some ideas and useful avenues to pursue on your path to a solution.
I am currently developing a video conference web based application. My project is based upon the following example : http://www.codeproject.com/Articles/65190/Your-First-Step-to-the-Silverlight-Voice-Video-Cha ..I am currently experiencing a 'lag' in the video conference per se..
First of all i, what could be the cause of such lag? Could it be due to codec reasons? if not what could be the other possibilities?
As regards to the codec, the current project, takes a frame (i.e. JPG) encodes it and sends it.. What other codec are recommended for encoding a JPG frame and decoding it on the receiving end?
Thanks, Any kind of help is highly appreciated..
I have no experience with the work referenced on CodeProject. But, with that said, if the approach is indeed using frame grabs and sending jpg files, then it will be extremely inefficient, resulting in very low frame rates as you describe.
An application like this should rely on a highly efficient encoding mechanism, based on moving video, and not still images. This will create economies based on deltas in the source image, rather than digitizing a new frame from scratch every time.
Additionally. H.264 is not intended as a real-time transport, whereas H.263 is intended for precisely this type of application.
A simple Google search will reveal a wealth of references, including the topic here on stackoverflow - See this
Good luck.
I am solving a problem of transferring images from a camera in a loop from a client (a robot with camera) to a server (PC).
I am trying to come up with ideas how to maximize the transfer speed so I can get the best possible FPS (that is because I want to create a live video stream out of the transferred images). Disregarding the physical limitations of WIFI stick on the robot, what would you suggest?
So far I have decided:
to use YUV colorspace instead of RGB
to use UDP protocol instead of TCP/IP
Is there anything else I could do to get the maximum fps possible?
This might be quite a bit of work but if your client can handle the computations in real time you could use the same method that video encoders use. Send a key frame every say 5 frames and in between only send the information that changed not the whole frame. I don't know the details of how this is done, but try Googling p-frames or video compression.
Compress the difference between successive images. Add some checksum. Provide some way for the receiver to request full image data for the case where things get out of synch.
There are probably a host of protocols doing that already.
So, search for live video stream protocols.
Cheers & hth.,
I am looking to create an application that will allow me to record from my mic and playback the recording through other pc's. At this point however I would just like it to play back on my own computer so I can get it working.
I have been looking at NAudio for the past few hours and it seems like it may be able to help me achieve this goal.
I am just wondering if anyone else has had any experience with this and if it is at all possible?
Thanks,
Stuart
There is an example project on codeproject doing this:
http://www.codeproject.com/KB/cs/Streaming_wave_audio.aspx
I don't know how low the latency is.
As a codec I'd recommend Speex(at least for speech). It's free, open source and offers low latency and low bandwidth.
Bass Audio Library is another solid option worth looking into.
It is possible to do, but you are unlikely to get low latency with WaveIn/WaveOut (possibly better results with WASAPI). You could use the BufferedWaveProvider (in the latest source code) to store up the audio being recorded from the microphone and supplying the output to soundcard.
NAudio is great as a starting point for audio capture and playback but as Mark pointed out the latency might be an issue.
If you take the next step and want to sent the audio data across the network you will need a codec to compress the data as PCM or WAV are uncompressed and for voice you only need a small part of the bandwidth needed for WAV.
As you are working with C# there is a C# port of Speex available, called NSpeex, which might be worth having a look at..
I'm looking to develop a Silverlight application which will take a stream of data (not an audio stream as such) from a web server.
The data stream would then be manipulated to give audio of a certain format (G.711 a-Law for example) which would then be converted into PCM so that additional effects can be applied (such as boosting the volume).
I'm OK up to this point. I've got my data, converted the G.711 into PCM but my problem is being able to output this PCM audio to the sound card.
I basing a solution on some C# code intended for a .Net application but in Silverlight there is a problem with trying to take a copy of a delegate (function pointer) which will be the topic of a separate question once I've produced a simple code sample.
So, the question is... How can I output the PCM audio that I have held in a data structure (currently an array) in my Silverlight to the user? (Please don't say write the byte values to a text box)
If it were a MP3 or WMA file I would play it using a MediaElement but I don't want to have to make it into a file as this would put a crimp on applying dynamic effects to the audio.
I've seen a few posts from people saying low level audio support is poor/non-existant in Silverlight so I'm open to any suggestions/ideas people may have.
The simple answer is that there is no support for PCM playback from Silverlight in version 2. So unless you want to write a fully managed PCM to MP3 converter you are stuck. Even then I'm not sure you could get the MediaElement to play from isolated storage.
Is there any chance you could use a web service to perform the conversion?
See also this question:
Where's the sound API in Silverlight? Or, how do I write a music app to run in the browser?
Update: Silverlight 3 supports your custom audio sources. However, it won't let you intercept samples to perform effects on WMA or MP3, presumably for DRM reasons, so you would still potentially need to write your own decoder.
Short answer is use a MediaElement + a MediaStreamSource
Check out these:
http://blogs.msdn.com/gillesk/archive/2009/03/23/playing-back-wave-files-in-silverlight.aspx
http://code.msdn.microsoft.com/wavmss/Release/ProjectReleases.aspx?ReleaseId=2417
Basically, write a decoder in managed code to convert G.711 a-Law to PCM, then do whatever modifications you want to the raw values, then pass those into a MediaStreamSource.
Looks like Silverlight 3 supports direct PCM output now, or will when released. I don't see anything in the docs about the raw AV pipeline yet.
Mark Heath's answer is correct - only certain formats are supported - mp3 and certain flavours of WMA (unfortunately not WMA lossless which would be 'closer' to PCM).
To play PCM data in Silverlight, you could do the following:
* Convert the PCM into mp3 data, and store it in memory.
* Play the mp3 data using the technique presented at ManagedMediaHelpers. The idea here involves a class called Mp3MediaStreamSource (derived from System.Windows.Media.MediaStreamSource) that provides mp3 chunks to a MediaElement to play. The chunks will need to be in a stream, but of course a memory stream will do.
I initially thought you might be able to provide PCM chunks via MediaStreamSource, but this does not work. It's a real shame as it would solve your problem (and the one I was facing - making a Speex audio file player) really easily!