I'm working on an application C# which consist in collect images (format bayer8) from a camera and then stream it over the local network by rtsp.
I looked for a framework which allow me to send over the network bitmap images.
Every time I find something, it's a file which is streamed, but i need to stream bitmap images.
Thanks for your help
If you are proficient enough to use native code (either through COM or P/Invoke), Live555 can be used to achieve what you need. I am not aware if it has as C# wrapper (IMHO most likely no). There is also this answer that mentions a codeproject page on RTSP in C# alongside Live555.
That being said I think there is a more important bit of information in your case (which is why I did not simply mark the question as a duplicate). RTP has a predefined set of payload types (so that the clients know how to remux the stream). RFC4175 lists the uncompressed video formats for RTP. As far as I understand bayer8 is not just another name for some kind of RGB compression - it is a totally different one (correct me if I'm wrong)? In that case you just can't put your data in a stream that standard clients (VLC, FFmpeg, etc.) will be able to demux. Then it's up to you to understand if you can recompress or if you don't need RTP at all since it will simply not work.
Related
I get a stream of frames, an initial SPS and PPS h264 data packet and then packets for the I and P frames.
Using c# .NET I want to convert into a series of JPEGs? Has anyone done this?
I have tried AForge.NET FFMpeg wrapper, but can only go from MP4 file to JPEGs?
Also looked at DirectShow.
I can't seem to find an examples that even come close to doing this?
Thanks
In Directshow, it sounds like you need to introduce a SampleGrabber filter into your filter graph. Insert this after the H264 decoder.
This is a pass through filter which can receive a callback containing each video frame. You can then obviously choose what you do with the frame, is save it to disk as a jpeg etc.
Rather than regurgitate MSDN, there is a great page explaining its usage here:
https://msdn.microsoft.com/en-us/library/windows/desktop/dd407288(v=vs.85).aspx
Update taking account of Roman's comments:
SampleGrabber is deprecated as its part of Directshow Editing Services. However, it is quite self contained. If it was removed from a later version of Windows it would be straight forward to replace with an alternative filter. I still use it in one of my consumer applications. Roman is correct though - it has a steep learning curve.
I've built a Windows Phone class to convert some WriteableBitmap into a AVI Full Frame (Uncompressed). Videos are really huge. Is there a simple codec implementation existing, like a codec that just zip images, or something that is making a next/previous XOR then zip it in jpeg?
Windows Phone doesn't allow any unsafe code and most DLL cannot be wrapped into a C# WP library. So that's why I'm coding something from scratch. Note that I'm more efficient at coding from scratch than in studying C++ existing sources (I'm not a C++ coder), so what I'm searching is infos about a compressed AVI format that can be achieved without writing 100000 lines. I've used AVI because the specs are simple.
[EDIT]
I've found something very interesting here on codeproject, from a 2004 article. It's a 100% C# source to convert frames to mpeg-1. Sadly that's i frame, and not p frame, so files are 3 times larger than an expected mpeg-1 average file size.
[EDIT]
To describe more my project, what I'll do is to apply some effects on a captured movie. This movie will then be uploaded on Youtube or some other websites. Thus, the user expect the exact resolution used on the phone, at least 25 frames/s, a decent quality, and a short upload time. So I can't stop with a Mpeg-1 I-Frames. I'll need to study about prediction in mpeg-1.
I take this to be a continuation/re-post of your previous question (but with a few more details). As I mentioned in the comments of that post, there is a whole universe of video codecs out there. One reason for the proliferation is that a lot of people like to re-invent wheels. However, a more salient reason is that there are a lot of different use cases for video.
You seem to be asking for a lot, yet there are a lot of variables you have not presented:
You express a need for a video encoder that will run purely in software on a Windows Phone device, which is necessarily a fairly low-powered machine; do you need it to run in real-time? I.e., do you expect a frame of video to be compressed almost immediately after to send in the uncompressed frame (within a few milliseconds)? Or can you let the device think about the compression for awhile?
How large are the video frames? Are you doing screen capture on a WP device, i.e., computer-generated data? Or are you reading raw frames from the camera and hoping to compress those?
Following from the previous point, what type of video data? Computer-generated data will look better with a certain class of codecs. Photo-quality images (from camera) implies a different family of codecs.
What bitrate are you aiming for? If you have 1 second of video, what's the max amount of bytes it should occupy (or so you hope)?
Who is the eventual consumer of the video? In the last post, you indicated you wanted to upload to YouTube. If that's the case, you're in luck, since YouTube -- backed by FFmpeg -- handles nearly every codec in the universe, so you would have a lot of options.
I don't know much about Windows Phone programming. However, any WP device is going to technically have hardware video encoding capabilities. I've done some cursory Googling to determine if you get any access to that at the application programming level but I can't find any evidence that it's possible (and this SO answer states that the functionality is not there).
I hope to impress upon you that writing a video encoder is a LOT of work (look at my username; I know from whence I speak). Generally, they require quite a lot of CPU horsepower (and, consequently, battery power, especially when implemented in pure software). However, you have already made some guesses about a codec that uses standard zlib. In fact, there are a few video codecs based on straight zlib, namely MSZH and ZLIB, collectively the Lossless Codec Libraries. That wiki page has a basic bitstream description (disclosure: I operate that wiki site). I'm confident the WP libraries include access to zlib encoding, so this might be a starting point, and YouTube should be able to digest the resulting files.
There is also a video codec that combines XOR and zlib as you guessed (Dosbox Capture Codec), but it's probably not appropriate for your application.
Do the libraries provide access to standard JPEG (i.e., can it encode JPEG files)? Another option (depending on the video type) would be successive frames of still JPEG images stuffed in the AVI file. This is known as Motion JPEG or MJPEG. However, it's roughly equivalent in bitrate to intra-only MPEG-1, which you expressed as being inadequate.
I hope I have given you some ideas and useful avenues to pursue on your path to a solution.
Recently, I finished my conference application. People can talk and watch to each other. Therefore I capture images (IntPtr of buffer converted to JPEG) from the webcam (DirectShow library). Right now I do not have any problems, since the program was used in a LAN only. But I'm planning to implement a internet version of it.
So my question is: Should I use something else than JPEG? Should I compare image x and image x+1 and only send differences? Should I use Motion-JPEG? (Sorry, I do not know anything about motion-jpeg, but it sounds relevant).
you are on the right track with recognizing that images change little from frame to frame, and that sending a sequence of jpegs is not the way to go. I believe mjpeg sends a sequence of jpegs, and is a poor choice. I do not use c#, but i believe that ffmpeg (a video compression library) makes a c# wrapper.
FFmpeg is extremely fast, but is not really well documented and is pure ANSI-C. I think that a better approach in your case is, as you already thought, to compress the difference between image x and image x-1, this should be enough to provide a significant bandwidth saving.
You should also include a method to compress the whole frame every once in a while, or compress the whole image when the difference with the previous one is above a certain threshold
How i can get stream from Ip Camera, Its using RTP, stream is MPEG4, i have multicast address and port,and i have ip camera's IP address and Port Number. And I cant reach via http forexample (http://ip/jpeg) And I cant reach stream with VLC Player too. forexample (rtp://ipadressofcam:port) and (rtp://multicastaddress:port)
So What is ffmpeg command of that?
I have windows OS, I only write code with C# right now.
But producer created their own ocx which used for viewing cam,that plugin can work on .net but i dont want to use it becouse it doesnt have much funcionality, I mean you cant get current picture or snapshot of cams, thats why i have to do it myself.
Are you sure the stream is not password-protected?
Try to see why your camera rejects VLC requests. And this is how to do it: Install Wireshark, start it, and put a filter for the camera address, like: ip.addr == camera_ip (xxxx.xxxx.xxxx.xxxx). Then, try to connect with VLC, and look there at the messages exchanged. If you see a DROP, UNAUTH, or something that tells you to use a passwd, introduce it. (VLC will first try to connect using an unauthenticated procedure, and if it fails, will ask for a passwd.)
Wireshark will give you clues if the failure reason is something different.
And keep in mind that if VLC can't access it, the chance to find some other way to do it is almost zero. Unless you're a video guru.
Hope it helps!
You can use VLC for such a thing ,and it's ActiveX control which is available for .NET also ,just need to install VLC Media Player and you can set it's control on VS toolbox
http://forum.videolan.org/viewtopic.php?f=32&t=54969
UPDATE
If you are ready to pay for this stuff you can use http://www.mainconcept.com/products/sdks/video.html this Company product's to advance with Decoding and Encoding ,where you can find a huge Library .
So you want to receive video stream from camera and convert individual frame into JPEG file. This sounds good and actually sounds natural: why not? there is a video feed being sent on network and we don't need much, just to pick individual frame.
The original stream is MPEG-4 (Part 2) and desired target encoding is JPEG. This breaks the task into parts of getting MPEG-4 video feed, decoding it into uncompressed images, and encoding into JPEG. Let us go through these from the last one backwards.
JPEG is a popular encoding and there are various codecs with different interface capable of compressing into JPEG. Options include GDI+ API, IJG JPEG library libjpeg, JPEG libraries and codecs for video with DirectShow and other interfaces.
MPEG-4 decoder is a complicated component, however is luckily well standardized and available in different interfaces and from several sources. In particular MPEG-4 Part 2 Video Decoder is shipped as DMO (and also through ) with Windows 7. Decoder is also available as DirectShow filter in ffdshow, Intel Media SDK.
Back to the first step, you need to reach MPEG-4 stream from network. First of all you indicated that the camera is available on multi-cast UDP address. This makes you open a socket and put into onto multi-cast group in order to start receiving RTP data. The data you would receive is not pure MPEG-4 yet, and is formatted according to RTP Payload Format for MPEG-4 Audio/Visual Streams and as you receive RTP stream of messages you will have to be prepared to receive out of order fragments, packet losses etc. You are supposed to receive what you can receive, check the RTP headers, and do your best in reconstructing MPEG-4 feed.
Eventually you need to cover these steps (not necessarily directly, you would rather use third party libraries including those mentioned in other answers) from your application and build a pipeline which stitches together receiving from network and transcoding.
So until now, I saw VLC cant open it but if we create .sdp file and play it with ffplay only a black screen appears.
vlc -vvv dshow:// --sout-keep --sout-all --sout=#rtp{dst=multicastaddress,port=portNo,sdp=file:///C:/test/my.sdp}
Of course this is not pure solution but there is little bit hope.
But obsolute solution can be like that;
There is no way to use just c# to achive what i intend. There are few c++ library components that i can use, but i can use it along with Managed c++ to write interop services and use those dlls in my c# code.Here is what i need:
1-I need an RTP Library, http://research.edm.uhasselt.be/~jori/page/index.php?n=CS.Jrtplib or live555 but live555 has certain limitations.
2-RTP Library is the transport for the MPEG4 stream i pull from my encoder. But i need ways to control the stream, those are provided producer's WSDLs.
3-After that i need a decoder; Libavcodec, I can use libavcodec to convert an Iframe to a jpeg image in .net.
I'm looking to develop a Silverlight application which will take a stream of data (not an audio stream as such) from a web server.
The data stream would then be manipulated to give audio of a certain format (G.711 a-Law for example) which would then be converted into PCM so that additional effects can be applied (such as boosting the volume).
I'm OK up to this point. I've got my data, converted the G.711 into PCM but my problem is being able to output this PCM audio to the sound card.
I basing a solution on some C# code intended for a .Net application but in Silverlight there is a problem with trying to take a copy of a delegate (function pointer) which will be the topic of a separate question once I've produced a simple code sample.
So, the question is... How can I output the PCM audio that I have held in a data structure (currently an array) in my Silverlight to the user? (Please don't say write the byte values to a text box)
If it were a MP3 or WMA file I would play it using a MediaElement but I don't want to have to make it into a file as this would put a crimp on applying dynamic effects to the audio.
I've seen a few posts from people saying low level audio support is poor/non-existant in Silverlight so I'm open to any suggestions/ideas people may have.
The simple answer is that there is no support for PCM playback from Silverlight in version 2. So unless you want to write a fully managed PCM to MP3 converter you are stuck. Even then I'm not sure you could get the MediaElement to play from isolated storage.
Is there any chance you could use a web service to perform the conversion?
See also this question:
Where's the sound API in Silverlight? Or, how do I write a music app to run in the browser?
Update: Silverlight 3 supports your custom audio sources. However, it won't let you intercept samples to perform effects on WMA or MP3, presumably for DRM reasons, so you would still potentially need to write your own decoder.
Short answer is use a MediaElement + a MediaStreamSource
Check out these:
http://blogs.msdn.com/gillesk/archive/2009/03/23/playing-back-wave-files-in-silverlight.aspx
http://code.msdn.microsoft.com/wavmss/Release/ProjectReleases.aspx?ReleaseId=2417
Basically, write a decoder in managed code to convert G.711 a-Law to PCM, then do whatever modifications you want to the raw values, then pass those into a MediaStreamSource.
Looks like Silverlight 3 supports direct PCM output now, or will when released. I don't see anything in the docs about the raw AV pipeline yet.
Mark Heath's answer is correct - only certain formats are supported - mp3 and certain flavours of WMA (unfortunately not WMA lossless which would be 'closer' to PCM).
To play PCM data in Silverlight, you could do the following:
* Convert the PCM into mp3 data, and store it in memory.
* Play the mp3 data using the technique presented at ManagedMediaHelpers. The idea here involves a class called Mp3MediaStreamSource (derived from System.Windows.Media.MediaStreamSource) that provides mp3 chunks to a MediaElement to play. The chunks will need to be in a stream, but of course a memory stream will do.
I initially thought you might be able to provide PCM chunks via MediaStreamSource, but this does not work. It's a real shame as it would solve your problem (and the one I was facing - making a Speex audio file player) really easily!