Intercept video frames in a WebRTC session for effects processing? - c#

I have a C#/C++ app that captures a video stream from the camera connected to a user's PC. It then does user segmentation using the Intel RealSense SDK to automatically remove the background from the user. What I need to know is how to
insert myself into the video delivery chain so I get each frame, process it, and send it on to the WebRTC module.
The whole desired effect is to make the user look like they are superimposed over the web page. Note, the only browser I need to support is Chrome since I am running the Chromium DLLs in an embedded browser, thanks to the CefSharp project.
The one piece I can't figure out is to put myself into the video so I can get notified when a new video frame is available, modify it, and then pass it on to WebRTC in Chromium. I downloaded the Chromium source and can't find the keyword getUserMedia anywhere.

Since you are using a port of Chromium, this means that you should have access to WebGL calls through WebKit.
Instead of doing it on the app-side, try doing it on your HTML.
Since the question is "Intercept video frames in a WebRTC session for effects processing?" I think this post and the demo it has, which is on github, does exactly what you are asking for:
Using WebGL to apply effects to WebRTC video frames
And since WebGL shaders are basically written in C++ (GLSL), you could easily create a fragment/pixel shader which removes the background. Odds are that there's probably already one out there, it'd be worth trying to google for that one.
Also, in case performance concerns you, doing it this way would be just as fast since shaders work off of the GPU; even in mobile.

Related

Create a livestream by XSplit Encoder (RTMP Server)

I'm making a program for do a livestream of the XSplit Encoder (RTMP Server). I need to have a site to watch that stream, and this program, the player need to have a button to the spectator can choose your video quality, and the stream have to be fluid and have a good quality. Can someone explain this or send me a link to do that? Please
(C#)
This is a VERY large undertaking. And an impossible question to answer unless you narrow the scope. You need a ingest server that takes in RTMP, You need a machine with enough CPU power to do all the transcodes. You need site to playback on. You also need enough bandwidth (CDN) for all your viewers. How many viewers do you need to support? What platforms do you want to play back on? iOS? then you need HLS. Web? then you need RTMP. Or you can use DASH if your ok with limiting it to modern browsers. Do you want it to play in firefox? Then it MUST be flash, because firefox does not have native support for the h.264 codec. But flash wont play in iOS. You can use JWPlaver premium, that will play HLS in flash. Actually is h.264 the codec you intend to use? Have you looked into services such as Zencoder live transcoding? Or Wowza with the transcoding module? Amazon offers preconfigured wowza instances. What is your budget for this project? Why not just use twitch?
Edit: You can probably string something together using ffmpeg:
https://trac.ffmpeg.org/wiki/StreamingGuide

Video calling in desktop application .net

I am working on a wpf application which is suppose to have video calling feature. So I am going for Skype to provide me with the solution.
I am using Skype4COM library to do it. I got all the friends from Skype in my application, I can make call to my friends and even can do video call, but I cant control streaming. Video is being shown in Skype window, I want to show it inside my application like a part of my application.
I don't even know which control should I use to handle the video streaming.
If there is another better way or free way to add video calling in a WPF application. I am ready to change my preferences.
I had previous plans to use Web browser control and go for WebRTC but that does not support IE and believe there is no web browser control from Mozilla and Chrome.
Videoconferencing is not trivial to implement. You have to first look at a signalling protocol like SIP or XMPP which would provide infrastructure to maintain a list of 'friends' and whether they are online or offline. The signalling infrastructure would also let you place a call and alert you when you receive a call. And then there is the question of actually flowing video/audio between two endpoints. There is no single solution that you can plug into your app.
You can look at Google's Libjingle or ConferenceXP for a starting point.
To establish a video conference you would need the following information:
Whom can you call? This is your friends list. You would need a mechanism to add friends to your list and ensure that you can only add friends that are willing to communicate with you
How do you establish a communication channel with your friends? For example what is their IP Address, can they support a particular video codec and so on.
Once you have established a way to communicate with your friend, then there is the question of receiving audio and video information and displaying it with correct timing.
For points 1 and 2 you would need what is called a signalling and presence protocol. SIP and XMPP are two very popular open protocol.
For point [3] you would look at a protocol like RTP.
You can google SIP, XMPP and RTP. You would receive a wide variety of literature. Look at the RFC documents for exact information they they can be a bit arcane.
Libjingle is a library written in C++ implements XMPP and RTP
ConferenceXP is a RTP implementation in C# with some basic signalling but I think you can get a start with the examples without really getting into details.
You may or maynot be able to control that screen. If skype decides to share that level of control with you.
But it looks like skype4com library is on a dead end. It will stop working soon. :(
http://gigaom.com/2013/07/13/skype-says-it-will-kill-desktop-api-by-end-of-2013/
http://aragonresearch.com/microsoft-kills-skype-desktop-apis-leaves-developers-scrambling/
I'd recommend you not to invest time on skype4com, but look to other solutions to get the video chat working.
Please go through the following thread, as this discussess few alternative ways for video chatting.
Developing a Video Chat Application with high quality video streaming
Another promising thing is the Bittorent Live. But you will have to wait. It claims
BitTorrent Live is a powerful new web-based live streaming technology.
Designed to eliminate barriers to broadcast, Live is an entirely new
protocol, designed to deliver high quality video to large audiences -
with significant reduction in infrastructure cost and network delays.
http://live.bittorrent.com/
Its not open yet, let see :)

Make entire screen monochrome and other full screen effects

Does anyone know how to apply effects to the entire screen, in c# or any programming language.
I'm mostly interested in making the screen monochrome (specifically green-white instead of black white).
I know a cross-graphic card solution is possible because I found a program that can do it:
http://www.freedomscientific.com/products/lv/magic-bl-product-page.asp
Anyone knows how to accomplish something this or where to look?
Thanks !!
There is no easy Windows API to modify the entire screen contents. But this could be done at the device driver level.
Otherwise you have to resort to some Windows API tricks: place a "fake" window over the entire desktop, in a loop: grab the entire screen contents without grabbing fake window contents, do your processing to get the monochrome effect, then display that on the fake window. Yes, it's hacky and slow, but possible. Even more hacky, when you get mouse clicks to "go through" the fake window (lots of SetWindowsRgn calls).
So cross-platform here means using GDI, though some older DirectDraw APIs might work, in that case, you have a much easier time with hardware overlays (and better performance). Though I'm not sure how many cards actually support hardware overlays, and if newer versions of windows support the older DirectDraw APIs.
One more possibility is if the video card has a C# or C++ or C API, then you can do whatever you want with the card without writing device driver code.
Then there's CUDA, but I haven't yet tried that out. I know it's for stream processing on nVidia boards, but I wonder if it could get you an easy backdoor into the video display stuff.
To help people in the future who are interested in this:
This is possible with the Magnification API's color effect method. This allows one to use a matrix that can be applied to the whole screen.
NegativeScreen is an open source project that implements the feature you are describing in C#.
Unfortunately, this only works with affine transformations, as the API takes only an augmented matrices rather than a delegate or something.

Simulating Screen capturing as a webcam?

Do we have a way to simulate a webcam driver, that will provide realtime captured screen (30 frames per sec) as it's output?
This is one of several features of ManyCam (free). It is a virtual webcam driver through which you can stream your real webcam video (with optional real-time video effects), video or image files, or your full/partial desktop.
Yes, just google video2webcam. It works quite well and will loop a video or picture as output.
The driver's job is to provide a level of abstraction between the software and hardware. The driver is supposed to issue commands to the hardware. It's not responsible for taking pictures and turning it into an animated GIF for instance. It's going to do low level stuff like, turn the device on and off, send raw data to a socket.
That being said, if you need to create a virtual device driver. Here's an overview of VDD's. Windows Programming/Device Driver Introduction
Generally these are not written in higher-level languages such as C#. Rather, they are written in languages such as C/C++. You will need the KMDF, or Kernel-Mode Driver Framework.
If you just need to access a webcam from a .NET application on a system with a webcam, you just need an API.
Open your browser.. go to google.com and type ".NET webcam API"
You will see something like this:
Webcam in your own application
It appears that this is a wrapper for the DirectShow class.

Using webcam's button in my own program

My webcam has a button, with which you take a photo of current frame
when using it's original software what came with it when I bought it.
Is there any way to use that button to take photos in my program.
I would be using C#.
I was thinking of using Windows Image Acquisition (WIA) or avicap32
for using webcam in my software.
Any help would be appreciated. Where do I start?
EDIT:
I see that you didn't understand me. I need to know if my phisycal camera button is pressed.
I am using DigiMicro 1.3 camera (Manufacturer is DNT), which isn't really a webcam but a USB microscope, but
it behaves like a webcam.
Does the camera actually require that the software be running when you press the button? I know some webcams implement this feature at the driver level. If that's the case, I don't think there's much you can do without interacting directly with the USB port, which would be incredibly difficult and likely not worth the trouble. I can't find any technical documentation on the vendor's website with that kind of information.
The "MicroCapture" software doesn't come with any DLLs that have exported functions, so P/Invoking doesn't appear to be an option. Unfortunately, I can't get the driver installed, so I can't check if it comes with any P/Invoke-able DLLs.
Sorry, but since there's no standard interface for that kind of functionality and the vendor doesn't provide any specific information on it, I don't think this is possible. I would suggest contacting DNT directly about it. They may have some component that will do what you need.
This Coding4Fun post guides you through pretty much everything you need to do to get started using WIA, including adding the reference to a project, displaying the select a device dialog, and of course actually taking the picture :P.
Coding4Fun - Look at me! Windows Image Acquisition
I suggest that you take a look at the DirectShow.NET library. It has great functionality for capturing images/videos off connected devices. It also comes with several samples to get you started quickly.

Categories