I am solving a problem of transferring images from a camera in a loop from a client (a robot with camera) to a server (PC).
I am trying to come up with ideas how to maximize the transfer speed so I can get the best possible FPS (that is because I want to create a live video stream out of the transferred images). Disregarding the physical limitations of WIFI stick on the robot, what would you suggest?
So far I have decided:
to use YUV colorspace instead of RGB
to use UDP protocol instead of TCP/IP
Is there anything else I could do to get the maximum fps possible?
This might be quite a bit of work but if your client can handle the computations in real time you could use the same method that video encoders use. Send a key frame every say 5 frames and in between only send the information that changed not the whole frame. I don't know the details of how this is done, but try Googling p-frames or video compression.
Compress the difference between successive images. Add some checksum. Provide some way for the receiver to request full image data for the case where things get out of synch.
There are probably a host of protocols doing that already.
So, search for live video stream protocols.
Cheers & hth.,
Related
I am using MagicLeap Headset and MLCamera API to capture a rawvideocapture which the output is YUV_420_888 which I am assuming is YUV420P. API returns yBuffer, uBuffer and vBuffer separately. I am having trouble combining these channels on c# without bitmap since I am using unity I am using Mono. What I am trying to do is to combine these channels and send it to my remote python server to process the image that I have captured. To process the image, it needs to be a full image. I have tried just using the Y plane and creating a gray-scale image but the server couldn't process it so I need to combine all 3 channels on the client and then compress it to preferable jpeg since the size decreases drastically and I am processing the images at 420x420 size although the camera output is 1920x1080. I am trying different methods for the last week and half but couldn't find something decent. There are a couple methods especially for Android but I don't want to convert it to NV21 if I don't have to. I have also seen one with ARCore but I also can't use that one since I am using MagicLeap.
PS: The latency and the processing time is super important so if there is a way to convert YCbCr to jpeg directly without converting it to RGB, I think it would help my case better but I don't know if it's possible. In general I think I lack some basic knowledge that prevents me from going further.
Any help is greatly appreciated!
I've tried something similar in the past, was beating my head on the YUV420 stuff for weeks, but couldn't solve it. In the end, I bought this library OpenCV for Unity. It has custom parts just for the MagicLeap, including reading frames from the Camera in reduced resolution for speed up.
I'm not sure how ever if it managed real time. Maybe in the reduced resolution, yes.
I'm working on an application C# which consist in collect images (format bayer8) from a camera and then stream it over the local network by rtsp.
I looked for a framework which allow me to send over the network bitmap images.
Every time I find something, it's a file which is streamed, but i need to stream bitmap images.
Thanks for your help
If you are proficient enough to use native code (either through COM or P/Invoke), Live555 can be used to achieve what you need. I am not aware if it has as C# wrapper (IMHO most likely no). There is also this answer that mentions a codeproject page on RTSP in C# alongside Live555.
That being said I think there is a more important bit of information in your case (which is why I did not simply mark the question as a duplicate). RTP has a predefined set of payload types (so that the clients know how to remux the stream). RFC4175 lists the uncompressed video formats for RTP. As far as I understand bayer8 is not just another name for some kind of RGB compression - it is a totally different one (correct me if I'm wrong)? In that case you just can't put your data in a stream that standard clients (VLC, FFmpeg, etc.) will be able to demux. Then it's up to you to understand if you can recompress or if you don't need RTP at all since it will simply not work.
Recently, I finished my conference application. People can talk and watch to each other. Therefore I capture images (IntPtr of buffer converted to JPEG) from the webcam (DirectShow library). Right now I do not have any problems, since the program was used in a LAN only. But I'm planning to implement a internet version of it.
So my question is: Should I use something else than JPEG? Should I compare image x and image x+1 and only send differences? Should I use Motion-JPEG? (Sorry, I do not know anything about motion-jpeg, but it sounds relevant).
you are on the right track with recognizing that images change little from frame to frame, and that sending a sequence of jpegs is not the way to go. I believe mjpeg sends a sequence of jpegs, and is a poor choice. I do not use c#, but i believe that ffmpeg (a video compression library) makes a c# wrapper.
FFmpeg is extremely fast, but is not really well documented and is pure ANSI-C. I think that a better approach in your case is, as you already thought, to compress the difference between image x and image x-1, this should be enough to provide a significant bandwidth saving.
You should also include a method to compress the whole frame every once in a while, or compress the whole image when the difference with the previous one is above a certain threshold
I am in the process of creating a TCP remote desktop broadcasting application. (Something like Team Viewer or VNC)
the server application will
1. run on a PC listening for multiple clients on one Thread
2. and on another thread it will record the desktop every second
3. and it will broadcast the desktop for each connected client.
i need to make this application possible to run on a connections with a 12KBps upload and 50KBps download DSL connection (client's and server).
so.. i have to reduce the size of the data/image i send per second.
i tried to reduce by doing the following.
I. first i send a Bitmap frame of the desktop and each other time i send only the difference of the previously sent frame.
II. the second way i tried was, each time i send a JPEG frame.
i was unsuccessful to send a JPEG frame and then each next time send the difference of the previously sent JPEG frame.
i tried using lzma compression (7zip SDK) for the when i was transmitting the difference of the Bitmap.
But i was unsuccessful to reduce the data into 12KBps. the maximum i was able to achieve was around 50KBps.
Can someone advice me an algorithm/procedure for doing this?
What you want to do is do what image compression formats do, but in a custom way (Send only the changes, not the whole image over and over). Here is what I would do, in two phases (phase 1: get it done, prove it works, phase 2: optimize)
Proof of concept phase
1) Capture an image of the screen in bitmap format
2) Section the image into blocks of contiguous bytes. You need to play around to find out what the optimal block size is; it will vary by uplink/downlink speed.
3) Get a short hash (crc32, maybe md5, experiment with this as well) for each block
4) Compress (don't forget to do this!) and transfer each changed block (If the hash changed, the block changed and needs to be transferred). Stitch the image together at the receiving end to display it.
5) Use UDP packets for data transfer.
Optimization phase
These are things you can do to optimize for speed:
1) Gather stats and hard code transfer speed vs frame size and hash method for optimal transfer speed
2) Make a self-adjusting mechanism for #1
3) Images compress better in square areas rather then contiguous blocks of bytes, as I explained in #2 of the first phase above. Change your algorithm so you are getting a visual square area rather than sequential blocks of lines. This square method is how the image and video compression people do it.
4) Play around with the compression algorithm. This will give you lots of variables to play with (CPU load vs internet access speed vs compression algorithm choice vs frequency of screen updates)
This is basically a summary of how (roughly) compressed video streaming works (you can see the similarities with your task if you think about it), so it's not an unproven concept.
HTH
EDIT: One more thing you can experiment with: After you capture a bitmap of the screen, reduce the number of colors in it. You can save half the image size if you go from 32 bit color depth to 16 bit, for example.
We're building an app which requires really fast video streaming.
We've never done nothing like it so during the research we thought to ask the pros :P
Which codecs support fast encoding/decoding for real time video streaming (<150ms) with around 30fps and low bandwidth?
Edit:
If you can list some apis for either java,c# or c++ it would be nice
Question is: what's the task? You want to encode video on the fly?
Most codecs with compression (or even all) require complete video stream for analysis and encoding. Any video stream sent without any compression (RGB) will be to much too handle with low bandwidth. That's why any pseudo real-time digital sources send picture in some format like mpeg-2 or other and provide 20 fps (and only pretty good hardware gives 25-30 fps).
For your task the best way is to receive encoded media stream (web camera, some video file) sent over with some restrictions and preliminary checks made, and then schedule its conversion to some other standard format.
This way you have a binary stream stored first-hand and then processed normally.