I am creating an application which records the desktop screen and sends it over a network. I was using TCP and it was working, however there was huge frame stutters even when doing it on the same machine. When the screen would change it would require more data to be sent, this usually caused the TCP client to take an abornal amount of time sending the data
Client
byte[] encoded = Encoder.Encode(frame); // Takes in a bitmap image and only keeps
// the part of the image that has changed
byte[] compressed = DataCompressor.Compress(encoded); // GZIP Compresses image
byte[][] slices = ByteManipulator.SliceBytes(compressed, 15000); // Divides the
// image slices that are 15000 bytes in length
foreach (byte[] slice in slices)
{
SendTo(slice, HostIPEP); // Sends to the server (The main issue)
}
The issue with the above code is that data does not receive in the correct order, due to it being UDP. How does one get around this issue to stream video like this over UDP?
Well, the stutters come from the huge amount of data to transfer, at the begging, and when something change in screen content.
I think if you use professional encoder for image(screen content), it should be much better, for example, x264/x265/av1 encoder, especially AV1 Real-Time Screen Content Coding.
For the transport, recommend to use WebRTC server, such as SRS or MediaSoup etc, please read more detail from this post.
If use C# to build the app, there is also some native binding to use WebRTC in C#.
Related
I'm making a project using c# 2013, windows forms and this project will use an IP camera to display a video for a long time using CGI Commands.
I know from the articles I've read that the return of the streaming video of the IP camera is a continuous multi-part stream. and I found some samples to display the video like this one Writing an IP Camera Viewer in C# 5.0
but I see a lot of code to extract the single part that represents a single image and displays it and so on.
Also I tried to take continuous snap shots from the camera using the following code.
HttpWebRequest req=(HttpWebRequest)WebRequest.Create("http://192.168.1.200/snap1080");
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
Stream strm = res.GetResponseStream();
image.Image = Image.FromStream(strm);
and I repeated this code in a loop that remains for a second and counts the no. of snapshots that were taken in a second and it gives me a number between 88 and 114 snapshots per second
IMHO the first example that displays the video makes a lot of processing to extract the single part of the multi-part response and displays it which may be as slow as the other method of taking a continuous snapshots.
So I ask for other developers' experiences in this issue if they see other difference between the 2 methods of displaying the video. Also I want to know the effect of receiving a continuous multi-part stream on the memory is it safe or will generate an out of memory errors.
Thanks in advance
If you are taking more than 1 jpeg per 1-3 seconds, better capture H264 video stream, it will take less bandwidth and cpu.
Usually mjpeg stream is 10-20 times bigger than the same h264 stream. So 80 snapshots per second is a really big amount.
As long as you dispose of the image and stream correctly, you should not have memory issues. I have done a similar thing in the past with an IP Camera, even converting all the images that I take as a snapshot back into a video using ffmpeg (I think it was).
Currently developping an application where I must send a picture every X seconds to my server and the server will upload it to my FTP. To not make the picture heavy, its format is JPEG and my last image was 135Ko, which is 135000 bytes.
Usually I send packets of max 8192 bytes, but I need this picture sending mechanism in my application, so I'm here to ask you guys what would be the best way to send those 135000 bytes to my server? A fast way too.
All at once?
Slice it, 8192 bytes a piece ?
Other method that I miss?
EDIT : I use TCP
Thanks for your time.
A TCP Packet Size can go up to 64K (65535 bytes) so:
You will have Three options (yes):
Assuming you already convert your image to bytes[], send each image via two packets and then combine on the server side. You will need to watch out for their order.
Resize your image so you reach the same packet size or less than it.
Search for a library that does it for you (split an image down to several packets and combines it back together)
I am using directshow sample grabber in order to take pictures with rate of 25 fps from a web cam. Using pic resolution of 640x480. Pic size is around the 25500 bytes after converting it to jpeg. I am sending the frame using the rtp protocol. Also sending voice encoded with g711 with rtp protocol on different port. I am struggling a delay issue with the video from time to time. Maybe the jpeg size is too big? Do I need some how to compress the to mjpeg before sending?
When I recieve the frame on the client side, I am showing it in a picturebox. Changing the pictrue in the picturebox in small period of time give us the illusion of video.
Is this the right way?
https://net7mma.codeplex.com/ has an implementation of this, you would use the RtspServer and just put the new images in a directory and use the class RFC2435Stream which does this for you by monitoring that directory.
I am trying to get image from a stream (MemoryStream to be more precise). I can not find anything from Microsoft that can solve my problem.
I am getting my streams from SQL so if there is some way to get an image from there, it will be OK.
I have checked ffmpeg and the problem is that I need to save the video files. The files can reach up to 2GB and if there is a way not writing to the disk it will be helpful. If there is a way to read only the first 10MB or other limited size and read the image from it, that can also be a solution.
Video feed might be as simple as raw uncompressed video frames side by side to more complex multiplexed file format compatible chunk of data, e.g. .MP4 file. While the former case might be pretty simple, the latter requires you to demultiplex the file, seek within the stream, start decoding, possibly skip a few frames, then grab the frame of interest. The point is that it might be not as simple as it seems.
Video processing APIs in Windows are DirectShow, Media Foundation. With DirectShow it is possible to create a custom data source on top of SQL backed data stream and stream from there fetching DB data on demand, using API interfaces components (stock and third party) to do the rest of the task.
It is possible to capture frames with free VideoConverter for .NET that actually is a wrapper to FFMpeg tool. The idea is using live streaming capabilities (to C# Stream) of VideoConverter for special FFMpeg format "rawvideo" that actually is bitmap stream that can be processed by C# program, something like that:
var videoConv = new FFMpegConverter();
var ffMpegTask = videoConv.ConvertLiveMedia(
"input.mp4",
null, // autodetect live stream format
rawBmpOutputStream, // this is your special stream that will capture bitmaps
"rawvideo",
new ConvertSettings() {
VideoFrameSize = "320x200", // lets resize to exact frame size
CustomOutputArgs = " -pix_fmt bgr24 ", // windows bitmap pixel format
VideoFrameRate = 5, // lets consume 5 frames per second
MaxDuration = 5 // lets consume live stream for first 5 seconds
});
VideoConverter can read live streams from another .NET Stream (if input format can be used with live stream conversion).
I am solving a problem of transferring images from a camera in a loop from a client (a robot with camera) to a server (PC).
I am trying to come up with ideas how to maximize the transfer speed so I can get the best possible FPS (that is because I want to create a live video stream out of the transferred images). Disregarding the physical limitations of WIFI stick on the robot, what would you suggest?
So far I have decided:
to use YUV colorspace instead of RGB
to use UDP protocol instead of TCP/IP
Is there anything else I could do to get the maximum fps possible?
This might be quite a bit of work but if your client can handle the computations in real time you could use the same method that video encoders use. Send a key frame every say 5 frames and in between only send the information that changed not the whole frame. I don't know the details of how this is done, but try Googling p-frames or video compression.
Compress the difference between successive images. Add some checksum. Provide some way for the receiver to request full image data for the case where things get out of synch.
There are probably a host of protocols doing that already.
So, search for live video stream protocols.
Cheers & hth.,