I am working on screen sharing project. I am sending only screen differences over socket comparing previous and actual buffer. It working
I am sending 8 to 9 FPS to client using Format16bppRgb555 to reduce overall bytes size of Bitmap
byte[] wholescreensize= new byte[1360 * 768 * 2];// Its around 2 Mb
My problem Is when full screen is changed.
I am getting about 45-60 kb of PNG image using below function
45kb * 10 (FPS) = 450 kb
It is possible to reduce beyond 45 kb.
I am not interested to reduce FPS as it live screen sharing app.
JPEG Compression or LZ4/GZIP also not making much difference as PNG image already compressed
private void SendImgDiffToClient(byte[] contents,Rectangle rectangle)
{
//Converting Small Portion to Bitmap.Bcoz Image.FromStrem not working here error Parameter is not Valid
byte[] byteArrayout = new byte[contents.Length];
var bitmap = new Bitmap(rectangle.Width, rectangle.Height, PixelFormat.Format16bppRgb555);
var bitmap_data = bitmap.LockBits(new Rectangle(0, 0, bitmap.Width, bitmap.Height), ImageLockMode.WriteOnly, PixelFormat.Format16bppRgb555);
Marshal.Copy(contents, 0, bitmap_data.Scan0, byteArrayout.Length);
bitmap.UnlockBits(bitmap_data);
//Converting Small Bitmap to Png Byte Array and Sending to Client
using (MemoryStream ms = new MemoryStream())
{
Image msImage = (Image)bitmap;
msImage.Save(ms, ImageFormat.Png);
msImage.Dispose();
byteArrayout = ms.ToArray();
}
SendtoClient(byteArrayout);
}
My Questing is what is a best approach to reduce bytes in such scenario.
Video streaming is essentially what you're doing; and modern video compression algorithms have lots of enhancements. Perhaps they can track or move an artifact, or otherwise distort said artifact as part of their functionality. Perhaps they can stream the data in a progressively building manner, so that static items eventually acquire more detail (similar to progressive jpeg images.) They do lots of things all at the same time. You can try to research them further, and take inspiration from them, or you could pick and use one.
This is to say that many people here seem to prefer the solution of using a readily available video compression library. Especially if you are worried about streaming bandwidth.
If you don't want to use an existing video library, then you have to decide how much effort you want to put in, versus how sloppy you want to be with consuming more bandwidth than otherwise necessary.
Related
I'm trying to send as fast as possible images over socket tried to compress... compare the images... it's still working pretty slow...
btw I tried to save the image before and after the compression and size was the same.... 1 or 2 kb les...
have a look in client side code:
Bitmap pre;
private void Form2_Load(object sender, EventArgs e)
{
pre = GetDesktopImage();
prev = Compress(ImageToByte(pre)).Length;
theThread = new Thread(new ThreadStart(startSend));
theThread.Start();
}
Bitmap curr;
byte[] compressed;
private void startSend()
{
sck = client.Client;
s = new NetworkStream(sck);
while (true)
{
curr = GetDesktopImage();
compressed = Compress(ImageToByte(curr));
if (Math.Abs(compressed.Length - prev) > 500)
{
bFormat.Serialize(s, compressed);
prev = compressed.Length;
count++;
}
}
}
compression methods:
byte[] Compress(byte[] b)
{
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream z = new GZipStream(ms, CompressionMode.Compress, true))
z.Write(b, 0, b.Length);
return ms.ToArray();
}
}
byte[] ImageToByte(Image img)
{
ImageConverter converter = new ImageConverter();
return (byte[])converter.ConvertTo(img, typeof(byte[]));
}
and this is the server side:
while (true)
{
try
{
bFormat = new BinaryFormatter();
inBytes = bFormat.Deserialize(stream) as byte[];
inImage = ByteToImage(Decompress(inBytes));
theImage.Image = (Image)inImage;
count++;
label1.Invoke(new Action(() => label1.Text = count.ToString()));
}
catch { }
}
btw I've seen some people who used socket.send and didn't save the image to stream.... may u guys explain the difference? and suggest me what wrong in my code and how can I improve my algorithm?
Your question is really pushing the limits in terms of "too broad" as a close reason. The general problem of sending image data over a network is a very broad area of research, with a large number of different techniques, the specific application/user-scenario determining which technique is actually best.
That said, there is one very obvious change you can make to you code that is needed, and which might speed it up, depending on where the bottleneck is.
Specifically, you are using ImageConverter.ConvertTo() to convert the Bitmap object to a byte[], and then you are using GzipStream to compress that array of bytes. The problem with this is that ConvertTo() is already compressing the data; the byte[] it returns contains the original bitmap represented as PNG format, which is a fairly good, lossless compression algorithm for images.
So not only does compressing it again accomplish practically nothing, it costs you a lot of CPU to do that nothing. Don't do that. Just send the byte[] data as-is, without running it through GzipStream.
Now, all that said…
As I mentioned, whether that change will really help all that much depends on other things, including how large the bitmaps are, and how fast the network you are using is. If you are already saturating the network even with the inefficient code you posted in your question, then speeding that code up isn't going to help.
Techniques that are used to deal with network bandwidth as a bottleneck include (but are not limited to):
Using lossy compression (e.g. JPEG, MPEG, etc.), and so simply discarding information that costs too much to send.
Using a differential compression technique (e.g. MPEG, MP4, Quicktime, etc.), which takes advantage of the fact that when dealing with motion picture video, most of the pixels from one frame to the next are unchanged or at least are very similar.
Sending rendering commands instead of bitmap data. This is commonly used for things like VNC or Microsoft's Remote Desktop/Terminal Server APIs, and takes advantage of the fact that on-screen drawing is very commonly affecting a large number of pixels using relatively simple drawing commands (filling/outlining rectangles, drawing text, painting small bitmaps, etc.).
In many cases, these techniques are combined in varying ways to achieve maximum performance.
If you want to use these kinds of techniques, you need to do a bit more than just asking a question on Stack Overflow. It is well beyond the scope of this site to provide broad documentation and tutorials on those techniques. You'll need to research them yourself, or even better just use existing implementations to achieve your goals.
Edit: SOLVED! Please see my answer down below for details.
I was unable to find an answer to the original question but I found an alternate solution
This question may be asked somewhere else but I have been searching for days and can't find anything that helps.
Question: I need to convert "Stream" to "image(bgr, byte)" in one go, Is there a way/command to convert directly from System.Drawing.Image.FromStream to Emgu.CV.Image(Bgr, Byte) without converting from stream to image to bitmap to image(bgr, byte)?
Information: I'm coding in c# in Visual Studio 2010 as part of my dissertation project.
I am taking a image stream from an IP camera on a network and applying many algorithms to detect faces/extract facial features and recognise an individuals face. On my laptops local camera I can achieve FPS of about 25~ (give or take) including algorithms because I don't have to convert the image. For an IP camera stream I need to convert it many times to achieve the desired format and the result is around 5-8fps.
(I know my current method is extremely inefficient which is why I'm here, I'm actually converting an image 5 times total (even gray scaling too), actually only using half of my processors memory (i7, 8gb RAM)). It does have to be image(bgr, byte) as that is the only format the algorithms will function with.
The code I'm using to get the image:
//headers
using System.IO
using System.Threading;
using System.Net;
//request a connection
req = (HttpWebRequest)HttpWebRequest.Create(cameraUrl);
//gives chance for timeout for errors to occur or loss of connection
req.AllowWriteStreamBuffering = true;
req.Timeout = 20000;
//retrieve response (if successfull)
res = req.GetResponse();
//image returned
stream = res.GetResponseStream();
I have alot of stuff in the background managing connections, data, security etc which I have shortened to the above code.
My current code to covert the image to the desired output:
//Convert stream to image then to bitmap
Bitmap bmpImage = new Bitmap(System.Drawing.Image.FromStream(stream));
//Convert to emgu image (desired goal)
currentFrame = new Emgu.CV.Image<Bgr, Byte>(bmpImage);
//gray scale for other uses
gray = currentFrame.Convert<Gray, Byte>();
I understand there is a method to save an image locally temporarily but I would need to avoid that for security purposes. I'm looking more for a direct conversion to help save processing power.
Am I overlooking something? All help is appreciated.
Thanks for reading. (I will update this if anyone requests any more details)
-Dave
You've got a couple potential bottlenecks, not the least of which is that you're probably jpeg decoding the stream into an image and then converting that into a bitmap and then into an openCV image.
One way around this is to bypass the .NET imaging entirely. This would involve trying to use libjpeg directly. There's a free port of it here in C#, and IIRC you can hook into it to get called on a per-scanline basis to fill up a buffer.
The downside is that you're decoding JPEG data in managed code which will run at least 1.5X slower than equivalent the C, although quite frankly I would expect network speed to dwarf this immensely.
OpenCV should be able to read jpeg images directly (wanna guess what they use under the hood? Survey says: libjpeg), which means that you can buffer up the entire stream and hand it to OpenCV and bypass the .NET layer entirely.
I believe I found the answer to my problem. I have dabbled using Vano Maisuradze's idea of processing in memory which improved the fps a tiny margin (not immediately noticable without testing). And also thanks to Plinths answer I have a understanding of Multi-Threading and I can optimise this as I progress as I can split the algorithms up to work in parallel.
What I think is my cause is the networking speed! not the actual algorithm delay. As pointed out by Vano with the stopwatch to find the speed the algorithms didn't actually consume that much. So with and without the algorithms the speed is about the same if I optimise using threading so the next frame is being collected as the previous one finishes processing.
I did some testing on some physical Cisco routers and got the same result if a bit slower messing round with clock speeds and bandwidths which was noticeable. So I need to find out a way to retrieve frames over networks faster, Very big thank you to everyone who answered who helped me understand better!
Conclusion:
Multi-threading to optimise
Processing in memory instead of converting constantly
Better networking solutions (Higher bandwidth and speeds)
Edit: The code to retrieve an image and process in memory for anyone who finds this looking for help
public void getFrames(object sender, EventArgs e)
{//Gets a frame from the IP cam
//Replace "IPADDRESS", "USERNAME", "PASSWORD"
//with respective data for your camera
string sourceURL = "http://IPADDRESS/snapshot.cgi?user=USERNAME&pwd=PASSWORD";
//used to store the image retrieved in memory
byte[] buffer = new byte[640 * 480];
int read, total = 0;
//Send a request to the peripheral via HTTP
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(sourceURL);
WebResponse resp = req.GetResponse();
//Get the image capture after recieving a request
//Note: just a screenshot not a steady stream
Stream stream = resp.GetResponseStream();
while ((read = stream.Read(buffer, total, 1000)) != 0)
{
total += read;
}//While End
//Convert memory (byte) to bitmap and store in a picturebox
pictureBox1.Image = (Bitmap)Bitmap.FromStream(new MemoryStream(buffer, 0, total));
}//getFrames End
private void button1_Click(object sender, EventArgs e)
{//Trigger an event to start running the function when possible
Application.Idle += new EventHandler(getFrames);
}//Button1_Click End
You can save several image in memory (buffer) and then start processing from buffer.
Something like this:
//Convert stream to image then to bitmap
Bitmap bmpImage = new Bitmap(System.Drawing.Image.FromStream(stream));
//Convert to emgu image (desired goal)
currentFrame = new Emgu.CV.Image<Bgr, Byte>(bmpImage);
//gray scale for later use
gray = currentFrame.Convert<Gray, Byte>();
SaveToBuffer(gray);
Queue<Emgu.CV.Image<Gray, Byte>> buffer = new Queue<Emgu.CV.Image<Gray, Byte>>();
bool canProcess = false;
// ...
private void SaveToBuffer(Emgu.CV.Image<Gray, Byte> img)
{
buffer.Enqueue(img);
canProcess = buffer.Count > 100;
}
private void Process()
{
if(canProcess)
{
buffer.Dequeue();
// Processing logic goes here...
}
else
{
// Buffer is still loading...
}
}
But note that you will need enough RAM to store images in memory and also you should adjust buffer size to meat your requirements.
I'm trying to render a bitmap in Memory using mono. This image should be displayed on Adafruits 2.8" touch TFT (320*240). The Programm is developed with Visual Studio 2013 Community Edition. I want to host a ASP.NET Web Api and Show
some data on the Display. The ASP.NET part is working fine and the image is rendered. My idea was to write the Image to the framebuffer Input, but doing this I get an Exception saying that file is to large. I'm just writing raw data without BMP Header. Has someone managed doing this? Maybe creation of image is
wrong.
It seems as something is happening because the display changes and I can see white areas which might be from my image.
I don't want to use any extra libraries to keep it simple. So my idea is to use FBI directly. Does anyone know this problem and the solution?
Here is some of my code:
using (Bitmap bmp = new Bitmap(240, 320, PixelFormat.Format16bppRgb555))
{
[...]
Byte[] image = null;
using(MemoryStream memoryStream = new MemoryStream())
{
bitmap.Save(memoryStream, ImageFormat.Bmp);
Byte[] imageTemp = memoryStream.GetBuffer();
//Remove BMP header
image = new Byte[imageTemp.Length - 54];
Buffer.BlockCopy(imageTemp, 54, image, 0, image.Length);
//153600 byte
using (FileStream fb1 = new FileStream("/dev/fb1", FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
{
fb1.Write(image, 0, image.Length);
fb1.Close();
}
}
}
Take a look at http://computerstruggles.blogspot.de/2013/02/how-to-program-directfb-in-c-on.html - the idea is to install the directfb library and use it from C# with PInvoke. The blog's author uses a mini wrapper in C to make using it even easier. BTW why don't you like to install additional libraries and to profit from the work others have done for you?
You may be running out of memory when the MemoryStream reallocates memory. When it needs to grow, it doubles in size. With this large of a write, the internal buffer is probably exceeding available memory. See Why does C# memory stream reserve so much memory? for more information.
In my program, I compress a BMP into a JPEG like this:
private void ConvertBmpToStreamJPG30(Bitmap b, Stream s)
{
s.Flush();
EncoderParameters encoderParameters = new EncoderParameters(1);
encoderParameters.Param[0] = new EncoderParameter(System.Drawing.Imaging.Encoder.Quality, 30L);
b.Save(s, GetEncoder(ImageFormat.Jpeg), encoderParameters);
}
Then a function is receiving the JPEG in a MemoryStream, I transform it into a Bitmap by doing
Bitmap b = new Bitmap(stream);
When I display the image, there are a lot of lines like this :
What am I doing wrong, people?
Edit 1
Here a small visual studio solution showing the problem: http://www.fast-files.com/getfile.aspx?file=79311
It is the beginning of a screen sharing software. What it does: It takes screenshots, compare them, compress the difference and send it to another part of the program that decompress it and recompose an image with everything received. It opens a window displaying what is "sent" on the left and the recomposed image on the right.
Three things come to mind:
Try setting a better quality than 30 and see if that helps;
Check your RAM (and possibly video RAM, though I doubt that GDI+ might use VGA for compression) for hardware problems;
I've had a similar weird problem where I loaded some JPEG file, modified it a bit, and then saved it again. That produced an exception. The solution was to make a new bitmap based on the old one and save the copy. Try that.
I want to build a Screen Sharing program in C#.(with TCP)
I sniffed around the web and found out that the most efficient way to do it is by sending alot of screenshots from the client to the server.
The point is - how can I compress a Bitmap to Jpeg - receive it on the server and decompress again to Bitmap (so I can show it in a form) ?
I've tried using the JpegBitmapEncoder with no luck, here's my code:
Bitmap screen = TakeScreenshot();
MemoryStream ms = new MemoryStream();
byte[] Bytes = BmpToBytes_Unsafe(screen);
ms.Write(Bytes, 0, Bytes.Length);
Jpeg = new JpegBitmapEncoder();
Jpeg.Frames.Add(BitmapFrame.Create(ms));
Jpeg.QualityLevel = 40;
Jpeg.Save(ms);
BinaryReader br = new BinaryReader(ms);
SendMessage(br.ReadBytes((int)ms.Length));
It throws an NotSupportedException at Jpeg.Frames.Add(BitmapFrame.Create(ms));
No imaging component suitable to complete this operation was found.
So I need a way to convert a Bitmap to Jpeg, then to byte[], then send it over TCP.
And on the other end, do the exact opposite. Any suggestions ?
Thank you.
JPEG was designed for photographs, not for screen captures. Also, most of the screen doesn't change so better to just send the changed portions and only a full screen when much of the screen has changed.
Unless you're just doing this for fun, you are going about this all wrong. VNC has been doing this for years and the source code is free so you could look to see how that's done.