How to change MediaCapture to Byte[] - c#

How to change MediaCapture to byte[] in Windows Store App for Windows 8.1.
From lib:
Windows.Media.Capture.MediaCapture asd = new
Windows.Media.Capture.MediaCapture();
Thans!

I assume you want to get a byte array from what the camera is seeing at the moment, although it's hard to interpret from your question.
There is a sample on the Microsoft github page that is relevant, although they target Windows 10. You may be interested in migrating your project to get this functionality.
GetPreviewFrame: This sample will capture preview frames as opposed to full-blown photos, but it should be a good starting point. Once it has a preview frame, it can edit the pixels on it.
Here is the relevant part:
private async Task GetPreviewFrameAsSoftwareBitmapAsync()
{
// Get information about the preview
var previewProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
// Create the video frame to request a SoftwareBitmap preview frame
var videoFrame = new VideoFrame(BitmapPixelFormat.Bgra8, (int)previewProperties.Width, (int)previewProperties.Height);
// Capture the preview frame
using (var currentFrame = await _mediaCapture.GetPreviewFrameAsync(videoFrame))
{
// Collect the resulting frame
SoftwareBitmap previewFrame = currentFrame.SoftwareBitmap;
// Add a simple green filter effect to the SoftwareBitmap
EditPixels(previewFrame);
}
}
private unsafe void EditPixels(SoftwareBitmap bitmap)
{
// Effect is hard-coded to operate on BGRA8 format only
if (bitmap.BitmapPixelFormat == BitmapPixelFormat.Bgra8)
{
// In BGRA8 format, each pixel is defined by 4 bytes
const int BYTES_PER_PIXEL = 4;
using (var buffer = bitmap.LockBuffer(BitmapBufferAccessMode.ReadWrite))
using (var reference = buffer.CreateReference())
{
// Get a pointer to the pixel buffer
byte* data;
uint capacity;
((IMemoryBufferByteAccess)reference).GetBuffer(out data, out capacity);
// Get information about the BitmapBuffer
var desc = buffer.GetPlaneDescription(0);
// Iterate over all pixels
for (uint row = 0; row < desc.Height; row++)
{
for (uint col = 0; col < desc.Width; col++)
{
// Index of the current pixel in the buffer (defined by the next 4 bytes, BGRA8)
var currPixel = desc.StartIndex + desc.Stride * row + BYTES_PER_PIXEL * col;
// Read the current pixel information into b,g,r channels (leave out alpha channel)
var b = data[currPixel + 0]; // Blue
var g = data[currPixel + 1]; // Green
var r = data[currPixel + 2]; // Red
// Boost the green channel, leave the other two untouched
data[currPixel + 0] = b;
data[currPixel + 1] = (byte)Math.Min(g + 80, 255);
data[currPixel + 2] = r;
}
}
}
}
}
And declare this outside your class:
[ComImport]
[Guid("5b0d3235-4dba-4d44-865e-8f1d0e4fd04d")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe interface IMemoryBufferByteAccess
{
void GetBuffer(out byte* buffer, out uint capacity);
}
Have a closer look at the sample to see how to get all the details. Or, to have a walkthrough, you can watch the camera session from the recent //build/ conference, which includes a little bit of a walkthrough through some camera samples.

Related

Convert 32bpp to 8bpp UWP

I'm trying to draw InkCanvas to an 8bpp image but when I try to do so the image convert itself to 32bpp, the lower I got was 24bpp but not 8bpp. Anyone can help me out? The image I am giving as input is an 8bpp BMP image created with paint.
Image imgToEdit;
InkCanvas inkCanvas;
file = await Windows.ApplicationModel.Package.Current.InstalledLocation.GetFileAsync(Ambiente.imgBlankFirma);
await file.CopyAsync(photoFolder, NomeFile, NameCollisionOption.ReplaceExisting);
file = await photoFolder.GetFileAsync(NomeFile);
imgToEdit = imgFirma;
inkCanvas = inkCanvasFirma;
if (inkCanvas.InkPresenter.StrokeContainer.GetStrokes().Count <= 0)
{
errore = true;
return;
}
var randomAccessStream = await file.OpenReadAsync();
CanvasDevice device = CanvasDevice.GetSharedDevice();
CanvasRenderTarget renderTarget = new CanvasRenderTarget(device, (int)inkCanvas.ActualWidth, (int)inkCanvas.ActualHeight, 96); //inkCanvas.ActualWidth inkCanvas.ActualHeight
using (var ds = renderTarget.CreateDrawingSession())
{
var image = await CanvasBitmap.LoadAsync(device, randomAccessStream);
// draw your image first
ds.DrawImage(image);
// then draw contents of your ink canvas over it
ds.DrawInk(inkCanvas.InkPresenter.StrokeContainer.GetStrokes());
}
randomAccessStream.Dispose();
// save results
using (var fileStream = await file.OpenAsync(FileAccessMode.ReadWrite))
{
await renderTarget.SaveAsync(fileStream, CanvasBitmapFileFormat.Tiff, 1f);
}
This shows how to do the conversion by hand. It requires direct pointer access to the image data. So just create a new 8bpp (i.e. Format8bppIndexed) bitmap with the correct size for the conversion target. So converting the data should look something like this:
public static unsafe void Bgr24ToMono8(byte* source, byte* target, int sourceStride, int targetStride, int width, int height)
{
for(var y = 0; y < height; y++)
{
var sourceRow = source + y * sourceStride;
var targetRow = y * targetStride;
for (int x = 0; x < width; x++)
{
var sourceIndex = (sourceRow + x * 3);
var value = (byte)(sourceIndex[0] * 0.11f + sourceIndex[1] * 0.59f + sourceIndex[2] * 0.3f);
target[targetRow + x] = value;
}
}
}
if you have 32bpp data it should just be a issue of changing x * 3 to x * 4. Note that there is some confusion regarding Bgr vs Rbg, different contexts uses different terms for the same thing.
Note that this converts the bitmap to 8bpp grayscale. If you need 8bpp color indexed this will be much more work since you would ideally need to find a optimal color-map, find the closest colors in said map, and apply dithering to avoid banding. For this I would recommend some image processing library. I do not think there is any built in functions for this, and it is way to much work to demonstrate here.

C# Drawing.Imaging dropping GIF frames if they are identical

I need to load GIF animations and convert them frame by frame to bitmaps. To do that, I am extracting my GIF file frame by frame using Drawing.Imaging library and then casting each frame to bitmap.
Everything works just fine except the times when the consecutive frames are the same, when there is no pixel difference. The library seems to be dropping such frames.
I came to that conslusion with a simple test. I have created an animation of a circle that is growing and shrinking with a pause between the moment the last circle dissapears and the new one is not yet shown. When I play the animation consisting of my extracted bitmaps that pause is not present. If I compare the GIFs of the same frame-length but different amounts of identical frames, the returned totalframescount value is different. I have also observed that webbrowsers display identical consecutive frames correctly.
public void DrawGif(Image img)
{
FrameDimension dimension = new FrameDimension(img.FrameDimensionsList[0]);
int frameCountTotal = img.GetFrameCount(dimension);
for (int framecount = 0; framecount < frameCountTotal; framecount++)
{
img.SelectActiveFrame(dimension, framecount);
Bitmap bmp = new Bitmap(img); //cast Image type to Bitmap
for (int i = 0; i < 16; i++)
{
for (int j = 0; j < 16; j++)
{
Color color = bmp.GetPixel(i, j);
DrawPixel(i, j, 0, color.R, color.G, color.B);
}
}
Does someone encountered such a problem?
As I am pretty new to C# - is there a way to modify .NET lib ?
Maybe there is a solution to my problem that I am not aware of, that does not involve changing the library?
Updated code - the result is the same
public void DrawGif(img)
{
int frameCountTotal = img.GetFrameCount(FrameDimension.Time);
for (int framecount = 0; framecount < frameCountTotal; framecount++)
{
img.SelectActiveFrame(FrameDimension.Time, framecount);
Bitmap bmp = new Bitmap(img);
for (int i = 0; i < 16; i++)
{
for (int j = 0; j < 16; j++)
{
Color color = bmp.GetPixel(i, j);
DrawPixel(i, j, 0, color.R, color.G, color.B);
}
The Image is not saving duplicate frames, so you have to take the time of each frame into account. Here is some sample code based on this book in Windows Programming on how to get all the frames and the correct duration. An example:
public class Gif
{
public static List<Frame> LoadAnimatedGif(string path)
{
//If path is not found, we should throw an IO exception
if (!File.Exists(path))
throw new IOException("File does not exist");
//Load the image
var img = Image.FromFile(path);
//Count the frames
var frameCount = img.GetFrameCount(FrameDimension.Time);
//If the image is not an animated gif, we should throw an
//argument exception
if (frameCount <= 1)
throw new ArgumentException("Image is not animated");
//List that will hold all the frames
var frames = new List<Frame>();
//Get the times stored in the gif
//PropertyTagFrameDelay ((PROPID) 0x5100) comes from gdiplusimaging.h
//More info on http://msdn.microsoft.com/en-us/library/windows/desktop/ms534416(v=vs.85).aspx
var times = img.GetPropertyItem(0x5100).Value;
//Convert the 4bit duration chunk into an int
for (int i = 0; i < frameCount; i++)
{
//convert 4 bit value to integer
var duration = BitConverter.ToInt32(times, 4*i);
//Add a new frame to our list of frames
frames.Add(
new Frame()
{
Image = new Bitmap(img),
Duration = duration
});
//Set the write frame before we save it
img.SelectActiveFrame(FrameDimension.Time, i);
}
//Dispose the image when we're done
img.Dispose();
return frames;
}
}
We need a structure to save the Bitmap and duration for each Frame
//Class to store each frame
public class Frame
{
public Bitmap Image { get; set; }
public int Duration { get; set;}
}
The code will load a Bitmap, check whether it is a multi-frame animated GIF. It then loops though all the frames to construct a list of separate Frame objects that hold the bitmap and duration of each frame. Simple use:
var frameList = Gif.LoadAnimatedGif ("a.gif");
var i = 0;
foreach(var frame in frameList)
frame.Image.Save ("frame_" + i++ + ".png");

Two image compare using win32 api c#

i have two images and i want to compare two image and want to get difference. i search google and found a link from where i copy paste the code for image comparison using win32 api.
so this is the url
http://blog.bobcravens.com/2009/04/create-a-remote-desktop-viewer-using-c-and-wcf/
here i am pasting the code.
private void button1_Click(object sender, EventArgs e)
{
Bitmap _prevBitmap = new Bitmap(#"d:\prev.jpg");
Bitmap _newBitmap = new Bitmap(#"d:\current.jpg");
Rectangle bounds = GetBoundingBoxForChanges(_prevBitmap, _newBitmap);
if (bounds == Rectangle.Empty)
{
}
Bitmap diff = new Bitmap(bounds.Width, bounds.Height);
Graphics g = Graphics.FromImage(diff);
g.DrawImage(_newBitmap, 0, 0, bounds, GraphicsUnit.Pixel);
g.Dispose();
// Set the current bitmap as the previous to prepare
// for the next screen capture.
//
diff.Save(#"d:\diff.bmp");
//return diff;
}
private Rectangle GetBoundingBoxForChanges(Bitmap _prevBitmap, Bitmap _newBitmap)
{
// The search algorithm starts by looking
// for the top and left bounds. The search
// starts in the upper-left corner and scans
// left to right and then top to bottom. It uses
// an adaptive approach on the pixels it
// searches. Another pass is looks for the
// lower and right bounds. The search starts
// in the lower-right corner and scans right
// to left and then bottom to top. Again, an
// adaptive approach on the search area is used.
//
// Note: The GetPixel member of the Bitmap class
// is too slow for this purpose. This is a good
// case of using unsafe code to access pointers
// to increase the speed.
//
// Validate the images are the same shape and type.
//
if (_prevBitmap.Width != _newBitmap.Width ||
_prevBitmap.Height != _newBitmap.Height ||
_prevBitmap.PixelFormat != _newBitmap.PixelFormat)
{
// Not the same shape...can't do the search.
//
return Rectangle.Empty;
}
// Init the search parameters.
//
int width = _newBitmap.Width;
int height = _newBitmap.Height;
int left = width;
int right = 0;
int top = height;
int bottom = 0;
BitmapData bmNewData = null;
BitmapData bmPrevData = null;
try
{
// Lock the bits into memory.
//
bmNewData = _newBitmap.LockBits(
new Rectangle(0, 0, _newBitmap.Width, _newBitmap.Height),
ImageLockMode.ReadOnly, _newBitmap.PixelFormat);
bmPrevData = _prevBitmap.LockBits(
new Rectangle(0, 0, _prevBitmap.Width, _prevBitmap.Height),
ImageLockMode.ReadOnly, _prevBitmap.PixelFormat);
// The images are ARGB (4 bytes)
//
int numBytesPerPixel = 4;
// Get the number of integers (4 bytes) in each row
// of the image.
//
int strideNew = bmNewData.Stride / numBytesPerPixel;
int stridePrev = bmPrevData.Stride / numBytesPerPixel;
// Get a pointer to the first pixel.
//
// Note: Another speed up implemented is that I don't
// need the ARGB elements. I am only trying to detect
// change. So this algorithm reads the 4 bytes as an
// integer and compares the two numbers.
//
System.IntPtr scanNew0 = bmNewData.Scan0;
System.IntPtr scanPrev0 = bmPrevData.Scan0;
// Enter the unsafe code.
//
unsafe
{
// Cast the safe pointers into unsafe pointers.
//
int* pNew = (int*)(void*)scanNew0;
int* pPrev = (int*)(void*)scanPrev0;
// First Pass - Find the left and top bounds
// of the minimum bounding rectangle. Adapt the
// number of pixels scanned from left to right so
// we only scan up to the current bound. We also
// initialize the bottom & right. This helps optimize
// the second pass.
//
// For all rows of pixels (top to bottom)
//
for (int y = 0; y < _newBitmap.Height; ++y)
{
// For pixels up to the current bound (left to right)
//
for (int x = 0; x < left; ++x)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x < left)
{
left = x;
}
if (x > right)
{
right = x;
}
if (y < top)
{
top = y;
}
if (y > bottom)
{
bottom = y;
}
}
}
// Move the pointers to the next row.
//
pNew += strideNew;
pPrev += stridePrev;
}
// If we did not find any changed pixels
// then no need to do a second pass.
//
if (left != width)
{
// Second Pass - The first pass found at
// least one different pixel and has set
// the left & top bounds. In addition, the
// right & bottom bounds have been initialized.
// Adapt the number of pixels scanned from right
// to left so we only scan up to the current bound.
// In addition, there is no need to scan past
// the top bound.
//
// Set the pointers to the first element of the
// bottom row.
//
pNew = (int*)(void*)scanNew0;
pPrev = (int*)(void*)scanPrev0;
pNew += (_newBitmap.Height - 1) * strideNew;
pPrev += (_prevBitmap.Height - 1) * stridePrev;
// For each row (bottom to top)
//
for (int y = _newBitmap.Height - 1; y > top; y--)
{
// For each column (right to left)
//
for (int x = _newBitmap.Width - 1; x > right; x--)
{
// Use pointer arithmetic to index the
// next pixel in this row.
//
if ((pNew + x)[0] != (pPrev + x)[0])
{
// Found a change.
//
if (x > right)
{
right = x;
}
if (y > bottom)
{
bottom = y;
}
}
}
// Move up one row.
//
pNew -= strideNew;
pPrev -= stridePrev;
}
}
}
}
catch (Exception ex)
{
int xxx = 0;
}
finally
{
// Unlock the bits of the image.
//
if (bmNewData != null)
{
_newBitmap.UnlockBits(bmNewData);
}
if (bmPrevData != null)
{
_prevBitmap.UnlockBits(bmPrevData);
}
}
// Validate we found a bounding box. If not
// return an empty rectangle.
//
int diffImgWidth = right - left + 1;
int diffImgHeight = bottom - top + 1;
if (diffImgHeight < 0 || diffImgWidth < 0)
{
// Nothing changed
return Rectangle.Empty;
}
// Return the bounding box.
//
return new Rectangle(left, top, diffImgWidth, diffImgHeight);
}
when GetBoundingBoxForChanges() call then i am getting error and error message is Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
error occur at this code if ((pNew + x)[0] != (pPrev + x)[0])
so i am not being able to find out the reason. how to fix this error. please guide. thanks
bmNewData = _newBitmap.LockBits(...., _newBitmap.PixelFormat);
This algorithm implicitly assumes that a pixel has 4 bytes and can be addressed with an int*. It however fails to provide that guarantee. Asking for _newBitmap.PixelFormat in LockBits() is not sufficient, that just asks for the same format that the original image used. You'll get a hard crash if the images are 24bpp for example, very common.
Explicitly ask for 32bppArgb instead.
Instead of the Win32 API you could use a managed image processing library such as AForge.NET. In the documentation look for the AForge.Imaging.Filters.Difference class. It works with Bitmap objects so you will have to make minimal changes to your program.
Bitmap overlayImage;
Bitmap sourceImage;
//ToDo: Load the two images.
// Create filter.
Difference filter = new Difference(overlayImage);
// Apply the filter and return a new bitmap that is the difference between the source and overlay images.
Bitmap resultImage = filter.Apply(sourceImage);
// If you don't want a new image the you can apply the filter directly to the source image.
filter.ApplyInPlace(sourceImage);
This is the mechanism that I use to calculate image differences in C#. Note that it need to be compiled with the unsafe directive. Hope it helps:
using System;
using System.Collections.Generic;
using System.Text;
using System.Drawing;
using System.Drawing.Imaging;
using log4net;
namespace ImageDiff
{
public class ImageDifferences
{
private static ILog mLog = LogManager.GetLogger("ImageDifferences");
public static unsafe Bitmap PixelDiff(Image a, Image b)
{
if (!a.Size.Equals(b.Size)) return null;
if (!(a is Bitmap) || !(b is Bitmap)) return null;
return PixelDiff(a as Bitmap, b as Bitmap);
}
public static unsafe Bitmap PixelDiff(Bitmap a, Bitmap b)
{
Bitmap output = new Bitmap(
Math.Max(a.Width, b.Width),
Math.Max(a.Height, b.Height),
PixelFormat.Format32bppArgb);
Rectangle recta = new Rectangle(Point.Empty, a.Size);
Rectangle rectb = new Rectangle(Point.Empty, b.Size);
Rectangle rectOutput = new Rectangle(Point.Empty, output.Size);
BitmapData aData = a.LockBits(recta, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
BitmapData bData = b.LockBits(rectb, ImageLockMode.ReadOnly, PixelFormat.Format32bppArgb);
BitmapData outputData = output.LockBits(rectOutput, ImageLockMode.ReadWrite, PixelFormat.Format32bppArgb);
try
{
byte* aPtr = (byte*)aData.Scan0;
byte* bPtr = (byte*)bData.Scan0;
byte* outputPtr = (byte*)outputData.Scan0;
int len = aData.Stride * aData.Height;
for (int i = 0; i < len; i++)
{
// For alpha use the average of both images (otherwise pixels with the same alpha won't be visible)
if ((i + 1) % 4 == 0)
*outputPtr = (byte)((*aPtr + *bPtr) / 2);
else
*outputPtr = (byte)~(*aPtr ^ *bPtr);
outputPtr++;
aPtr++;
bPtr++;
}
return output;
}
catch (Exception ex)
{
mLog.Error("Error calculating image differences: " + ex.Message);
return null;
}
finally
{
a.UnlockBits(aData);
b.UnlockBits(bData);
output.UnlockBits(outputData);
}
}
}
}

Kinect Depth FPS slows Significantly when Human Detected

I have a application where I create my own Depth Frame (using the Kinect SDK). The problem is when a human is detected the FPS of the depth (and then the color too) slows down significantly. Here is a movie of when the frame slows down. The code I am using:
using (DepthImageFrame DepthFrame = e.OpenDepthImageFrame())
{
depthFrame = DepthFrame;
pixels1 = GenerateColoredBytes(DepthFrame);
depthImage = BitmapSource.Create(
depthFrame.Width, depthFrame.Height, 96, 96, PixelFormats.Bgr32, null, pixels1,
depthFrame.Width * 4);
depth.Source = depthImage;
}
...
private byte[] GenerateColoredBytes(DepthImageFrame depthFrame2)
{
short[] rawDepthData = new short[depthFrame2.PixelDataLength];
depthFrame.CopyPixelDataTo(rawDepthData);
byte[] pixels = new byte[depthFrame2.Height * depthFrame2.Width * 4];
const int BlueIndex = 0;
const int GreenIndex = 1;
const int RedIndex = 2;
for (int depthIndex = 0, colorIndex = 0;
depthIndex < rawDepthData.Length && colorIndex < pixels.Length;
depthIndex++, colorIndex += 4)
{
int player = rawDepthData[depthIndex] & DepthImageFrame.PlayerIndexBitmask;
int depth = rawDepthData[depthIndex] >> DepthImageFrame.PlayerIndexBitmaskWidth;
byte intensity = CalculateIntensityFromDepth(depth);
pixels[colorIndex + BlueIndex] = intensity;
pixels[colorIndex + GreenIndex] = intensity;
pixels[colorIndex + RedIndex] = intensity;
if (player > 0)
{
pixels[colorIndex + BlueIndex] = Colors.Gold.B;
pixels[colorIndex + GreenIndex] = Colors.Gold.G;
pixels[colorIndex + RedIndex] = Colors.Gold.R;
}
}
return pixels;
}
FPS is quite crucial to me since I am making an app that saves pictures of people when they are detected. How can I maintain a faster FPS? Why is my application doing this?
G.Y is correct that you're not disposing properly. You should refactor your code so the DepthImageFrame is disposed of ASAP.
...
private short[] rawDepthData = new short[640*480]; // assuming your resolution is 640*480
using (DepthImageFrame depthFrame = e.OpenDepthImageFrame())
{
depthFrame.CopyPixelDataTo(rawDepthData);
}
pixels1 = GenerateColoredBytes(rawDepthData);
...
private byte[] GenerateColoredBytes(short[] rawDepthData){...}
You said that you're using the depth frame elsewhere in the application. This is bad. If you need some specific data from the depth frame, save it separately.
dowhilefor is also correct that you should look at using a WriteableBitmap, it's super simple.
private WriteableBitmap wBitmap;
//somewhere in your initialization
wBitmap = new WriteableBitmap(...);
depth.Source = wBitmap;
//Then to update the image:
wBitmap.WritePixels(...);
Also, you're creating new arrays to store pixel data again and again on every frame. You should create these arrays as global variables, create them a single time, and then just overwrite them on every frame.
Finally, although this shouldn't make a huge difference, I'm curious about your CalculateIntensityFromDepth method. If the compiler isn't inlining that method, that's a lot of extraneous method calls. Try to remove that method and just write the code where the method call is right now.

RFB/VNC Receiving FrameBuffer and Byte Data issue C#

I'm hoping someone that's had experience with the RFB protocol will give me an answer.
Following the RFB protocol, I've implemented a 3.3 client, the handshake and all is fine. What I don't understand / having issues with, is the FrameUpdateRequest and FrameUpdate using Raw data.
I've read and implemented the documentation verbatim # contents 6.4.3 and 6.5.1 from http://www.realvnc.com/docs/rfbproto.pdf
It's a bit messy as I've been playing left, right and center with it. But here's what I'm doing:
public int beginNormalOperation()
{
byte[] fbUp = new byte[10];
fbUp[0] = 0003; //Message type, 3= FrameBufferUpdate
fbUp[1] = 0001; //Incremental 0=true, 1=false
fbUp[2] = 0000; //X Position High Byte
fbUp[3] = 0000; //X Position Low Byte
fbUp[4] = 0000; //Y Position High Byte
fbUp[5] = 0000; //Y Position Low Byte
fbUp[6] = INTtoU16(serverSetWidth)[0];
fbUp[7] = INTtoU16(serverSetWidth)[1];
fbUp[8] = INTtoU16(serverSetHeight)[0];
fbUp[9] = INTtoU16(serverSetHeight)[1];
//fbUp[6] = 0000;
//fbUp[7] = 100;
//fbUp[8] = 0000;
//fbUp[9] = 100;
sock.Send(fbUp);
System.Drawing.Image img;
byte[] bufferInfo = new byte[4];
try
{
sock.Receive(bufferInfo);
textBox4.AppendText("\r\n" + Encoding.Default.GetString(bufferInfo));
}
catch (SocketException ex) { MessageBox.Show("" + ex); }
return U16toINT(bufferInfo[2], bufferInfo[3]);
}
The return value is the number of rectangles, because I'm calling this method from a button click, then passing it to:
public void drawImage(int numRectangles)
{
//Now within the class
//int xPos = 0;
//int yPos = 0;
//int fWidth = 0;
//int fHeight = 0;
if (myBmp == null)
{
myBmp = new Bitmap(serverSetWidth, serverSetHeight); //I'm requesting full size, so using server bounds atm
}
for (int i = 0; i < numRectangles; i++)
{
byte[] bufferData = new byte[12];
int headerLen = 0;
if (!gotRectangleHeader)
{
try
{
sock.Receive(bufferData);
}
catch (SocketException ex) { MessageBox.Show("" + ex); }
xPos = U16toINT(bufferData[0], bufferData[1]);
yPos = U16toINT(bufferData[2], bufferData[3]);
fWidth = U16toINT(bufferData[4], bufferData[5]);
fHeight = U16toINT(bufferData[6], bufferData[7]);
//headerLen = 12; //I'm now reading the first 12 bytes first so no need for this
gotRectangleHeader = true;
}
bufferData = new byte[((fWidth * fHeight)*4)];
try
{
sock.Receive(bufferData);
}
catch (SocketException ex) { MessageBox.Show("" + ex); }
//Testing to see where the actual data is ending
//byte[] end = new byte[1000];
//Array.Copy(bufferData, 16125, end, 0, 1000);
//for(int f=0; f<bufferData.Length;f++)
//{
// if (Convert.ToInt32(bufferData[f].ToString()) == 0 &&
// Convert.ToInt32(bufferData[f + 1].ToString()) == 0 &&
// Convert.ToInt32(bufferData[f + 2].ToString()) == 0 &&
// Convert.ToInt32(bufferData[f + 3].ToString()) == 0)
// {
// Array.Copy(bufferData, f-30, end, 0, 500);
// int o = 1;
// }
//}
int curRow = 0;
int curCol = 0;
for (int curBit = 0; curBit < (bufferData.Length - headerLen) / 4; curBit++)
{
int caret = (curBit * 4) + headerLen;
if (curRow == 200)
{
int ss = 4;
}
Color pixCol = System.Drawing.Color.FromArgb(Convert.ToInt32(bufferData[caret+3].ToString()), Convert.ToInt32(bufferData[caret+2].ToString()), Convert.ToInt32(bufferData[caret+1].ToString()), Convert.ToInt32(bufferData[caret].ToString()));
myBmp.SetPixel(curCol, curRow, pixCol);
if (curCol == (fWidth - 1))
{
curRow++;
curCol = 0;
}
else
{
curCol++;
}
}
}
imgForm.Show();
imgForm.updateImg(myBmp);
}
I'm sorry for the code, I've gone through so many permutations messing about it's become a mess.
This is what I'm trying to do and the way I imagine that it should work according to the protocol:
I request a FrameBufferUpdateRequest, incremental is false (1, according to the Doc's), X and Y position set to 0 and width & height both U16 set to 1366 x 768 respectively.
I receive a FrameBufferUpdate with Number of Rectangles
I call drawImage passing Number of Rectangles in.
I assume from the docs, for each rectangle then create a buffer to that rectangles height and width. And set the pixels on a BMP to the rectangles bounds.
The first rectangle always has a header, and within that header the requested width and height. The following rectangle doesn't have any header information. So i'm missing something here. I'm guessing I haven't received all of the first rectangles data even though I have set the sockets buffer size to width*height*bytes.
Sometimes I get say the top 200 pixels or so and full width though a quarter of the right hand screen is shown on the left hand side in my BMP. Sometimes I've had the full screen and that's what I want but mostly I get a slither say 10px of the top of the screen then nothing.
I'm doing something wrong, I know I am. But what??? The documentation isn't great. If someone could hold my hand through the FrameBufferUpdateRequest -> FrameBufferUpdate -> Display Raw Pixel Data!!
Thanks for any input
craig
I suggest you refer to http://tigervnc.org/cgi-bin/rfbproto which I've found to be a much better reference on the protocol. Specifically the sections on framebufferupdaterequest and framebufferupdate
Some other notes:
you have your incremental values swapped. 1 is incremental and 0 is a full request. With incremental set to 1 you are only requesting rectangles that have changed since the last round.
don't assume that framebuffer updates comes synchronously right after you send a framebufferupdate. You really need to read the first byte first to determine what sort of message the server has sent and process the message accordingly.
you should really create your bitmap based on that actual size of each rectangle (i.e. after you read the size of the rectangle you are processing.
you need to take into account the encoding format (you're currently ignoring it). That will determine how large the image data for the rectangle is.
Also, I'm not familiar with how C# Socket.Receive works. Unless it is always guaranteed to block until the buffer is filled, you might need to check how much data was actually read since the servers don't always send the whole framebufferupdate message all at once and even if they do, the messages might get fragmented and not arrive all at once.

Categories