We Keep Coding

Convert IMFMediaBuffer data having data type of YUY2 to RGB24 or RGB32

I am reading a frames from my web cam using MediaFoundation APIS.
IMFMediaType mediatype = null;
Hresult hr= mSourceReaderAsync.GetNativeMediaType((int)MF_SOURCE_READER.FirstAudioStream, i, out mediatype);
returns only YUY2 media types.So i am getting the output of ReadSample gives YUY2 frame. I need to convert YUY2 to RGB24 or BitmapSource to show in WPF window.
This is my OnRead callback method
public HResult OnReadSample(HResult hrStatus, int dwStreamIndex, MF_SOURCE_READER_FLAG dwStreamFlags, long llTimestamp, IMFSample pSample)
{
HResult hr = hrStatus;
IMFMediaBuffer pBuffer = null;
Stream s = null;
JpegBitmapDecoder jpgdecoder = null;
BitmapSource cameraframe = null;
lock (this)
{
try
{
if (Succeeded(hr))
{
if (pSample != null)
{
// Get the video frame buffer from the sample.
hr = pSample.GetBufferByIndex(0, out pBuffer);
}
}
if (pBuffer != null)
{
int maxlen, curlen;
pBuffer.GetMaxLength(out maxlen);
pBuffer.GetCurrentLength(out curlen);
var arr = new byte[maxlen - 1];
pBuffer.Lock(out IntPtr ptr, out int maxLen, out int curLen);
if (arr == null)
arr = new byte[maxlen - 1];
var writable = (maxlen > 0) ? true : false;
if (s == null)
s = new MemoryStream(arr, writable);
System.Runtime.InteropServices.Marshal.Copy(ptr, arr, 0, curlen);
s.Flush();
s.Seek(0, SeekOrigin.Begin);
if (jpgdecoder == null)
jpgdecoder = new JpegBitmapDecoder(s, BitmapCreateOptions.None, BitmapCacheOption.OnLoad);
var frame = jpgdecoder.Frames[0];
cameraframe = frame;
}
dispatcher.Invoke(() =>
{
OnCapture.Invoke(this, cameraframe);
});
// Request the next frame.
if (Succeeded(hr))
{
// Ask for the first sample.
}
}
catch (Exception ex)
{
Console.WriteLine(ex.ToString());
}
finally
{
SafeRelease(pBuffer);
SafeRelease(pSample);
dispatcher.Invoke(() =>
{
hr = mSourceReaderAsync.ReadSample((int)MF_SOURCE_READER.FirstVideoStream, 0, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero);
});
}
}
return hr;
}
now it raises exception that {"No imaging component suitable to complete this operation was found."}

I think you can use Media foundation transform to convert between types.You can refer samples from MF.net by snarfle
enter link description here

You have several options for converting frames.
On your CPU. I do not recommend this for obvious reasons, but if the image preview is not your main feature, you can get away with this. Extract the buffer like you are doing and convert the data array manually, preferably with array operations.
On hardware. I have two options that come to mind. The first is OpenGL. Assign your pixel buffer to a buffer in OpenGL and with a shader, you can convert the pixels from yuy2 to RGB. To do so, use the yuy2 spec.
The other option on hardware is using MFTs. I do not really know why you do not want to do this; it is fast. I see that you are implementing this in C#, which is quite the challenge, but I have done this before, proving it is possible. I can add a small code sample if you prefer this, but I suggest figuring this out with the docs. The key is to configure your transforms, then start the transform (set the pipeline to take in frames), and then pipe buffers from the samples you read into the transform, after which you extract the result from the transform.
I would recommend MFTs in your scenario. Maybe try to set up DirectX or OpenGL for displaying; your solution with BitmapSources can be slow for higher resolutions.

How Do I Pass a CImg Image Object from C++/CLI to C#

I have some C++/CLI code that creates a simple CImg image and draws a circle on it. I want to pass it to C#, but I'm not sure how. I thought of using a byte array to pass it to C#, but I can't get the length of the array, which is needed for any conversion from byte* to byte[] or for passing into the unmanaged memory stream. I've tried using strlen, but that just returns 0.
Here is my C++ code:
unsigned char* calculateFrame::ReadImage() {
CImg<unsigned char> testImage(1920, 1080, 1, 3, 0);
const unsigned char white[3] = { 255,255,255 };
testImage.draw_circle(256, 256, 200, white, 1.0f, ~0U);
testImage.draw_point(500, 500, white, 255);
unsigned char* charArray = (unsigned char*)testImage;
return charArray;
}
C# code:
Bitmap testBmp;
using(var test = new FrameCalculator.calculateFrame())
{
Console.WriteLine(test.calculateOneFrame(3));
unsafe
{
byte* imageArray = test.ReadImage();
using(var ms = new UnmanagedMemoryStream(imageArray , /* length of byte* (unknown) */))
{
testBmp = new Bitmap(ms);
}
}
}
If you have any tricks to get around unsafe code without sacrificing performance, that would be nice, but I'm not opposed to using unsafe code if it's necessary.

I ended up deciding that in the future, I would need a frame buffer, which necessitated writing the frames to the disk, so that they weren't lost in a restart.
Anyways, my solution was to write the image to disk as a .BMP and access it using Image.FromFile in C#. This isn't a great way to do this in most cases, because it adds a lot of overhead, but it made sense in my program.

How to get a byte** from managed byte[] buffer

I've been using FFmpeg.AutoGen https://github.com/Ruslan-B/FFmpeg.AutoGen wrapper to decode my H264 video for sometime with great success and now have to add AAC audio decoding (previous I was using G711 and NAudio for this).
I have the AAC stream decoding using avcodec_decode_audio4, however the output buffer or frame is in floating point format FLT and I need it to be in S16. For this I have found unmanaged examples using swr_convert and FFmpeg.AutoGen does have this function P/Invoked as;
[DllImport(SWRESAMPLE_LIBRARY, EntryPoint="swr_convert", CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
public static extern int swr_convert(SwrContext* s, byte** #out, int out_count, byte** #in, int in_count);
My trouble is that I can't find a successful way of converting/fixing/casting my managed byte[] in to a byte** to provide this as the destination buffer.
Has anyone doing this before?
My non-working code...
packet.ResetBuffer(m_avFrame->linesize[0]*2);
fixed (byte* pData = packet.Payload)
{
byte** src = &m_avFrame->data_0;
//byte** dst = *pData;
IntPtr d = new IntPtr(pData);
FFmpegInvoke.swr_convert(m_pConvertContext, (byte**)d.ToPointer(), packet.Length, src, (int)m_avFrame->linesize[0]);
}
Thanks for any help.
Cheers
Dave

The function you are trying to call is documented here: http://www.ffmpeg.org/doxygen/2.0/swresample_8c.html#a81af226d8969df314222218c56396f6a
The out_arg parameter is declare like this:
uint8_t* out_arg[SWR_CH_MAX]
That is an length SWR_CH_MAX array of byte arrays. Your translation renders that as byte** and so forces you to use unsafe code. Personally I think I would avoid that. I would declare the parameter like this:
[MarshalAs(UnmanagedType.LPArray)]
IntPtr[] out_arg
Declare the array like this:
IntPtr[] out_arg = new IntPtr[channelCount];
I am guessing that the CH in SWR_CH_MAX is short-hand for channel.
Then you need to allocate memory for the output buffer. I'm not sure how you want to do that. You could allocate one byte array per channel and pin those arrays to get hold of a pointer to pass down to the native code. That would be my preferred approach because upon return you'd have your channels in nice managed arrays. Another way would be a call to Marshal.AllocHGlobal.
The input buffer would need to be handled in the same way.
I would not use the automated pinvoke translation that you are currently using. It seems he'll bent on forcing you to use pointers and unsafe code. Not massively helpful. I'd translate it by hand.
I'm sorry not to give more specific details but it's a little hard because your question did not contain any information about the types used in your code samples. I hope the general advice is useful.

Thanks to #david-heffernan answer I've managed to get the following working and I'm posting as an answer as examples of managed use of FFmpeg are very rare.
fixed (byte* pData = packet.Payload)
{
IntPtr[] in_buffs = new IntPtr[2];
in_buffs[0] = new IntPtr(m_avFrame->data_0);
in_buffs[1] = new IntPtr(m_avFrame->data_1);
IntPtr[] out_buffs = new IntPtr[1];
out_buffs[0] = new IntPtr(pData);
FFmpegInvoke.swr_convert(m_pConvertContext, out_buffs, m_avFrame->nb_samples, in_buffs, m_avFrame->nb_samples);
}
In in the complete context of decoding a buffer of AAC audio...
protected override void DecodePacket(MediaPacket packet)
{
int frameFinished = 0;
AVPacket avPacket = new AVPacket();
FFmpegInvoke.av_init_packet(ref avPacket);
byte[] payload = packet.Payload;
fixed (byte* pData = payload)
{
avPacket.data = pData;
avPacket.size = packet.Length;
if (packet.KeyFrame)
{
avPacket.flags |= FFmpegInvoke.AV_PKT_FLAG_KEY;
}
int in_len = packet.Length;
int count = FFmpegInvoke.avcodec_decode_audio4(CodecContext, m_avFrame, out frameFinished, &avPacket);
if (count != packet.Length)
{
}
if (count < 0)
{
throw new Exception("Can't decode frame!");
}
}
FFmpegInvoke.av_free_packet(ref avPacket);
if (frameFinished > 0)
{
if (!mConverstionContextInitialised)
{
InitialiseConverstionContext();
}
packet.ResetBuffer(m_avFrame->nb_samples*4); // need to find a better way of getting the out buff size
fixed (byte* pData = packet.Payload)
{
IntPtr[] in_buffs = new IntPtr[2];
in_buffs[0] = new IntPtr(m_avFrame->data_0);
in_buffs[1] = new IntPtr(m_avFrame->data_1);
IntPtr[] out_buffs = new IntPtr[1];
out_buffs[0] = new IntPtr(pData);
FFmpegInvoke.swr_convert(m_pConvertContext, out_buffs, m_avFrame->nb_samples, in_buffs, m_avFrame->nb_samples);
}
packet.Type = PacketType.Decoded;
if (mFlushRequest)
{
//mRenderQueue.Clear();
packet.Flush = true;
mFlushRequest = false;
}
mFirstFrame = true;
}
}

c++ Get raw pixel data from hbitmap

I am fairly new to using p/invoke calls and am wondering if someone can guide me on how to retrieve the raw pixel data (unsigned char*) from an hbitmap.
This is my scenario:
I am loading a .NET Bitmap object on the C# side and sending it's IntPtr to my unmanaged c++ method. Once I receive the hbitmap ptr on the C++ side, I would like to access the Bitmaps' pixel data. I already made a method that accepts an unsigned char* which represents the raw pixel data from c# however I found extracting the byte[] from the c# is fairly slow. This is why I want to send in the Bitmap ptr instead of converting the Bitmap into a byte[] and sending that to my C++ method.
C# code for getting Bitmap IntPtr
Bitmap srcBitmap = new Bitmap(m_testImage);
IntPtr hbitmap = srcBitmap.GetHbitmap();
C# code for importing c++ method
[SuppressUnmanagedCodeSecurityAttribute()]
[DllImport("MyDll.dll", CharSet = CharSet.Unicode, CallingConvention = CallingConvention.Cdecl)]
public static extern int ResizeImage(IntPtr srcImg);
C++ method that will receive the Hbitmap handler
int Resize::ResizeImage(unsigned char* srcImg){
//access srcImgs raw pixel data (preferably in unsigned char* format)
//do work with that
return status;
}
Questions:
1) Since I am sending in an IntPrt, can my C++ method parameter be an unsigned char* ?
2) If not, how can I access the bitmap's raw data from c++?

The GetHbitmap method does not retrieve pixel data. It yields a GDI bitmap handle, of type HBITMAP. Your unmanaged code would receive that as a parameter of type HBITMAP. You can obtain the pixel data from that using GDI calls. But it is not, in itself, the raw pixels.
In fact, I'm pretty sure you are attacking this problem the wrong way. You are probably heading this way because GetPixel and SetPixel are slow. This quite true. Indeed, their GDI equivalents are too. What you need to do is to use LockBits. This will allow you to operate on the entire pixel data in C# in an efficient way. A good description of the subject can be found here: https://web.archive.org/web/20141229164101/http://bobpowell.net/lockingbits.aspx. Note that, for efficiency, this is one type of C# code where unsafe code and pointers is often the best solution.
If, for whatever reason, you still wish to operate on the pixel data using C++ code, then you can still use LockBits as the simplest way to get a pointer to the pixel data. It's certainly much easier than the unmanaged GDI equivalents.

First, an HBITMAP shouldn't be a unsigned char*. If you are passing an HBITMAP to C++ then the parameter should be an HBITMAP:
int Resize::ResizeImage(HBITMAP hBmp)
Next to convert from HBITMAP to pixels:
std::vector<unsigned char> ToPixels(HBITMAP BitmapHandle, int &width, int &height)
{
BITMAP Bmp = {0};
BITMAPINFO Info = {0};
std::vector<unsigned char> Pixels = std::vector<unsigned char>();
HDC DC = CreateCompatibleDC(NULL);
std::memset(&Info, 0, sizeof(BITMAPINFO)); //not necessary really..
HBITMAP OldBitmap = (HBITMAP)SelectObject(DC, BitmapHandle);
GetObject(BitmapHandle, sizeof(Bmp), &Bmp);
Info.bmiHeader.biSize = sizeof(BITMAPINFOHEADER);
Info.bmiHeader.biWidth = width = Bmp.bmWidth;
Info.bmiHeader.biHeight = height = Bmp.bmHeight;
Info.bmiHeader.biPlanes = 1;
Info.bmiHeader.biBitCount = Bmp.bmBitsPixel;
Info.bmiHeader.biCompression = BI_RGB;
Info.bmiHeader.biSizeImage = ((width * Bmp.bmBitsPixel + 31) / 32) * 4 * height;
Pixels.resize(Info.bmiHeader.biSizeImage);
GetDIBits(DC, BitmapHandle, 0, height, &Pixels[0], &Info, DIB_RGB_COLORS);
SelectObject(DC, OldBitmap);
height = std::abs(height);
DeleteDC(DC);
return Pixels;
}

Apparently sending in the Pointer from Scan0 is equivalent to what I was searching for. I am able to manipulate the data as expected by sending in an IntPtr retrieved from the bitmapData.Scan0 method.
Bitmap srcBitmap = new Bitmap(m_testImage);
Rectangle rect = new Rectangle(0, 0, srcBitmap.Width, srcBitmap.Height);
BitmapData bmpData = srcBitmap.LockBits(rect, ImageLockMode.ReadWrite, srcBitmap.PixelFormat);
//Get ptr to pixel data of image
IntPtr ptr = bmpData.Scan0;
//Call c++ method
int status = myDll.ResizeImage(ptr);
srcBitmap.UnlockBits(bmpData);
To further help clarify, the only code I changed from my initial post was the first block of code. All the rest remained the same. (C++ method still accepts unsigned char * as a param)

PhotoCaptureDevice in native code

On Windows Phone 8, I wish to take a camera shot in native code, but I'm blocked on the final stage not being able to extract information from IOutputStream.
in C# we code:
MemoryStream image = new MemoryStream();
MemoryStream imagePreview = new MemoryStream();
cameraCaptureSequence.Frames[0].CaptureStream = image.AsOutputStream();
cameraCaptureSequence.Frames[0].ThumbnailStream = imagePreview.AsOutputStream();
await cameraCaptureSequence.StartCaptureAsync();
from now image stream has information of captured image and I can render it.
In C++ / Cx I need to do the same thing but more until to catch the byte* of captured image, here my code:
Windows::Phone::Media::Capture::CameraCaptureSequence^ cameraCaptureSequence;
IBuffer^ image;
return concurrency::create_async([this]()
{
cameraCaptureSequence->Frames->GetAt(0)->CaptureStream = reinterpret_cast<IOutputStream^>(image);
create_task( m_camera->PrepareCaptureSequenceAsync(cameraCaptureSequence) ).wait();
create_task( cameraCaptureSequence->StartCaptureAsync() ).then([this]()
{
}
}
Starting from the most basic thing I wish to understand how to "save" into an IBuffer^ the result of captured image stream, better how to get the internal byte* buffer.
Thanks

You can access the pixel data from a captured image in Native code through the ICameraCaptureFrameNative. The object that implements the interface is obtained through COM. Once you have obtained the object, use MapBuffer() to access the BYTE * array.
Note that the pixel data obtained this way is in NV12 format, not a JPEG or RGB as one would expect.
#include <Windows.Phone.Media.Capture.Native.h>
CameraCaptureFrame^ frame = m_cameraCaptureSequence->Frames->GetAt(0);
pNativeFrame = NULL;
HRESULT hr = reinterpret_cast<IUnknown*>(frame)->QueryInterface(__uuidof(ICameraCaptureFrameNative ), (void**) &pNativeFrame);
create_task( m_camera->PrepareCaptureSequenceAsync(m_cameraCaptureSequence) ).wait();
create_task( m_cameraCaptureSequence->StartCaptureAsync() ).then([this]()
{
DWORD bufferSize =0;
BYTE * pBuffer = NULL;
pNativeFrame->MapBuffer(&bufferSize, &pBuffer); // Pixels are in pBuffer.
// Unmap() the buffer before capturing another image.

ICameraCaptureFrameNative doesn't give access to a texture containing the preview?
If you want acces data from a IBuffer look here : http://msdn.microsoft.com/en-us/library/windows/apps/dn182761.aspx
For your case, i thinks you need a class which implement IOutputStream. Maybe InMemoryRandomAccessStream ?