I am building a screen recording app in C# using Windows Graphics Capture API. I am using this script. I can select monitor and can record it to mp4 file. I can also select a Window and record it too. But how can we record a region with this? Ideally I need to give x,y coordinates along with width and height to record that specific region.
Here are the functions which return GraphicsCaptureItem for Window or Monitor hwnd, which can be used to record.
public static GraphicsCaptureItem CreateItemForWindow(IntPtr hwnd)
{
var factory = WindowsRuntimeMarshal.GetActivationFactory(typeof(GraphicsCaptureItem));
var interop = (IGraphicsCaptureItemInterop)factory;
var temp = typeof(GraphicsCaptureItem);
var itemPointer = interop.CreateForWindow(hwnd, GraphicsCaptureItemGuid);
var item = Marshal.GetObjectForIUnknown(itemPointer) as GraphicsCaptureItem;
Marshal.Release(itemPointer);
return item;
}
public static GraphicsCaptureItem CreateItemForMonitor(IntPtr hmon)
{
var factory = WindowsRuntimeMarshal.GetActivationFactory(typeof(GraphicsCaptureItem));
var interop = (IGraphicsCaptureItemInterop)factory;
var temp = typeof(GraphicsCaptureItem);
var itemPointer = interop.CreateForMonitor(hmon, GraphicsCaptureItemGuid);
var item = Marshal.GetObjectForIUnknown(itemPointer) as GraphicsCaptureItem;
Marshal.Release(itemPointer);
return item;
}
And this is Recording function
private async void RecordScreen(GraphicsCaptureItem item)
{
_device = Direct3D11Helpers.CreateDevice();
// Get our encoder properties
uint frameRate = 30;
uint bitrate = 3 * 1000000;
var width = (uint)item.Size.Width;
var height = (uint)item.Size.Height;
// Kick off the encoding
try
{
newFile = GetTempFile();
using (var stream = new FileStream(newFile, FileMode.CreateNew).AsRandomAccessStream())
using (_encoder = new Encoder(_device, item))
{
await _encoder.EncodeAsync(
stream,
width, height, bitrate,
frameRate);
}
}
catch (Exception ex)
{}
}
I achieved this by passing a custom region to CopySubresourceRegion in WaitForNewFrame method.
public SurfaceWithInfo WaitForNewFrame()
{
.....
using (var multithreadLock = new MultithreadLock(_multithread))
using (var sourceTexture = Direct3D11Helpers.CreateSharpDXTexture2D(_currentFrame.Surface))
{
.....
using (var copyTexture = new SharpDX.Direct3D11.Texture2D(_d3dDevice, description))
{
.....
var region = new SharpDX.Direct3D11.ResourceRegion(
_region.Left,
_region.Top,
0,
_region.Left + _region.Width,
_region.Top + _region.Height,
1
);
_d3dDevice.ImmediateContext.CopyResource(_blankTexture, copyTexture);
_d3dDevice.ImmediateContext.CopySubresourceRegion(sourceTexture, 0, region, copyTexture, 0);
result.Surface = Direct3D11Helpers.CreateDirect3DSurfaceFromSharpDXTexture(copyTexture);
}
}
....
}
Related
The following question answers how to resize a printscreen taken with SharpDX by a power of two Resizing a DXGI Resource or Texture2D in SharpDX. I'm trying to resize the printscreen by a variable amount (e.g. 80% of original size - not necessarily a power of two). Right now I found "a way to make my goal work" by resizing the bitmap generated by the printscreen. I achieve this by first converting into a WicImage:
private void button1_Click(object sender, EventArgs e)
{
Stopwatch stopWatchInstance = Stopwatch.StartNew();
//or Bitmap.save(new filestream)
var stream = File.OpenRead("c:\\test\\pc.png");
var test = DrawResizedImage(stream);
stopWatchInstance.Stop();
File.WriteAllBytes("c:\\test\\result.png", test.ToArray());
int previousCalculationTimeServer = (int)(stopWatchInstance.ElapsedMilliseconds % Int32.MaxValue);
}
MemoryStream DrawResizedImage(Stream fileName)
{
ImagingFactory wic = new WIC.ImagingFactory();
D2D.Factory d2d = new D2D.Factory();
FormatConverter image = CreateWicImage(wic, fileName);
var wicBitmap = new WIC.Bitmap(wic, image.Size.Width, image.Size.Height, WIC.PixelFormat.Format32bppPBGRA, WIC.BitmapCreateCacheOption.CacheOnDemand);
var target = new D2D.WicRenderTarget(d2d, wicBitmap, new D2D.RenderTargetProperties());
var bmpPicture = D2D.Bitmap.FromWicBitmap(target, image);
target.BeginDraw();
{
target.DrawBitmap(bmpPicture, new SharpDX.RectangleF(0, 0, target.Size.Width, target.Size.Height), 1.0f, D2D.BitmapInterpolationMode.Linear);
}
target.EndDraw();
var ms = new MemoryStream();
SaveD2DBitmap(wic, wicBitmap, ms);
return ms;
}
void SaveD2DBitmap(WIC.ImagingFactory wicFactory, WIC.Bitmap wicBitmap, Stream outputStream)
{
var encoder = new WIC.BitmapEncoder(wicFactory, WIC.ContainerFormatGuids.Png);
encoder.Initialize(outputStream);
var frame = new WIC.BitmapFrameEncode(encoder);
frame.Initialize();
frame.SetSize(wicBitmap.Size.Width, wicBitmap.Size.Height);
var pixelFormat = wicBitmap.PixelFormat;
frame.SetPixelFormat(ref pixelFormat);
frame.WriteSource(wicBitmap);
frame.Commit();
encoder.Commit();
}
WIC.FormatConverter CreateWicImage(WIC.ImagingFactory wicFactory, Stream stream)
{
var decoder = new WIC.PngBitmapDecoder(wicFactory);
var decodeStream = new WIC.WICStream(wicFactory, stream);
decoder.Initialize(decodeStream, WIC.DecodeOptions.CacheOnLoad);
var decodeFrame = decoder.GetFrame(0);
var scaler = new BitmapScaler(wicFactory);
scaler.Initialize(decodeFrame, 2000, 2000, SharpDX.WIC.BitmapInterpolationMode.Fant);
var test = (BitmapSource)scaler;
var converter = new WIC.FormatConverter(wicFactory);
converter.Initialize(test, WIC.PixelFormat.Format32bppPBGRA);
return converter;
}
Upon clicking on button, the above code resizes a bitmap (containing the printscreen) to 2000x2000. However, the above code is very slow, it takes about 200ms (not taking into account the fileread and filewrite time). I use BitmapScaler to do the resizing.
Does anyone know how to variably resize the output produced from the Resizing a DXGI Resource or Texture2D in SharpDX question, so the resizing becomes much faster? I tried to look for documentation to apply bitmapscaler directly to any of the objects in the answered code, but didn't succeed.
I've uploaded the above code can be found as a small Visual Studio Project which compiles
Here is a rewritten and commented version of your program that gets a video frame from the desktop using DXGI's Output Duplication, resizes it using any ratio using Direct2D, and saves it to a .jpeg file using WIC.
It works only in the GPU until the image is saved to a file (stream) using WIC. On my PC, I get something like 10-15 ms for the capture and resize, 30-40 ms for WIC save to file.
I've not used the D2D Scale effect I talked about in my comment because the ID2D1DeviceContext::DrawBitmap method can do resize that with various interpolation factors, without using any effect. But you can use the same code to apply Hardware accelerated effects.
Note some objects I create and dispose in button1_Click could be created in the constructor (like factories, etc.) and reused.
using System;
using System.Windows.Forms;
using System.IO;
using DXGI = SharpDX.DXGI;
using D3D11 = SharpDX.Direct3D11;
using D2D = SharpDX.Direct2D1;
using WIC = SharpDX.WIC;
using Interop = SharpDX.Mathematics.Interop;
namespace WindowsFormsApp1
{
public partial class Form1 : Form
{
private readonly D3D11.Device _device;
private readonly DXGI.OutputDuplication _outputDuplication;
public Form1()
{
InitializeComponent();
var adapterIndex = 0; // adapter index
var outputIndex = 0; // output index
using (var dxgiFactory = new DXGI.Factory1())
using (var dxgiAdapter = dxgiFactory.GetAdapter1(adapterIndex))
using (var output = dxgiAdapter.GetOutput(outputIndex))
using (var dxgiOutput = output.QueryInterface<DXGI.Output1>())
{
_device = new D3D11.Device(dxgiAdapter,
#if DEBUG
D3D11.DeviceCreationFlags.Debug |
#endif
D3D11.DeviceCreationFlags.BgraSupport); // for D2D support
_outputDuplication = dxgiOutput.DuplicateOutput(_device);
}
}
protected override void Dispose(bool disposing) // remove from Designer.cs
{
if (disposing && components != null)
{
components.Dispose();
_outputDuplication?.Dispose();
_device?.Dispose();
}
base.Dispose(disposing);
}
private void button1_Click(object sender, EventArgs e)
{
var ratio = 0.8; // resize ratio
using (var dxgiDevice = _device.QueryInterface<DXGI.Device>())
using (var d2dFactory = new D2D.Factory1())
using (var d2dDevice = new D2D.Device(d2dFactory, dxgiDevice))
{
// acquire frame
_outputDuplication.AcquireNextFrame(10000, out var _, out var frame);
using (frame)
{
// get DXGI surface/bitmap from resource
using (var frameDc = new D2D.DeviceContext(d2dDevice, D2D.DeviceContextOptions.None))
using (var frameSurface = frame.QueryInterface<DXGI.Surface>())
using (var frameBitmap = new D2D.Bitmap1(frameDc, frameSurface))
{
// create a GPU resized texture/surface/bitmap
var desc = new D3D11.Texture2DDescription
{
CpuAccessFlags = D3D11.CpuAccessFlags.None, // only GPU
BindFlags = D3D11.BindFlags.RenderTarget, // to use D2D
Format = DXGI.Format.B8G8R8A8_UNorm,
Width = (int)(frameSurface.Description.Width * ratio),
Height = (int)(frameSurface.Description.Height * ratio),
OptionFlags = D3D11.ResourceOptionFlags.None,
MipLevels = 1,
ArraySize = 1,
SampleDescription = { Count = 1, Quality = 0 },
Usage = D3D11.ResourceUsage.Default
};
using (var texture = new D3D11.Texture2D(_device, desc))
using (var textureDc = new D2D.DeviceContext(d2dDevice, D2D.DeviceContextOptions.None)) // create a D2D device context
using (var textureSurface = texture.QueryInterface<DXGI.Surface>()) // this texture is a DXGI surface
using (var textureBitmap = new D2D.Bitmap1(textureDc, textureSurface)) // we can create a GPU bitmap on a DXGI surface
{
// associate the DC with the GPU texture/surface/bitmap
textureDc.Target = textureBitmap;
// this is were we draw on the GPU texture/surface
textureDc.BeginDraw();
// this will automatically resize
textureDc.DrawBitmap(
frameBitmap,
new Interop.RawRectangleF(0, 0, desc.Width, desc.Height),
1,
D2D.InterpolationMode.HighQualityCubic, // change this for quality vs speed
null,
null);
// commit draw
textureDc.EndDraw();
// now save the file, create a WIC (jpeg) encoder
using (var file = File.OpenWrite("test.jpg"))
using (var wic = new WIC.ImagingFactory2())
using (var jpegEncoder = new WIC.BitmapEncoder(wic, WIC.ContainerFormatGuids.Jpeg))
{
jpegEncoder.Initialize(file);
using (var jpegFrame = new WIC.BitmapFrameEncode(jpegEncoder))
{
jpegFrame.Initialize();
// here we use the ImageEncoder (IWICImageEncoder)
// that can write any D2D bitmap directly
using (var imageEncoder = new WIC.ImageEncoder(wic, d2dDevice))
{
imageEncoder.WriteFrame(textureBitmap, jpegFrame, new WIC.ImageParameters(
new D2D.PixelFormat(desc.Format, D2D.AlphaMode.Premultiplied),
textureDc.DotsPerInch.Width,
textureDc.DotsPerInch.Height,
0,
0,
desc.Width,
desc.Height));
}
// commit
jpegFrame.Commit();
jpegEncoder.Commit();
}
}
}
}
}
_outputDuplication.ReleaseFrame();
}
}
}
}
I'm running the following code in LinqPad 5:
var client = new MongoClient(#"mongodb://192.168.0.108:27017");
var db = client.GetDatabase("Stfc");
var fitCollection = db.GetCollection<ModelFit>("RecentFits");
var fits = fitCollection.AsQueryable();
var captureCollection = db.GetCollection<Capture>("Captures");
var captures = captureCollection.AsQueryable();
var classificationCollection = db.GetCollection<Classification>("Classifications");
var classifications = classificationCollection.AsQueryable();
var modelsDir = new DirectoryInfo(#"\\iansdesktop\Shared\Stfc\mymodels");
var imagesDir = new DirectoryInfo(#"\\iansdesktop\Shared\Stfc\Images");
var classificationDir = new DirectoryInfo(#"C:\Users\Ian\Documents\Projects\Output\StfcBot\Classification");
var capturesById = captures.ToDictionary(x => x.Id);
var systems = classifications
.Where(x => x.Label == "system");
var count = systems.Count();
var i = 0;
var pen = new Pen(Color.FromArgb(255, 255, 0, 0));
foreach (var classification in systems)
{
var capture = capturesById[classification.CaptureId];
var img = imagesDir.File(capture.ImageName);
var srcFile = imagesDir.File(capture.ImageName);
var destFile = classificationDir.File(capture.ImageName);
while (!destFile.Exists)
{
try
{
using (var bmp = Bitmap.FromFile(srcFile.FullName))
using (var dest = new Bitmap(bmp))
{
using (var g = Graphics.FromImage(dest))
{
g.DrawEllipse(pen, capture.X - 20, capture.Y - 20, 40, 40);
}
dest.Save(destFile.FullName);
dest.Dispose();
bmp.Dispose();
}
destFile.Refresh();
destFile.Name.Dump();
}
catch (IOException ex)
{
ex.Dump();
Thread.Sleep(30_000);
}
}
++i;
if (i % 10 == 0)
{
i.ToProgressSummary(count).Dump();
}
}
Am I missing anything, or could this be a bug in LinqPad?
Turns out this is because the bitmap was being loaded from a network path, and the network was occasionally disconnecting.
The documentation states:
You must keep the stream open for the lifetime of the Bitmap.
Bitmap Constructors (See Remarks)
The OOM exception obfuscates what is going on for some reason, but the underlying stream was being closed.
The solution is to copy the file locally and operate on that local copy:
var tmpFile = new DirectoryInfo(Path.GetTempPath()).File(srcFile.Name);
while (!destFile.Exists)
{
srcFile.CopyTo(tmpFile.FullName);
try
{
using (var bmp = Bitmap.FromFile(tmpFile.FullName))
using (var dest = new Bitmap(bmp))
{
.
}
destFile.Refresh();
destFile.Name.Dump();
}
catch (IOException ex)
{
...
}
finally
{
tmpFile.Delete();
}
}
Of course if the network still disconnects an exception occurs, but at least it's a sensible and understandable error instead of OOM.
The following question answers how to resize a printscreen taken with SharpDX by a power of two Resizing a DXGI Resource or Texture2D in SharpDX. I'm trying to resize the printscreen by a variable amount (e.g. 80% of original size - not necessarily a power of two). Right now I found "a way to make my goal work" by resizing the bitmap generated by the printscreen. I achieve this by first converting into a WicImage:
private void button1_Click(object sender, EventArgs e)
{
Stopwatch stopWatchInstance = Stopwatch.StartNew();
//or Bitmap.save(new filestream)
var stream = File.OpenRead("c:\\test\\pc.png");
var test = DrawResizedImage(stream);
stopWatchInstance.Stop();
File.WriteAllBytes("c:\\test\\result.png", test.ToArray());
int previousCalculationTimeServer = (int)(stopWatchInstance.ElapsedMilliseconds % Int32.MaxValue);
}
MemoryStream DrawResizedImage(Stream fileName)
{
ImagingFactory wic = new WIC.ImagingFactory();
D2D.Factory d2d = new D2D.Factory();
FormatConverter image = CreateWicImage(wic, fileName);
var wicBitmap = new WIC.Bitmap(wic, image.Size.Width, image.Size.Height, WIC.PixelFormat.Format32bppPBGRA, WIC.BitmapCreateCacheOption.CacheOnDemand);
var target = new D2D.WicRenderTarget(d2d, wicBitmap, new D2D.RenderTargetProperties());
var bmpPicture = D2D.Bitmap.FromWicBitmap(target, image);
target.BeginDraw();
{
target.DrawBitmap(bmpPicture, new SharpDX.RectangleF(0, 0, target.Size.Width, target.Size.Height), 1.0f, D2D.BitmapInterpolationMode.Linear);
}
target.EndDraw();
var ms = new MemoryStream();
SaveD2DBitmap(wic, wicBitmap, ms);
return ms;
}
void SaveD2DBitmap(WIC.ImagingFactory wicFactory, WIC.Bitmap wicBitmap, Stream outputStream)
{
var encoder = new WIC.BitmapEncoder(wicFactory, WIC.ContainerFormatGuids.Png);
encoder.Initialize(outputStream);
var frame = new WIC.BitmapFrameEncode(encoder);
frame.Initialize();
frame.SetSize(wicBitmap.Size.Width, wicBitmap.Size.Height);
var pixelFormat = wicBitmap.PixelFormat;
frame.SetPixelFormat(ref pixelFormat);
frame.WriteSource(wicBitmap);
frame.Commit();
encoder.Commit();
}
WIC.FormatConverter CreateWicImage(WIC.ImagingFactory wicFactory, Stream stream)
{
var decoder = new WIC.PngBitmapDecoder(wicFactory);
var decodeStream = new WIC.WICStream(wicFactory, stream);
decoder.Initialize(decodeStream, WIC.DecodeOptions.CacheOnLoad);
var decodeFrame = decoder.GetFrame(0);
var scaler = new BitmapScaler(wicFactory);
scaler.Initialize(decodeFrame, 2000, 2000, SharpDX.WIC.BitmapInterpolationMode.Fant);
var test = (BitmapSource)scaler;
var converter = new WIC.FormatConverter(wicFactory);
converter.Initialize(test, WIC.PixelFormat.Format32bppPBGRA);
return converter;
}
Upon clicking on button, the above code resizes a bitmap (containing the printscreen) to 2000x2000. However, the above code is very slow, it takes about 200ms (not taking into account the fileread and filewrite time). I use BitmapScaler to do the resizing.
Does anyone know how to variably resize the output produced from the Resizing a DXGI Resource or Texture2D in SharpDX question, so the resizing becomes much faster? I tried to look for documentation to apply bitmapscaler directly to any of the objects in the answered code, but didn't succeed.
I've uploaded the above code can be found as a small Visual Studio Project which compiles
Here is a rewritten and commented version of your program that gets a video frame from the desktop using DXGI's Output Duplication, resizes it using any ratio using Direct2D, and saves it to a .jpeg file using WIC.
It works only in the GPU until the image is saved to a file (stream) using WIC. On my PC, I get something like 10-15 ms for the capture and resize, 30-40 ms for WIC save to file.
I've not used the D2D Scale effect I talked about in my comment because the ID2D1DeviceContext::DrawBitmap method can do resize that with various interpolation factors, without using any effect. But you can use the same code to apply Hardware accelerated effects.
Note some objects I create and dispose in button1_Click could be created in the constructor (like factories, etc.) and reused.
using System;
using System.Windows.Forms;
using System.IO;
using DXGI = SharpDX.DXGI;
using D3D11 = SharpDX.Direct3D11;
using D2D = SharpDX.Direct2D1;
using WIC = SharpDX.WIC;
using Interop = SharpDX.Mathematics.Interop;
namespace WindowsFormsApp1
{
public partial class Form1 : Form
{
private readonly D3D11.Device _device;
private readonly DXGI.OutputDuplication _outputDuplication;
public Form1()
{
InitializeComponent();
var adapterIndex = 0; // adapter index
var outputIndex = 0; // output index
using (var dxgiFactory = new DXGI.Factory1())
using (var dxgiAdapter = dxgiFactory.GetAdapter1(adapterIndex))
using (var output = dxgiAdapter.GetOutput(outputIndex))
using (var dxgiOutput = output.QueryInterface<DXGI.Output1>())
{
_device = new D3D11.Device(dxgiAdapter,
#if DEBUG
D3D11.DeviceCreationFlags.Debug |
#endif
D3D11.DeviceCreationFlags.BgraSupport); // for D2D support
_outputDuplication = dxgiOutput.DuplicateOutput(_device);
}
}
protected override void Dispose(bool disposing) // remove from Designer.cs
{
if (disposing && components != null)
{
components.Dispose();
_outputDuplication?.Dispose();
_device?.Dispose();
}
base.Dispose(disposing);
}
private void button1_Click(object sender, EventArgs e)
{
var ratio = 0.8; // resize ratio
using (var dxgiDevice = _device.QueryInterface<DXGI.Device>())
using (var d2dFactory = new D2D.Factory1())
using (var d2dDevice = new D2D.Device(d2dFactory, dxgiDevice))
{
// acquire frame
_outputDuplication.AcquireNextFrame(10000, out var _, out var frame);
using (frame)
{
// get DXGI surface/bitmap from resource
using (var frameDc = new D2D.DeviceContext(d2dDevice, D2D.DeviceContextOptions.None))
using (var frameSurface = frame.QueryInterface<DXGI.Surface>())
using (var frameBitmap = new D2D.Bitmap1(frameDc, frameSurface))
{
// create a GPU resized texture/surface/bitmap
var desc = new D3D11.Texture2DDescription
{
CpuAccessFlags = D3D11.CpuAccessFlags.None, // only GPU
BindFlags = D3D11.BindFlags.RenderTarget, // to use D2D
Format = DXGI.Format.B8G8R8A8_UNorm,
Width = (int)(frameSurface.Description.Width * ratio),
Height = (int)(frameSurface.Description.Height * ratio),
OptionFlags = D3D11.ResourceOptionFlags.None,
MipLevels = 1,
ArraySize = 1,
SampleDescription = { Count = 1, Quality = 0 },
Usage = D3D11.ResourceUsage.Default
};
using (var texture = new D3D11.Texture2D(_device, desc))
using (var textureDc = new D2D.DeviceContext(d2dDevice, D2D.DeviceContextOptions.None)) // create a D2D device context
using (var textureSurface = texture.QueryInterface<DXGI.Surface>()) // this texture is a DXGI surface
using (var textureBitmap = new D2D.Bitmap1(textureDc, textureSurface)) // we can create a GPU bitmap on a DXGI surface
{
// associate the DC with the GPU texture/surface/bitmap
textureDc.Target = textureBitmap;
// this is were we draw on the GPU texture/surface
textureDc.BeginDraw();
// this will automatically resize
textureDc.DrawBitmap(
frameBitmap,
new Interop.RawRectangleF(0, 0, desc.Width, desc.Height),
1,
D2D.InterpolationMode.HighQualityCubic, // change this for quality vs speed
null,
null);
// commit draw
textureDc.EndDraw();
// now save the file, create a WIC (jpeg) encoder
using (var file = File.OpenWrite("test.jpg"))
using (var wic = new WIC.ImagingFactory2())
using (var jpegEncoder = new WIC.BitmapEncoder(wic, WIC.ContainerFormatGuids.Jpeg))
{
jpegEncoder.Initialize(file);
using (var jpegFrame = new WIC.BitmapFrameEncode(jpegEncoder))
{
jpegFrame.Initialize();
// here we use the ImageEncoder (IWICImageEncoder)
// that can write any D2D bitmap directly
using (var imageEncoder = new WIC.ImageEncoder(wic, d2dDevice))
{
imageEncoder.WriteFrame(textureBitmap, jpegFrame, new WIC.ImageParameters(
new D2D.PixelFormat(desc.Format, D2D.AlphaMode.Premultiplied),
textureDc.DotsPerInch.Width,
textureDc.DotsPerInch.Height,
0,
0,
desc.Width,
desc.Height));
}
// commit
jpegFrame.Commit();
jpegEncoder.Commit();
}
}
}
}
}
_outputDuplication.ReleaseFrame();
}
}
}
}
I'm trying to find a working sample to record videos with IOS (using xamarin) but there's always something missing or not working for me.
My best try using several forum posts and samples is the following :
using System;
using CoreGraphics;
using Foundation;
using UIKit;
using AVFoundation;
using CoreVideo;
using CoreMedia;
using CoreFoundation;
using System.IO;
using AssetsLibrary;
namespace avcaptureframes {
public partial class AppDelegate : UIApplicationDelegate {
public static UIImageView ImageView;
UIViewController vc;
AVCaptureSession session;
OutputRecorder outputRecorder;
DispatchQueue queue;
public override bool FinishedLaunching (UIApplication application, NSDictionary launchOptions)
{
ImageView = new UIImageView (new CGRect (10f, 10f, 200f, 200f));
ImageView.ContentMode = UIViewContentMode.Top;
vc = new UIViewController {
View = ImageView
};
window.RootViewController = vc;
window.MakeKeyAndVisible ();
window.BackgroundColor = UIColor.Black;
if (!SetupCaptureSession ())
window.AddSubview (new UILabel (new CGRect (20f, 20f, 200f, 60f)) {
Text = "No input device"
});
return true;
}
bool SetupCaptureSession ()
{
// configure the capture session for low resolution, change this if your code
// can cope with more data or volume
session = new AVCaptureSession {
SessionPreset = AVCaptureSession.PresetMedium
};
// create a device input and attach it to the session
var captureDevice = AVCaptureDevice.DefaultDeviceWithMediaType (AVMediaType.Video);
if (captureDevice == null) {
Console.WriteLine ("No captureDevice - this won't work on the simulator, try a physical device");
return false;
}
//Configure for 15 FPS. Note use of LockForConigfuration()/UnlockForConfiguration()
NSError error = null;
captureDevice.LockForConfiguration (out error);
if (error != null) {
Console.WriteLine (error);
captureDevice.UnlockForConfiguration ();
return false;
}
if (UIDevice.CurrentDevice.CheckSystemVersion (7, 0))
captureDevice.ActiveVideoMinFrameDuration = new CMTime (1, 15);
captureDevice.UnlockForConfiguration ();
var input = AVCaptureDeviceInput.FromDevice (captureDevice);
if (input == null) {
Console.WriteLine ("No input - this won't work on the simulator, try a physical device");
return false;
}
session.AddInput (input);
// create a VideoDataOutput and add it to the sesion
var settings = new CVPixelBufferAttributes {
PixelFormatType = CVPixelFormatType.CV32BGRA
};
using (var output = new AVCaptureVideoDataOutput { WeakVideoSettings = settings.Dictionary }) {
queue = new DispatchQueue ("myQueue");
outputRecorder = new OutputRecorder ();
output.SetSampleBufferDelegate (outputRecorder, queue);
session.AddOutput (output);
}
session.StartRunning ();
return true;
}
public override void OnActivated (UIApplication application)
{
}
public class OutputRecorder : AVCaptureVideoDataOutputSampleBufferDelegate
{
AVAssetWriter writer=null;
AVAssetWriterInput writerinput= null;
CMTime lastSampleTime;
int frame=0;
NSUrl url;
public OutputRecorder()
{
string tempFile = Path.Combine(Path.GetTempPath(), "NewVideo.mp4");
if (File.Exists(tempFile)) File.Delete(tempFile);
url = NSUrl.FromFilename(tempFile);
NSError assetWriterError;
writer = new AVAssetWriter(url, AVFileType.Mpeg4, out assetWriterError);
var outputSettings = new AVVideoSettingsCompressed()
{
Height = 300,
Width = 300,
Codec = AVVideoCodec.H264,
CodecSettings = new AVVideoCodecSettings
{
AverageBitRate = 1000000
}
};
writerinput = new AVAssetWriterInput(mediaType: AVMediaType.Video, outputSettings: outputSettings);
writerinput.ExpectsMediaDataInRealTime = false;
writer.AddInput(writerinput);
}
public override void DidOutputSampleBuffer (AVCaptureOutput captureOutput, CMSampleBuffer sampleBuffer, AVCaptureConnection connection)
{
try
{
lastSampleTime = sampleBuffer.PresentationTimeStamp;
var image = ImageFromSampleBuffer(sampleBuffer);
if (frame == 0)
{
writer.StartWriting();
writer.StartSessionAtSourceTime(lastSampleTime);
frame = 1;
}
String infoString = "";
if (writerinput.ReadyForMoreMediaData)
{
if (!writerinput.AppendSampleBuffer(sampleBuffer))
{
infoString = "Failed to append sample buffer";
}
else
{
infoString = String.Format("{0} frames captured", frame++);
}
}
else
{
infoString = "Writer not ready";
}
Console.WriteLine(infoString);
ImageView.BeginInvokeOnMainThread(() => ImageView.Image = image);
}
catch (Exception e)
{
Console.WriteLine(e);
}
finally
{
sampleBuffer.Dispose();
}
}
UIImage ImageFromSampleBuffer (CMSampleBuffer sampleBuffer)
{
// Get the CoreVideo image
using (var pixelBuffer = sampleBuffer.GetImageBuffer () as CVPixelBuffer)
{
// Lock the base address
pixelBuffer.Lock (CVOptionFlags.None);
// Get the number of bytes per row for the pixel buffer
var baseAddress = pixelBuffer.BaseAddress;
var bytesPerRow = (int)pixelBuffer.BytesPerRow;
var width = (int)pixelBuffer.Width;
var height = (int)pixelBuffer.Height;
var flags = CGBitmapFlags.PremultipliedFirst | CGBitmapFlags.ByteOrder32Little;
// Create a CGImage on the RGB colorspace from the configured parameter above
using (var cs = CGColorSpace.CreateDeviceRGB ())
{
using (var context = new CGBitmapContext (baseAddress, width, height, 8, bytesPerRow, cs, (CGImageAlphaInfo)flags))
{
using (CGImage cgImage = context.ToImage ())
{
pixelBuffer.Unlock (CVOptionFlags.None);
return UIImage.FromImage (cgImage);
}
}
}
}
}
void TryDispose (IDisposable obj)
{
if (obj != null)
obj.Dispose ();
}
}
}
}
This works displaying live camera image and I get "frames captured" message in consol but I don't find how to record to file.
I read somewhere about adding VideoCapture but I don't know how to link with my code.
Any help will is welcome.
From your code, in the construct of class OutputRecorder you have defined the url where you want to save the recording:
string tempFile = Path.Combine(Path.GetTempPath(), "NewVideo.mp4");
if (File.Exists(tempFile)) File.Delete(tempFile);
url = NSUrl.FromFilename(tempFile);
It means you want to save the video in the tmp folder in the app's sandbox. If you want to use the video sometime later, I recommend you to change the folder to documents by using:
string filePath = Path.Combine(NSSearchPath.GetDirectories(NSSearchPathDirectory.DocumentDirectory, NSSearchPathDomain.User)[0], "NewVideo.mp4");
I notice that you have called session.StartRunning(); in the method bool SetupCaptureSession() to start recording. please add session.StopRunning(); to end recording then the video will be saved in the path we just defined above.
Moreover, you can retrieve the video with the path like:
NSData videoData = NSData.FromFile(filePath);
i am beginning in develop winphone and nokia imaging sdk. i have two function.
firstly, i call the function below to change image to gray color
private async void PickImageCallback(object sender, PhotoResult e)
{
if (e.TaskResult != TaskResult.OK || e.ChosenPhoto == null)
{
return;
}
using (var source = new StreamImageSource(e.ChosenPhoto))
{
using (var filters = new FilterEffect(source))
{
var sampleFilter = new GrayscaleFilter();
filters.Filters = new IFilter[] { sampleFilter };
var target = new WriteableBitmap((int)CartoonImage.ActualWidth, (int)CartoonImage.ActualHeight);
var renderer = new WriteableBitmapRenderer(filters, target);
{
await renderer.RenderAsync();
_thumbnailImageBitmap = target;
CartoonImage.Source = target;
}
}
}
SaveButton.IsEnabled = true;
}
then i call function to change image to binary color
private async void Binary(WriteableBitmap bm_image)
{
var target = new WriteableBitmap((int)CartoonImage.ActualWidth, (int)CartoonImage.ActualHeight);
MemoryStream stream= new MemoryStream();
bm_image.SaveJpeg(stream, bm_image.PixelWidth, bm_image.PixelHeight, 0, 100);
using (var source = new StreamImageSource(stream))
{
using (var filters = new FilterEffect(source))
{
var sampleFilter = new StampFilter(5, 0.7);
filters.Filters = new IFilter[] { sampleFilter };
var renderer1 =new WriteableBitmapRenderer(filters, target);
{
await renderer1.RenderAsync();
CartoonImage.Source = target;
}
}
}
}
but when it run to " await renderer1.RenderAsync();" in the second function, it doesn't work. How can i solve it. And you can explain for me about how "await" and "async" work ?
thank you very much!
I'm mostly guessing here since I do not know what error you get, but I'm pretty sure your problem lies in setting up the source. Have you made sure the memory stream position is set to the beginning (0) before creating an StreamImageSource?
Try adding:
stream.Position = 0;
before creating the StreamImageSource.
Instead of trying to create a memory stream from the writeable bitmap I suggest doing:
using Nokia.InteropServices.WindowsRuntime;
...
using (var source = new BitmapImageSource(bm_image.AsBitmap())
{
...
}