Good morning,
I have been teaching myself a bit of Direct2D programming in C#, utilizing native wrappers that are available (currently using d2dSharp, but have also tried SharpDX). I'm running into problems with efficiency, though, where the basic drawing Direct2D drawing methods are taking approximately 250 ms to draw 45,000 basic polygons. The performance I am seeing is on par, or even slower than, Windows GDI+. I'm hoping that someone can take a look at what I've done and propose a way(s) that I can dramatically improve the time it takes to draw.
The background to this is that I have a personal project in which I am developing a basic but functional CAD interface capable of performing a variety of tasks, including 2D finite element analysis. In order to make it at all useful, the interface needs to be able to display tens-of-thousands of primitive elements (polygons, circles, rectangles, points, arcs, etc.).
I initially wrote the drawing methods using Windows GDI+ (System.Drawing), and performance is pretty good until I reach about 3,000 elements on screen at any given time. The screen must be updated any time the user pans, zooms, draws new elements, deletes elements, moves, rotates, etc. Now, in order to improve efficiency, I utilize a quad tree data structure to store my elements, and I only draw elements that actually fall within the bounds of the control window. This helped significantly when zoomed in, but obviously, when fully zoomed out and displaying all elements, it makes no difference. I also use a timer and tick events to update the screen at the refresh rate (60 Hz), so I'm not trying to update thousands of times per second or on every mouse event.
This is my first time programming with DirectX and Direct2D, so I'm definitely learning here. That being said, I've spent days reviewing tutorials, examples, and forums, and could not find much that helped. I've tried a dozen different methods of drawing, pre-processing, multi-threading, etc. My code is below
Code to Loop Through and Draw Elements
List<IDrawingElement> elementsInBounds = GetElementsInDraftingWindow();
_d2dContainer.Target.BeginDraw();
_d2dContainer.Target.Clear(ColorD2D.FromKnown(Colors.White, 1));
if (elementsInBounds.Count > 0)
{
Stopwatch watch = new Stopwatch();
watch.Start();
#region Using Drawing Element DrawDX Method
foreach (IDrawingElement elem in elementsInBounds)
{
elem.DrawDX(ref _d2dContainer.Target, ref _d2dContainer.Factory, ZeroPoint, DrawingScale, _selectedElementBrush, _selectedElementPointBrush);
}
#endregion
watch.Stop();
double drawingTime = watch.ElapsedMilliseconds;
Console.WriteLine("DirectX drawing time = " + drawingTime);
watch.Reset();
watch.Start();
Matrix3x2 scale = Matrix3x2.Scale(new SizeFD2D((float)DrawingScale, (float)DrawingScale), new PointFD2D(0, 0));
Matrix3x2 translate = Matrix3x2.Translation((float)ZeroPoint.X, (float)ZeroPoint.Y);
_d2dContainer.Target.Transform = scale * translate;
watch.Stop();
double transformTime = watch.ElapsedMilliseconds;
Console.WriteLine("DirectX transform time = " + transformTime);
}
DrawDX Function for Polygon
public override void DrawDX(ref WindowRenderTarget rt, ref Direct2DFactory fac, Point zeroPoint, double drawingScale, SolidColorBrush selectedLineBrush, SolidColorBrush selectedPointBrush)
{
if (_pathGeometry == null)
{
CreatePathGeometry(ref fac);
}
float brushWidth = (float)(Layer.Width / (drawingScale));
brushWidth = (float)(brushWidth * 2);
if (Selected == false)
{
rt.DrawGeometry(Layer.Direct2DBrush, brushWidth, _pathGeometry);
//Note that _pathGeometry is a PathGeometry
}
else
{
rt.DrawGeometry(selectedLineBrush, brushWidth, _pathGeometry);
}
}
Code to Create Direct2D Factory & Render Target
private void CreateD2DResources(float dpiX, float dpiY)
{
Factory = Direct2DFactory.CreateFactory(FactoryType.SingleThreaded, DebugLevel.None, FactoryVersion.Auto);
RenderTargetProperties props = new RenderTargetProperties(
RenderTargetType.Default, new PixelFormat(DxgiFormat.B8G8R8A8_UNORM,
AlphaMode.Premultiplied), dpiX, dpiY, RenderTargetUsage.None, FeatureLevel.Default);
Target = Factory.CreateWindowRenderTarget(_targetPanel, PresentOptions.None, props);
Target.AntialiasMode = AntialiasMode.Aliased;
if (_selectionBoxLeftStrokeStyle != null)
{
_selectionBoxLeftStrokeStyle.Dispose();
}
_selectionBoxLeftStrokeStyle = Factory.CreateStrokeStyle(new StrokeStyleProperties1(LineCapStyle.Flat,
LineCapStyle.Flat, LineCapStyle.Flat, LineJoin.Bevel, 10, DashStyle.Dash, 0, StrokeTransformType.Normal), null);
}
I create a Direct2D factory and render target once and keep references to them at all times (that way I'm not recreating each time). I also create all of the brushes when the drawing layer (which describes color, width, etc.) is created. As such, I am not creating a new brush every time I draw, simply referencing a brush that already exists. Same with the geometry, as can be seen in the second code-snippet. I create the geometry once, and only update the geometry if the element itself is moved or rotated. Otherwise, I simply apply a transform to the render target after drawing.
Based on my stopwatches, the time taken to loop through and call the elem.DrawDX methods takes about 225-250 ms (for 45,000 polygons). The time taken to apply the transform is 0-1 ms, so it appears that the bottleneck is in the RenderTarget.DrawGeometry() function.
I've done the same tests with RenderTarget.DrawEllipse() or RenderTarget.DrawRectangle(), as I've read that using DrawGeometry is slower than DrawRectangle or DrawEllipse as the rectangle / ellipse geometry is known beforehand. However, in all of my tests, it hasn't mattered which draw function I use, the time for the same number of elements is always about equal.
I've tried building a multi-threaded Direct2D factory and running the draw functions through tasks, but that is much slower (about two times slower). The Direct2D methods appear to be utilizing my graphics card (hardware accelerated is enabled), as when I monitor my graphics card usage, it spikes when the screen is updating (my laptop has an NVIDIA Quadro mobile graphics card).
Apologies for the long-winded post. I hope this was enough background and description of things I've tried. Thanks in advance for any help!
Edit #1
So changed the code from iterating over a list using foreach to iterating over an array using for and that cut the drawing time down by half! I hadn't realized how much slower lists were than arrays (I knew there was some performance advantage, but didn't realize this much!). It still, however, takes 125 ms to draw. This is much better, but still not smooth. Any other suggestions?
Direct2D can be used with P/Invoke
See the sample "VB Direct2D Pixel Perfect Collision"
from https://social.msdn.microsoft.com/Forums/en-US/cea42526-4b82-454d-9d79-2e1d94083552/collisions?forum=vbgeneral
the animation is perfect, even done in VB
My background is C++ and I have been trying to translate one of my apps that uses OpenCV to a C# app that uses EmguCV. I have read the EmguCV documentation and Wiki as well as StackOverflow Q/As as deeply as humanly possible with no luck in solving some of the basic problems. At this point I wonder if the way I think about EmguCV is fundamentally flawed.
I was under the impression that when EmguCV has provided an interface to the C++ library for let's say FloodFill using CvInvoke.FloodFill(IInputOutputArray, IInputOutputArray, ...), underneath it takes care of its own memory management to the point that the returning IInputOutputArrays (in my case a Mat) is all taken care of. In practice however, A call to CvInvoke.FloodFill fails with an exception from within the dll.
This is what I have:
int i = ...
int j = ...
int intensity = ...
int height = img.Rows;
int width = img.Cols;
Mat outputMask = new Mat(height + 2, width + 2, DepthType.Cv8U, 1);
Rectangle dummy = new Rectangle();
CvInvoke.FloodFill(img, outputMask, new Point(i, j), new MCvScalar(255), out dummy, new MCvScalar(intensity), new MCvScalar(intensity), Connectivity.EightConnected, FloodFillType.MaskOnly);
But the call fails with an exception description that is not helpful. I have used GCHandle allocation and pinning for other methods to be able to access the raw data underneath, but I believe CvInvoke.FloodFill should be safe when called on some local Mats.
My question is whether I'm completely off track with the way EmguCV is supposed to be used given my background with C++ and OpenCV. If so, what is the correct way of calling such function?
This is for anyone who may find themselves having the same problem with EmguCV 3.1 FloodFill. After debugging my code for quite a long time, I am convinced that EmguCV 3.1 implementation of CvInvoke.FloodFill is buggy. There was no way to get a mask out of that method using FloodFillType.MaskOnly flag.
Instead I ended up editing my original image to have values between 0 and 254, then did a normal FloodFill using fill value of 255. Then used Image.InRange() method to separate all 255 (filled) pixels from the original image. I will report the bug to EmguCV team and see what happens.
I have in my application a 3D world and data for this 3D world. The UI around the application is done with WPF and so far it seems to be working ok. But now I am implementing the following functionality: If you click on the terrain in the 3D view it will show the textures used in this chunk of terrain in a WPF control. The image data of the textures is compressed (S3TC) and I handle creation of BGRA8 data in a separate thread. Once its ready I'm using the main windows dispatcher to do the WPF related tasks. Now to show you this in code:
foreach (var pair in loadTasks)
{
var img = pair.Item2;
var loadInfo = TextureLoader.LoadToArgbImage(pair.Item1);
if (loadInfo == null)
continue;
EditorWindowController.Instance.WindowDispatcher.BeginInvoke(new Action(img =>
{
var watch = Stopwatch.StartNew();
var source = BitmapSource.Create(loadInfo.Width, loadInfo.Height, 96, 96, PixelFormats.Bgra32,
null,
loadInfo.Layers[0], loadInfo.Width * 4);
watch.Stop();
img.Source = source;
Log.Debug(watch.ElapsedMilliseconds);
}));
}
While I cant argue with the visual output there is a weird performance issue. As you can see I have added a stopwatch to check where the time is consumed and I found the culprit: BitmapSource.Create.
Typically I have 5-6 elemets in loadTasks and the images are 256x256 pixels. Interestingly now the first invocation shows 280-285ms for BitmapSource.Create. The next 4-5 all are below 1ms. This consistently happens every time I click the terrain and the loop is started. The only way to avoid the penalty in the first element is to click on the terrain constantly but as soon as I don't click the terrain (and therefore do not invoke the code above) for 1-2 seconds the next call to BitmapSource.Create gets the 280ms penalty again.
Since anything above 5ms is far beyond any reasonable or acceptable time to create 256x256 bitmap (my S3TC decompression does all 10(!) mip layers in less than 2 ms) I guess there has to be something else going on here?
FYI: All properties of loadInfo are static properties and do not perform any calculations you cant see in the code.
I am trying to apply image manipulation effects in Windows 8 app on camera feeds directly.
I have tried a way using canvas and redrawing images after applying effects getting from webcam directly. But this approach works fine for basic effects but for effects like edge detection its creating large lag and flickering while using canvas approach.
Other way is to create MFT(media foundation transform) but it can be implemented in C about which i have no idea.
Can anyone tell me how can i achieve my purpose of applying effects on webcam stream directly in Windows 8 metro style app either by improving canvas approach so that large effects like edge detection don not have any issues or how can i apply MFT in C# since i have worked on C# language or by some other approach?
I have just played quite a bit in this area the last week and even considered writing a blog post about it. I guess this answer can be just as good.
You can go the MFT way, which needs to be done in C++, but the things you would need to write would not be much different between C# and C++. The only thing of note is that I think the MFT works in YUV color space, so your typical convolution filters/effects might behave a bit differently or require conversion to RGB. If you decide to go that route On the C# application side the only thing you would need to do is to call MediaCapture.AddEffectAsync(). Well that and you need to edit your Package.appxmanifest etc., but let's go with first things first.
If you look at the Media capture using webcam sample - it already does what you need. It applies a grayscale effect to your camera feed. It includes a C++ MFT project that is used in an application that is available in C# version. I had to apply the effect to a MediaElement which might not be what you need, but is just as simple - call MediaElement.AddVideoEffect() and your video file playback now applies the grayscale effect. To be able to use the MFT - you need to simply add a reference to the GrayscaleTransform project and add following lines to your appxmanifest:
<Extensions>
<Extension Category="windows.activatableClass.inProcessServer">
<InProcessServer>
<Path>GrayscaleTransform.dll</Path>
<ActivatableClass ActivatableClassId="GrayscaleTransform.GrayscaleEffect" ThreadingModel="both" />
</InProcessServer>
</Extension>
</Extensions>
How the MFT code works:
The following lines create a pixel color transformation matrix
float scale = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_SATURATION, 0.0f);
float angle = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_CHROMA_ROTATION, 0.0f);
m_transform = D2D1::Matrix3x2F::Scale(scale, scale) * D2D1::Matrix3x2F::Rotation(angle);
Depending on the pixel format of the video feed - a different transformation method is selected to scan the pixels. Look for these lines:
m_pTransformFn = TransformImage_YUY2;
m_pTransformFn = TransformImage_UYVY;
m_pTransformFn = TransformImage_NV12;
For my sample m4v file - the format is detected as NV12, so it is calling TransformImage_NV12.
For pixels within the specified range (m_rcDest) or within the entire screen if no range was specified - the TransformImage_~ methods call TransformChroma(mat, &u, &v).
For other pixels - the values from original frame are copied.
TransformChroma transforms the pixels using m_transform. If you want to change the effect - you can simply change the m_transform matrix or if you need access to neighboring pixels as in an edge detection filter - modify the TransformImage_ methods to process these pixels.
This is one way to do it. I think it is quite CPU intensive, so personally I prefer to write a pixel shader for such operations. How do you apply a pixel shader to a video stream though? Well, I am not quite there yet, but I believe you can transfer video frames to a DirectX surface fairly easily and call a pixel shader on them later. So far - I was able to transfer the video frames and I am hoping to apply the shaders next week. I might write a blog post about it. I took the meplayer class from the Media engine native C++ playback sample and moved it to a template C++ DirectX project converted to a WinRTComponent library, then used it with a C#/XAML application, associating the swapchain the meplayer class creates with the SwapChainBackgroundPanel that I use in the C# project to display the video. I had to make a few changes in the meplayer class. First - I had to move it to a public namespace that would make it available to other assembly. Then I had to modify the swapchain it creates to a format accepted for use with a SwapChainBackgroundPanel:
DXGI_SWAP_CHAIN_DESC1 swapChainDesc = {0};
swapChainDesc.Width = m_rcTarget.right;
swapChainDesc.Height = m_rcTarget.bottom;
// Most common swapchain format is DXGI_FORMAT_R8G8B8A8-UNORM
swapChainDesc.Format = m_d3dFormat;
swapChainDesc.Stereo = false;
// Don't use Multi-sampling
swapChainDesc.SampleDesc.Count = 1;
swapChainDesc.SampleDesc.Quality = 0;
//swapChainDesc.BufferUsage = DXGI_USAGE_BACK_BUFFER | DXGI_USAGE_RENDER_TARGET_OUTPUT;
swapChainDesc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT; // Allow it to be used as a render target.
// Use more than 1 buffer to enable Flip effect.
//swapChainDesc.BufferCount = 4;
swapChainDesc.BufferCount = 2;
//swapChainDesc.Scaling = DXGI_SCALING_NONE;
swapChainDesc.Scaling = DXGI_SCALING_STRETCH;
swapChainDesc.SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL;
swapChainDesc.Flags = 0;
Finally - instead of calling CreateSwapChainForCoreWindow - I am calling CreateSwapChainForComposition and associating the swapchain with my SwapChainBackgroundPanel:
// Create the swap chain and then associate it with the SwapChainBackgroundPanel.
DX::ThrowIfFailed(
spDXGIFactory.Get()->CreateSwapChainForComposition(
spDevice.Get(),
&swapChainDesc,
nullptr, // allow on all displays
&m_spDX11SwapChain)
);
ComPtr<ISwapChainBackgroundPanelNative> dxRootPanelAsSwapChainBackgroundPanel;
// Set the swap chain on the SwapChainBackgroundPanel.
reinterpret_cast<IUnknown*>(m_swapChainPanel)->QueryInterface(
IID_PPV_ARGS(&dxRootPanelAsSwapChainBackgroundPanel)
);
DX::ThrowIfFailed(
dxRootPanelAsSwapChainBackgroundPanel->SetSwapChain(m_spDX11SwapChain.Get())
);
*EDIT follows
Forgot about one more thing. If your goal is to stay in pure C# - if you figure out how to capture frames to a WriteableBitmap (maybe by calling MediaCapture.CapturePhotoToStreamAsync() with a MemoryStream and then calling WriteableBitmap.SetSource() on the stream) - you can use WriteableBitmapEx to process your images. It might not be top performance, but if your resolution is not too high or your frame-rate requirements are not high - it might just be enough. The project on CodePlex does not officially support WinRT yet, but I have a version that should work that you can try here (Dropbox).
As far as I know, MFTs need to be implemented in C++. I believe that there is a media transform SDK sample which shows implementing some straightforward transforms from a metro style application.
I'm attempting to capture a rendered screen from a Managed DirectX application. Typically, the way to do this is as follows:
Surface renderTarget = device.GetRenderTarget(0);
SurfaceLoader.Save(snapshotName, ImageFileFormat.Bmp, renderTarget);
Which is (in my understanding) shorthand for something like:
Surface renderTarget = device.GetRenderTarget(0);
Surface destTarget = device.CreateOffscreenPlainSurface(ClientRectangle.Width, ClientRectangle.Height, graphicsSettings.WindowedDisplayMode.Format, Pool.SystemMemory);
device.GetRenderTargetData(renderTarget,destTarget);
SurfaceLoader.Save(snapshotName,ImageFileFormat.Bmp, destTarget);
The problem is that on older video cards which don't support non-power-of-two dimension textures, the above fails. I've tried a number of workarounds, but nothing seems to accomplish this seemingly simple task of saving arbitrary-dimensioned screen captures. For example, the following fails on new Bitmap() with an invalid parameter exception (note that this requires creating the device with PresentFlag.LockableBackBuffer):
Surface surf = m_device.GetRenderTarget(0);
GraphicsStream gs = surf.LockRectangle(LockFlags.ReadOnly);
Bitmap bmp = new Bitmap(gs);
bmp.Save(snapshotName, ImageFormat.Png);
surf.UnlockRectangle();
Any tips would be greatly appreciated...I've pretty much exhausted everything I can think of (or turn up on Google)...
Why not create a texture which is the next highest power of 2 and then copy a sub rect? It would get round your issues even if the image saved has a whole load of blank space.
I'm surprised Bitmap has issues, tbh. However .. if thats the case then the above will work even it its not ideal.