I am currently writing a small app which shows the preview from the phone camera using a SharpDX sprite batch. For those who have an nokia developer account, the code is mainly from this article.
Problem
Occasionally, it seems like previous frames are drawn to the screeb (the "video" jumps back and forth), for the fracture of a second, which looks like oscillation/flicker.
I thought of a threading problem (since the PreviewFrameAvailable event handler is called by a different thread than the method which is responsible for rendering), but inserting a lock statement into both methods makes the code too slow (the frame rate drops below 15 frames/sec).
Does anyone have an idea how to resolve this issue or how to accoplish thread synchronization in this case without loosing too much performance?
Code
First, all resources are created, whereas device is a valid instance of GraphicsDevice:
spriteBatch = new SpriteBatch(device);
photoDevice = await PhotoCaptureDevice.OpenAsync(CameraSensorLocation.Back, captureSize);
photoDevice.FocusRegion = null;
width = (int)photoDevice.PreviewResolution.Width;
height = (int)photoDevice.PreviewResolution.Height;
previewData = new int[width * height];
cameraTexture = Texture2D.New(device, width, height, PixelFormat.B8G8R8A8.UNorm);
photoDevice.PreviewFrameAvailable += photoDevice_PreviewFrameAvailable;
Then, whenever the preview frame changes, I set the data to the texture:
void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
{
sender.GetPreviewBufferArgb(previewData);
cameraTexture.SetData(previewData);
}
Finally, the Texture is drawn using a SpriteBatch whereas the parameters backBufferCenter, textureCenter, textureScaling and Math.Pi / 2 are used to center and adjust the texture in landscape orientation:
spriteBatch.Begin();
spriteBatch.Draw(cameraTexture, backBufferCenter, null, Color.White, (float)Math.PI / 2, textureCenter, textureScaling, SpriteEffects.None, 1.0f);
spriteBatch.End();
The render method is called by the SharpDX game class, which basically uses the IDrawingSurfaceBackgroundContentProvider interface, which is called by the DrawingSurfaceBackgroundGrid component of the Windows Phone 8 runtime.
Solution
Additional to Olydis solution (see below), I also had to set Game.IsFixedTimeStep to false, due to a SharpDX bug (see this issue on GitHub for details).
Furthermore, it is not safe to call sender.GetPreviewBufferArgb(previewData) inside the handler for PreviewFrameAvailable, due to cross thread access. See the corresponding thread in the windows phone developer community.
My Guess
As you guessed, I'm also pretty sure this may be due to threading. I suspect that, for example, the relatively lengthy SetData call may be intercepted by the Draw call, leading to unexpected output.
Solution
The following solution does not use synchronization, but instead moves "critical" parts (access to textures) to the same context.
Also, let's allocate two int[] instead of one, which we will use in an alternating fashion.
Code Fragments
void photoDevice_PreviewFrameAvailable(ICameraCaptureDevice sender, object args)
{
sender.GetPreviewBufferArgb(previewData2);
// swap buffers
var previewDataTemp = previewData1;
previewData1 = previewData2;
previewData2 = previewDataTemp;
}
Then add this to your Draw call (or equal context):
cameraTexture.SetData(previewData1);
Conclusion
This should practically prevent your problem since only "fully updated" textures are drawn and there is no concurrenct access to them. The use of two int[] reduces the risk of having SetData and GetPreviewBufferArgb access the same array concurrently - however, it does not eliminate the risk (but no idea if concurrent access to the int[] can result in weird behaviour in the first place).
Related
Using MonoGame (Basically XNA) I have some code which allows you to host a DirectX11 window inside of a System.Windows.Controls.Image, the purpose of which is to allow you to display the window as a standard WPF control.
I created this code by looking at a number of online code examples which demonstrated similar functionality (as I am a complete newbie to game dev). Among some of the code that I have leveraged there is a method of specific interest to me which looks like this:
private static void InitializeGraphicsDevice(D3D11Host game, int width, int height)
{
lock (GraphicsDeviceLock)
{
_ReferenceCount++;
if (_ReferenceCount == 1)
{
// Create Direct3D 11 device.
_GraphicsDeviceManager = new WpfGraphicsDeviceManager(game, width, height);
_GraphicsDeviceManager.CreateDevice();
}
}
}
This code is called on the creation of the hosting object (i.e. System.Windows.Controls.Image) and clearly it appears the intent is to limit the creation of multiple GraphicsDeviceManagers. However I have ended up in the situation where this code prevents me from creating multiple game windows, as needed.
I have changed this code from static to instance and removed the counter and everything seems to be working fine BUT I am concerned that there is something fundamental I don't understand which might come up later.
So, why does the above code prevent creating multiple DeviceManagers? Is it legal for me to create multiple graphics device managers in XNA (MonoGame)? I have to assume there must have been a reason for it?
I think it's because of the fundamental design thought behind xna. You have one game loop, one window for graphic output and so on.
If I remember correctly it should be no problem to create multiple graphic devices on different handles (in your case different windows).
There appears to be a memory leak with WriteableBitmaps when writing to the backbuffer directly and using the AddDirtyRect function multiple times within a single Lock/Unlock. The rectangles need to define different regions within the bitmap. The memory will then leak when you try to discard the WriteableBitmap.
You can recreate it by inserting the following code into a new WPF application. When the application starts, resize the window to create new WriteableBitmaps and watch the memory rise.
public partial class MainWindow : Window
{
public MainWindow()
{
InitializeComponent();
Image m = new Image();
m.Stretch = Stretch.Fill;
this.Content = m;
this.SizeChanged += OnSizeChanged;
}
private void OnSizeChanged(object sender, SizeChangedEventArgs args)
{
WriteableBitmap bm = new WriteableBitmap((int)args.NewSize.Width, (int)args.NewSize.Height, 96, 96, PixelFormats.Bgra32, null);
bm.Lock();
bm.AddDirtyRect(new Int32Rect(1, 1, 1, 1));
bm.AddDirtyRect(new Int32Rect(2, 2, 1, 1));
bm.Unlock();
((Image)this.Content).Source = bm;
}
}
We need to be able to discard the bitmap so keeping the same one around and reusing it is not an option. We could also not write to the backbuffer directly and instead use WritableBitmap.WritePixels() but it's slower and speed is an issue.
UPDATE:
I've tested the WritePixels method and it leaks all the same. It may be an issue of calling too many writes too quickly in different regions.
We've contacted Microsoft on this issue and it appears to be a problem with the underlying c++ library backing WPF. We haven't been given an promise of when (or if) a fix will come but it is still a bug as of .NET 4.5.1.
There are currently only two ways we have found to work around this problem and they are mutually exclusive. You can either:
Never dirty any subregion of the bitmap, only dirty the full bitmap
The problem with this method is performance. You can try to counteract this by making your bitmaps smaller but there likely many situations where this isn't possible.
Never discard your bitmap
If you're going to dirty multiple subsections of the bitmap then you must ensure it will never be garbage collected unless you're about to close the application. This comes with it's own host of problems as you have to make sure the bitmap is large enough the first time you create it. If users are allowed to resize the window then you'll have to make it fit the entire desktop, but even this is problematic as users can change their desktop resolution or add/remove monitors meaning you'll either have to leak memory or not have enough bitmap to cover the entire possible size of the window.
Hopefully Microsoft will release a fix for this in the future but for the mean time be very careful with WriteableBitmap as it's very prone to leaking memory.
I have the following code as part of a game:
protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.Black);
spriteBatch.Begin();
terrainSprite.Draw(spriteBatch);
if (resourceMap.pixels.IsDisposed == false)
{
resources.Draw(spriteBatch, spriteFont);
}
spriteBatch.End();
base.Draw(gameTime);
//Disposes the texture here:
resources.pixels.Dispose();
}
//In the resources class
public void Update()
{
//gD = graphics device
pixels = new Texture2D(gD, 800, 353);
//big update method
//manipulate the pixels texture
}
When I open task manager and look at the resource monitor, the memory usage for 'myGame.exe' is constantly going up by about 8 KB (I realize this is small, but my game holds a LOT of data, so saving every bit I can is important, and it builds up fairly quickly). This is after all other code is commented out except for what is shown here. Then, when I comment out the code: "pixels = new Texture2D(gD, 800, 353);", the memory usage stays constant. I also tried GC.Collect(), but no dice.
Is there anything else I can do to try and stop it? (Sorry, getting rid of the code is not an option :p, renewing the texture is much faster than any other method I've come across to make the texture go blank)
Depending on your Game configuration and really, many other factors, such as how slow everything is running, etc., Update and Draw are not perfectly synchronous with each other and are not guaranteed to be run in the following fashion:
Update
Draw
Update
Draw
Update
Draw
Update
Draw
....
Therefore, since you're Disposeing in one and creating a brand new one in the other, something like this can definitely happen:
Update: create new
Update: create new //PREVIOUS ONE LEAKED!
Draw: disposes only current
Update: create new
Update: create new //AGAIN LEAK
Draw: disposes only current
...
Thus, do not Dispose separately in this fashion; Dispose one time for each new one created, no matter what.
I should also add on that textures, along with some other XNA classes (sound and music, and Effects, to name a few) are unmanaged resources, meaning the GC does not see them at all. You must call Dispose on these.
As Andrew points out in his comment, the best way to avoid these pitfalls is not to recreate textures so often - simply reuse the same one and modify it as you see fit.
It appears that Texture2D are not fully handled by the garbage collector.
So when you stop using it (when reusing a variable referencing it, like here, or during the OnDestroy callback), you have to manually destroy the texture. Here :
if(pixels != null) {
Destroy(pixels);
}
pixels = new Texture2D(gD, 800, 353);
I am part of a team that has created a tool to view and interact with very large and heavily interconnected graphs in C#/WPF. Viewing and interacting with the graph is done through a custom control that takes in a set of DrawingVisuals and displays them on a canvas. Nodes in the graph may have a custom shape created with our editor. The current control works very well and is fairly coupled with our program but there are legitimate worries about performance when considering much larger graphs (20,000+ nodes and lots of connection).
After doing a bit of research it seems the two approaches are:
A GDI+ route where graphics are drawn to a WriteableBitmap or InteropBitmap.
SlimDX or DirectX variant (hosted in a D3DImage)
Given these two extremely different approaches which route would be best to take considering:
Interacting with the graph must be fast even while viewing the whole graph.
Updating the visuals should be fast (color or size change)
Hit testing must be fast (point and rectangle).
Development must be completed in a timely manner.
Which method would you use and why?
EDIT:
It looks like a similar question was asked but not answered.
I use GDI for my cartographic application. While GDI+ is slower than, say, DirectX, I find that there are a lot of things and tricks that can be used to speed things up. A lot of CPU is used for preparing the data before drawing it itself, so GDI should not be the only bottleneck there.
Things to look out for (and these are general enough to apply to other graphics engines, too):
First of all: measure. Use a profiler to see what is the real bottleneck in your code.
Reuse GDI primitives. This is vital. If you have to draw 100,000 graphics objects that look the same or similar, use the same Pen, Brush etc. Creating these primitives is expensive.
Cache the rendering data - for example: don't recalculate gfx element positions if you don't need to.
When panning/zooming, draw the scene with lower GDI+ quality (and without expensive GDI operations). There are a number of Graphics object settings to lower the quality. After the user stops panning, draw the scene with the high quality.
A lot and lot of little things that improve performance. I've been developing this app for 2-3 years (or is it 4 already hmm?) now and I still find ways to improve things :). This is why profiling is important - the code changes and this can affect the performance, so you need to profile the new code.
One other thing: I haven't used SlimDX, but I did try Direct2D (I'm referring to Microsoft.WindowsAPICodePack.DirectX.Direct2D1). The performance was considerably faster than GDI+ in my case, but I had some issues with rendering bitmaps and never had the time to find the proper solution.
I have recently ported some drawing code over to DirectX and have been very pleased with the results. We were mainly rendering bitmaps using bit-bashing and are seeing render times that could be measured in minutes reduced to around 1-2 seconds.
This can't be directly compared to you usage, as we've gone from bit-bashing in C++ to Direct3D in C# using SlimDX, but I imagine you will see performance benefits, even if they're not the orders of magnitude we're seeing.
I would advise you to take a look at using Direct2D with SlimDX. You will need to use DirectX 10.1 as Direct2D isn't compatible with DirectX 11 for some reason. If you have used the drawing API in WPF then you will already be familiar with Direct2D as its API is based on the WPF drawing API as far as I can tell. The main problems with Direct2D are a lack of documentation and the fact it only works in Vista onwards.
I've not experimented with DirectX 10/WPF interop, but I believe it is possible (http://stackoverflow.com/questions/1252780/d3dimage-using-dx10)
EDIT: I thought I'd give you a comparison from our code of drawing a simple polygon. First the WPF version:
StreamGeometry geometry = new StreamGeometry();
using (StreamGeometryContext ctx = geometry.Open())
{
foreach (Polygon polygon in mask.Polygons)
{
bool first = true;
foreach (Vector2 p in polygon.Points)
{
Point point = new Point(p.X, p.Y);
if (first)
{
ctx.BeginFigure(point, true, true);
first = false;
}
else
{
ctx.LineTo(point, false, false);
}
}
}
}
Now the Direct2D version:
Texture2D maskTexture = helper.CreateRenderTexture(width, height);
RenderTargetProperties props = new RenderTargetProperties
{
HorizontalDpi = 96,
PixelFormat = new PixelFormat(SlimDX.DXGI.Format.Unknown, AlphaMode.Premultiplied),
Type = RenderTargetType.Default,
Usage = RenderTargetUsage.None,
VerticalDpi = 96,
};
using (SlimDX.Direct2D.Factory factory = new SlimDX.Direct2D.Factory())
using (SlimDX.DXGI.Surface surface = maskTexture.AsSurface())
using (RenderTarget target = RenderTarget.FromDXGI(factory, surface, props))
using (SlimDX.Direct2D.Brush brush = new SolidColorBrush(target, new SlimDX.Color4(System.Drawing.Color.Red)))
using (PathGeometry geometry = new PathGeometry(factory))
using (SimplifiedGeometrySink sink = geometry.Open())
{
foreach (Polygon polygon in mask.Polygons)
{
PointF[] points = new PointF[polygon.Points.Count()];
int i = 0;
foreach (Vector2 p in polygon.Points)
{
points[i++] = new PointF(p.X * width, p.Y * height);
}
sink.BeginFigure(points[0], FigureBegin.Filled);
sink.AddLines(points);
sink.EndFigure(FigureEnd.Closed);
}
sink.Close();
target.BeginDraw();
target.FillGeometry(geometry, brush);
target.EndDraw();
}
As you can see, the Direct2D version is slightly more work (and relies on a few helper functions I've written) but it's actually pretty similar.
Let me try and list the pros and cons of each approach - which will perhaps give you some idea about which to use.
GDI Pros
Easy to draw vector shapes with
No need to include extra libraries
GDI Cons
Slower than DX
Need to limit "fancy" drawing (gradients and the like) or it might slow things down
If the diagram needs to be interactive - might not be a great option
SlimDX Pros
Can do some fancy drawing while being faster than GDI
If the drawing is interactive - this approach will be MUCH better
Since you draw the primitives you can control quality at each zoom level
SlimDX Cons
Not very easy to draw simple shapes with - you'll need to write your own abstractions or use a library that helps you draw shapes
Not as simple to use a GDI especially if you've not used it before
And perhaps more I forgot to put in here, but perhaps these will do for starters?
-A.
Attempting to do SetData() on Texture2D that has been recently Draw()-n by SpriteBatch leads to following exception:
The operation was aborted. You may not modify a resource that has been set on a device, or after it has been used within a tiling bracket.
Can I determine in advance if executing SetData() will throw this exception?
Basically, you have three options:
1) See if SpriteBatch has finished it's operation and call SetData() afterwards. The drawing methods usually are asynchronous. That means that, they just get added to the please-render-me queue and the method returns immediately. What you need is either a callback notification when the drawing has finished or a synchronous call to Draw().
2) Avoiding that SetData() gets interrupted. You can do that by putting it into a critical section which I would not recommend. It should be possible to lock the texture data. It's called LockRect() in Direct3D, it should be similar in XNA.
3) There should be some method like Flush() somewhere that waits until all graphics-related operation have finished.
Sorry for the rather vague help, but you should be able to find the method names from the XNA doc.
Basically no.
The easiest thing to do is to only call SetData in your Update method.
It's bad practice to use SetData within your Draw method, as the device could be doing all kinds of voodoo magic with the old data. This is explained in detail in the "Caution" box on its MSDN page.
Now, up until XNA 3.1 you could use SetDataOptions with Texture2D.SetData. But it looks like that functionality has been removed in XNA 4.0 for textures.
Shawn Hargreaves explains here how SetDataOptions could be used tell the GPU "actually yes I do want to overwrite that data you might be using, don't complain". And why it's difficult to get right.
I've solved a problem like this by creating two textures, and switching between the active one, essentially double-buffering:
void CreateTextures()
{
depth_1 = new Texture2D(this.GraphicsDevice, width, height, false, SurfaceFormat.Single);
depth_2 = new Texture2D(this.GraphicsDevice, width, height, false, SurfaceFormat.Single);
depth_current = depth_1;
...
}
void Draw(GameTime gt)
{
depth_current = depth_current == depth_1 ? depth_2 : depth_1;
Depth.SetData(this.DepthBuffer);
...
}
In my case it wasn't possible to move the SetData outside of Draw, but I suppose that's the nicest way to do it.