We Keep Coding

I want to get all objects except text object as an image from PDF using iTextSharp

I am developing a program to convert PDF to PPTX for specific reasons using iTextSharp.
What I've done so far is to get all text objects and image objects and locations.
But I'm feeling difficult to get Table objects without texts.
Actually it would be better if I can get them as images.
My plan is to merge all objects except text objects as a background image and put text objects at proper locations.
I tried to find similar questions here but no luck so far.
If anyone knows how to do this particular job, please answer.
Thanks.

You say
What I've done so far is to get all text objects and image objects and locations.
but you don't go into detail how you do so. I assume you use a matching IRenderListener implementation.
But IRenderListener, as you found out yourself,
only extracts images and texts.
The main missing objects are paths and their usages.
To extract them, too, you should implement IExtRenderListener which extends IRenderListener but also retrieves information about paths. To understand the callback methods, please first be aware how path related instructions work in PDFs:
First there are instructions for building the actual path; these instructions essentially
move to some position,
add a line to some position from the previous position,
add a Bézier curve to some position from the previous position using some control points, or
add an upright rectangle at some position using some width and height information.
Then there is an optional instruction to intersect the current clip path with the generated path.
Finally, there is a drawing instruction for any combination of filling the inside of the path and stroking along the path, i.e. for doing both, either one, or neither one.
This corresponds to the callbacks you retrieve in your IExtRenderListener implementation:
/**
* Called when the current path is being modified. E.g. new segment is being added,
* new subpath is being started etc.
*
* #param renderInfo Contains information about the path segment being added to the current path.
*/
void ModifyPath(PathConstructionRenderInfo renderInfo);
is called once or more often to build the actual path, PathConstructionRenderInfo containing the actual instruction type in its Operation property (compare to the PathConstructionRenderInfo constant members MOVETO, LINETO, etc. to determine the operation type) and the required coordinates / dimensions in its SegmentData property. The Ctm property additionally returns the affine transformation that currently is set to be applied to all drawing operations.
Then
/**
* Called when the current path should be set as a new clipping path.
*
* #param rule Either {#link PathPaintingRenderInfo#EVEN_ODD_RULE} or {#link PathPaintingRenderInfo#NONZERO_WINDING_RULE}
*/
void ClipPath(int rule);
is called if the current clip path shall be intersected with the constructed path.
Finally
/**
* Called when the current path should be rendered.
*
* #param renderInfo Contains information about the current path which should be rendered.
* #return The path which can be used as a new clipping path.
*/
Path RenderPath(PathPaintingRenderInfo renderInfo);
is called, PathPaintingRenderInfo containing the drawing operation in its Operation property (any combination of the PathPaintingRenderInfo constants STROKE and FILL), the rule for determining what "inside the path" means in its Rule property (NONZERO_WINDING_RULE or EVEN_ODD_RULE), and some other drawing details in the Ctm, LineWidth, LineCapStyle, LineJoinStyle, MiterLimit, and LineDashPattern properties.

try to implement IRenderListener
internal class ImageExtractor : IRenderListener
{
private int _currentPage = 1;
private int _imageCount = 0;
private readonly string _outputFilePrefix;
private readonly string _outputFolder;
private readonly bool _overwriteExistingFiles;
private ImageExtractor(string outputFilePrefix, string outputFolder, bool overwriteExistingFiles)
{
_outputFilePrefix = outputFilePrefix;
_outputFolder = outputFolder;
_overwriteExistingFiles = overwriteExistingFiles;
}
/// <summary>
/// Extract all images from a PDF file
/// </summary>
/// <param name="pdfPath">Full path and file name of PDF file</param>
/// <param name="outputFilePrefix">Basic name of exported files. If null then uses same name as PDF file.</param>
/// <param name="outputFolder">Where to save images. If null or empty then uses same folder as PDF file.</param>
/// <param name="overwriteExistingFiles">True to overwrite existing image files, false to skip past them</param>
/// <returns>Count of number of images extracted.</returns>
public static int ExtractImagesFromFile(string pdfPath, string outputFilePrefix, string outputFolder, bool overwriteExistingFiles)
{
// Handle setting of any default values
outputFilePrefix = outputFilePrefix ?? System.IO.Path.GetFileNameWithoutExtension(pdfPath);
outputFolder = String.IsNullOrEmpty(outputFolder) ? System.IO.Path.GetDirectoryName(pdfPath) : outputFolder;
var instance = new ImageExtractor(outputFilePrefix, outputFolder, overwriteExistingFiles);
using (var pdfReader = new PdfReader(pdfPath))
{
if (pdfReader.IsEncrypted())
throw new ApplicationException(pdfPath + " is encrypted.");
var pdfParser = new PdfReaderContentParser(pdfReader);
while (instance._currentPage <= pdfReader.NumberOfPages)
{
pdfParser.ProcessContent(instance._currentPage, instance);
instance._currentPage++;
}
}
return instance._imageCount;
}
#region Implementation of IRenderListener
public void BeginTextBlock() { }
public void EndTextBlock() { }
public void RenderText(TextRenderInfo renderInfo) { }
public void RenderImage(ImageRenderInfo renderInfo)
{
if (_imageCount == 0)
{
var imageObject = renderInfo.GetImage();
var imageFileName = _outputFilePrefix + _imageCount; //to get multiple file (you should add .jpg or .png ...)
var imagePath = System.IO.Path.Combine(_outputFolder, imageFileName);
if (_overwriteExistingFiles || !File.Exists(imagePath))
{
var imageRawBytes = imageObject.GetImageAsBytes();
//create a new file ()
File.WriteAllBytes(imagePath, imageRawBytes);
}
}
_imageCount++;
}
#endregion // Implementation of IRenderListener
}

How to Convert a WPF inch unit to Winforms pixels and vice versa?

I have a window that designed at WPF and I used that on center of a WinForms owner.
Now, I want to move the owner form and at the moment my WPF window has also to be moved in the center of the form!
But I have a problem, Only when window is in the center of the form that form in the center of the screen. And otherwise act in a different form than the Windows coordinates.
I just add the displacement values of the form to window location.
Now I have come to the conclusion that the coordinates of the pixels on WPF Windows are different by WinForms!
How to convert WPF window location to WinForms base location and vice versa ?
Owner Form codes is:
public partial class Form1 : Form
{
private WPF_Window.WPF win;
public Form1()
{
InitializeComponent();
win = new WPF();
win.Show();
CenterToParent(win);
}
private void CenterToParent(System.Windows.Window win)
{
win.Left = this.Left + (this.Width - win.Width) / 2;
win.Top = this.Top + (this.Height - win.Height) / 2;
}
protected override void OnMove(EventArgs e)
{
base.OnMove(e);
CenterToParent(win);
}
}

Best way to get DPI value in WPF
Method 1
It’s the same way you did that in Windows Forms. System.Drawing.Graphics object provides convenient properties to get horizontal and vertical DPI. Let’s sketch up a helper method:
/// <summary>
/// Transforms device independent units (1/96 of an inch)
/// to pixels
/// </summary>
/// <param name="unitX">a device independent unit value X</param>
/// <param name="unitY">a device independent unit value Y</param>
/// <param name="pixelX">returns the X value in pixels</param>
/// <param name="pixelY">returns the Y value in pixels</param>
public void TransformToPixels(double unitX,
double unitY,
out int pixelX,
out int pixelY)
{
using (Graphics g = Graphics.FromHwnd(IntPtr.Zero))
{
pixelX = (int)((g.DpiX / 96) * unitX);
pixelY = (int)((g.DpiY / 96) * unitY);
}
// alternative:
// using (Graphics g = Graphics.FromHdc(IntPtr.Zero)) { }
}
You can use it transforms both coordinates as well as size values. It’s pretty simple and robust and completely in managed code (at least as far as you, the consumer, is concerned). Passing IntPtr.Zero as HWND or HDC parameter results in a Graphics object that wraps a device context of the entire screen.
There is one problem with this approach though. It has a dependency on Windows Forms/GDI+ infrastructure. You are going to have to add a reference to System.Drawing assembly. Big deal? Not sure about you, but for me this is an issue to avoid.
Method 2
Let’s take it one step deeper and do it the Win API way. GetDeviceCaps function retrieves various information for the specified device and is able to retrieve horizontal and vertical DPI’s when we pass it LOGPIXELSX and LOGPIXELSY parameters respectively.
GetDeviceCaps function is defined in gdi32.dll and is probably what System.Drawing.Graphics uses under the hood.
Let’s have a look at what our helper has become:
[DllImport("gdi32.dll")]
public static extern int GetDeviceCaps(IntPtr hDc, int nIndex);
[DllImport("user32.dll")]
public static extern IntPtr GetDC(IntPtr hWnd);
[DllImport("user32.dll")]
public static extern int ReleaseDC(IntPtr hWnd, IntPtr hDc);
public const int LOGPIXELSX = 88;
public const int LOGPIXELSY = 90;
/// <summary>
/// Transforms device independent units (1/96 of an inch)
/// to pixels
/// </summary>
/// <param name="unitX">a device independent unit value X</param>
/// <param name="unitY">a device independent unit value Y</param>
/// <param name="pixelX">returns the X value in pixels</param>
/// <param name="pixelY">returns the Y value in pixels</param>
public void TransformToPixels(double unitX,
double unitY,
out int pixelX,
out int pixelY)
{
IntPtr hDc = GetDC(IntPtr.Zero);
if (hDc != IntPtr.Zero)
{
int dpiX = GetDeviceCaps(hDc, LOGPIXELSX);
int dpiY = GetDeviceCaps(hDc, LOGPIXELSY);
ReleaseDC(IntPtr.Zero, hDc);
pixelX = (int)(((double)dpiX / 96) * unitX);
pixelY = (int)(((double)dpiY / 96) * unitY);
}
else
throw new ArgumentNullException("Failed to get DC.");
}
So we have exchanged a dependency on managed GDI+ for the dependency on fancy Win API calls. Is that an improvement? In my opinion yes, as long as we run on Windows Win API is a least common denominator. It is lightweight. On other platforms we wouldn’t probably have this dilemma in the first place.
And don’t get fooled by that ArgumentNullException. This solution is as robust as the first one. System.Drawing.Graphics will throw this same exception if it can’t obtain a device context too.
Method 3
As officially documented here there is a special key in the registry: HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\FontDPI. It stores a DWORD value which is exactly what the user chooses for DPI in the display settings dialog (it’s called a font size there).
Reading it is a no-brainer but I wouldn’t recommend it. You see there is a difference between an official API and a storage for various settings. The API is a public contract that stays the same even if the internal logic is totally rewritten (If it doesn’t the whole platform sucks, doesn’t it?).
But nobody guarantees that the internal storage will remain the same. It may have lasted for a couple of decades but a crucial design document that describes its relocation may already be pending an approval. You never know.
Always stick to API (whatever it is, native, Windows Forms, WPF, etc). Even if the underlying code reads the value from the location you know.
Method 4
This is a pretty elegant WPF approach that I’ve found documented in this blog post. It is based on the functionality provided by System.Windows.Media.CompositionTarget class that ultimately represents the display surface on which the WPF application is drawn. The class provides 2 useful methods:
TransformFromDevice
TransformToDevice
The names are self-explanatory and in both cases we get a System.Windows.Media.Matrix object that contains the mapping coefficients between device units (pixels) and independent units. M11 will contain a coefficient for the X axis and M22 – for the Y axis.
As we have been considering a units->pixels direction so far let’s re-write our helper with CompositionTarget.TransformToDevice. When calling this method M11 and M22 will contain values that we calculated as:
dpiX / 96
dpiY / 96
So on a machine with DPI set to 120 the coefficients will be 1.25.
Here’s the new helper:
/// <summary>
/// Transforms device independent units (1/96 of an inch)
/// to pixels
/// </summary>
/// <param name="visual">a visual object</param>
/// <param name="unitX">a device independent unit value X</param>
/// <param name="unitY">a device independent unit value Y</param>
/// <param name="pixelX">returns the X value in pixels</param>
/// <param name="pixelY">returns the Y value in pixels</param>
public void TransformToPixels(Visual visual,
double unitX,
double unitY,
out int pixelX,
out int pixelY)
{
Matrix matrix;
var source = PresentationSource.FromVisual(visual);
if (source != null)
{
matrix = source.CompositionTarget.TransformToDevice;
}
else
{
using (var src = new HwndSource(new HwndSourceParameters()))
{
matrix = src.CompositionTarget.TransformToDevice;
}
}
pixelX = (int)(matrix.M11 * unitX);
pixelY = (int)(matrix.M22 * unitY);
}
I had to add one more parameter to the method, the Visual. We need it as a base for calculations (previous samples used the device context of the entire screen for that). I don’t think it’s a big issue as you are more than likely to have a Visual at hand when running your WPF application (otherwise, why would you need to translate pixel coordinates?). However, if your visual hasn't been attached to a presentation source (that is, it hasn't been shown yet) you can't get the presentation source (thus, we have a check for NULL and construct a new HwndSource).
Reference

I've just uncovered and tested this (in VB):
formVal = Me.LogicalToDeviceUnits(WPFval)
formVal and WPFval can be either Integers or Sizes.

How do I draw the outline of a collection of rectangles?

As part of a project I'm working on I have to store and restore magic wand regions from an image. To obtain the data for storage, I'm utilizing the GetRegionData method. As the specification specifies, this method:
Returns a RegionData that represents the information that describes this Region.
I store the byte[] kept in the RegionData.Data property in a base64 string, so I can retrieve the RegionData later through a somewhat unconventional method:
// This "instantiates" a RegionData object by simply initiating an object and setting the object type pointer to the specified type.
// Constructors don't run, but in this very specific case - they don't have to. The RegionData's only member is a "data" byte array, which we set right after.
var regionData =
(RegionData)FormatterServices.GetUninitializedObject(typeof(RegionData));
regionData.Data = bytes;
I then create a Region and pass the above RegionData object in the constructor and call GetRegionScans to obtain the rectangle objects which comprise the region:
var region = new Region(regionData);
RectangleF[] rectangles = region.GetRegionScans(new Matrix());
This way I end up with a collection of rectangles that I use to draw and reconstruct the region. I have isolated the entire drawing process to a WinForms application, and I'm using the following code to draw this collection of rectangles on an image control:
using (var g = Graphics.FromImage(picBox.Image))
{
var p = new Pen(Color.Black, 1f);
var alternatePen = new Pen(Color.BlueViolet, 1f);
var b = new SolidBrush(picBox.BackColor);
var niceBrush = new SolidBrush(Color.Orange);
foreach (var r in rectangles)
{
g.DrawRectangle(p,
new Rectangle(new Point((int)r.Location.X, (int)r.Location.Y),
new Size((int)r.Width, (int)r.Height)));
}
}
The above code results in the following being rendered in my picture control:
The outline here is correct - that's exactly what I originally marked with my magic wand tool. However, as I'm drawing rectangles, I also end up with horizontal lines, which weren't a part of the original magic wand selection. Because of this, I can't view the actual image anymore, and my wand tool now makes the image useless.
I figured I'd only draw the left and right edges of each of the rectangles on the screen, so I'd end up with a bunch of points on the screen which outline the image for me. To do this, I tried the following code:
var pointPair = new[]
{
new Point((int) r.Left, (int) r.Y),
new Point((int) r.Right, (int) r.Y)
};
g.DrawRectangle(p, pointPair[0].X, pointPair[0].Y, 1, 1);
g.DrawRectangle(p, pointPair[1].X, pointPair[1].Y, 1, 1);
And the result can be observed below:
Though closer, it's still no cigar.
I'm looking for a way to connect the dots in order to create an outline of the rectangles in my region. Something that's trivial for humans to do, but I can not figure out for the life of me how to instruct a computer to do this for me.
I've already tried creating new points from each of the rectangles by calculating the most adjacent points on both the Left and Right points of each rectangle, and rendering those as individual rectangles, but to no avail.
Any help would be greatly appreciated, as I'm really at a loss here.
Thanks!
Solution
Thanks to Peter Duniho's answer, I managed to solve this problem. I'm including the SafeHandle wrapper classes and the code I've used to make this work below, for the sake of completeness.
Drawing code
The code below is what I've used to draw the region outline.
private void DrawRegionOutline(Graphics graphics, Color color, Region region)
{
var regionHandle = new SafeRegionHandle(region, region.GetHrgn(graphics));
var deviceContext = new SafeDeviceContextHandle(graphics);
var brushHandle = new SafeBrushHandle(color);
using (regionHandle)
using (deviceContext)
using (brushHandle)
FrameRgn(deviceContext.DangerousGetHandle(), regionHandle.DangerousGetHandle(), brushHandle.DangerousGetHandle(), 1, 1);
}
SafeHandleNative
Small wrapper around SafeHandleZeroOrMinusOneIsInvalid to ensure cleanup after we're done with the handles.
[HostProtection(MayLeakOnAbort = true)]
[SuppressUnmanagedCodeSecurity]
public abstract class SafeNativeHandle : SafeHandleZeroOrMinusOneIsInvalid
{
#region Platform Invoke
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
protected internal static extern bool CloseHandle(IntPtr hObject);
#endregion
/// <summary>
/// Initializes a new instance of the <see cref="SafeNativeHandle"/> class.
/// </summary>
protected SafeNativeHandle() : base(true)
{}
/// <summary>
/// Initializes a new instance of the <see cref="SafeNativeHandle"/> class.
/// </summary>
/// <param name="handle">The handle.</param>
protected SafeNativeHandle(IntPtr handle)
: base(true)
{
SetHandle(handle);
}
}
I've created three other wrappers, one for the Region object, one for the Brush and one for the device context handles. These all inherit from SafeNativeHandle, but in order not to spam I'll only provide the one I've used for the region below. The other two wrappers are virtually identical, but use the respective Win32 API required to clean up their own resources.
public class SafeRegionHandle : SafeNativeHandle
{
private readonly Region _region;
/// <summary>
/// Initializes a new instance of the <see cref="SafeRegionHandle" /> class.
/// </summary>
/// <param name="region">The region.</param>
/// <param name="handle">The handle.</param>
public SafeRegionHandle(Region region, IntPtr handle)
{
_region = region;
base.handle = handle;
}
/// <summary>
/// When overridden in a derived class, executes the code required to free the handle.
/// </summary>
/// <returns>
/// true if the handle is released successfully; otherwise, in the event of a catastrophic failure, false. In this case, it generates a releaseHandleFailed MDA Managed Debugging Assistant.
/// </returns>
[ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]
protected override bool ReleaseHandle()
{
try
{
_region.ReleaseHrgn(handle);
}
catch
{
return false;
}
return true;
}
}

I'm still not entirely sure I understand the question. However, it sounds to me as though you simply want to draw the given region by outlining it, rather than filling it.
Unfortunately, as far as I know the .NET API does not support this. However, the native Windows API does. Here is some code that should do what you want:
[DllImport("gdi32")]
static extern bool FrameRgn(System.IntPtr hDC, System.IntPtr hRgn, IntPtr hBrush, int nWidth, int nHeight);
[DllImport("gdi32")]
static extern IntPtr CreateSolidBrush(uint colorref);
[DllImport("gdi32.dll")]
static extern bool DeleteObject([In] IntPtr hObject);
[StructLayout(LayoutKind.Explicit)]
struct COLORREF
{
[FieldOffset(0)]
public uint colorref;
[FieldOffset(0)]
public byte red;
[FieldOffset(1)]
public byte green;
[FieldOffset(2)]
public byte blue;
public COLORREF(Color color)
: this()
{
red = color.R;
green = color.G;
blue = color.B;
}
}
void DrawRegion(Graphics graphics, Color color, Region region)
{
COLORREF colorref = new COLORREF(color);
IntPtr hdc = IntPtr.Zero, hbrush = IntPtr.Zero, hrgn = IntPtr.Zero;
try
{
hrgn = region.GetHrgn(graphics);
hdc = graphics.GetHdc();
hbrush = CreateSolidBrush(colorref.colorref);
FrameRgn(hdc, hrgn, hbrush, 1, 1);
}
finally
{
if (hrgn != IntPtr.Zero)
{
region.ReleaseHrgn(hrgn);
}
if (hbrush != IntPtr.Zero)
{
DeleteObject(hbrush);
}
if (hdc != IntPtr.Zero)
{
graphics.ReleaseHdc(hdc);
}
}
}
Call the DrawRegion() method from your Paint event handler or other appropriate context where you have a Graphics instance, such as drawing into an Image object as in your example.
Obviously you could make this an extension method for more convenience. Also, while in this example I am dealing with the initialization and releasing of the handles directly, a better implementation would wrap the handles in appropriate SafeHandle subclasses, so that you can conveniently use using instead of try/finally, and to get the backup of finalization (in case you forget to dispose).

Using a method to get Embedded Font Causes Protected Memory Error

I am using this code to get an embedded font:
/// <summary>
/// Returns an Embedded Font
/// </summary>
/// <param name="ImagePath">String begins with namespace e.g MyProgram.Image.png</param>
/// <returns></returns>
public static Font GetEmbeddedFont(string FontPath, float Size)
{
Font _font = null;
Thread getFontThread = new Thread(() => GetFont(FontPath, Size, out _font));
getFontThread.Start();
getFontThread.Join();
return _font;
}
#region GetFont
private static void GetFont(string FontPath, float Size, out Font FontOut)
{
Font fnt = null;
Assembly asm = Assembly.GetExecutingAssembly();
Stream resStream = asm.GetManifestResourceStream(FontPath);
if (null != resStream)
{
//
// GDI+ wants a pointer to memory, GDI wants the memory.
// We will make them both happy.
//
// First read the font into a buffer
byte[] rgbyt = new Byte[resStream.Length];
resStream.Read(rgbyt, 0, rgbyt.Length);
resStream.Close();
// Then do the unmanaged font (Windows 2000 and later)
// The reason this works is that GDI+ will create a font object for
// controls like the RichTextBox and this call will make sure that GDI
// recognizes the font name, later.
uint cFonts;
AddFontMemResourceEx(rgbyt, rgbyt.Length, IntPtr.Zero, out cFonts);
// Now do the managed font
IntPtr pbyt = Marshal.AllocCoTaskMem(rgbyt.Length);
if (null != pbyt)
{
Marshal.Copy(rgbyt, 0, pbyt, rgbyt.Length);
m_pfc = new PrivateFontCollection();
m_pfc.AddMemoryFont(pbyt, rgbyt.Length);
Marshal.FreeCoTaskMem(pbyt);
}
}
if (m_pfc.Families.Length > 0)
{
// Handy how one of the Font constructors takes a
// FontFamily object, huh? :-)
fnt = new Font(m_pfc.Families[0], Size);
}
m_pfc.Dispose();
FontOut = fnt;
}
I use a Thread to attempt to wait for it to finish, because the error seems to occur, if this method is called multiple times within a short space of time.
How can I stop this error occuring, I think it has something to do with the method being called within quick succession of each other.
I get the exception here:
_font = value;
using (Graphics g = _parent.CreateGraphics())
{
SizeF soize = g.MeasureString(_text, _font);
_size = new Size((int)soize.Width, (int)soize.Height);
_width = _size.Width;
_height = _size.Height;
}
On the line g.MeasureString(_text, _font);
However I know that the error is in the GetEmbeddedFont Method, as it only throws an error if the font is set using the GetEmbeddedFont method.
It will work fine once, but if it is used a second time to shortly after the first, it will throw the error.
And if I debug the code, _font returns this:
{Name = '((System.Drawing.Font)(_font)).fontFamily.Name' threw an exception of type 'System.ArgumentException' Size=15.0}

There's a documentation problem with AddMemoryFont(), it doesn't specify how long the pointer needs to remain valid. I've always chosen the conservative route and made sure to not release the memory until after the program stops using the private font. That has worked well, no access violations.
I thus strongly recommend you remove the Marshal.FreeCoTaskMem() call. Either entirely, Windows automatically cleans up when the program terminates, or move it to, say, the FormClosed event. The same goes for your m_pfc.Dispose() call. Don't release resources until you're done with them.

.NET and Direct2D rapid draw crash

General purpose
I am developing a .NET user control that can be used for multipurpose graphics rendering using Direct2D. This control is based on the ID2D1HwndRenderTarget interface with a ID2D1BitmapRenderTarget back-buffer/double-buffer.
Background
I have written a wrapper using C++/CLI that exposes a set of managed wrapper objects for Direct2D, and I have a designed a Singleton resource manager for Direct2D bitmaps that code can send to the binary data to, and get a resource key back from, which code can then pass in a message to the user control to render a bitmap.
Usage so far
I have been able to load bitmaps, JPEGs, custom imaging formats and the like and send them to the Direct2D render control all with no issue. I've written a custom video decoder that could load frames from a 15-fps video (decoding in a background thread) and render them from events raised by a Win32 multimedia timer, no problem.
Issue
My issue is that when I attempted to expand out from a single multimedia format into something more flexible (specifically connecting to LibAV/ffmpeg), the Direct2D render control starts to crash display drivers. This does not happen when frames are rendered sequentially (using a button to render the next frame, rather than a timer). It also does not happen straightaway. It also does not happen when I block inside the Win32 timer callback, but does when I use a Mutex to raise a thread that will free the timer callback up and let the interim thread instruct the control to render.
Symptoms
If I start a timer and attach to its elapsed event methods that will cause the next render, it will typically play fine for around 2-20 seconds. Using smaller video, I can get longer playback before the issue starts. If I play back 1080p video, I can get the crash usually within 5 seconds without contest. Playback will start fine, maybe have a hiccup here or there before catching up.
Eventually, I will start to see flickering, as if the background color is rendered but the frame is not. Additional frames may or may not later render, but this is very short-lived.
After the flicker, if I stop the timer quickly enough there will be no crash.
If I do not stop in time:
If I'm lucky (maybe 10% of the time), frames will stop rendering, be stuck on one frame until I resize the control, at which point only black is drawn to the entire control. (I've read that this may indicate a lost device context for DirectX rendering, but have not seen much more.)
If I'm unlucky, the display driver will crash, Windows recovers and restarts the driver, after which point the EndDraw will finally tell me that an error has occurred and return D2DERR_RECREATE_TARGET.
If I am really unlucky, the display will start to look like a kaleidoscope and I'll have to power off my machine, and I just wasted 5 minutes booting, logging on, loading Visual Studio and my solution, loading the video and lost all my debugging data.
I want to think that there is some sort of race condition that I am missing, but every time I run through the rendering code, it appears that it should be properly locked.
Render Control's Code
using System;
using System.Drawing;
using System.Windows.Forms;
using Direct2D = Bardez.Projects.DirectX.Direct2D;
using ExternalPixelEnums = Bardez.Projects.FileFormats.MediaBase.Video.Pixels.Enums;
using Bardez.Projects.DirectX.Direct2D;
using Bardez.Projects.FileFormats.MediaBase.Video;
using Bardez.Projects.FileFormats.MediaBase.Video.Enums;
using Bardez.Projects.Win32;
namespace Bardez.Projects.Output.Visual
{
/// <summary>Represents a rendering target for Direct2D.</summary>
/// <remarks>
/// To use this control, an external component will need to supply bitmap data. This does not need to be a GDI+ Bitmap class.
/// However, the container for this control will need to push data into this control.
/// So, in the case of a movie player, we'd see the following model:
/// * decompress a frame.
/// * push frame to control
/// * invoke Invalidate
/// * control will render the bitmap
/// * sleep just a little bit
/// * go back to first step
/// </remarks>
public class Direct2dRenderControl : VisualRenderControl
{
/*
* Locking orientation:
* There are two rendering targets: a GDI display and a bitmap back buffer.
* There are 5 'real' locking operations:
* Rendering to the GDI display (OnPaint)
* Rendering to the back buffer (DrawBitmapToBuffer, DiscardCurrentBuffer)
* Setting the current displayed frame (SetRenderFrame)
* Resource Freeing (FreeFrameResource)
* Resizing (OnResize)
*
* Briefly, the overarching effects of these five are:
* Rendering
* Utilizes the buffer and the display
* Back Buffer
* Utilizes the buffer and if mid-render could affect this display
* Set Frame
* Sets the buffer's displayed image
* Resource Freeing
* Affects the buffer if a resource is refernced
* Resizing
* Resizes the render control and the bitmap, uses the frame set by set
*
* Locking plan:
* Resize should block set, free and back buffer
* Render should block back buffer, control
* Set frame should block resize and back buffer
* Free should block back buffer,
* Back buffer should block rendering, setting, resizing, freeing
*
* Basically, lock everything at the process level, and not at the access level.
*/
#region Fields
/// <summary>Represents a Win32 HWND render target for Direct2D</summary>
private ControlRenderTarget ctrlRenderTarget;
/// <summary>Represents a drawing buffer</summary>
private BitmapRenderTarget bmpRenderTarget;
/// <summary>Represents a drawing buffer, used solely to create bitmaps</summary>
private BitmapRenderTarget resourceBmpRenderTarget;
/// <summary>Represents a buffer drawing command lock</summary>
private Object controlBufferLock;
/// <summary>Represents a paint/render drawing command lock</summary>
private Object controlPaintRenderLock;
/// <summary>Locking object reference for resource management (memory bitmaps, etc.)</summary>
private Object resourceLock;
/// <summary>Represents the key to accessing the currently set key</summary>
protected Int32 currentFrameKey;
#endregion
#region Properties
/// <summary>Indicates whether there is a frame set for this control</summary>
protected Boolean HasFrameSet
{
get { return currentFrameKey > -1; }
}
/// <summary>Exposes a wrapper for the bitmap render target</summary>
protected BitmapRenderTarget BmpRenderTarget
{
get { return this.bmpRenderTarget; }
set
{
lock (this.controlBufferLock)
{
if (this.bmpRenderTarget != null)
this.bmpRenderTarget.Dispose();
this.bmpRenderTarget = value;
}
}
}
/// <summary>Exposes a wrapper for the bitmap render target</summary>
protected BitmapRenderTarget ResourceBmpRenderTarget
{
get { return this.resourceBmpRenderTarget; }
set
{
lock (this.resourceLock)
{
if (this.resourceBmpRenderTarget != null)
this.resourceBmpRenderTarget.Dispose();
this.resourceBmpRenderTarget = value;
}
}
}
/// <summary>Represents a Win32 HWND render target for Direct2D</summary>
protected ControlRenderTarget CtrlRenderTarget
{
get { return this.ctrlRenderTarget; }
set
{
lock (this.controlPaintRenderLock)
{
if (this.ctrlRenderTarget != null)
this.ctrlRenderTarget.Dispose();
this.ctrlRenderTarget = value;
}
}
}
#endregion
#region Construction
/// <summary>Default constructor</summary>
public Direct2dRenderControl() : base()
{
this.controlBufferLock = new Object();
this.controlPaintRenderLock = new Object();
this.resourceLock = new Object();
this.currentFrameKey = -1;
this.InitializeControlDirect2D();
}
/// <summary>Initializes the Direct2D</summary>
protected void InitializeControlDirect2D()
{
//disable Windows background draw
this.SetStyle(ControlStyles.UserPaint | ControlStyles.AllPaintingInWmPaint, true);
// Build options on the Control render target
PixelFormat format = new PixelFormat(DXGI_ChannelFormat.FORMAT_B8G8R8A8_UNORM, AlphaMode.Unknown); //32-bit color, pure alpha
DpiResolution res = Direct2dResourceManager.Instance.Factory.GetDesktopDpi();
RenderTargetProperties rtProp = new RenderTargetProperties(RenderTargetType.Default, format, res, RenderTargetUsage.GdiCompatible, DirectXVersion.DirectX9);
//Build out control render target properties
HwndRenderTargetProperties hwndProp = new HwndRenderTargetProperties(this.Handle, new SizeU(this.Size), PresentationOptions.RetainContents);
lock (this.controlPaintRenderLock)
{
// populate the Control rendering target
ResultCode result = Direct2dResourceManager.Instance.Factory.CreateHwndRenderTarget(rtProp, hwndProp, out this.ctrlRenderTarget);
lock (this.controlBufferLock)
{
// create a bitmap rendering targets
this.CtrlRenderTarget.CreateCompatibleRenderTarget(out this.bmpRenderTarget);
lock (this.resourceLock)
this.CtrlRenderTarget.CreateCompatibleRenderTarget(out this.resourceBmpRenderTarget);
}
}
}
#endregion
#region Destruction
/// <summary>Disposal code; releases unmanaged resources</summary>
/// <param name="disposing">True indicates to dispose managed resources</param>
protected override void Dispose(Boolean disposing)
{
this.ResourceBmpRenderTarget = null; //property disposes
this.BmpRenderTarget = null; //property disposes
this.CtrlRenderTarget = null; //property disposes
base.Dispose(disposing);
}
/// <summary>Disposal</summary>
~Direct2dRenderControl()
{
this.Dispose();
}
#endregion
#region Event Raising
/// <summary>Draws the output, then raises the paint event</summary>
/// <param name="e">Painting Event arguments</param>
protected override void OnPaint(PaintEventArgs e)
{
lock (this.controlPaintRenderLock)
{
lock (this.controlBufferLock)
{
this.SuspendLayout();
this.OnPaintBackground(e);
Direct2D.Bitmap bmp;
ResultCode result;
result = this.BmpRenderTarget.GetBitmap(out bmp);
Direct2D.RectangleF rect = new Direct2D.RectangleF(e.ClipRectangle);
this.CtrlRenderTarget.BeginDraw();
this.CtrlRenderTarget.DrawBitmap(bmp, rect, 1.0F, BitmapInterpolationMode.Linear, rect);
result = this.CtrlRenderTarget.EndDraw();
bmp.Dispose();
if (result != ResultCode.Success_OK)
throw new ApplicationException(String.Format("Error encountered during draw: '{0}'", result.ToString()));
base.OnPaint(e);
this.ResumeLayout();
}
}
}
/// <summary>Overides the resize method</summary>
/// <param name="e">Parameters for the resize event</param>
protected override void OnResize(EventArgs e)
{
lock (this.controlPaintRenderLock)
{
lock (this.controlBufferLock)
{
//Null check since resize fires before it is constructed, with the size event during parent's InitializeComponent
if (this.CtrlRenderTarget != null)
{
this.SuspendLayout();
//resize the existing control rendering target
this.CtrlRenderTarget.Resize(new SizeU(this.Size));
//I also need to resize the buffer, but can't. Instead, create a new one, then copy the existing one. Kind of lame.
this.ResizeBitmapRenderTarget();
base.OnResize(e);
//cause another draw
this.Invalidate(new Rectangle(new Point(0, 0), this.Size));
this.ResumeLayout();
}
}
}
}
/// <summary>Overridden Paint background method</summary>
/// <param name="e">Paint event arguments</param>
/// <remarks>
/// Made empty to avoid GDI and Direct2D writing to the same control; when moving the control around
/// or rendering at high speeds, the dreaded WinForms flicker was introduced.
/// The background painting can be found in the <see cref="FillBufferRenderTarget"/> method,
/// which fills the bitmap back buffer with the control's background color.
/// </remarks>
protected override void OnPaintBackground(PaintEventArgs e) { }
#endregion
#region Drawing
/// <summary>Resizes the bitmap rendering target buffer</summary>
/// <remarks>Does not lock the GDI render target. The OnResize or other callers lock instead.</remarks>
protected void ResizeBitmapRenderTarget()
{
lock (this.controlPaintRenderLock)
{
lock (this.controlBufferLock)
{
using (BitmapRenderTarget bmpCurr = this.BmpRenderTarget)
{
BitmapRenderTarget bmpNew;
ResultCode result = this.CtrlRenderTarget.CreateCompatibleRenderTarget(out bmpNew);
this.DuplicateDoubleBufferContents(bmpNew);
//Property disposes and locks
this.BmpRenderTarget = bmpNew;
}
}
}
}
/// <summary>Draws a Bitmap to the render buffer</summary>
/// <param name="bmp">Direct2D bitmap to draw to the buffer.</param>
protected void DrawBitmapToBuffer(Direct2D.Bitmap bmp, Point2dF origin)
{
lock (this.controlBufferLock)
{
lock (this.resourceLock)
{
Direct2D.SizeF bmpSize = bmp.GetSize();
Single width = bmpSize.Width > this.BmpRenderTarget.Size.Width ? this.BmpRenderTarget.Size.Width : bmpSize.Width;
Single height = bmpSize.Height > this.BmpRenderTarget.Size.Height ? this.BmpRenderTarget.Size.Height : bmpSize.Height;
Direct2D.RectangleF destRect = new Direct2D.RectangleF(origin.X, origin.X + width, origin.Y, origin.Y + height);
Direct2D.RectangleF srcRect = new Direct2D.RectangleF(0.0F, width, 0.0F, height);
this.BmpRenderTarget.BeginDraw();
this.FillBufferRenderTarget(this.BmpRenderTarget);
// do the actual draw
this.BmpRenderTarget.DrawBitmap(bmp, destRect, 1.0F, BitmapInterpolationMode.Linear, srcRect);
//tell Direct2D that a paint operation is ending
ResultCode result = this.BmpRenderTarget.EndDraw();
}
}
}
/// <summary>Draws a Bitmap to the render buffer</summary>
/// <param name="bmp">Direct2D bitmap to draw to the buffer.</param>
protected void DrawBitmapToBuffer(Direct2D.Bitmap bmp)
{
this.DrawBitmapToBuffer(bmp, new Point2dF(0.0F, 0.0F));
}
/// <summary>Duplicates the bitmap behind the existing rendering target, and drawing it to a new one, discarding the current and setting the new.</summary>
/// <remarks>Does not lock any references, as the outside method locks</remarks>
protected void DuplicateDoubleBufferContents(BitmapRenderTarget bmpNew)
{
Direct2D.Bitmap bmp = null;
ResultCode result = ResultCode.Success_OK;
if (this.HasFrameSet)
bmp = Direct2dResourceManager.Instance.GetBitmapResource(this.currentFrameKey);
else
result = this.BmpRenderTarget.GetBitmap(out bmp);
bmpNew.BeginDraw();
this.FillBufferRenderTarget(bmpNew);
//calculate the size to copy
Direct2D.SizeF bmpSize = bmp.GetSize();
Single width = bmpSize.Width > this.CtrlRenderTarget.Size.Width ? this.CtrlRenderTarget.Size.Width : bmpSize.Width;
Single height = bmpSize.Height > this.CtrlRenderTarget.Size.Height ? this.CtrlRenderTarget.Size.Height : bmpSize.Height;
//Determine the copy rectangle
Direct2D.RectangleF rect = new Direct2D.RectangleF(0, width, 0, height);
//Copy
bmpNew.DrawBitmap(bmp, rect, 1.0F, BitmapInterpolationMode.Linear, rect);
//conditionally disose the bitmap, don't if it is in the manager
if (!this.HasFrameSet)
bmp.Dispose();
result = bmpNew.EndDraw();
}
/// <summary>Discards the current buffer and replaces it with a blank one.</summary>
protected void DiscardCurrentBuffer()
{
lock (this.controlBufferLock)
{
BitmapRenderTarget bmpNew;
// create a bitmap rendering target
ResultCode result = this.CtrlRenderTarget.CreateCompatibleRenderTarget(out bmpNew);
bmpNew.BeginDraw();
this.FillBufferRenderTarget(bmpNew);
result = bmpNew.EndDraw();
//property locks, so no lock here
this.BmpRenderTarget = bmpNew; //replace the old buffer
}
}
/// <summary>Fills the buffer render target with the background color</summary>
/// <param name="renderTarget">Bitmap render target to fill</param>
protected void FillBufferRenderTarget(BitmapRenderTarget renderTarget)
{
SolidColorBrush brush = null;
ResultCode result = renderTarget.CreateSolidColorBrush(new ColorF(this.BackColor), out brush);
renderTarget.FillRectangle(new Direct2D.RectangleF(new Rectangle(new Point(0, 0), this.Size)), brush);
brush.Dispose();
}
#endregion
#region Frame Resources
/// <summary>Posts a Frame resource to the resource manager and returns a unique key to access it.</summary>
/// <param name="resource">Frame to be posted.</param>
/// <returns>A unique Int32 key</returns>
public override Int32 AddFrameResource(Frame resource)
{
lock (this.resourceLock)
{
//create the bitmap
BitmapProperties properties = new BitmapProperties(new PixelFormat(DXGI_ChannelFormat.FORMAT_B8G8R8A8_UNORM, AlphaMode.PreMultiplied), Direct2dResourceManager.Instance.Factory.GetDesktopDpi());
SizeU dimensions = new SizeU(Convert.ToUInt32(resource.Pixels.Metadata.Width), Convert.ToUInt32(resource.Pixels.Metadata.Height));
Direct2D.Bitmap bmp = null;
ResultCode result = this.ResourceBmpRenderTarget.CreateBitmap(dimensions, properties, out bmp);
Byte[] data = resource.Pixels.GetPixelData(ExternalPixelEnums.PixelFormat.RGBA_B8G8R8A8, ScanLineOrder.TopDown, 0, 0);
result = bmp.CopyFromMemory(new RectangleU(dimensions), data, Convert.ToUInt32(resource.Pixels.Metadata.Width * 4));
return Direct2dResourceManager.Instance.AddFrameResource(bmp);
}
}
/// <summary>Frees a Bitmap resource in the resource manager and Disposes of it.</summary>
/// <param name="frameKey">Direct2D Bitmap key to be Disposed.</param>
public override void FreeFrameResource(Int32 frameKey)
{
lock (this.resourceLock)
{
if (frameKey > -1)
Direct2dResourceManager.Instance.FreeFrameResource(frameKey);
}
}
#endregion
#region Frame Setting
/// <summary>Sets the frame to be rendered to the User Control</summary>
/// <param name="key">Frame key to set as current image</param>
public override void SetRenderFrame(Int32 key)
{
this.SetRenderFrame(key, 0, 0);
}
/// <summary>Sets the frame to be rendered to the User Control</summary>
/// <param name="key">Frame key to set as current image</param>
/// <param name="originX">X coordinate to start drawing from</param>
/// <param name="originY">Y coordinate to start drawing from</param>
public override void SetRenderFrame(Int32 key, Int64 originX, Int64 originY)
{
lock (this.controlBufferLock)
{
if (key > -1)
this.DrawBitmapToBuffer(Direct2dResourceManager.Instance.GetBitmapResource(key), new Point2dF(Convert.ToSingle(originX), Convert.ToSingle(originY)));
else
this.DiscardCurrentBuffer();
this.currentFrameKey = key;
}
}
/// <summary>This method invokes the rendering. For use by the appliation to tell the control to change images on demand.</summary>
public void Render()
{
this.Invalidate();
}
/// <summary>Sets the frame to be rendered to the User Control and then renders it</summary>
/// <param name="key">Frame key to set as current image</param>
public virtual void SetRenderFrameAndRender(Int32 key)
{
this.SetRenderFrameAndRender(key, false);
}
/// <summary>Sets the frame to be rendered to the User Control and then renders it</summary>
/// <param name="key">Frame key to set as current image</param>
/// <param name="freePreviousFrame">
/// Flag indicating whether to dispose of the previous image set
/// (in the case of transient images, such as composite images for a game or high-frame video playback)
/// </param>
public virtual void SetRenderFrameAndRender(Int32 key, Boolean freePreviousFrame)
{
this.SetRenderFrameAndRender(key, 0, 0, freePreviousFrame);
}
/// <summary>Sets the frame to be rendered to the User Control and then renders it</summary>
/// <param name="key">Frame key to set as current image</param>
/// <param name="originX">X coordinate to start drawing from</param>
/// <param name="originY">Y coordinate to start drawing from</param>
public virtual void SetRenderFrameAndRender(Int32 key, Int64 originX, Int64 originY)
{
this.SetRenderFrameAndRender(key, originX, originY, false);
}
/// <summary>Sets the frame to be rendered to the User Control and then renders it</summary>
/// <param name="key">Frame key to set as current image</param>
/// <param name="originX">X coordinate to start drawing from</param>
/// <param name="originY">Y coordinate to start drawing from</param>
/// <param name="freePreviousFrame">
/// Flag indicating whether to dispose of the previous image set
/// </param>
public virtual void SetRenderFrameAndRender(Int32 key, Int64 originX, Int64 originY, Boolean freePreviousFrame)
{
lock (this.controlBufferLock)
{
Int32 previousFrameKey = this.currentFrameKey;
this.SetRenderFrame(key, originX, originY);
this.Render();
if (freePreviousFrame)
this.FreeFrameResource(previousFrameKey);
}
}
#endregion
}
}
Win32 Multimedia Timer Callback's Code
Excluded for size, but here is the relevant code, linking to "winmm.dll", EntryPoint = "timeSetEvent"
/// <summary>Callback method for WIn32 API</summary>
protected virtual void Win32Callback(UInt32 timerId, UInt32 message, IntPtr user, IntPtr param1, IntPtr param2)
{
TimeSpan raise;
lock (this.timerLock)
{
//get system time for start time
UInt32 uptime = Win32MultimediaTimeFunctions.GetEnvironmentUptime();
Int32 upTimeSec = (Int32)(uptime / 1000);
Int32 upTimeMilliSec = (Int32)(uptime % 1000);
TimeSpan WinUpTime = new TimeSpan(0, 0, 0, upTimeSec, upTimeMilliSec);
raise = WinUpTime - this.StartTime;
//this.elapsed(raise);
this.RaiseTimer(raise);
}
//Note: outside the lock, this.elapse(raise) kills the display driver
//inside, without threading, it is fine.
//inside, with threading, the display driver will crash after a bit of playback
}
protected void RaiseTimer(TimeSpan elapsedTime)
{
Thread timerEvent = new Thread(() => { this.RunThread(elapsedTime); });
timerEvent.IsBackground = true;
timerEvent.Name = "Timer callback";
timerEvent.Start();
}
/// <summary>Raises the timer on a separate thread</summary>
protected void RunThread(TimeSpan elapsedTime)
{
if (this.threadLock.WaitOne(0))
{
this.elapsed(elapsedTime);
this.threadLock.ReleaseMutex();
}
}
Conclusion
After 3 weeks, I'm not ashamed to say I'm baffled by this behavior, and looking for any sort of input for identifying a race condition, something that I might be missing, architecturally speaking, about Direc2D or where I'm just showing my ignorance.

All locks statements in the code seems to be an over complicated solution. Also creating COM objects (like Direct2D1) from constructors and using them from thread UI methods is usually not a good practice.
I would simplify your application by performing all device/objects creation and rendering on the same thread, also by keeping your rendering code outside the form. Ideally, this should be totally independent, as you could render to any kind of windows/controls via the HWND or to a DXGI surface, or to a Windows 8 Metro surface.
Also, as Aren suggest, instead of using your own Direct2D1 wrapper, you should rely on an existing robust wrapper like SharpDX. It will save you to maintain a costly layer and help you to focus on your specific application.