We Keep Coding

Why are C# callbacks only executed when the python module uses cv.imshow()?

I couldn't come up with a better more descriptive title as it involves 3 languages which I'll explain now.
I wrote a C++ wrapper around a Python module, which works just fine in C++ by the way. I made a DLL out of this wrapper and exposed some functionalities as a C and used them in a C# application.
The issue is, the C# application just hangs if I do not display the webcam feed.
That is in Python module there is this condition:
if self.debug_show_feed:
cv2.imshow('service core face Capture', frame)
and when set True, will display the webcam feed.
This is mostly a debug thing I put and for actual production it needs to be disabled. On C++ its fine
I can set this to false (through the constructor) and all is fine.
However, On C#, this behavior doesn't happen, if I try to use the module without setting the webcam feed to true, The C# app hangs, and that's because the Start() which calls the main operation becomes a blocking call and none of the callbacks are returned.
my DllImport are as follows by the way:
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern int Initialize(bool showFeed);
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void Start(bool async);
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void Stop();
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void SetCpuAffinity(int mask);
public delegate void CallbackDelegate(bool status, string message);
[MethodImplAttribute(MethodImplOptions.InternalCall)]
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void AddCallback(IntPtr fn);
[DllImport(#"Core_DLL.dll", CallingConvention = CallingConvention.Cdecl)]
public static extern void RemoveCallback(IntPtr fn);
And this is my C# callback:
private CallbackDelegate del;
public void SetUpCallback()
{
txtLog.Text += "Registering C# callback...\r\n";
del = new CallbackDelegate(callback01);
AddCallback(Marshal.GetFunctionPointerForDelegate(del));
txtLog.Text += "Calling passed C++ callback...\r\n";
}
bool status;
string id;
public void callback01(bool status, string id)
{
this.status = status;
this.id = id;
}
And this is the main python modules that are executed :
def start(self):
try:
self.is_running = True
self._main_loop()
except Exception as ex:
path='exceptions-servicecore.log'
track = traceback.format_exc()
exception_time = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
with open(path, 'a') as f:
f.writelines(f'\n{exception_time} : exception occured {ex.args} \n{track}')
def start_async(self):
st = threading.Thread(target=self.start)
st.start()
def _main_loop(self):
name = None
is_valid = False
while self.is_running and self.cap.isOpened():
is_success, frame = self.cap.read()
if is_success:
name="sth"
is_valid=True
self._execute_callbacks(is_valid, name, frame)
self._execute_c_callbacks(is_valid, name)
if self.debug_show_feed:
cv2.imshow('service core face Capture', frame)
if self.save:
self.video_writer.write(frame)
if (cv2.waitKey(1)&0xFF == ord('q')) or (not self.is_running):
break
self.cap.release()
if self.save:
self.video_writer.release()
cv2.destroyAllWindows()
Knowing this only happens in C# and not C++, I may have some issues in my marshaling or the way I'm trying to use the C# callback in this scenario.
Here is a Visual Studio with a minimal example that demonstrates this: https://workupload.com/file/epsgzmMMVMY
What is the problem here? Why is cv.imshow() causes this behavior?

I found the reason why callbacks on C# side didn't output anything. The callbacks are all executing as they should, but since the main loop on Python side is a blocking method, they only start their execution when the blocking method is over (just like a recursive function where the outputs don't get to be returned until the very end).
I then noticed cv2.imshow() creates a short pause and in that time, the C# client gets the chance to update the output and what it has been sent to. I first tried to pause the current running thread in Python, and it actually kind of worked, the output started to popup on C# side, but still the app was not responsive.
I noticed I can actually make the callback outputs show up in C# by simply using a cv2.waitkey(1) or cv2.imread('') in the else clause when showFeed is False:
while (self.is_running):
...
if self.showFeed:
cv2.imshow("image", frame)
else:
#cv2.imread('')
# or
cv2.waitkey(1)
...
And by writing :
while (self.is_running):
...
if self.showFeed:
cv2.imshow("image", frame)
else:
cv2.namedWindow('image', cv2.WINDOW_OPENGL)
cv2.waitKey(1)
cv2.destroyAllWindows()
...
The outputs are showing just fine and the application is responsive again, However, constantly creating and destroying an empty Opencv window is not a solution as it flickers and is just very bad.
I need to add that, using a timer() control event on c# to print the output and keep the app responsive doesn't work, creating a new thread also doesn't seem to work. it seems the marshaled callbacks this way can't be used like this (I am happy to hear that I am wrong though).
I'll update this answer when I find better solutions than these both C# wise (running the callbacks on a thread) or on Python side, making a visible window or altogether fix this issue.
Update
I removed the changes on Python side and implemented the threading on C# part. The reason the initial threading didn't work was because, all interops had to be called in the same thread, meaning all of the imported methods such as Initialize, Start and AddCallback had to be setup and run from within the same thread.

QueryInterface fails with E_NOINTERFACE on C#

Hi Stack Overflow members,
I'm a newbie to C# programming. I am developing a basic camera streaming and still capture application. Once user takes still, I will be displaying it on overlay using VMR9's bitmap mixing concept.
What I did?
I am making use of C# direct show library from here
First I get all required filters interfaces. Find the attached capture device. Called Render stream with source filter and vmr9 for PREVIEW pin. Source filter, sample grabber and null renderer for STILL PIN.
I am having three menu buttons -> take still, show overlay and hide overlay.
I am making use of bitmap mixer sample provided in that library.
Each time user presses Take Still menu, image will be saved in desktop and will be re-sized to small resolution and displayed on video overlay.
Show Overlay and hide overlay calls ShowHideBitmap() which perform operation of querying VMR9BitmapMixer interface from vmr9 filter, fills VMR9AlphaBitmap structure and then calls IVMRMixerBitmap9.SetAlphaBitmap function.
What issue I face?
After taking still, if I call ShowHideBitmap() through menu option, the still image taken is being updated perfectly on overlay.
This is another option that performs automatic update of overlay as soon as still image is saved. I create event based thread and made it to wait for update event created using EventWaitHandle. Before returning from samplegrabber BufferCB function, I set this update event. Which in turn proceeds with waiting thread. Inside thread I call ShowHideBitmap function. In this scenario, I receive error message as follows.
Unable to case COM object of type 'DirectShowLib.VideoMixingRenderer9' to interface type 'DirectShowLib.IVMRMixerBitmap9'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{ced175e5-1935-4820-81bd-ff6ad00c9108}' failed due to the following error: No such interface supported (Exception from HRESULT: 0X80040002 (E_NOINTERFACE)
Here is the code block of ShowHideBitmap function
//Declarations
private static IBaseFilter vmr9 = null;
private static IVMRMixerBitmap9 vmr9mixerBitmap = null;
private IVMRWindowlessControl9 vmr9windowlessCtrl = null;
private static void ShowHideBitmap(Boolean bEnable)
{
int hr = 0;
VMR9AlphaBitmap alphaBmp;
if (!bEnable)
{
if (vmr9mixerBitmap != null)
{
// Get current Alpha Bitmap Parameters
hr = vmr9mixerBitmap.GetAlphaBitmapParameters(out alphaBmp);
DsError.ThrowExceptionForHR(hr);
// Disable them
alphaBmp.dwFlags = VMR9AlphaBitmapFlags.Disable;
// Update the Alpha Bitmap Parameters
hr = vmr9mixerBitmap.UpdateAlphaBitmapParameters(ref alphaBmp);
DsError.ThrowExceptionForHR(hr);
// Create a surface from our alpha bitmap
surface.Dispose();
vmr9mixerBitmap = null;
//Release this alpha bitmap source.
if (alphaBitmap != null)
{
alphaBitmap.Dispose();
}
}
return;
}
else
{
try
{
alphaBitmap = BitmapGenerator.GenerateAlphaBitmap();
// Create a surface from our alpha bitmap
if(surface == null)
surface = new Surface(device, alphaBitmap, Pool.SystemMemory);
// Get the unmanaged pointer
unmanagedSurface = surface.GetObjectByValue(DxMagicNumber);
if (vmr9mixerBitmap == null)
vmr9mixerBitmap = (IVMRMixerBitmap9)vmr9;
// Set Alpha Bitmap Parameters for using a Direct3D surface
alphaBmp = new VMR9AlphaBitmap();
alphaBmp.dwFlags = VMR9AlphaBitmapFlags.EntireDDS;
alphaBmp.pDDS = unmanagedSurface;
alphaBmp.rDest = GetDestRectangle();
alphaBmp.fAlpha = 1.0f;
// Set Alpha Bitmap Parameters
hr = vmr9mixerBitmap.SetAlphaBitmap(ref alphaBmp);
DsError.ThrowExceptionForHR(hr);
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
}
And here is the thread the waits for update event.
Thread overlayupdatethreadhandle = new Thread(new ThreadStart(overlayupdatethread));
overlayupdatethreadhandle.Start();
private void overlayupdatethread()
{
do
{
overlayupdateeventhandle.WaitOne();
ShowHideBitmap(GlobalVar.m_ShowOverlay);
} while (true);
}
I have tried updating this overlay using timer which was running at background with interval of 100ms. Using timer was working good, but for this operation, using timer is of bad choice. So i moved with threading concept.
Why is that getting interface failed while calling from thread and works good when calling from menu options? Should I have to take care of any special thing? I have even tried parametrized thread, but no luck.
Thanks in advance for your help.
EDIT: If ShowHideBitmap is called from Main Thread, every thing works fine. If ShowHideBitmap is called from worker thread, COM object creates Exception. How to handle this cross-thread operation?

The exception is ratty, not uncommon in COM. What it really means is "I have no idea how to give you an interface reference that you can use from a worker thread". Which is a common kind of mishap, these kind of COM components just are not thread-safe at all. And they enforce that by either taking care of it, marshaling the calls from the worker thread to the owner thread automatically. Or by not letting you use them from another thread at all because marshaling would be pointless, making it way too slow. VMR falls in the latter category.
This is very unlike .NET, it also has a lot of classes that are completely thread-unsafe. Basic stuff too, none of the collection classes are for example. But it lets you use these classes in a thread anyway, leaving it up to you to make it thread-safe. This quite often goes wrong of course, using proper locking is a skill.
COM has always been thread-aware by design. With the philosophy that threading is very hard to get right so it should be taken care of by the smart people. Which works fantastically 95% of the time. And gives you a major migraine the rest of the time. The kind of migraine induced by the hard to diagnose poor perf when COM takes care of threading. And the crappy error reporting when it doesn't.
Well, no can do, you really do have to use that interface from the same thread that created the VMR instance. No way around that.

I had error E_NOINTERFACE when trying to use Listener / Event handler object from Delphi library. To overcome issue with marshaling & different threads I saved dispatcher of thread that assigns listener and then use it to fire events.
Interfaces:
[ComVisible(true)]
[Guid("2FFC2C20-A27B-4D67-AEA3-350223D3655F")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IDataSystemInterfaceEventListener
{
void OnIntializeCompleted(int status);
void OnTerminateCompleted(int status);
void OnRunCompleted(int status);
}
[ComVisible(true)]
[Guid("B9953413-A8C9-4CE2-9263-B488CA02E7EC")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
public interface IDataSystemInterface
{
void Intialize(string config);
void StartRun(string conditions);
void StopRun();
void Terminate();
IDataSystemInterfaceEventListener Listener { get; set; }
}
Then implementation (notice Dispatcher.CurrentDispatcher stored)
[ComVisible(true)]
[Guid("0818F830-DA37-4167-BF31-3A2C55A9BF2B")]
public class DataSystemModule : IDataSystemInterface
{
private Dispatcher m_dispatcherListener = null;
private IDataSystemInterfaceEventListener m_listener = null;
public IDataSystemInterfaceEventListener Listener
{
get
{
return m_listener;
}
set
{
m_dispatcherListener = Dispatcher.CurrentDispatcher;
m_listener = value;
}
}
}
Then in code:
if (Listener != null)
{
m_dispatcherListener.Invoke((Action)delegate()
{
Listener.OnTerminateCompleted((int)TerminateStatus.Completed);
});
}
Without dispacther if Listener is called in different thread it will produce error

Exception only occuring on DragDrop

I have a WinForms application that makes use of a TaskDialog library that leverages the Vista style dialogs from ComCtl32.dll and for lesser OS's it uses an emulated win form...
But that's not the problem... This library works fine and we've never had issues with it. Until now... In deed if we launch a dialog under normal circumstances, then it looks fine.
However, I've added a drag-drop handler on my main form to capture file paths dropped from other sources (say Windows Explorer). If that drag-drop handler is the first time the dialog has been shown then we get the following exception:
Unable to find an entry point named 'TaskDialogIndirect' in DLL 'ComCtl32'.
This occurs on the third party library's call to:
/// <summary>
/// TaskDialogIndirect taken from commctl.h
/// </summary>
/// <param name="pTaskConfig">All the parameters about the Task Dialog to Show.</param>
/// <param name="pnButton">The push button pressed.</param>
/// <param name="pnRadioButton">The radio button that was selected.</param>
/// <param name="pfVerificationFlagChecked">The state of the verification checkbox on dismiss of the Task Dialog.</param>
[DllImport ( "ComCtl32", CharSet = CharSet.Unicode, PreserveSig = false )]
internal static extern void TaskDialogIndirect (
[In] ref TASKDIALOGCONFIG pTaskConfig,
[Out] out int pnButton,
[Out] out int pnRadioButton,
[Out] out bool pfVerificationFlagChecked );
If a dialog has already been shown, then the handler will run OK.
The DragDrop handler for the form does not show InvokeRequired and but I was careful to raise the dialog via Form.Invoke anyway.
private void MainForm_DragDrop(object sender, DragEventArgs e)
{
if (e.Data.GetDataPresent(DataFormats.FileDrop))
{
Array fileNames = (Array)e.Data.GetData(DataFormats.FileDrop);
if (fileNames != null && fileNames.OfType<string>().Any())
{
foreach (var fileName in fileNames.OfType<string>())
{
this.Invoke(new Action<string>(this.AttemptOpenFromPath), fileName);
}
}
}
}
As a side: I am compiling (and running) it on a 64-bit, Windows 7 machine but with the "AnyCPU" architecture flag.
Any thoughts/solutions as to why the exception is only raised when the first call to TaskDialogIndirect is via the DragDrop handler???

[DllImport ( "ComCtl32", ...)]
The library is taking a pretty heavy shortcut to use the comctl32.dll Windows dll. This tends to come to a good end by accident, but it falls over in your code. The full explanation is rather long-winded, I'll try to keep it short.
The core problem is that Windows has two versions of comctl32.dll. The one in c:\windows\system32 is a legacy version, implementing the common controls the way they looked and worked back in Windows 2000 and earlier. Windows XP acquired visual styles, making those controls look very different. There's another DLL that implements those visual styles, it is stored in the Windows side-by-side cache (c:\windows\winsxs).
An application must explicitly tell Windows that it supports the new version of the DLL. There are two ways to do it, you can do so in a manifest (the way WPF does it) or you can make an operating system call, the CreateActCtx() function (the way Winforms does it).
The way the library works is that it hopes that somebody has done one of those two things. And loaded the correct version of comctl32.dll so that pinvoking the [DllImport] function doesn't actually load the c:\windows\system32 version. The old one that doesn't implement TaskDialogIndirect(). This works by accident because some code usually does. And the fact that Windows only cares about the DLL name and not where it came from to determine if it needs to load a DLL.
I can somewhat guess how you ran out of luck. You are using Control.Invoke(), something you only ever need to do when you are using threads. Clearly you are displaying this form on another thread, not the main UI thread. This is in general a Really Bad Idea, the UI thread was already designed to be able to handle multiple windows. The one thing that didn't happen that normally happens on the UI thread is the Application.EnableVisualStyles() call. The one that tells Windows that you want the new version of comctl32.
You can try calling it on your worker thread. Might work, no idea. By far the best solution is to not create windows on worker threads. You can get rid of the wonky library by using the Windows API Code Pack, it provides a wrapper for task dialogs.

It turned out that within the DragDrop handler, I should be using BeginInvoke to asynchronously queue the call onto the Form's UI thread as opposed to synchronously waiting for it to complete within the handler...
Therefore, it was resolved with:
private void MainForm_DragDrop(object sender, DragEventArgs e)
{
if (e.Data.GetDataPresent(DataFormats.FileDrop))
{
Array fileNames = (Array)e.Data.GetData(DataFormats.FileDrop);
if (fileNames != null && fileNames.OfType<string>().Any())
{
foreach (var fileName in fileNames.OfType<string>())
{
this.BeginInvoke(new Action<string>(this.AttemptOpenFromPath), fileName);
}
}
}
}
I'm not sure why though!?? Can a commenter perhaps provide a reason?

WinForm not receiving messages except right after creation

I've got some unmanaged code sitting in a DLL. It publishes some methods that my calling (managed) code uses to hook into some COM notifications. Rather than deal with unmanaged code calling back into managed code, I've created a hidden Control derived object and am passing its handle property which the unmanaged code then uses as a parameter to SendMessage.
My Control derived class:
class InteropWindow : Control
{
//delegate
private Handler m_callback;
//window message
private uint m_message;
public InteropWindow(Handler callback, uint message)
: base()
{
m_callback = callback;
m_message = message;
}
protected override void WndProc(ref Message m)
{
if (m.Msg == m_message)
{
m_callback(new IntPtr((int)m.WParam));
}
base.WndProc(ref m);
}
}
Relevant line in unmanaged code:
SendMessage(m_notify, m_window_message, (WPARAM)pData, 0);
m_window_message & m_message are the same (both from RegisterWindowMessage), and m_notify == InteropWindow.Handle (pData varies, but is used as an opaque handle in the managed code). The unmanaged code is being invoked. These facts have been confirmed via debugging.
Shortly after I create the InteropWindow, the calls to SendMessage succeed. Afterwards (seconds later) the messages stop getting to WndProc, though there is no indication of any error.
The question is, what am I doing wrong here?
I've ruled out lifecycle issues (to the best of knowledge anyway), and played with HandleRef to no avail.
Edit the second.
I've re-written this to use function calls instead, which while fraught with its own perils, works a bit more like I'd expect. I've come to suspect this is a COM threading issue, but that's just a gut feeling.

Did you try passing your managed window's handle as a HandleRef? C# can marshal a HandleRef as an IntPtr and vice versa, I've seen Microsoft use that trick quite a bit when decompiling some of their stuff.
You can also load up a .Net profiler and watch the GC. It would be nice to know if your app is breaking right after a collect.

C# program (process) will not unload

I have a C# program that uses a class from another assembly, and this class calls an unmanaged DLL to do some processing. Here is a snippet:
public class Util
{
const string dllName = "unmanaged.dll";
[DllImport(dllName, EntryPoint = "ExFunc")]
unsafe static extern bool ExFunc(StringBuilder path, uint field);
public bool Func(string path, uint field)
{
return ExFunc(new StringBuilder(path), field);
}
...
}
Util util = new Util();
bool val = util.Func("/path/to/something/", 1);
The problem I'm having is that if I call "Func" my main C# program will not unload. When I call Close() inside my main form the process will still be there if I look in Task Manager. If I remove the call to "Func" the program unloads fine. I have done some testing and the programs Main function definitely returns so I'm not sure what's going on here.

It looks like your unmanaged library is spawning a thread for asynchronous processing.
Odds are it supports a cancel function of some sort; I suggest that you attempt to call that at program shutdown. If your program is just completing before the asynchronous call happens to complete, look for a "wait for completion" function and call that before returning from your "Func" method.

It might dispatch a non background thread that is not letting go when your main application closes. Can't say for sure without seeing the code but that is what I would assume.
It's probably less then ideal, but if you need a workaround you could probably use:
System.Diagnostics.Process.GetCurrentProcess().Kill();
This will end your app at the process level and kill all threads that are spawned through the process.

Do you have the source code to unmanaged.dll ? It must be doing something, either starting another thread and not exiting, or blocking in it's DllMain, etc.