We Keep Coding

WinForms window changes dimensions when it encounters an async call

I have a WinForms project which is several years old and has been retro-fitted with async event-handlers:
private async void dgvNewOrders_CellClick(object sender, DataGridViewCellEventArgs e)
Inside this method is an async call:
var projectTemplate = await GetProjectTemplateFile(companyId, sourceLang, targetLang);
When the program runs on a normal resolution screen, it runs as expected. However, when run on a high-DPI screen the window's dimensions - as well as those of all child controls - jump to half-size as soon as it encounters that inner async call. It's as if the program is suddenly run in a compatibility mode or the scaling has been disabled.
Currently, in an effort to debug the problem, the GetProjectTemplateFile method consists simply of
private async Task<ProjectTemplateFile> GetProjectTemplateFile(long companyId, string sourceLanguage, string targetLanguage)
{
return null;
}
It makes no difference whether GetProjectTemplateFile performs an async operation or not.
If I comment-out that async call to GetProjectTemplateFile then the program runs as expected without any jump in dimensions, even though there are still other async calls made in the CellClick event.
I've tried appending .ConfigureAwait(true) to the async call, which makes no difference. Nor does running the call synchronously with .GetAwaiter().GetResult().
Can anyone explain why the window's dimensions are changing with this particular async call, and/or how to prevent this from happening?
Update
As per a request, here is a code sample which elicits the explained behaviour. There's nothing unusual happening here that I can see but I assure you, this very code is causing the explained behaviour.
private async void dgvNewOrders_CellClick(object sender, DataGridViewCellEventArgs e)
{
var result = await _templateInteraction.GetProjectTemplateFile(1,
"en-US",
"de-CH");
return;
}
public class TemplateInteraction : ITemplateInteraction
{
public async Task<ProjectTemplateFile> GetProjectTemplateFile(long companyId, string sourceLanguage, string targetLanguage)
{
return null;
// elided code
}
// other methods
}
Some other information which might be relevant:
The FormBorderStyle of the window is "FixedToolWindow"
The window is given an explicit width in a startup method
AutoSize = False
AutoSizeMode = GrowOnly
The computer which it's being developed on does not have the Windows 10 1703
(Creator's) update which has new scaling logic
If the GetprojectTemplateFile method is not async, i.e. has signature public ProjectTemplateFile GetProjecttemplateFile(...) then there is no problem. This problem appears to exist only when the method call is async - even if I make it a blocking call.
UPDATE 2:
I've found the specific line(s) of code which cause this problem:
MessageBox.Show(...);
The inner async call, GetProjectTemplateFile, calls an API and then checks the response:
var responseMessage = await client.GetAsync(uri);
if (!responseMessage.IsSuccessStatusCode)
{
MessageBox.Show(...);
return null;
}
If I comment-out the MessageBox.Show(...) call then everything is normal, no scaling problems, no jump in dimensions.
But the problem occurs when the MessageBox.Show(...) call is in-place.
Furthermore, the API responds with a 200 (OK) so the MessageBox code isn't even being used. My guess is that the JIT compiler sees it as a possibility so... it re-renders the form?
Also, importantly, this code is not in the form's code-behind, it's in a class which the form is given an instance of in its constructor.

I guess you are using MessageBox from System.Windows namespace, referenced from PresentationFramework.dll, instead of System.Windows.Forms namespace?
// Causes DPI scaling problems:
System.Windows.MessageBox.Show() // loads WPF version from PresentationFramework.dll
// no DPI scaling issues:
System.Windows.Forms.MessageBox.Show() // uses standard winforms messagebox
So try using the standard MessageBox instead.
I've found that whenever any WPF-targeted dll gets loaded into memory, the DPI autoscaling gets reset. The specific function doesn't even need to be actually called - the dll's are loaded as soon as the parent function is called.
I had same problem by just having System.Windows.Input.Keyboard.IsKeyToggled(), which loaded PresentationCore.dll. Thought I was going mad as well...

FatalExecutionEngineError in C# / WSC (COM) interop

I'm about to start a migration project at work for a legacy system written in VBScript. It has an interesting structure in that much of it was segregated by writing various components as "WSC" files, which are effectively a way of exposing VBScript code in a COM-like manner. The boundary interface from the "core" to these components is fairly tight and well known so I was hoping that I would be able to tackle writing a new core and reuse the WSCs, postponing their rewriting.
It's possible to load a WSC by adding a reference to "Microsoft.VisualBasic" and calling
var component = (dynamic)Microsoft.VisualBasic.Interaction.GetObject("script:" + controlFilename, null);
where "controlFilename" is the full file path. GetObject returns a reference of type "System.__ComObject" but the properties and methods can be accessed using .net's "dynamic" type.
This appeared to work fine initially, but I've encountered problems when quite a specific set of circumstances are combined - my worry is that this may happen in other cases or, even worse, that bad things are happening much of the time and being masked, just waiting to blow up when I least expect it.
The raised exception is of type "System.ExecutionEngineException", which sounds particularly scary (and vague)!
I've cobbled together what I believe to be the minimum reproduce case and was hoping that someone could cast a little light on what the problem could be. I've also identified some tweaks that can be made that seem to prevent it, though I can't explain why.
Create a new empty "ASP.NET Web Application" called "WSCErrorExample" (I've done this in VS 2013 / .net 4.5 and VS 2010 / .net 4.0, it makes no difference)
Add a reference to "Microsoft.VisualBasic" to the project
Add a new "Web Form" called "Default.aspx" and paste the following over the top of "Default.aspx.cs"
using System;
using System.IO;
using System.Reflection;
using System.Runtime.InteropServices;
using Microsoft.VisualBasic;
namespace WSCErrorExample
{
public partial class Default : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
var currentFolder = GetCurrentDirectory();
var logFile = new FileInfo(Path.Combine(currentFolder, "Log.txt"));
Action<string> logger = message =>
{
// The try..catch is to avoid IO exceptions when reproducing by requesting the page many times
try { File.AppendAllText(logFile.FullName, message + Environment.NewLine); }
catch { }
};
var controlFilename = Path.Combine(currentFolder, "TestComponent.wsc");
var control = (dynamic)Interaction.GetObject("script:" + controlFilename, null);
logger("About to call Go");
control.Go(new DataProvider(logger));
logger("Completed");
}
private static string GetCurrentDirectory()
{
// This is a way to get the working path that works within ASP.Net web projects as well as Console apps
var path = Path.GetDirectoryName(Assembly.GetExecutingAssembly().GetName().CodeBase);
if (path.StartsWith(#"file:\", StringComparison.InvariantCultureIgnoreCase))
path = path.Substring(6);
return path;
}
[ComVisible(true)]
public class DataProvider
{
private readonly Action<string> _logger;
public DataProvider(Action<string> logger)
{
_logger = logger;
}
public DataContainer GetDataContainer()
{
return new DataContainer();
}
public void Log(string content)
{
_logger(content);
}
}
[ComVisible(true)]
public class DataContainer
{
public object this[string fieldName]
{
get { return "Item:" + fieldName; }
}
}
}
}
Add a new "Text File" called "TestComponent.wsc", open its properties window and change "Copy to Output Directory" to "Copy if newer" then paste the following in as its content
<?xml version="1.0" ?>
<?component error="false" debug="false" ?>
<package>
<component id="TestComponent">
<registration progid="TestComponent" description="TestComponent" version="1" />
<public>
<method name="Go" />
</public>
<script language="VBScript">
<![CDATA[
Function Go(objDataProvider)
Dim objDataContainer: Set objDataContainer = objDataProvider.GetDataContainer()
If IsEmpty(objDataContainer) Then
mDataProvider.Log "No data provided"
End If
End Function
]]>
</script>
</component>
</package>
Running this once should cause no apparent issue, the "Log.txt" file will be written to the "bin" folder. Refreshing the page, however, normally results in an exception
Managed Debugging Assistant 'FatalExecutionEngineError' has detected a problem in 'C:\Program Files (x86)\IIS Express\iisexpress.exe'.
Additional information: The runtime has encountered a fatal error. The address of the error was at 0x733c3512, on thread 0x1e10. The error code is 0xc0000005. This error may be a bug in the CLR or in the unsafe or non-verifiable portions of user code. Common sources of this bug include user marshaling errors for COM-> interop or PInvoke, which may corrupt the stack.
Occasionally, the second request does not result in this exception, but holding down F5 in the browser window for a couple of seconds will ensure that it rears its ugly head. The exception, so far as I can tell, happens at the "If IsEmpty" check (other versions of this reproduce case had more logging calls, which pointed to that line being the source of the problem).
I've tried various things to try to get to the bottom of this, I've tried to recreate in a console app and the problem does not occur, even if I spin up hundreds of threads and get them to process the work above. I've tried an ASP.Net MVC web application, rather than using a Web Form and the same issue DOES occur. I've tried changing the apartment state from the default MTA to STA (I was clutching at straws a bit at that point!) and it made no change to the behavour. I've tried building a web project that uses Microsoft's OWIN implementation and the issue occurs in that scenario as well.
Two interesting things that I have noticed - if the "DataContainer" class does not have an indexed property (or a default method / property, decorated with a [DispId(0)] attribute - not illustrated in this example) then the error does not occur. If the "logger" closure does not contain a "FileInfo" reference (if a string "logFilePath" was maintained, rather than the FileInfo instance "logFile") then the error does not occur. I suppose it sounds like one approach would be to avoid doing these things! But I would be concerned that there could be other ways to trigger this scenario that I don't currently know about and trying to enforce the rule of not-doing-these-things could get complicated as the code base grows, I can imagine this error creeping back in without it being immediately obvious why.
On one run (through Katana), I got additional call stack information:
This thread is stopped with only external code frames on the call stack. External code frames are typically from framework code but can also include other optimized modules which are loaded in the target process.
Call stack with external code
mscorlib.dll!System.Variant.Variant(object obj)
mscorlib.dll!System.OleAutBinder.ChangeType(object value, System.Type type, System.Globalization.CultureInfo cultureInfo)
mscorlib.dll!System.RuntimeType.TryChangeType(object value, System.Reflection.Binder binder, System.Globalization.CultureInfo culture, bool needsSpecialCast)
mscorlib.dll!System.RuntimeType.CheckValue(object value, System.Reflection.Binder binder, System.Globalization.CultureInfo culture, System.Reflection.BindingFlags invokeAttr)
mscorlib.dll!System.Reflection.MethodBase.CheckArguments(object[] parameters, System.Reflection.Binder binder, System.Reflection.BindingFlags invokeAttr, System.Globalization.CultureInfo culture, System.Signature sig)
[Native to Managed Transition]
One final note: if I create a wrapper for the "DataProvider" class, using IReflect and map the calls over IDispatch on to calls to the underlying "DataProvider" instance then the issue goes away. But again, deciding that this is somehow the answer seems dangerous to me - if I have to be meticulous about ensuring that any reference passed to the components has such a wrapper then errors could creep in that could be difficult to track down. What if a reference that IS encased in an IReflect-implementing wrapper returns a reference from a method or property call that isn't wrapped up in the same way? I suppose the wrapper could try to do something like ensuring it only returns "safe" reference (ie. those without indexed properties or DispId=0 methods or properties) without wrapping them in a further IReflect wrapper.. but it all seems a bit hacky.
I really have no clue where next to go with this problem, does anyone have any idea?

My guess is, the error you're seeing is caused by the fact that WSC script components are COM STA objects by their nature. They're implemented by the underlying VBScript Active Scripting Engine, which itself is an STA COM object. As such, they require an STA thread to be created and accessed on, and such thread should remain the same for the lifetime of any particular WSC object (the object requires thread affinity).
ASP.NET threads are not STA. They're ThreadPool threads, and they implicitly become COM MTA threads when you start using COM objects on them (for differences between STA and MTA, refer to INFO: Descriptions and Workings of OLE Threading Models). COM then creates a separate implicit STA apartment for your WSC objects and marshal calls there from your ASP.NET request thread. The whole thing may or may not go well in the ASP.NET environment.
Ideally, you should get rid of WSC script components and replace them with .NET assemblies. If that's not feasible short-term, I'd recommend that you run your own explicitly controlled STA thread(s) to host the WSC components. The following may help:
How to use non-thread-safe async/await APIs and patterns with ASP.NET Web API?
StaTaskScheduler and STA thread message pumping
Updated, why not give this a try? Your code would look like this:
// create a global instance of ThreadAffinityTaskScheduler - per web app
public static class GlobalState
{
public static ThreadAffinityTaskScheduler TaScheduler { get; private set; }
public static GlobalState()
{
GlobalState.TaScheduler = new ThreadAffinityTaskScheduler(
numberOfThreads: 10,
staThreads: true,
waitHelper: WaitHelpers.WaitWithMessageLoop);
}
}
// ... inside Page_Load
GlobalState.TaScheduler.Run(() =>
{
var control = (dynamic)Interaction.GetObject("script:" + controlFilename, null);
logger("About to call Go");
control.Go(new DataProvider(logger));
logger("Completed");
}, CancellationToken.None).Wait();
If that works, you can somewhat improve the web app's scalabilty by using PageAsyncTask and async/await instead of the blocking Wait().

How to implement single instance per machine application?

I have to restrict my .net 4 WPF application so that it can be run only once per machine. Note that I said per machine, not per session.
I implemented single instance applications using a simple mutex until now, but unfortunately such a mutex is per session.
Is there a way to create a machine wide mutex or is there any other solution to implement a single instance per machine application?

I would do this with a global Mutex object that must be kept for the life of your application.
MutexSecurity oMutexSecurity;
//Set the security object
oMutexSecurity = new MutexSecurity();
oMutexSecurity.AddAccessRule(new MutexAccessRule(new SecurityIdentifier(WellKnownSidType.BuiltinUsersSid, null), MutexRights.FullControl, AccessControlType.Allow));
//Create the global mutex and set its security
moGlobalMutex = new Mutex(True, "Global\\{5076d41c-a40a-4f4d-9eed-bf274a5bedcb}", bFirstInstance);
moGlobalMutex.SetAccessControl(oMutexSecurity);
Where bFirstInstance returns if this is the first instance of your application running globally. If you omited the Global part of the mutex or replaced it with Local then the mutex would only be per session (this is proberbly how your current code is working).
I believe that I got this technique first from Jon Skeet.
The MSDN topic on the Mutex object explains about the two scopes for a Mutex object and highlights why this is important when using terminal services (see second to last note).

I think what you need to do is use a system sempahore to track the instances of your application.
If you create a Semaphore object using a constructor that accepts a name, it is associated with an operating-system semaphore of that name.
Named system semaphores are visible throughout the operating system, and can be used to synchronize the activities of processes.
EDIT: Note that I am not aware if this approach works across multiple windows sessions on a machine. I think it should as its an OS level construct but I cant say for sure as i havent tested it that way.
EDIT 2: I did not know this but after reading Stevo2000's answer, i did some looking up as well and I think that the "Global\" prefixing to make the the object applicable to the global namespace would apply to semaphores as well and semaphore, if created this way, should work.

You could open a file with exclusive rights somewhere in %PROGRAMDATA%
The second instance that starts will try to open the same file and fail if it's already open.

How about using the registry?
You can create a registry entry under HKEY_LOCAL_MACHINE.
Let the value be the flag if the application is started or not.
Encrypt the key using some standard symmetric key encryption method so that no one else can tamper with the value.
On application start-up check for the key and abort\continue accordingly.
Do not forget to obfuscate your assembly, which does this encryption\decryption part, so that no one can hack the key in registry by looking at the code in reflector.

I did something similar once.
When staring up the application list, I checked all running processes for a process with identical name, and if it existed I would not allow to start the program.
This is not bulletproof of course, since if another application have the exact same process name, your application will never start, but if you use a non-generic name it will probably be more than good enough.

For the sake of completeness, I'd like to add the following which I just found now:
This web site has an interesting approach in sending Win32 messages to other processes. This would fix the problem of the user renaming the assembly to bypass the test and of other assemblies with the same name.
They're using the message to activate the main window of the other process, but it seems like the message could be a dummy message only used to see whether the other process is responding to it to know whether it is our process or not.
Note that I haven't tested it yet.

See below for full example of how a single instace app is done in WPF 3.5
public class SingleInstanceApplicationWrapper :
Microsoft.VisualBasic.ApplicationServices.WindowsFormsApplicationBase
{
public SingleInstanceApplicationWrapper()
{
// Enable single-instance mode.
this.IsSingleInstance = true;
}
// Create the WPF application class.
private WpfApp app;
protected override bool OnStartup(
Microsoft.VisualBasic.ApplicationServices.StartupEventArgs e)
{
app = new WpfApp();
app.Run();
return false;
}
// Direct multiple instances.
protected override void OnStartupNextInstance(
Microsoft.VisualBasic.ApplicationServices.StartupNextInstanceEventArgs e)
{
if (e.CommandLine.Count > 0)
{
app.ShowDocument(e.CommandLine[0]);
}
}
}
Second part:
public class WpfApp : System.Windows.Application
{
protected override void OnStartup(System.Windows.StartupEventArgs e)
{
base.OnStartup(e);
WpfApp.current = this;
// Load the main window.
DocumentList list = new DocumentList();
this.MainWindow = list;
list.Show();
// Load the document that was specified as an argument.
if (e.Args.Length > 0) ShowDocument(e.Args[0]);
}
public void ShowDocument(string filename)
{
try
{
Document doc = new Document();
doc.LoadFile(filename);
doc.Owner = this.MainWindow;
doc.Show();
// If the application is already loaded, it may not be visible.
// This attempts to give focus to the new window.
doc.Activate();
}
catch
{
MessageBox.Show("Could not load document.");
}
}
}
Third part:
public class Startup
{
[STAThread]
public static void Main(string[] args)
{
SingleInstanceApplicationWrapper wrapper =
new SingleInstanceApplicationWrapper();
wrapper.Run(args);
}
}
You may need to add soem references and add some using statements but it shoudl work.
You can also download a VS example complete solution by downloading the source code of the book from here.
Taken From "Pro WPF in C#3 2008 , Apress , Matthew MacDonald" , buy the book is gold. I did.

VSTO in VBA: AddIn.Object returns Nothing (null) sometimes

Given:
A VSTO Add-In
An override object RequestComAddInAutomationService() which returns an instance of a class which is called Facade in my scenario.
A VBA macro in Excel 2007 which accesses the AddIn.Object to get the Facade and uses it.
A plenty of times where this works perfectly fine.
A couple of times where out of the blue, this doesn't seem to work.
Update: Turns out that it's a particular user that has the problem. She has it all the time, others never have it (? never say "never")
In this "couple of times" I get
Error: Object variable or With block variable not set
at the line of code which tries to access a property of Facade. In short I can tell you that the code in RequestComAddInAutomationService() doesn't have any error-prone magic in it, and the VBA code to access the add-in has been taken from the web and looks fine, too. The longer version is yet to come, for those who'll take the time to read it :-)
Question: Does anyone have a clue why this can happen? Is it an Excel issue?
Details as promised:
MyAddIn.cs:
public partial class MyAddIn
{
public Facade Facade { get; private set; }
protected override object RequestComAddInAutomationService()
{
if (this.Facade == null)
this.Facade = new Facade(Controller.Instance);
return this.Facade;
}
}
Facade.cs:
[ComVisible(true)]
[Guid("1972781C-A71A-48cd-9675-AE47EACE95E8")]
[InterfaceType(ComInterfaceType.InterfaceIsDual)]
public interface IFacade
{
// some methods
}
[ComVisible(true)]
[ClassInterface(ClassInterfaceType.None)]
public class Facade : IFacade
{
private Controller Controller { get; set; }
public Facade(Controller controller)
{
this.Controller = controller;
}
}
Facade has some methods but not a single field.
Controller.cs:
public class Controller
{
private static Controller instance = null;
public static Controller Instance
{
get
{
if (instance == null) instance = new Controller();
return instance;
}
}
private Controller() { }
}
Controller has some private fields. Since the fields assignments are executed on creation, I reviewed them. Most of them are not initialized at all, or they are set to null, so the constructor does virtually nothing.
The VBA code:
Dim addin As Office.COMAddIn
Dim automationObject As Object
Set addin = Application.COMAddIns("My AddIn")
Set automationObject = addin.Object
Dim oResult As Object
Set oResult = automationObject.SomeMethodThatReturnsAnObject()
The last line is where the error happens. Although the method called returns an object, I am pretty sure that it cannot be the source of the error: If the reference returned was null, then the statement would simply evaluate to Set oResult = Nothing which is still valid. VBA rather throws this type of error whenever a method is executed on an reference that is Nothing, which is automationObject in my case.
On the other hand, if the add-in wasn't there at all, the Application.COMAddIns(...) would raise an index out of bounds error, I've seen that before.

Working most of the time and failing sometimes looks like a race-condition. Andrew Whitechapel has written about a race condition related to RequestComAddInAutomationService1:
COMAddIns Race Condition
Although he says that race conditions should not be a problem with in-process VBA macros, it could be that the problem might happen in your specific scenario.
Try the suggested workaround and loop until your Addin.Object is valid (C# code, similar in VBA):
while (utils == null)
{
utils = (ComServiceOleMarshal.IAddinUtilities)addin.Object;
System.Threading.Thread.Sleep(100);
}
utils.DoSomething();
1There's lots of useful information on his blog for the things you are doing, so don't miss the related articles.

Turned out that Excel disabled the COM add-in. This is known to sometimes happen silently, without Excel complaining about anything.
So, since the add-in was registered with excel, the following line succeeded:
Set addin = Application.COMAddIns("My AddIn")
But since it was disabled, the object was not created and
Set automationObject = addin.Object
resulted in Nothing.

I've had a similar problem, often but not always, so I can't say for certain but the thing that seemed to fix it was going to Project / Application / Assembly Information... and checking Make assembly COM-Visible, then creating the object (in Excel VBA) with:
Set automationObject = CreateObject("PlugInDllName.PlugInClass")
No problems since - fingers crossed.

Replacing Process.Start with AppDomains

Background
I have a Windows service that uses various third-party DLLs to perform work on PDF files. These operations can use quite a bit of system resources, and occasionally seem to suffer from memory leaks when errors occur. The DLLs are managed wrappers around other unmanaged DLLs.
Current Solution
I'm already mitigating this issue in one case by wrapping a call to one of the DLLs in a dedicated console app and calling that app via Process.Start(). If the operation fails and there are memory leaks or unreleased file handles, it doesn't really matter. The process will end and the OS will recover the handles.
I'd like to apply this same logic to the other places in my app that use these DLLs. However, I'm not terribly excited about adding more console projects to my solution, and writing even more boiler-plate code that calls Process.Start() and parses the output of the console apps.
New Solution
An elegant alternative to dedicated console apps and Process.Start() seems to be the use of AppDomains, like this: http://blogs.geekdojo.net/richard/archive/2003/12/10/428.aspx
I've implemented similar code in my application, but the unit tests have not been promising. I create a FileStream to a test file in a separate AppDomain, but don't dispose it. I then attempt to create another FileStream in the main domain, and it fails due to the unreleased file lock.
Interestingly, adding an empty DomainUnload event to the worker domain makes the unit test pass. Regardless, I'm concerned that maybe creating "worker" AppDomains won't solve my problem.
Thoughts?
The Code
/// <summary>
/// Executes a method in a separate AppDomain. This should serve as a simple replacement
/// of running code in a separate process via a console app.
/// </summary>
public T RunInAppDomain<T>( Func<T> func )
{
AppDomain domain = AppDomain.CreateDomain ( "Delegate Executor " + func.GetHashCode (), null,
new AppDomainSetup { ApplicationBase = Environment.CurrentDirectory } );
domain.DomainUnload += ( sender, e ) =>
{
// this empty event handler fixes the unit test, but I don't know why
};
try
{
domain.DoCallBack ( new AppDomainDelegateWrapper ( domain, func ).Invoke );
return (T)domain.GetData ( "result" );
}
finally
{
AppDomain.Unload ( domain );
}
}
public void RunInAppDomain( Action func )
{
RunInAppDomain ( () => { func (); return 0; } );
}
/// <summary>
/// Provides a serializable wrapper around a delegate.
/// </summary>
[Serializable]
private class AppDomainDelegateWrapper : MarshalByRefObject
{
private readonly AppDomain _domain;
private readonly Delegate _delegate;
public AppDomainDelegateWrapper( AppDomain domain, Delegate func )
{
_domain = domain;
_delegate = func;
}
public void Invoke()
{
_domain.SetData ( "result", _delegate.DynamicInvoke () );
}
}
The unit test
[Test]
public void RunInAppDomainCleanupCheck()
{
const string path = #"../../Output/appdomain-hanging-file.txt";
using( var file = File.CreateText ( path ) )
{
file.WriteLine( "test" );
}
// verify that file handles that aren't closed in an AppDomain-wrapped call are cleaned up after the call returns
Portal.ProcessService.RunInAppDomain ( () =>
{
// open a test file, but don't release it. The handle should be released when the AppDomain is unloaded
new FileStream ( path, FileMode.Open, FileAccess.ReadWrite, FileShare.None );
} );
// sleeping for a while doesn't make a difference
//Thread.Sleep ( 10000 );
// creating a new FileStream will fail if the DomainUnload event is not bound
using( var file = new FileStream ( path, FileMode.Open, FileAccess.ReadWrite, FileShare.None ) )
{
}
}

Application domains and cross-domain interaction is a very thin matter, so one should make sure he really understands how thing work before doing anything... Mmm... Let's say, "non-standard" :-)
First of all, your stream-creating method actually executes on your "default" domain (surprise-surprise!). Why? Simple: the method that you pass into AppDomain.DoCallBack is defined on an AppDomainDelegateWrapper object, and that object exists on your default domain, so that is where its method gets executed. MSDN doesn't say about this little "feature", but it's easy enough to check: just set a breakpoint in AppDomainDelegateWrapper.Invoke.
So, basically, you have to make do without a "wrapper" object. Use static method for DoCallBack's argument.
But how do you pass your "func" argument into the other domain so that your static method can pick it up and execute?
The most evident way is to use AppDomain.SetData, or you can roll your own, but regardless of how exactly you do it, there is another problem: if "func" is a non-static method, then the object that it's defined on must be somehow passed into the other appdomain. It may be passed either by value (whereas it gets copied, field by field) or by reference (creating a cross-domain object reference with all the beauty of Remoting). To do former, the class has to be marked with a [Serializable] attribute. To do latter, it has to inherit from MarshalByRefObject. If the class is neither, an exception will be thrown upon attempt to pass the object to the other domain. Keep in mind, though, that passing by reference pretty much kills the whole idea, because your method will still be called on the same domain that the object exists on - that is, the default one.
Concluding the above paragraph, you are left with two options: either pass a method defined on a class marked with a [Serializable] attribute (and keep in mind that the object will be copied), or pass a static method. I suspect that, for your purposes, you will need the former.
And just in case it has escaped your attention, I would like to point out that your second overload of RunInAppDomain (the one that takes Action) passes a method defined on a class that isn't marked [Serializable]. Don't see any class there? You don't have to: with anonymous delegates containing bound variables, the compiler will create one for you. And it just so happens that the compiler doesn't bother to mark that autogenerated class [Serializable]. Unfortunate, but this is life :-)
Having said all that (a lot of words, isn't it? :-), and assuming your vow not to pass any non-static and non-[Serializable] methods, here are your new RunInAppDomain methods:
/// <summary>
/// Executes a method in a separate AppDomain. This should serve as a simple replacement
/// of running code in a separate process via a console app.
/// </summary>
public static T RunInAppDomain<T>(Func<T> func)
{
AppDomain domain = AppDomain.CreateDomain("Delegate Executor " + func.GetHashCode(), null,
new AppDomainSetup { ApplicationBase = Environment.CurrentDirectory });
try
{
domain.SetData("toInvoke", func);
domain.DoCallBack(() =>
{
var f = AppDomain.CurrentDomain.GetData("toInvoke") as Func<T>;
AppDomain.CurrentDomain.SetData("result", f());
});
return (T)domain.GetData("result");
}
finally
{
AppDomain.Unload(domain);
}
}
[Serializable]
private class ActionDelegateWrapper
{
public Action Func;
public int Invoke()
{
Func();
return 0;
}
}
public static void RunInAppDomain(Action func)
{
RunInAppDomain<int>( new ActionDelegateWrapper { Func = func }.Invoke );
}
If you're still with me, I appreciate :-)
Now, after spending so much time on fixing that mechanism, I am going to tell you that is was purposeless anyway.
The thing is, AppDomains won't help you for your purposes. They only take care of managed objects, while unmanaged code can leak and crash all it wants. Unmanaged code doesn't even know there are such things as appdomains. It only knows about processes.
So, in the end, your best option remains your current solution: just spawn another process and be happy about it. And, I would agree with the previous answers, you don't have to write another console app for each case. Just pass a fully qualified name of a static method, and have the console app load your assembly, load your type, and invoke the method. You can actually package it pretty neatly in a very much the same way as you tried with AppDomains. You can create a method called something like "RunInAnotherProcess", which will examine the argument, get the full type name and method name out of it (while making sure the method is static) and spawn the console app, which will do the rest.

You don't have to create many console applications, you can create a single application that will receive as parameter the full qualified type name. The application will load that type and execute it.
Separating everything into tiny processes is the best method to really dispose all the resources. An application domain cannot do full resources disposing, but a process can.

Have you considered opening a pipe between the main application and the sub applications? This way you could pass more structured information between the two applications without parsing standard output.