We Keep Coding

How do I Integrate one application’s UI into another?

I apologize for the length of the question, but I believe it is difficult to understand the “why” without the background.
Background: I have two applications running in a Windows Embedded Standard 7 environment. They should be the only two applications running on the machine. One, called “Controller”, is written in C++ the other, “DBconnector”, is written in c#. This is not new code. It has been in active use and development for almost 20 years.
The purpose of the software is to run a manufacturing machine for producing parts. These machines are big and dangerous if the program crashes. Long ago, I discovered that if the network went down for some reason, all the threads in the application would stall – not just the network thread. This was disastrous since leaving the controller in a state with the wrong relays on in extremely rare circumstances could cause the machine to literally explode. Note: Several things have been added to the software and hardware to prevent this now. While this danger doesn’t really exist anymore, stability is still extremely important. I never want the operator to be stuck in a state where they can’t hit the reset button. My solution at the time was to move the networking tasks into a separate application. The OS was windows XP based at the time. I have no idea if the problem still exists in windows 10 since I really don’t want to rewrite hundreds of thousands of lines of code to try and merge the two programs now.
The development of the two programs diverged such that the one that controlled the machine, Controller, was designed for extreme stability and the other, DBconnector, was where dangerous things like networking and most file I/O happened. Communication between the two programs is facilitated using a memory mapped file that they both can access. I have no problem sharing window handles or process id’s or any other data that might be needed between the two programs.
Here is my question. How can I make the Controller application display the GUI of DBconnector? For example, I have started to add functionality to Controller that requires DBconnector to display the quality control sheets that are held on a web site on company servers. I want for an operator to be able to pull up the quality control sheet directly on the machine. The operator currently only interacts with the Controller application. I don’t want Controller to be able to access the network. Also, C# has some tools to make displaying a web page easy. It seems to me that the place to do this is DBconnector. The problem is that DBconnector runs in the background and cannot currently be seen or accessed by a user. So, the question is how to solve this.
First option I have tried is to tell DBconnector to come forward and put Controller in the background. Then, when the user is done, Controller comes back to the front. I have made this to work using some hacks, but it is inconsistent. The trick I used was to minimize and then maximize DBconnector which seems to bring it to the front most of the time and try to hold focus on one or the other. There still might be a way to do it this way, but it needs to be something that is consistent.
The second option is to run the DBconnector application inside of one of Controller’s windows. I have no idea how to do this. I thought about using ATL or COM, but I think these run as threads within Controllers process rather than as a separate application.
The third option I’ve considered is to create a window inside Controller that intercepts and passes all user input messages directly to Dbconnector using a windows message handle and takes a screenshot of DBconnector whenever the it is invalidated and passes it through the memory mapped file. Currently, this is what I am swaying towards.
Are there any suggestions on how to do the first and last option better, or how to do the second option at all, or another solution that I have missed? Keep in mind that our current hardware is running Windows Embedded Standard 7. The project is currently in visual studio 2015. The C++ window technology is MFC implemented using libraries originally from around 2003 I think. DBconnector is in .NET framework 4 in C#.

C# Windows Service queue with Pool

EDIT: Context
I have to develop a web asp.net application which will allow user to create "Conversion request" for one or several CAO files.
My application should just upload files to a directory where I can convert them.
I then want to have a service that will check the database updated by the web application to see if a conversion is waiting to be done. Then I have to call a batch command on this file with some arguments.
The thing is that those conversion can freeze if the user give a CAO file which has been done wrongly. In this case, we want the user or an admin to be able to cancel the conversion process.
The batch command use to convert is a third party tool that I can't change. It need a token to convert, and I can multithread as long as I have more than one token available. An other application can use those token at the same moment so I can't just have a pre-sized pool of thread according to the number of token I have.
The only way I have to know if I can convert is to start the conversion with the command and see if in the logs it tells me that I have a licence problem which mean either "No token available" or "Current licence doesn't accept the given input format". As I allow only available input formats on the web application, I can't have the second problem.
The web application is almost done, I mean that I can upload file and download results and conversion logs at the end. Now I need to do the service that will take input files, convert them, update convert status in database and lastly add those files in the correct download dirrectory.
I have to work on a service which will look in a database at a high frequency (maybe 5 or 10 seconds) if a row is set as "Ready to convert" or "Must be canceled".
If the row is set to "ready to convert" I must try to convert it, but I do it using a third party dll that have a licence token system that allow me to do only 5 converts simultaneously atm.
If the row is set to "Must be canceled" I must kill the conversion because it's either freeze and the admin had to kill it or because the user canceled his own task.
Also, conversion times can be very long, from 1 or 2 seconds to several hours depending on the file size and how it has been created.
I was thinking about a pooling system, as I saw here :
Stackoverflow answer
Pooling system give me the advantage to isolate the reading database part to the conversion process. But I have the feeling that I loose a kind of control on background process. Which is maybe juste because I'm not used to them.
But I'm not very used to services and even if this pool system seems good, I don't know how I can cancel any task if needed ?
The tool I use to convert work as a simple batch command that will just return me an error if no licence are available now, but using a pool how can I make the convert thread wait for the convert to be done if No licence are available is a simple infinite while loop an appropriate answer ? It seems quite bad to me.
Finally, I can't just use a "5 threads pool" as thoses licences are also used by 2 others applications which doesn't let me know at any time how many of them are available.
The idea of using pool can also be wrong, as I said, I'm not very used to services and before starting something stupid, I prefer ask abotu the good way to do it.
Moreover, about the database reading/writing, even if I think that the second option is better, should I:
Use big models files that I already use on my ASP.NET web application which will create a lot of objects (one for each row as it's entities models).
Don't use entities models but lighter models which will be less object entities oriented but will probably be less ressources demanding. This will also be harder to maintain.
So I'm more asking about some advices on how I should do it than a pure code answer, but some example could be very useful.
EDIT: to be more precise, I need to find a way to:
(For the moment, I stay with only one licence available, I will evolve it later if needed)
Have a service that run as a loop and will if possible start a new thread for the given request. The service must still be running as the status can be set to "Require to be cancel".
At the moment I was thinking about a task with a cancellation token. Which would probably achive this.
But if the task find that the token isn't currently available, how can I say to my main loop of the service that it can't process now ? I was thinking about having just an integer value as a return where the return code would be an indicator on the ending reason: Cancellation / No token / Ended... Is that a good way to do ?

What I'm hearing is that the biggest bottleneck in your process is the conversion... pooling / object mapping / direct sql doesn't sound as important as the conversion bottleneck.
There are several ways to solve this depending on your environment and what your constraints are. Good, fast, and cheap... pick 2.
As far as "best practice" go, there are large scale solutions (ESBs, Message Queue based etc), there are small scale solutions (console apps, batch files, powershell scripts on Windows scheduler, etc) and the in-between solutions (this is typically where the "Good Enough" answer is found). The volume of the stuff you need to process should decide which one is the best fit for you.
Regardless of which you choose above...
My first step will be to eliminate variables...
How much volume will you be processing? If you wrote something that's not optimized but works, will that be enough to process your current volume? (e.g. a console app to be run using the Windows Task Scheduler every 10 - 15 seconds and gets killed if it runs for more than say 15 minutes)
Does the third party dll support multi-threading? If no, that eliminates all your multi-threading related questions and narrows down your options on how to scale out.
You can then determine what approach will fit your problem domain...
will it be the same service deployed on several machines, each hitting the database every 10-15 seconds?, or
will it be one service on one machine using multi-threading?
will it be something else (pooling might or might not be in play)?
Once you get the answer above, the next question is.
will the work required fit within the allocated budget and time constraint of your project? if not, go back to the questions above and see if there questions above that you can answer differently that would change the answer to this question to yes.
I hope that these questions help you get to a good answer.

How to debug slow Office application interop constructor?

I have an application which deals with excel. Recently I encountered a problem with very slow creation of Excel object.
I've recreated the issue with this simple code:
Microsoft.Office.Interop.Excel.Application xlApp;
xlApp = new Microsoft.Office.Interop.Excel.Application();
The second line causes the delay.
In order to measure the time needed for new object allocation, above code has been extended with time tracking solution and the results are conclusive. In NORMAL situation, above code executes in 0.5s while in case of FAULTY-BEHAVIOR it can take up to 5 minutes.
There are no memory leaks and excel objects are being properly freed. My solution has been running 24/7 whole year without any issues. I'm not sure if it's important but the application is running on 20 separate user's sessions (server machine). So there are 20 copies of this application running at the same time and it may result in 20 copies of Excel running at the same time.
First time the issue has been noticed 2 months ago and has been solved by upgrade of Office (2010 -> 2013). This time I have more time to investigate and sadly results aren't promising.
Facts:
only one machine is currently affected by this issue (24 cpu cores, 24GB of Ram)
CPU isn't stressed at all when the "delay" happens
I've tried using "process monitor" application to verify what happens when we "new Excel.Application()" constructor (to see if there is any excessive disk/memory/cpu usage) - no signs of resources limitations. No sign of log files related to COM objects, etc.
The only issue here is this few minutes of delay. All other Excel Interop commands work as usual.
Main Question:
Is there a way to debug this Microsoft.Office.Interop.Excel.Application() constructor to see which part is an issue here?
External content
One guy with similar issue. His solution didn't help with my problem at all.
EDIT - additional test
PowerPoint constructor is not affected by the delay
ppApp = new Microsoft.Office.Interop.PowerPoint.Application();

I've found solution on my own. I'll post it as someone else may encounter similar problem and it can save him hours/days of investigation.
What i did to find solution?
I've analyzed test application (basically only one line where new excel application is being created) with Process Monitor and it didn't show anything important. Then I repeated analysis with newly started Excel process. It highlighted numerous reads of windows registry
HKEY_USERS\S-1-5-21-2929665075-1795331740-364918325-1024\Software\Microsoft\Office\15.0\Excel\Resiliency\DocumentRecovery
Under above location I've discovered tens of thousands of keys. They all were created by Excel's "auto-recovery" functionality. Because of the numbers, loading them when starting new Excel object was taking about 40 seconds. This number was additionally being multiplied by another 10-20 simultaneously loaded sessions (did I mention my application is running on 20 user sessions?).
Solution:
Removal of "Resilency" registry tree does the trick.
Why all these "auto-recovery" entries were there in a first place? I guess I don't handle closing of Excel very well and it "thinks" I'm having regular crashes and "tries" to help.
Now what's left is preventing it from happening all over again. I'll have a closer look at my ExcelClose() function.
Thanks for your attention - Adrian

I don't think the problem is with this constructor. Try to create the object dynamically:
var obj = Activator.CreateInstance(Type.GetTypeFromProgID("Excel.Application"));
Then cast it to Microsoft.Office.Interop.Excel.Application:
var xlApp = (Microsoft.Office.Interop.Excel.Application)obj;
MessageBox.Show(xlApp.Name);
I'd expect the slow-down to move to the Activator.CreateInstance call.
Anyway, you can try to work it around by placing the following into you app.config file (more details):
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
I'd also suggest to make sure you're running the latest VSTO Runtime and the latest Office PIAs.

Why does opening a form takes longer the first time?

I have made a simple search utility that will search for files in your computer.
It has a function search which searches for the files and creates the list of matched items to a mainloop function which in turn calls displayForm function which displays the results of the search in a new form.
Whenever, I run the application the first time after startup, although the search function completes the search in about 1 seconds, the time taken to display the result window takes considerable time(about 10 seconds) . This happens only for the first time you search after startup.
I am not providing any code for search function because I don't think search function matters because It takes almost same amount of time irrespective of running it the first time or subsequent times, and the working of displayForm is very simple.
public void displayForm()
{
// Do some stuff here
// Make a listbox and add items to display.
SearchForm.ShowDialog() ;
}
Also, by experimenting with a few cases , I must tell you that making a list box takes the same time irrespective of running it first time or subsequent times.
What could be the possible reasons for this ?

This is entirely normal, it has little to do with your code. Cold start time is dominated by the speed of the hard disk. Which can be disappointing when it has to locate the many DLLs that are required to start a .NET app. Not a problem exclusive to .NET apps, large unmanaged apps like Office apps and Adobe Reader have this issue as well. Which tend to cheat by installing an "optimizer", a program that runs at login and makes your machine slow by pre-loading the DLLs that the program needs so they are available in the file system cache, defeating SuperFetch in the process.
The operating system's file system cache is a pretty effective solution for the slow disk, but it is like a mile long freight train to get up to speed. Getting it filled from scratch with useful data takes a while, effective disk transfer rates when it has to seek is at best a few megabytes/sec. Also the core reason that users like an SSD so much, it provides a much more fundamental solution. Once you've experienced one, you can never go back.
Covered very well in many excellent articles, best way to find them is by googling ".NET cold start time".

Using the bittorrent protocol to distribute nightly and CI builds

This questions continues from what I learnt from my question yesterday titled using git to distribute nightly builds.
In the answers to the above questions it was clear that git would not suit my needs and was encouraged to re-examine using BitTorrent.
Short Version
Need to distribute nightly builds to 70+ people each morning, would like to use git BitTorrent to load balance the transfer.
Long Version
NB. You can skip the below paragraph if you have read my previous question.
Each morning we need to distribute our nightly build to the studio of 70+ people (artists, testers, programmers, production etc). Up until now we have copied the build to a server and have written a sync program that fetches it (using Robocopy underneath); even with setting up mirrors the transfer speed is unacceptably slow with it taking up-to an hour or longer to sync at peak times (off-peak times are roughly 15 minutes) which points to being hardware I/O bottleneck and possibly network bandwidth.
What I know so far
What I have found so far:
I have found the excellent entry on Wikipedia about the BitTorrent protocol which was an interesting read (I had only previously known the basics of how torrents worked). Also found this StackOverflow answer on the BITFIELD exchange that happens after the client-server handshake.
I have also found the MonoTorrent C# Library (GitHub Source) that I can use to write our own tracker and client. We cannot use off the shelf trackers or clients (e.g. uTorrent).
Questions
In my initial design, I have our build system creating a .torrent file and adding it to the tracker. I would super-seed the torrent using our existing mirrors of the build.
Using this design, would I need to create a new .torrent file for each new build? In other words, would it be possible to create a "rolling" .torrent where if the content of the build has only change 20% that is all that needs to be downloaded to get latest?
... Actually. In writing the above question, I think that I would need to create new file however I would be able download to the same location on the users machine and the hash will automatically determine what I already have. Is this correct?
In response to comments
For completely fresh sync the entire build (including: the game, source code, localized data, and disc images for PS3 and X360) ~37,000 files and coming in just under 50GB. This is going to increase as production continues. This sync took 29 minutes to complete at time when there is was only 2 other syncs happening, which low-peak if you consider that at 9am we would have 50+ people wanting to get latest.
We have investigated the disk I/O and network bandwidth with the IT dept; the conclusion was that the network storage was being saturated. We are also recording statistics to a database of syncs, these records show even with handful of users we are getting unacceptable transfer rates.
In regard not using off-the-shelf clients, it is a legal concern with having an application like uTorrent installed on users machines given that other items can be easily downloaded using that program. We also want to have a custom workflow for determining which build you want to get (e.g. only PS3 or X360 depending on what DEVKIT you have on your desk) and have notifications of new builds available etc. Creating a client using MonoTorrent is not the part that I'm concerned about.

To the question whether or not you need to create a new .torrent, the answer is: yes.
However, depending a bit on the layout of your data, you may be able to do some simple semi-delta-updates.
If the data you distribute is a large collection of individual files, with each build some files may have changed you can simply create a new .torrent file and have all clients download it to the same location as the old one (just like you suggest). The clients would first check the files that already existed on disk, update the ones that had changed and download new files. The main drawback is that removed files would not actually be deleted at the clients.
If you're writing your own client anyway, deleting files on the filesystem that aren't in the .torrent file is a fairly simple step that can be done separately.
This does not work if you distribute an image file, since the bits that stayed the same across the versions may have moved, and thus yielding different piece hashes.
I would not necessarily recommend using super-seeding. Depending on how strict the super seeding implementation you use is, it may actually harm transfer rates. Keep in mind that the purpose of super seeding is to minimize the number of bytes sent from the seed, not to maximize the transfer rate. If all your clients are behaving properly (i.e. using rarest first), the piece distribution shouldn't be a problem anyway.
Also, to create a torrent and to hash-check a 50 GiB torrent puts a lot of load on the drive, you may want to benchmark the bittorrent implementation you use for this, to make sure it's performant enough. At 50 GiB, the difference between different implementations may be significant.

Just wanted to add a few non-BitTorrent suggestions for your perusal:
If the delta between nightly builds is not significant, you may be able to use rsync to reduce your network traffic and decrease the time it takes to copy the build. At a previous company we used rsync to submit builds to our publisher, as we found our disc images didn't change much build-to-build.
Have you considered simply staggering the copy operations so that clients aren't slowing down the transfer for each other? We've been using a simple Python script internally when we do milestone branches: the script goes to sleep until a random time in a specified range, wakes up, downloads and checks-out the required repositories and runs a build. The user runs the script when leaving work for the day, when they return they have a fresh copy of everything ready to go.

You could use BitTorrent sync Which is somehow an alternative to dropbox but without a server in the cloud. It allows you to synchronize any number of folders and files of any size. with several people and it uses the same algorithms from the bit Torrent protocol. You can create a read-only folder and share the key with others. This method removes the need to create a new torrent file for each build.

Just to throw another option into the mix, have you considered BITS? Not used it myself but from reading the documentation it supports a distributed peer caching model which sounds like it will achieve what you want.
The downside is that it is a background service so it will give up network bandwidth in favour of user initiated activity - nice for your users but possibly not what you want if you need data on a machine in a hurry.
Still, it's another option.