We Keep Coding

Managing tons of 3rdparty dependencies in PInvoke for better compilation speed

The situation:
I'm working on a research project which, due to some constraints, has a C# user interface (used mostly for visualization) but does most of the processing with PInvoke and unmanaged C++ code. The unmanaged code has TONS of dependencies on various 3rdparty libraries: Boost, PCL, OpenCV, CGAL, VTK, Eigen, FLANN, OpenMesh, etc. (if you can name it, we probably depend on it!). The C# project interacts with a C++ project (which I simply refer to as "wrapper" from now on). Obviously, the wrapper is where all the 3rdparty dependencies are consumed and is where entry points for PInvokes are defined. The wrapper is compiled into a DLL and copied into the output directory of the C# project via a post-build event.
I am the sole developer of the project. My primary development platform is Windows 10 with Visual Studio 2015 and Git is my version control. I mostly develop on 3 differenct machines, but sometimes I need to develop on other machines which only have Visual Studio 2015 installed.
What I've done so far:
Obsiously, managing all those 3rdparty dependencies is a hassle for one person, and I'd hate to have to install those libraries on new development machines . What I've done is that I've compiled all those 3rdparty libraries from source into static lib files (except the header-only ones obviously). All sources are built once for Debug configuration and once for Release configuration. I spent some time and integrated them into my wrapper project (i.e. defining extra include directories, using lots of #pragma comment (lib, "blah.lib") which reference different builds depending on the build configuration, etc.). I also followed some of the advice in Microsoft's linker best practices, to reduce link times. Specifically, I'm using the incremental linker, I've disabled /LTCG and /OPT.
Now I have this gigantic "Dependencies" folder in my VS solution which is around 8GBs, and is version-controlled separately from the project (using a Git submodule). The wrapper project gets statically linked to all these, as a result and as mentioned above, only one DLL is produced after building the wrapper project. The upside of this approach is that on any new development machine, I clone the main repository, clone the Dependencies submodule and I'm ready to roll! But...
The worst part:
You've guessed it! Terrible link times. Even on a powerful computer, after I change a single line in the wrapper project, I would have to sit for a couple of minutes till the linker finishes. The thing I didn't see when I took the above approach was that I forgot how much I valued rapid prototyping: quick and dirty testing of some random feature in the wrapper project before exposing that feature to PInvoke. Ideally, I would like to be able to change something small in the wrapper project, quickly build, run and test that change and get on with exposing the feature to PInvoke.
My Question:
Clearly, I'm inexperienced in this! How should I have done things differently, specifically given the dependencies I mentioned above? Was building DLLs instead of static libraries better? But then I would've had to add the Dependencies to PATH everytime the C# program started (as mentioned here). As a side question, how do you evaluate my current approach?

Based on the comment by #silverscania, I decided to just take the DLL route. It was a bit of pain rebuilding all the dependencies, but I'm now super happy about the results.
Now, building the whole solution from scratch takes 36 seconds! It used to be about 4 minutes before, so I have nothing to complain about. Also, modifying a single file in the wrapper project and building again takes 3 seconds which is amazing! The fact that all the compiled dependencies are now about 1 GB (opposed to ~8GB with the static libraries) is a plus! I couldn't be happier.
A coupt of notes:
On the main machine where I do most of my development, I have a SanDisk SSD. I noticed that for some reason beyond my comprehension, building the project on that device was way slower compared to a regular HDD. I'm looking into this issue, but haven't found an reason for this (TRIM is enabled and the drive is in AHCI mode).
I played around with the flags a bit more. I noticed that the compiler flag /GL (Whole program optimization) caused considerable slowdown during linking. I disabled that option too.

Easiest way to use a managed library in Unity3d that's part of the solution

I have a Unity creates solution for UWP (hololens actually, not sure if it matters).
Most of my work is done on a class library that contains my logic. I want to call from the unity C# scripts into this class library I am using. Since the Assembly-csharp DLL gets recreated completely every time, I obviously can't just add a project reference in there.
What's the best way to achieve this?
What I've found so far is that I can pull the actual DLL binary of my library DLL into the Unity Assets folder, at which point it actually does get referenced in the Assembly-csharp project - however - the DLL actually gets copied to a separate location (\Unprocessed\MyLibrary.dll). That means I need to copy my project there everytime I make a change (or re-publish from Unity which takes a bunch of time).
I know I can have a post-build stage that will copy the file - however - is there any more "correct" way? The way this works now (dropping a DLL into the Assets) also has other disadvantages such as making building a certain flavor much harder than it should be.
Thanks

That is the correct way: you put the library DLL in Unity's Asset folder, and never touch the generated C# project.
Consider that in most common scenarios, both the library itself and the Unity code referencing it would change so a rebuild of the Unity project is a natural part of the process. If that takes too much time, it is usually advisable to create a dedicated, smaller Unity project to quickly iterate through the DLL development cycles.

Additional global include directory in Visual Studio for C#

There are a lot of little things I find myself re-writing here and there because they might be too large/complex to represent as a snippet, but realistically it doesn't make sense to make a stand-alone DLL out of it because we might only be talking a few dozen or a few hundred lines of code.
For example a little form which contains only a text box where the user enters a password and closes on {Enter}.
Or an extension method which can serialise/deserialise any object to/from a GZipped file assuming the object is marked as Serializable.
The list goes on. I have accumulated lots of little bits and pieces over the years and it's not organised in any neat way.
In C++ projects, I can write a lib file containing these bits of code which I can add to my compiler settings in such a way that any future C++ project I create has this lib included. I have done this with ATL and Boost.
I don't know of a way to do this for C# projects. Is it possible?
Edit:
If I make an assembly, I have to compile it to a DLL and distribute the DLL alongside my main executable. The DLL may be small or it may be quite large, I don't know. But I may only need to use a few tiny functions in that DLL for my project. In C++, only the functions I use are statically linked when I use the library, however if I distribute my software with a DLL then I have to distribute everything.
I know it is possible to merge the DLL with the main executable so that the user isn't aware that there is a separate library, however the whole DLL is still being packaged along with the executable.
Imagine I write a DLL with lots of my own maths, stats, file IO, image manipulation, serialisation, user IO, etc included. Nothing fancy, just some common things I find myself doing quite frequently. The DLL might be, say, 4MB.
Now I want to write a program which uses a tiny part of the DLL, and if I were to simply copy/paste the necessary code then my EXE would end up being, say, 700kB.
Are you saying that I either copy/paste the code I need, or I have to distribute a 4MB DLL along with my 700kB EXE?

Aside from using an assembly, the only way I know of is to create a link in your project to the source code in question. In visual studio the process is:
Project → Add → Existing File → Add As Link (the little down arrow:)

It is not possible at a source code level, although often requested (just Google c# #include equivalent). The only reasonable alternative that c# offers is compiling your common code as a DLL and adding a reference to it.
Note that although you can add a file to your project from another project, it will take a copy and therefore not maintain updates. I have used this to achieve the same effect 'manually' - when the common file is updated, I excluded it from the project 'referencing' it and then re-added to get a fresh copy.
UPDATE As commented below, you can add as a link - how cool! Why did nobody tell me.

We add a common directory to the overall includes path, then use
#include <somefile.cpp>
directly in our cpp files. It'll include the source straight in.

Where do you put your 3rd party libraries?

I've got a bunch of .dll assemblies, such as HtmlAgilityPack and MoreLinq. Where am I supposed to put these files? I usually toss them somewhere in my Projects folder, but then I'm always digging around for them. Is there a standard place to put them?

There's no standard place to put them, but make sure you:
Put them in one place
Include them in source control.
I put all my required dll's in a top level directory in my solution called "Dependencies", parallel to the project folders. I have them in source control such that when new developers check out the solution, it all compiles and works right off. It's the only way to go.
I include only the .dll files absolutely needed. This keeps it light, which is good, but then when I find some other part of MVC Contrib or whatever that I need, I have to go find the unzipped directory, which might not even be on my computer! Others put entire library directories (readme.txt and all) as part of their source control linked to the solution. This ensures you and future developers will have everything they need, but adds a little dead weight. Either is a good strategy.

Having a "Lib" folder at the same level as source projects is a common way.
To be honest, it's not the dependencies my projects have that I find hard to manage, it's the dependencies the dependencies have. I'd just like to mention NHibernate, Castle Windsor and the various Castle Windsor Facilities in particular. Getting all of those to play together on my last project cost me a lot of time.
For open source projects, I also like to have the source code handy because sometimes its useful to debug into the source code. (And sometimes because the documentation is so poor, you have to read the source code to find out how it works). I've seen VS projects arranged so that the project references the DLL yet at the same time, VS knows where to find the source code, as I write I can't quite remember how to do that.
So, a Lib folder for DLLs works for me; I often call it "Shared Dependencies".
As for open-source source code, I don't have a standard way to version that because each project is structured differently and has a different build process. I don't like to tinker with the open-source project structure or build method because then, I take responsibility for it. If for some reason, it won't build, or builds incorrectly, or produces a faulty DLL, the cause would be exceedingly difficult to track down, and I'd have to get deep into troubleshooting all of that which I dont care about at all.

In a folder UNDER your solution directory, e.g. "external" or "library". That way your continuous integration system (or other team members) can do a pull of one root from your source control system and have everything they need.
In SVN, use svn:externals to pull that directory from a different root so you can easily share library DLLS (and library projects) between solutions.

In the office we have a share on the network for referenced asseblies. These could be 3rd party or assemblies of our own that could be shared between projects.
I also, don't like the idea of putting the dll files in source control. If all the developers have access to the share all will work fine.

The visual studio directory in My Documents seems like a logical place to put them. I don't know if it's the best or anything wrong with it but at least all the libraries are found in one place.
%USERPROFILE%\My Documents\Visual Studio XXXX\Libraries

At my company we place all our shared DLL assemblies onto a network drive in a folder called Assemblies. From there, we use SyncToy to mirror changes between that folder and a folder on our local development machines (in my case C:\Assemblies with subfolders for different versions or useful third party assemblies). Using the "Reference Paths" feature of Visual Studio projects makes it very easy to select different assembly versions based only on locations.
For projects at home, I would definitely go with the idea mentioned by Jeff M of placing them in the Visual Studio folder under My Documents.

I don't have a hard and fast rule on the location. However, I would encourage consistency!
For example, I needed to to this for a small tool I'm writing for a client at the moment, so I checked their other code bases in Bitbucket which seemed to use a dependencies folder in the solution folder (alongside the other projects), so I copied that.

How do you package external libraries in your .Net projects?

A lot of my projects contain the Castle/NHibernate/Rhino-Tools stack. What's confusing about this is that Castle depends on some NHibernate libraries, NHibernate depends on some Castle libraries, and Rhino-Tools depends on both.
I've built all three projects on my machine, but I feel that copying the NHibernate/Castle libraries is a bit redundant since I built Rhino-Tools using the resulting libraries from my NHibernate and Castle builds.
Right now, I include all projects in seperate folders in my /thirdparty/libs folder in my project tree. Should I simply just have /thirdparty/libs/rhino-tools in my project and use the Castle/NHibernate libs from there? That would seem to make logical sense in not duplicating files, but I also like having each project in it's own distinct folder.
What are your views on this?

This is one of the problems that we're trying to tackle in the Refix open source project on CodePlex.
The idea is that Refix will parse all the projects in your solution, and before your project compiles, copy the necessary binaries from a single local repository on your machine into a folder within the solution tree and point the projects at them. This way, there's no need to commit the binaries. Your local Refix repository will pull binaries from a remote one (we're setting one up at repo.refixcentral.com), and you can set up an intermediate one for your team/department/company that can hold any additional software not held centrally.
It will also try to resolve conflicting version numbers - Visual Studio can be too forgiving of mismatched component version numbers, leading to solutions that compile but fall over at run time when they fail to load a dependency because two different versions would be needed.
So to answer the question "how do you package external libraries in your .Net projects", our vision is that you don't - you just include a Refix step in your build script, and let it worry about it for you.

I use a folder for each, which seems to be the convention.
Does it really make a difference if you're copying them?
What if you want to switch one out? Let's say you go with a new O/R mapper. It will be much easier to just delete the NHibernate folder than to selectively delete DLLs in your Rhino-Tools folder.
Take this to it's logical conclusion and you won't have any folder organization in your lib folder since everything uses log4net :)

Add additional probing paths to your app.config files to locate the dependency dlls. This way your can get away with having just one copy of everything you want. Though there are some quirks to using this feature (you must create the folder structure in a certain way). Look here for more details on the tag.

I will definetly recommend having a thirdparty or vendor folder in each of your project trees. If you find it annoying to have 32 copies of the rhino-tools package, you can have a single copy of it in your code repository, and do external references to it in your project tree.
Lets say you are using SVN, you can make a repository called "thirdparty libs" and in this have versioned copies of the libs. You then make an external property on your "thirdparty"-folder in your project tree which then in turn automaticly will do a check out of your centralized thirdparty libs. This way you for instance only have to update in one place if a security or a bugfix comes out, but each project is still in command of choosing which thirdparty libs, and which versions to use.
About the deps internally in thirdparty libs, i wouldn't mind those. The first time you compile your project, and some of the libs arent copied to your bin-folder because of implicit dependencies you can add an external attribute into your bin-folder, which will then automaticly check out the missing libs. That way you still only have to update your thirdparty libs in one place.