I am build a system that is similar to project builds. A project has set of inputs and outputs. A project may depend on other projects. I can use topological sort and find the sequence in which I must evaluate projects. But, how do I parallel build. Toposort does not give same ranks to projects that can build in parallel. Also, how do I do incremental builds.
You can parallel build all projects which have not any dependencies. After build remove builded project from graph and again build projects without dependencies. Repeat until graph is empty.
In pseudo code
Build project graph
Validate graph (e.g. circular dependencies)
Find all projects without dependencies and not under build
Start build all found in 3 and mark them under build
Wait for one of builds end and romove builded project from graph
if graph not empty go to 3
Wait for rest builds
Related
Recently we had this case:
We have multiple projects in our solution that depend on each other. With project dependencies we make sure the project is build in the right order.
In the case above a developer forgot to add a project dependency. Therefore 2 things could happen:
1. The build would fail because msbuild builds and outputs the projects in an order that does not work
2. The build would succeed because msbuild builds and outputs the projects in the right order (by accident)
As you can see above, the first nightly build succeeded, the second nightly did not, and the third succeeded again. This can be confusing. I would like to make the builds more repeatable, so that the consecutive builds either all fail or all succeed.
Is there an approach i can take to make this better?
I read about deterministic builds but i am unsure whether that would help my case:
https://github.com/dotnet/roslyn/blob/master/docs/compilers/Deterministic%20Inputs.md
Additional info;
- The code is checkout clean before each build
- We use a plugin-based model so projects are not stitched together with project references, but plugin projects are build separately without dependencies. Then we reference the output dll from our application projects. Therefore the plugin should be built before the application project. We manually add project dependencies to make sure this happens.
I have 2 solutions that contain test related code. One is for UI automation and another for API. I combined both solutions and I am able to build and run tests from both but:
Each one of those contains similar configs that I would like to combine under one project. No need to have the same thing in both obviously. I removed configs from project2 and created references from Proj1 to Proj2. I build Project1 and artifacts are generated fine from Project1, but it does not contain all artifacts from Proj2.
Previously, when I build projects separately, It would generate say 60 files in the build folder, but now its much less.
Question:
If I decide to add project #3 and add configs to it. How can I build project 3 and combine artifacts from all of them in bin folder of project #3.
I hope I'm making this clear.
Essentially I want to be able to build 1 project which will trigger builds for remaining 2 and pull all artifacts to together.
As Ziggler suggests, you can change the build output folder to combine all artifacts from all of them in bin folder of project #3.
And you can do this by passing $(OutputPath) property in command-line. It seems that you're using something like TeamCity, Azure Devops or what in your environment.
For these tools, we can add msbuild parameter like: /p:OutputPath="Path to project3's bin folder" if you're using msbuild to build whole solution(msbuild xx.sln). See Global Property.
And about the build order, you can add project dependency to control the build order. Msbuild will recognize project dependency and project reference and then choose the right order automatically. Hope it helps :)
I am currently migrating a large solution from the old csproj file format to the new csproj format. I am doing this a few projects at a time, so I have a mixed environment with some projects using the old project file format and some projects using the new project file format.
I have started to notice some builds failing because files are in use. My theory (based on this answer) is that MSBuild is building some projects twice because the properties are different (i.e. the new project file format specify the TargetFramework property while old projects do not).
The projects that seem to have concurrency issues are projects that are referenced by other projects, where the referencing projects are split between the project file formats.
The command I am using to build the project is:
msbuild.exe /maxcpucount:6 /property:Configuration=Debug;Platform=x64 /t:Rebuild my.sln
Is there a way to instruct MSBuild to only build a given project once (regardless of properties) until I am able to convert all of the projects in the solution over to the new project file format?
Note that building single threaded does correct the concurrency issues, but that significantly slows down the build and the projects are still built multiple times.
This is a bug in MSBuild when referencing C++ CLI projects from multi-targeting projects. It appears they are putting in a fix to address this.
To work around the bug, the property can be removed from the reference using the GlobalPropertiesToRemove attribute:
<ProjectReference Include="..\B.CppCLILibrary\B.CppCLILibrary.vcxproj" GlobalPropertiesToRemove="TargetFramework" />
Only you could have correctly solved this since we do not have access to your code. But generally MSBuild and all build systems operate under the rule that a build 'target' is only processed once. No matter how many projects refer to it.
A build system should provide language for the user to specify dependencies between different 'targets'. And then it is up to the build system to figure out in which order to build these things in. (i.e. called a topological sort). Beginning or starting with making the most independent targets all the way to the least independent target.
If something is getting built twice it could be:
1. A bug in the build system.
2. The user forcing the project to build twice.
Anyways glad you got it sorted out.
I have a small project with a lot of dependencies. I changed one of the files in the project and noticed that MSBuild re-compiles all the dependencies also. What might cause this? Is this the default behavior and if so how do I change it?
If it matters I'm compiling via the command line (not from visual studio)
It could be building them all, or it could just be checking the dependencies to determine if they need a rebuild.
First, what kinds of projects are in the solution? More obscure project types may have targets that were not written with incremental builds in mind and may have buggy behavior (such as 2012 SQL Projects always rebuilding because it attempts to resolve SQLVAR params to file locations).
Here's some advice to get you on your way:
Prefer a file logger allowing you to troubleshoot without relying on the console buffer.
msbuild.exe MySolution.sln /fl3 /flp3:warningsonly;logfile=msbuild.wrn /fl4 /flp4:errorsOnly;logfile=msbuild.err /fl5 /flp5:Verbosity=diag;logfile=msbuild.log
This will generate three logs, two for warnings and errors, and a log with the greatest level of detail. Open msbuild.log and identify the build for one of the dependencies.
Does the build log indicate targets were skipped?
Skipping target "CoreCompile" because all output files are up-to-date with respect to the input files.
or that they were rebuilt?
Building target "SqlBuild" completely.
MsBuild can build your projects in parallel.
Another trick to enhancing performance is to build your projects in parallel. MsBuild is the scheduler and can build your dependencies in parallel using the magical "/m" switch!
I am having problems getting my C# Solution to build "Fresh". If I clean the solution and build it again it will not build (I can do it a few times and it will build). It has an error about the azure project getting build before the worker and web projects that it is dependent on. Also about how most of the projects in the solution are looking for
WAT070 : The referenced assembly {...}/Worker.dll was not found.
Please make sure to build the role project that produces this assembly before building this Windows Azure Cloud Service Project.
{...}\VisualStudio\v12.0\Windows Azure Tools\2.2\Microsoft.WindowsAzure.targets 1252 5 AzureProjectName
Now if I build the projects in the order listed in the Project Dependencies -> Build Order everything works. Also, the web and work role are listed before Azure Project.
Solution is very simple.
You can set your project build order by right click on 'Project Solution' and select "Select Project Build Order" option.
For Example, I have a WpfFormApplicaiton1 and two class with title "ClassLibrary1" and "ClassLibrary2". By default Visual Studio sets it as follow:
My requirement is such as "ClassLibrary2" will be used by "ClassLibrary1" & "ClassLibrary1" will be used by "WpfFormApplication1". So, in order to fulfill this requirement I have to change the default project build order.
Go to Project Dependencies; Select the "ClassLibrary1" and set the "ClassLibrary2" as its dependency.
Similary, select the "WpfFormApplication1" and set the "ClassLibrary1" as its dependency.
Now, the desired project build order is set; confirmed by Project Build Order's options.
Solution is taken from my blog.
Your issue may be that the Dependencies are not defined. Even though the Build Order shows the order in which projects are built, if you do not define the dependencies for each one under Project Dependencies, msbuild will not know to wait for the dependencies build to complete before moving on.
To clarify: Unless you actually check the box that an item is a dependency, the projects in the build order list may build in parallel and not sequentially.
You can see under Tools->Options->Project and Solutions->Build and Runthat there is a default value for the number of parallel projects to build.
So to make the build process wait for dependencies to build make sure that all of the "Depends on" fields are checked for the projects needed under Project Dependencies -> Dependencies.
We had an issue where the project guids differed in case. Editing the project files solved the issue. We changed all Guids to uppercase
I just had this issue too. In my case the issue was that I had several project references within the solution. The other projects were using a different version of the framework to my Worker Role (4.5.1 vs 4.5).
When I changed all projects to use the same version of the framework the solution builds and runs successfully.
I had this issue. In my case, the solution's project build order, as determined by VS.Net, was not correct. Specifically, my web project was listed above three of its dependencies.
These three dependencies were listed under in the References node of the web project in the Solution Explorer. However, in the dialogue Project Dependencies, the web project did not depend on any project.
Also, I noticed that the web project had a small blue exclamation mark, with hoover-over message "The Web project '' requires SQL Server Express, which is not installed on this computer. ...". After fiddling with the web.config, based on ASP.NET Web Api: Project requires SQL Server Express and reloading the project, the exclamation disappeared and the project dependencies were correctly checked in the dialogue Project Dependencies and the build order reflected this correctly. However, when I reverted the changes to the web.config, as a test, the dependencies were not removed, so I am quite unsure what fixed my issues.
Anyway, in the dialogue Project Dependencies, you can manually check any project which was not automatically identified as a dependency.
Check to make sure there aren't any residual old files in your working folder. These can cause confusions with MSBuild. To avoid that, simply blow away all the old files in the path and get latest.