[cmake-developers] [CMake] Visual Studio <Version> <Arch> - Ninja Generator

[James Johnston]

Recently I have wondered if it would be useful if the Visual Studio generators in CMake were refactored somewhat (to what degree, I am not sure).  Not that I have time to work on it right now, and I have not studied this section of CMake code in detail, so I may be a little off base with some of this - please bear with me...  Example: instead of Visual C++ 2008 generator, we have a back-end generator for VC++2005/2008 projects.  Then instead of Visual C++ 2010 generator, we have back-end MSBuild generator.  (Perhaps the back-end generators could be tools reused by existing normal "Visual Studio 2008" CMake generators and so on).  If a VC++ 2008 project is needed, the back-end VC2008 generator is used.  If a C# 2008 project is needed, the back-end MSBuild generator is used.

There are some use cases:

 * The example given by Guillaume Dumont.  In this case, it would be useful to have Visual Studio available as an "Extra" CMake generator.  For example, specification of "Visual Studio 2015 - Ninja" (much like how you can do "CodeBlocks - Ninja" today, etc.).  This would run Ninja in background to do the building, while retaining the IDE.  Presumably this generator will have to share a lot of code with the regular VS2015 generator.  I would be very happy if this feature existed and would use it tomorrow.  The MSBuild system has serious concurrency issues resulting in serious over/undersubscription.

* Recently there have been mailings from Michael Stuermer about adding C# as a language CMake.  Another feature I would use tomorrow if it worked well.  I skimmed through some of the commits to see how he is doing it; while helpful, I think there are some shortcomings that may not be easy to fix without some preliminary work related to Visual Studio project generation.  IIRC the author's code works for generating Visual C# 2013 projects using the Visual C++ 2013 generator, but that's it.  Ideally you really want to be able to: (1) build C# 2005/2008 projects, (2) use more efficient tools like Ninja.  And to do that I think requires decoupling the language from the project file generator.  Right now, CMake makes the assumption that the language project file format of each VS version is basically identical (e.g. all MSBuild, or not), and this is not true.

To elaborate on the C# example some more, suppose the user wants to generate a Visual Studio 2008 solution.  So they pick the normal VS2008 generator.  In this situation: (1) C++ projects need to be generated using the C++ project generation code in the existing VS2008 generator, (2) C# projects need to be generated using an MSBuild generator, which is very different from the VS2008 C++ project format - i.e. these projects will need to use code from the current VC++ 2010 generator.  Summary: to do this right for VS2005/2008 requires using significant code from two different existing CMake generators: some code from existing VS2005/2008 generator, and some code from VS2010 generator.

The Ninja C# example is also tricky to get right.  It may not be practical to invoke csc.exe directly; there might be other tools involved and a lot of how things "should" work is currently largely undocumented and hidden away inside of the system MSBuild .targets files.  An example is resolving assembly references.  If you pass in an unqualified assembly reference directly to csc.exe, the compiler searches for the assembly in various directories using rules defined here: https://msdn.microsoft.com/en-us/library/s5bac5fx.aspx - however, these rules do not match what actually happens when you add an assembly reference to a Visual C# project and compile it with MSBuild.exe.  In that situation, MSBuild.exe resolves assemblies using a COMPLETELY DIFFERENT set of rules than csc.exe, and then passes absolute paths to csc.exe.  The correct rules to use are probably the ones inside of Microsoft.Common.targets file, which doesn't seem to be documented in MSDN, and only documented by a comment in the XML:

        <!--
        The SearchPaths property is set to find assemblies in the following order:

            (1) Files from current project - indicated by {CandidateAssemblyFiles}
            (2) $(ReferencePath) - the reference path property, which comes from the .USER file.
            (3) The hintpath from the referenced item itself, indicated by {HintPathFromItem}.
            (4) The directory of MSBuild's "target" runtime from GetFrameworkPath.
                The "target" runtime folder is the folder of the runtime that MSBuild is a part of.
            (5) Registered assembly folders, indicated by {Registry:*,*,*}
            (6) Legacy registered assembly folders, indicated by {AssemblyFolders}
            (7) Look in the application's output folder (like bin\debug)
            (8) Resolve to the GAC.
            (9) Treat the reference's Include as if it were a real file name.
        -->

For example, I found when building with MSBuild, absolute paths to assemblies in "C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\v3.5" were used, and this location is not used if I pass an unqualified path directly to csc.exe and use csc's rules.  This blog post suggests that these new assembly resolution rules are very important:  http://blogs.msdn.com/b/msbuild/archive/2007/04/12/new-reference-assemblies-location.aspx  ---  Therefore, I have to conclude that there are only two valid ways of compiling a C# project from CMake:

a.  Reverse engineer the MSBuild files for each Visual Studio project and teach CMake largely undocumented rules for things like assembly resolution and whatever else we run into.  If it's just assembly resolution it might be manageable, but I fear it's a rabbit hole...

b.  Have CMake generate a C# project file for each C# project - even when using non-VS generators like Ninja.  The Ninja build rule for a C# project would just be to invoke MSBuild.exe and build that one project.  (Only generate projects, not solutions.)  But this requires the ability for a CMake language to invoke the MSBuild-generating code inside CMake to spit out a Visual Studio project for each C# target.  Benefit: now CMake doesn't reimplement undocumented behavior of Microsoft's MSBuild targets; we just use them directly as a black box.

Best regards,

James Johnston

From: CMake [mailto:cmake-bounces at cmake.org] On Behalf Of Guillaume Dumont
Sent: Monday, August 31, 2015 16:49
To: Gonzalo
Cc: cmake at cmake.org
Subject: Re: [CMake] Visual Studio <Version> <Arch> - Ninja Generator

@Hendrik Sattler
I have not experimented with the /MP flag that much, but this won't solve my problem when the project contains a lot of CUDA files. The /MP flag as not effect there. 
Yes I could indeed create a custom build target and create additional build trees to build using ninja in there but this is suboptimal in my opinion.

@Gonzalo
Yes this is precisely what I do but then you only get the ninja build files. No solution for debugging etc.
My question is more about the difficulty of creating a new generator that makes the use of ninja as transparent as possible to the user instead of writing custom CMake code to do this in my own projects.

On Mon, Aug 31, 2015 at 12:29 PM, Gonzalo <ggarra13 at gmail.com> wrote:
s it has changed names a couple of times).  That shoul




-- 
Guillaume Dumont
=========================
dumont.guillaume at gmail.com