[cmake-developers] Some numbers on cmake+ninja+VS on Windows

[Óscar Fuentes]

Setup: Windows XP SP3 32 bits running on VMWare Workstation 8.0.1 with 4
real cores (no HT) assigned out of 4 available on the host machine,
which is Kubuntu 11.08 64 bits (kernel version 3.0.0-16-generic).

Visual Studio 2010 Ultimate. JOM 0.9.4 (that's an old version, but on my
experience newer versions are slower).

CMake was taken from https://github.com/syntheticpp/CMake.git, branch
ninja-generator-pr-win revision fa9ce5e, plus this patch:

diff --git a/Modules/Platform/Windows-cl.cmake b/Modules/Platform/Windows-cl.cmake
index ccccbc9..be6abb6 100644
--- a/Modules/Platform/Windows-cl.cmake
+++ b/Modules/Platform/Windows-cl.cmake
@@ -37,7 +37,7 @@ SET(CMAKE_COMPILE_RESOURCE "rc <FLAGS> /fo<OBJECT> <SOURCE>")
 # that is automatically copied into try_compile directories
 # by the global generator.
 SET(MSVC_IDE 1)
-IF(CMAKE_GENERATOR MATCHES "Makefiles")
+IF(CMAKE_GENERATOR MATCHES "Makefiles" OR CMAKE_GENERATOR MATCHES "Ninja")
   SET(MSVC_IDE 0)
   IF(NOT CMAKE_VC_COMPILER_TESTS_RUN)
     SET(CMAKE_VC_COMPILER_TESTS 1)
@@ -125,7 +125,7 @@ IF(CMAKE_GENERATOR MATCHES "Makefiles")
     ENDIF(CMAKE_COMPILER_RETURN)
     MAKE_DIRECTORY("${CMAKE_BINARY_DIR}${CMAKE_FILES_DIRECTORY}/CMakeTmp3")
   ENDIF(NOT CMAKE_VC_COMPILER_TESTS_RUN)
-ENDIF(CMAKE_GENERATOR MATCHES "Makefiles")
+ENDIF(CMAKE_GENERATOR MATCHES "Makefiles" OR CMAKE_GENERATOR MATCHES "Ninja")
 
 IF(MSVC_C_ARCHITECTURE_ID MATCHES 64)
   SET(CMAKE_CL_64 1)
diff --git a/Source/cmNinjaTargetGenerator.cxx b/Source/cmNinjaTargetGenerator.cxx
index e48ac12..034aefe 100644
--- a/Source/cmNinjaTargetGenerator.cxx
+++ b/Source/cmNinjaTargetGenerator.cxx
@@ -230,9 +230,13 @@ cmNinjaDeps cmNinjaTargetGenerator::ComputeLinkDeps() const
   if(!cli)
     return cmNinjaDeps();
 
-  const std::vector<std::string> &deps = cli->GetDepends();
-  cmNinjaDeps result(deps.size());
-  std::transform(deps.begin(), deps.end(), result.begin(), MapToNinjaPath());
+  cmNinjaDeps result;
+  for(unsigned i = 0; i < cli->GetItems().size(); ++i) {
+    if( cli->GetItems()[i].Target ) {
+      result.push_back( cli->GetItems()[i].Target->GetName() );
+    }
+  }
+
   return result;
 }


Ninja was taken from https://github.com/syntheticpp/ninja.git, branch
token-splitter revision 335e08e.

I worked with some internal projects and with LLVM revision f340a29
taken from LLVM's official git mirror: http://llvm.org/git/llvm.git

First, some quick recap of the build tool I used so far with VS:

NMake: no support for parallel jobs.

MSBuild (equivalent to VS IDE): platform checks very slow, tricky
support for parallel jobs (one knob /m:JOBS at MSBuild.exe level and
another indepent one /MP at the compiler driver level; it's easy to end
with starved cores or with dozens of compile jobs competing for the
available cores.) On addition, CMake+MSBuild suffers from a nasty bug
that consists on not being able to use the regenerated project files
when CMake is invoked from within the build because some CMakeLists.txt
changed. There more issues with MSBuild when it is used as a
replacement of `make' but those described so far are enough to motivate
anyone to search for a better alternative.

Jom: no support for building more than one top-level target in
parallel. This was reported to the author time ago as a feature request
and was closed as wontfix.

Now, for the numbers.

Build LLVM with -DCMAKE_BUILD_TYPE=Release:

ninja -j 4: 6m55s
MSBuild (cmake --build .): 38m17s (*)
jom: 10m45s

(*): this very long time may be due to the lack of top-level
parallelization (no /m:JOBS was used for invoking msbuild.exe) and/or a
slower console when executing on VMWare for showing all the text output
that msbuild emits.

No-op build (best of two consecutive runs):

ninja -j 4: 0.4s
MSBuild: 11s
jom: 53s

`ninja -t clean' takes less than a second.

On an internal project here, a full build with ninja requires half the
time of jom/msbuild (35s vs 1m20s) while a no-op build is instantaneous
with ninja (0.1s) while reasonably fast with msbuild (2.3s) and
annoyingly slow with jom (8.5s). That project consists on a shared
library with ~30 cpp files plus 10 small (typically 1 cpp file) shared
libraries or executables that depends on the big shared library. On this
scenario, Ninja's correct handling of parallel jobs makes a lot of a
difference over jom and msbuild.

CMake + Ninja looks like a perfect match for people building with VC++
outside of the IDE.