[cmake-developers] NMake Batch-Mode Rules (was: Inquiry on contribution to NMake generator)

[Florent Castelli]

At my company, we didn’t bother trying to use NMake.
We used the Visual Studio solution generator and the proper compilation flags and msbuild flags to build in parallel. This proved to speedup our builds quite a lot and was very easy to integrate.
The other solution you mentioned is “ninja” and sped up our builds even more. The msbuild tool is not smart enough and with a number of targets high enough and enough files per target, you can just generate n^2 parallel compilation processes, which can be counter productive. Though, you can manually specify the parallelism factor you want for both msbuild and cl.exe and you get decent results.

Give those a shot, NMake is deprecated in Visual Studio land and has been superseded by msbuild.

And if your devs complain about having to install some tools before compiling, just give them a script that downloads and setup CMake and Ninja on their system. It’s easy to do and should reduce the number of complaints!

Regards,
Florent

> On 01 Jun 2015, at 11:10, Nagy-Egri MĂĄtĂŠ Ferenc via cmake-developers <cmake-developers at cmake.org> wrote:
> 
> Thanks Brad for the detailed description. From what I’ve read about the 3 level makefile system, it should be pretty straightforward to implement batch-mode support.
> 
> Actual compilation takes place only in the last level, and throughout generation, this is the point where all the compilation options are present. Obtaining a list of files with identical compiler switches might be tricky. Depends on how CMake handles this in code internally. I’ll see once I dive into code. Deferred generation of the makefile is not nice, but might be the only way. Handling the exact location of the object files might be another property source files must match.
> 
> As far as I’ve seen NMake is not able to call multiple makefiles in parallel, so source-level parallelism might be the most NMake could do, but for large projects that could be sufficient.
> 
> Feladó: Brad King <mailto:brad.king at kitware.com>
> Elküldve: ‎szerda‎, ‎2015‎. ‎május‎ ‎27‎. ‎20‎:‎13
> Címzett: Nagy-Egri MC!tC) Ferenc <mailto:csiga.biga at aol.com>
> Másolat: cmake-developers at cmake.org <mailto:cmake-developers at cmake.org>
> 
> On 05/27/2015 01:33 PM, Nagy-Egri MC!tC) Ferenc via cmake-developers wrote:
> > like to see the NMake generator finally support multicore builds by
> > adding Batch Mode support to the generated makefiles.
> 
> For reference, Batch-Mode Rules are documented here:
> 
>  https://msdn.microsoft.com/en-us/library/f2x0zs74.aspx
> 
> They support specifying multiple source files to a single "cl" invocation,
> as in
> 
>  cl -c src1.cpp src2.cpp
> 
> instead of
> 
>  cl -c src1.cpp
>  cl -c src2.cpp
> 
> Any multi-core utilization occurs inside "cl" and NMake is not aware of it.
> 
> > I have read on the CMake user mailing list, that the problem with
> > Batch Mode support, is that it is not trivial to implement.
> 
> It is very hard to implement and may not be possible at all.  See below.
> 
> >   * What is the design of the NMake generator in CMake?
> >   * What is the pourpose of having so many makefiles?
> 
> http://www.cmake.org/Wiki/CMake_FAQ#Why_does_CMake_generate_recursive_Makefiles.3F
> 
> >   * What is the reason why it is not trivial to implement batch mode?
> 
> The main reasons are:
> 
> * CMake generates a separate compilation line for each source in each
>   target and the flags may differ across sources and targets.  CMake
>   does not use pattern rules for any make backend.
> 
> * CMake needs to control the location of the object files and "cl" does
>   not allow this when more than one source is given.
> 
> Implementing batch mode rules would require teaching the generator to
> somehow recognize when a group of sources are all built with the same
> flags and generate a batch mode rule that matches exactly those sources
> and no others (likely the hardest part).  This grouping would also have
> to account for the names that "cl" generates for the object files to avoid
> conflicts within a group.  I do not know if generating a proper batch mode
> rule is possible in general.
> 
> >   * Where should I start browsing the source files?
> 
> One place is Source/cmLocalUnixMakefileGenerator3.cxx but there is no
> specific spot to make a change like this.  It would likely involve
> major refactoring.
> 
> One can achieve multi-core builds without third-party tools by using
> the VS IDE generators.  It is still possible to build from the command
> line with these.
> 
> -Brad
> 
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20150601/3cb0407e/attachment-0001.html>