[cmake-developers] Fwd: NMake Batch-Mode Rules (was: Inquiry on contribution to NMake generator)

Paul Anton Letnes pa at letnes.com
Mon Jun 1 11:55:58 EDT 2015


Sorry David, my bad.

CMake combined with GNU Make does indeed handle dependencies for Fortran, so it can’t be impossible - but maybe it is hard enough that noone bothered implementing it for ninja.

Paul


> I have no experience with Fortran, and from what I understand,
> parallel builds of Fortran code are very difficult to get right
> because of the way Fortran module dependencies work.
> 
> You should re-send to the whole list instead of just me if you want
> somebody else to chime in about Fortran and parallel builds.
> 
> 
> D
> 
> 
> On Mon, Jun 1, 2015 at 10:35 AM, Paul Anton Letnes <pa at letnes.com> wrote:
>> To me, ninja sounds great. But, does ninja build Fortran sources yet? I’ve never managed to build my Fortran/C/C++ mixed-language projects correctly with jom/nmake (performance issues aside), and the Visual Studio builds are way, way slower than make builds on Linux (in my experience).
>> 
>> Cheers
>> Paul
>> 
>>> On 1. jun. 2015, at 13.28, David Cole via cmake-developers <cmake-developers at cmake.org> wrote:
>>> 
>>> CMake + Ninja is definitely the best move you can make for giving your
>>> developers the minimum possible build time on Windows.
>>> 
>>> NMake is basically useless, its underlying code hasn't been updated in
>>> years, and it does absolutely nothing to try to take advantage of
>>> multiple cores.
>>> 
>>> Fastest Windows build time in recent years has always been ninja
>>> driving the build in my experience. And I've tried the comparisons
>>> several times on projects small and large. It always gives at least
>>> some improvement over other techniques on real world projects.
>>> 
>>> 
>>> HTH,
>>> David C.
>>> 
>>> 
>>> 
>>> On Mon, Jun 1, 2015 at 6:55 AM, Florent Castelli
>>> <florent.castelli at gmail.com> wrote:
>>>> At my company, we didn’t bother trying to use NMake.
>>>> We used the Visual Studio solution generator and the proper compilation
>>>> flags and msbuild flags to build in parallel. This proved to speedup our
>>>> builds quite a lot and was very easy to integrate.
>>>> The other solution you mentioned is “ninja” and sped up our builds even
>>>> more. The msbuild tool is not smart enough and with a number of targets high
>>>> enough and enough files per target, you can just generate n^2 parallel
>>>> compilation processes, which can be counter productive. Though, you can
>>>> manually specify the parallelism factor you want for both msbuild and cl.exe
>>>> and you get decent results.
>>>> 
>>>> Give those a shot, NMake is deprecated in Visual Studio land and has been
>>>> superseded by msbuild.
>>>> 
>>>> And if your devs complain about having to install some tools before
>>>> compiling, just give them a script that downloads and setup CMake and Ninja
>>>> on their system. It’s easy to do and should reduce the number of complaints!
>>>> 
>>>> Regards,
>>>> Florent
>>>> 
>>>> On 01 Jun 2015, at 11:10, Nagy-Egri MĂĄtĂŠ Ferenc via cmake-developers
>>>> <cmake-developers at cmake.org> wrote:
>>>> 
>>>> Thanks Brad for the detailed description. From what I’ve read about the 3
>>>> level makefile system, it should be pretty straightforward to implement
>>>> batch-mode support.
>>>> 
>>>> Actual compilation takes place only in the last level, and throughout
>>>> generation, this is the point where all the compilation options are present.
>>>> Obtaining a list of files with identical compiler switches might be tricky.
>>>> Depends on how CMake handles this in code internally. I’ll see once I dive
>>>> into code. Deferred generation of the makefile is not nice, but might be the
>>>> only way. Handling the exact location of the object files might be another
>>>> property source files must match.
>>>> 
>>>> As far as I’ve seen NMake is not able to call multiple makefiles in
>>>> parallel, so source-level parallelism might be the most NMake could do, but
>>>> for large projects that could be sufficient.
>>>> 
>>>> Feladó: Brad King
>>>> Elküldve: ‎szerda‎, ‎2015‎. ‎május‎ ‎27‎. ‎20‎:‎13
>>>> Címzett: Nagy-Egri MC!tC) Ferenc
>>>> Másolat: cmake-developers at cmake.org
>>>> 
>>>> On 05/27/2015 01:33 PM, Nagy-Egri MC!tC) Ferenc via cmake-developers wrote:
>>>>> like to see the NMake generator finally support multicore builds by
>>>>> adding Batch Mode support to the generated makefiles.
>>>> 
>>>> For reference, Batch-Mode Rules are documented here:
>>>> 
>>>> https://msdn.microsoft.com/en-us/library/f2x0zs74.aspx
>>>> 
>>>> They support specifying multiple source files to a single "cl" invocation,
>>>> as in
>>>> 
>>>> cl -c src1.cpp src2.cpp
>>>> 
>>>> instead of
>>>> 
>>>> cl -c src1.cpp
>>>> cl -c src2.cpp
>>>> 
>>>> Any multi-core utilization occurs inside "cl" and NMake is not aware of it.
>>>> 
>>>>> I have read on the CMake user mailing list, that the problem with
>>>>> Batch Mode support, is that it is not trivial to implement.
>>>> 
>>>> It is very hard to implement and may not be possible at all.  See below.
>>>> 
>>>>> * What is the design of the NMake generator in CMake?
>>>>> * What is the pourpose of having so many makefiles?
>>>> 
>>>> http://www.cmake.org/Wiki/CMake_FAQ#Why_does_CMake_generate_recursive_Makefiles.3F
>>>> 
>>>>> * What is the reason why it is not trivial to implement batch mode?
>>>> 
>>>> The main reasons are:
>>>> 
>>>> * CMake generates a separate compilation line for each source in each
>>>> target and the flags may differ across sources and targets.  CMake
>>>> does not use pattern rules for any make backend.
>>>> 
>>>> * CMake needs to control the location of the object files and "cl" does
>>>> not allow this when more than one source is given.
>>>> 
>>>> Implementing batch mode rules would require teaching the generator to
>>>> somehow recognize when a group of sources are all built with the same
>>>> flags and generate a batch mode rule that matches exactly those sources
>>>> and no others (likely the hardest part).  This grouping would also have
>>>> to account for the names that "cl" generates for the object files to avoid
>>>> conflicts within a group.  I do not know if generating a proper batch mode
>>>> rule is possible in general.
>>>> 
>>>>> * Where should I start browsing the source files?
>>>> 
>>>> One place is Source/cmLocalUnixMakefileGenerator3.cxx but there is no
>>>> specific spot to make a change like this.  It would likely involve
>>>> major refactoring.
>>>> 
>>>> One can achieve multi-core builds without third-party tools by using
>>>> the VS IDE generators.  It is still possible to build from the command
>>>> line with these.
>>>> 
>>>> -Brad
>>>> 
>>>> --
>>>> 
>>>> Powered by www.kitware.com
>>>> 
>>>> Please keep messages on-topic and check the CMake FAQ at:
>>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>> 
>>>> Kitware offers various services to support the CMake community. For more
>>>> information on each offering, please visit:
>>>> 
>>>> CMake Support: http://cmake.org/cmake/help/support.html
>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>> 
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>> 
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/cmake-developers
>>>> 
>>>> 
>>>> 
>>>> --
>>>> 
>>>> Powered by www.kitware.com
>>>> 
>>>> Please keep messages on-topic and check the CMake FAQ at:
>>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>> 
>>>> Kitware offers various services to support the CMake community. For more
>>>> information on each offering, please visit:
>>>> 
>>>> CMake Support: http://cmake.org/cmake/help/support.html
>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>> 
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>> 
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://public.kitware.com/mailman/listinfo/cmake-developers
>>> --
>>> 
>>> Powered by www.kitware.com
>>> 
>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>> 
>>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>> 
>>> CMake Support: http://cmake.org/cmake/help/support.html
>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>> 
>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>> 
>>> Follow this link to subscribe/unsubscribe:
>>> http://public.kitware.com/mailman/listinfo/cmake-developers
>> 



More information about the cmake-developers mailing list