[CMake] Memory usage and configuration time for large project

[Isaiah Norton]

As one ballpark datapoint: a "superbuild" of 3D Slicer (slicer.org) has a
similar object and library count:

macbook-pro:s5nj inorton$ find ./ -name *.o | wc -l
>    14127

macbook-pro:s5nj inorton$ find ./ -name *.dylib -or -name *.so | wc -l
>     2158


Based on a few quick tests, the aggregate cmake time is probably about 6-8
minutes for the ninja generator, over the multi-hour build (each dependency
is configured and built as a separate sub-project via CMake's
ExternalProject mechanism) -- with the caveat that each of those cmake runs
is doing lengthy checks that should only be done once if your project
strictly uses `add_subdirectory`.

As a more concrete point of comparison, building VTK generates 5747 object
files, and a clean configure on my 2-core macbook takes about 90s.

Isaiah



On Fri, May 4, 2018 at 2:16 PM, Patrick E Gartung <gartung at fnal.gov> wrote:

> Just to be clear, the memory and time used are just to configure and
> generate the makefiles or Ninja file. The build itself can take several
> hours.
>
> On 4/30/18, 4:55 PM, "R0b0t1" <r030t1 at gmail.com> wrote:
>
>     On Mon, Apr 30, 2018 at 4:44 PM, Patrick E Gartung <gartung at fnal.gov>
> wrote:
>     >  Hi,
>     >
>     > We have a large c/c++/fortran project, CMSSW, that is built with a
> custom tool, scram.
>     >
>     > https://github.com/cms-sw/cmssw
>     >
>     > The project might move to a cmake based build in the future. The
> scripts to convert to CmakeLists.tx has been written
>     >
>     > https://github.com/cms-sw/cmssw2cmake
>     >
>     > Tests show that with the cmake files generated with this script,
> configuring the project uses up to 1.5GB of ram and takes 11 minutes when
> using -GMakefiles. Using -GNinja and using the latest cmake, this time can
> be reduced to 8 minutes.
>     >
>     > The project builds 14k object files, 2.2k libraries, ~600 binaries,
> 500 generated source files with links to ~100 external libraries.
>     >
>     > Is this amount of memory usage and time typical for a project of
> this size?
>     >
>
>     I'm inclined to say "yes" as many builds such as Firefox, its
>     supporting libraries, and Chrome all take lots of time and memory.
>     Chrome uses Ninja, I might add. But the issue is not CMake itself.
>     CMake tends to produce better builds.
>
>     As I am not intimately familiar with your project, I can't make good
>     concrete suggestions. You may enjoy reading
>     https://news.ycombinator.com/item?id=14733829 and searching for build
>     optimization strategies.
>
>     Keep in mind a lot of the blame falls on C++ and Windows, should you
>     be using Windows. If you aren't using Windows, then the advice in the
>     comments above should still be relevant, and give you something to go
>     on.
>
>     Cheers,
>          R0b0t1
>
>     > Patrick Gartung
>     > Fermilab
>     >
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180504/cb1f820a/attachment.html>