[cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS

[Craig Scott]

On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth <john at johnwordsworth.com>
wrote:

> I have recently been reviewing ways to improve build times for our
> project, which is comprised of a number of static libraries. I stumbled
> across this post on the CMake tracker from 2012/13 (
> https://cmake.org/Bug/view.php?id=13799). It suggests adding a
> COMPILE_DEPENDS target property to to explicitly set dependencies for
> STATIC libraries instead of always using all linked libraries as
> build-order dependencies.
>
> Having done a draft implementation in a local CMake repository it has
> shaved off  20% of our 120s build time. I expect the savings to be much
> more dramatic when I test with Incredibuild (approximately 50% based on
> tests done previously from just deleting dependencies manually in Visual
> Studio).
>
> I don’t really want to refactor our code to use “OBJECT” libraries as the
> inability to link with other targets means that propagating compile options
> / include directories etc down the chain of linked libs becomes painful.
> This method allows me to switch between static and shared libs using a
> config option and none of my CMake scripts need to change.
>

There's a couple more choices here. If your project consists of lots of
small (static) libraries, consider whether you can combine some of them to
result in a smaller number of larger libraries. This isn't always a gain,
but in terms of ability to compile sources in parallel, it will often lead
to more efficient builds. You just need to be careful you don't end up with
so many objects being combined into one library that you start to hit max
open file limits during linking/archiving (something I've hit on multiple
platforms lately, so it's not just a hypothetical example). Use of
target_sources() can be quite helpful if you want to try out this path (you
may find this article
<https://crascit.com/2016/01/31/enhanced-source-file-handling-with-target_sources/>
helpful).

Another choice is to go in the opposite direction and take advantage of the
optimisation made for the Ninja generator (if that's a choice open to you)
that was introduced in CMake 3.9.0
<https://gitlab.kitware.com/cmake/cmake/merge_requests/430> where if no
custom commands exist on a target A, then compilation steps of another
target B linking to A are allowed to proceed without waiting for A's link
step to complete. Only B's link step will depend on A's link step. In your
project, if you have custom commands, see if you can split up that target
into just those sources that need the results of the custom command and
another target that doesn't. The latter will then be able to compile
earlier, so fewer sources have to wait for earlier linking steps. This
might be hard to do, it really depends on how your project is structured.

Both of the above choices allow you to retain the automatic propagation of
compile options, include directories, etc. and to switch between
shared/static easily, but the latter is specific to the Ninja generator and
may not be an acceptable change for you.



>
> Anyway, I was wondering whether there was any interest in me pushing my
> solution back to Git / submitting a Pull request so that it might be merged
> in at some point. If there is - any advice on any gotchas I might watch for
> instead of just adding some fairly simple code to
> cmComputeTargetDepends.cxx would be gratefully received - especially as
> this is my first time poking around in CMake code.
>

The existing behaviour is conservative and any change would have to also be
conservative, meaning that it must not introduce any possibility of
breaking existing projects. If I'm understanding your proposed feature
correctly, it sounds like you want to relax the build-order dependencies by
default when a COMPILE_DEPENDS target property is defined. Basically, if
COMPILE_DEPENDS is defined, you are taking over responsibility for the
build-order dependencies. This would be something I'd usually discourage
projects from doing because such manual dependencies would be a prime
candidate for not being kept up to date as a project evolves, leading to
subtle, hard-to-trace build errors. Some judicious project restructuring
can normally give a pretty efficient parallel build without having to
resort to such measures, so I'm wary of adding a feature like this (though
I can understand the desire for it).

In my experience, you can get some considerable speedups using tools like
ccache (and its equivalents for other platforms/compilers). These obviously
only help for subsequent builds of things that have been built previously,
but for everyday development where you switch between branches or for CI
servers doing lots of similar builds, the savings can be impressively big.

-- 
Craig Scott
Melbourne, Australia
https://crascit.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake-developers/attachments/20180104/bf0690f5/attachment.html>