[cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS

[John Wordsworth]

I realise I wasn't very clear in my previous explanation. Hopefully this
paints a better picture;

- We have 40 or so developers spread across Visual Studio 2015 (MSBuild -
Windows), Xcode (macOS) and Make/Clang (Linux). 80% use Windows.
- We have a core set of libs in use by a number of projects (~30 static
libs and ~10 different projects), but the projects mix and match which libs
they use - merging the static libs (beyond a few of them) isn't something
we want to do.
- I have recently been revising our CMake structure and have boiled it down
to a fairly simple "add_library(x), target_link_libraries(x,
default_compile_settings, dependencies)" which means compile settings,
include directories and macros propagate through the libs nicely.
- We sometimes want to build and use certain libs as shared libs instead of
static libs.

With regards to the COMPILE_DEPENDS feature that I have mocked up and
started testing with our team;

- If and only if COMPILE_DEPENDS is set for a STATIC library, then when
building the target dependencies for that library, it uses the list of
provided targets instead of those that would have been inferred from
previous calls to "target_link_libraries" or "add_dependencies".

The reality is, the libs which are being built statically can nearly all
build in parallel with only one or two "real" dependencies. I understand
that our case is rather specific, but having implemented "COMPILE_DEPENDS"
and written ~10 lines of CMake in our project (hard coding the one or two
actual build order dependencies and disabling the rest), our compile graph
looks much nicer and saves a significant amount of time building on a
single machine (see https://www.johnwordsworth.com/temp/cmake_compile_
depends.jpg). We see similar savings on macOS using Xcode but have not
tested on Linux yet. When we allow Incredibuild to distribute building of
compilation units across 15-20 agents, the proportional build time drops
more dramatically between the two (the same project goes from ~2.6mins ->
~1.6mins).

I understand that just "hard overriding" the target dependencies is a bit
messy, but I'd also be happy to explore other ways I could add a similar
feature to CMake if there is a potentially better way to do this. What I am
ideally looking for in a solution is that I can continue to propagate
include directories / compile settings / preprocessor macros through the
chain of libs and that it improves building static libs in parallel across
all our build systems (MSVC / XCode / make). Ideally, I wouldn't have to
restructure my CMake project too much either - as the draft I have for our
new structure feels super clean now and refactoring it just to remove build
order dependencies would be shame. If there are any alternative ideas for
how to implement this, I'd love to discuss. I'm no CMake expert, but maybe
if we could use "target_link_libraries" with OBJECT libraries to grab
compile settings, or perhaps I could look into a mechanism for not adding
build order dependencies on VS2015/Xcode if there is no need too?

Thanks again for the feedback and discussion so far.

On Thu, Jan 4, 2018 at 12:16 PM, Eric Noulard <eric.noulard at gmail.com>
wrote:

>
> 2018-01-04 10:48 GMT+01:00 Craig Scott <craig.scott at crascit.com>:
>
>>
>>
>> On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth <john at johnwordsworth.com>
>> wrote:
>>
>>> I have recently been reviewing ways to improve build times for our
>>> project, which is comprised of a number of static libraries. I stumbled
>>> across this post on the CMake tracker from 2012/13 (
>>> https://cmake.org/Bug/view.php?id=13799). It suggests adding a
>>> COMPILE_DEPENDS target property to to explicitly set dependencies for
>>> STATIC libraries instead of always using all linked libraries as
>>> build-order dependencies.
>>>
>>> Having done a draft implementation in a local CMake repository it has
>>> shaved off  20% of our 120s build time. I expect the savings to be much
>>> more dramatic when I test with Incredibuild (approximately 50% based on
>>> tests done previously from just deleting dependencies manually in Visual
>>> Studio).
>>>
>>
> You said you tested with Incredibuild but with what kind of configuration?
> AFAIK Incredibuild takes its power from distributing the build? So does
> your CMake test uses something that distribute the build too ?
>
>
>>
>>> I don’t really want to refactor our code to use “OBJECT” libraries as
>>> the inability to link with other targets means that propagating compile
>>> options / include directories etc down the chain of linked libs becomes
>>> painful. This method allows me to switch between static and shared libs
>>> using a config option and none of my CMake scripts need to change.
>>>
>>
>> There's a couple more choices here. If your project consists of lots of
>> small (static) libraries, consider whether you can combine some of them to
>> result in a smaller number of larger libraries. This isn't always a gain,
>> but in terms of ability to compile sources in parallel, it will often lead
>> to more efficient builds. You just need to be careful you don't end up with
>> so many objects being combined into one library that you start to hit max
>> open file limits during linking/archiving (something I've hit on multiple
>> platforms lately, so it's not just a hypothetical example). Use of
>> target_sources() can be quite helpful if you want to try out this path (you
>> may find this article
>> <https://crascit.com/2016/01/31/enhanced-source-file-handling-with-target_sources/>
>> helpful).
>>
>> Another choice is to go in the opposite direction and take advantage of
>> the optimisation made for the Ninja generator (if that's a choice open to
>> you) that was introduced in CMake 3.9.0
>> <https://gitlab.kitware.com/cmake/cmake/merge_requests/430> where if no
>> custom commands exist on a target A, then compilation steps of another
>> target B linking to A are allowed to proceed without waiting for A's link
>> step to complete. Only B's link step will depend on A's link step. In your
>> project, if you have custom commands, see if you can split up that target
>> into just those sources that need the results of the custom command and
>> another target that doesn't. The latter will then be able to compile
>> earlier, so fewer sources have to wait for earlier linking steps. This
>> might be hard to do, it really depends on how your project is structured.
>>
>
> I fully agree with Craig here. We use CMake + Ninja and we get really
> efficient parallel build (on 10+ cores machines).
> This is on Linux though but it seems MS is putting effort to get CMake +
> ninja build work with Visual Studio:
> https://blogs.msdn.microsoft.com/vcblog/2017/05/10/cmake-
> support-in-visual-studio-whats-new-in-2017-15-3-update/
>
> Another way to improve your build may be to review your dependency graph
> carefully. You can use cmake --grahviz options to dump the graph and see
> whether you can cut
> some dependencies due to transitivity of dependency. I have seen many
> projects with "overspecified deps". I think this impair build time when
> using a lot of static libs because in that case you may get
> poor performance link time. Namelly if you have many unit tests linking to
> too many static libs because of overspecified deps.
>
> That said you did not specify whether if the 120s build time is only for
> the library or if this is a "global" build time which includes linking
> somes executables and possible unit tests programs.
>
> Note that even if this is not the case having a "big-picture" look at the
> dependency build graph generated by CMake may be insightful.
> Note that with CMake 3.10 you can now see difference between PUBLIC,
> PRIVATE and INTERFACE deps:
> https://cmake.org/cmake/help/v3.10/release/3.10.html
>
>
>
>>
>> Both of the above choices allow you to retain the automatic propagation
>> of compile options, include directories, etc. and to switch between
>> shared/static easily, but the latter is specific to the Ninja generator and
>> may not be an acceptable change for you.
>>
>>
>>>
>>> Anyway, I was wondering whether there was any interest in me pushing my
>>> solution back to Git / submitting a Pull request so that it might be merged
>>> in at some point. If there is - any advice on any gotchas I might watch for
>>> instead of just adding some fairly simple code to
>>> cmComputeTargetDepends.cxx would be gratefully received - especially as
>>> this is my first time poking around in CMake code.
>>>
>>
>> The existing behaviour is conservative and any change would have to also
>> be conservative, meaning that it must not introduce any possibility of
>> breaking existing projects. If I'm understanding your proposed feature
>> correctly, it sounds like you want to relax the build-order dependencies by
>> default when a COMPILE_DEPENDS target property is defined. Basically, if
>> COMPILE_DEPENDS is defined, you are taking over responsibility for the
>> build-order dependencies. This would be something I'd usually discourage
>> projects from doing because such manual dependencies would be a prime
>> candidate for not being kept up to date as a project evolves, leading to
>> subtle, hard-to-trace build errors. Some judicious project restructuring
>> can normally give a pretty efficient parallel build without having to
>> resort to such measures, so I'm wary of adding a feature like this (though
>> I can understand the desire for it).
>>
>> In my experience, you can get some considerable speedups using tools like
>> ccache (and its equivalents for other platforms/compilers). These obviously
>> only help for subsequent builds of things that have been built previously,
>> but for everyday development where you switch between branches or for CI
>> servers doing lots of similar builds, the savings can be impressively big.
>>
>
> +1 again.
>
> ccache boost our build. It can be more than 10x faster (depends on cache
> hit).
> You can even share (on a local network) the ccache directory between
> developers and the CI and get very high hit rate (> 80%).
>
> Now I'm not working on windows nor with Visual Studio and AFAIK ccache
> does not work with MSVC.
> There seem to be alternative, https://github.com/frerich/clcache, but as
> you guessed I did never used that.
>
> --
> Eric
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake-developers/attachments/20180105/05bfd9b1/attachment.html>