[CMake] Need ideas/opinions on third party library management

[Roger Leigh]

On 12/08/2016 19:59, Robert Dailey wrote:
> Hello,

> I've been thinking of a different approach for a while. I've done some
> toying around with the "Super Build" concept, where I have a separate
> CMake project that does nothing but use the ExternalProject module to
> build libraries in real time along with our project. So the order of
> operations would be as follows (for our automated build server):
>
> 1. Clone our "Third Party" repository
> 2. Use CMake to generate & build the "Super Build" project (this
> builds boost, openssl, freetype, etc for the current platform).
> 3. Clone the main code base's repository
> 4. Use CMake to generate & build, using find_package() to refer to
> interface targets exported by those third party libraries built in
> step 2

You can simplify this to two steps:

1. Clone the superbuild repository
2. Build the third-party and first-party packages

This is the approach we take with the OME super-build
https://github.com/ome/ome-cmake-superbuild/

See the packages in
https://github.com/ome/ome-cmake-superbuild/tree/develop/packages
all the ones with an "ome-" prefix are the first-party libraries;
everything else is third-party.  They are all external projects.

We have cmake options to select all packages, first-party packages only 
or select individual packages by hand; the recursive dependency 
resolution will add/ignore dependencies depending upon the chosen policy 
(defaulting to include third-party dependencies).  So when you're 
building on BSD or Linux, you can say "no third-party dependencies 
except for gtest", or choose everything, and on Windows where there is 
no package manager, you always build everything.  And on Windows you can 
build just the dependencies and reuse them later; it has support for 
using cached builds which we use on the CI side to cut down build times; 
we only rebuild the cache when the superbuild git tree hash changes; see 
the scripts/jenkins-build script

In addition to that, we have the means to build the first-party 
libraries in different ways

1. From source release tarball (last stable release)
2. From git (specify remote+branch as cmake options)
3. From a local git checkout (on whatever branch is currently checked out)

You can see the selection logic here: 
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L4

In practice (1) is for end-users to build the whole thing, and (3) is 
what I use in practice to develop locally where I have a full set of git 
repos checked out locally, and I tell the superbuild to build them, 
along with all the third-party deps; (3) is also what we use on the CI 
system, with jenkins cloning all the individual repos into the 
workspace, then doing pretty much the same thing.  (2) is OK for one-off 
builds, but the need for a full git clone for every package is too much 
for routine use

And you can see how first- and third-party dependencies are handled 
here: 
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/ome-files/superbuild.cmake#L48

I might rearrange that so that the classification is in the dependency 
rather than the dependeee later on though.  It kind of evolved this way 
as we added additional features and needs a little cleanup to make it 
perfect.

Everything in the above repository is BSD licensed, so feel free to use 
it for inspiration or take what you like.  The main point I wanted to 
make was that with the superbuild *everything* can be an external 
project, including your own code.  And from that point you can add in 
features like being able to selectively build just the third-party or 
just your own code, rather than forcing them to be separate manual steps.


Regards,
Roger