[CMake] ExternalProject_Add, dependencies and Make

[Alan W. Irwin]

On 2013-07-03 11:59+0800 Ben Pope wrote:

> Hi,
>
> I've got a number of libraries I depend on, I use ExternalProject_Add to 
> ./configure and build them, I set the DEPENDS field to another 
> ExternalProject_Add target.
>
> When I generate makefiles and then use -j to make parallel builds, multiple 
> instances try to build the targets in parallel, making a complete mess and 
> usually more often than not, breaking the configure or build steps.
>
> Any idea what I'm doing wrong?

Here are some possibilities.

1. A parallel build race condition in the external project.

For example, I am currently using ExternalProject_Add to build
wxwidgets (and several other projects I have configured with the
"build_projects" project), and parallel builds with that specific
project randomly error out depending on timing which I am virtually
positive is due to a parallel build race condition since that build is
completely reliable if I do not use a parallel build.  Another
example, is the jhbuild project (a very comprehensive build system for
the GTK stack of libraries) which has a special flag in its
configuration files for a few of the GTK stack of libraries that
indicates they just cannot be reliably built in parallel.

To check for this possibility, avoid using a parallel build for your
"parent" project, but use parallel builds for all the external
projects you configure with ExternalProject_Add.  That means each of
those projects will be built one at a time, but each of those builds
will be done in parallel so you get most/all of the efficiency you
would get if your own parent project was built in parallel.

2. A parallel build race condition in your own "parent" project that uses
ExternalProject_Add to build external projects.

I looked up the ExternalProject.cmake file, and apparently DEPENDS
simply establishes a target dependency (as opposed to a file
dependency) between the target for the project in question and the
targets associated with other projects being built with
ExternalProject_Add (and I guess arbitrary numbers of any other targets
you specify as dependencies as well).

I actually use that capability in build_projects where I build the
dependencies of PLplot first, then PLplot. But I have never tested
your scenario because each external project is built in parallel
already (with -j4), and building say the dependencies of plplot
simultaneously using -j4 for the parent project (build_projects) would
result in 16 jobs being done simultaneously which is probably quite
inefficient for my two-cpu computer.  So I have never tried the
scenario where the parent project was built in parallel.

Big Caveat.  I tend to use rote rules rather than any basic
fundamental understanding to attempt to avoid parallel build race
conditions in CMake-based build systems so I am not sure what could be
going wrong for parallel builds of the parent project.  However, just
to speculate, could two external projects that depend on a library
generated by a third external project generate such a race condition?

And I hope Brad King (the real expert on these matters) is willing to
chime in here with the answer to that question.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________