[CMake] Building all tests in parallel (whole subtrees)

[Alan W. Irwin]

On 2013-01-24 23:07+0100 Wojciech Knapik wrote:

> Anyway - yeah, the dependencies are there now and I don't think I'm
> missing any. If I create a target foo, that depends on bar and qux and
> both bar and qux do target_link_libraries(target fred), then when I try
> to build foo in parallel, I get a race when building fred as a
> dependency for bar and qux.
>
> Like I said above though, this is probably about the location of the
> targets foo, bar, baz, qux and fred in the source tree relative to each
> other.

If you are just using standard targets (libraries and executables)
with _no_ custom commands or targets, and you are using
target_link_libraries appropriately, then parallel builds should just
work.  Also, the subdirectly location of where targets are configured
should not matter.  Really!

So I am virtually positive the race conditions you are seeing are due
to some dependency screwup in your build system for add_custom_command
and/or add_custom_target.  Typically, the issue is that two different
custom targets file-depend on the same OUTPUT file from an
add_custom_command.  That's OK, but if those two targets have no
add_dependencies command relating them, then they can both be built at
the same time with a parallel-build ==> two instances of the custom
command (or one of its dependencies such as a build of a library) run
simultaneously ==> race.  So very often the solution is to add a
add_dependencies command as appropriate to insure the two targets with
common custom_command OUTPUT file dependencies aren't built at the
same time.

>
> I don't feel like researching this further, I've spent enough time on
> it, so I've just moved on to other options.
>
>> Note, CMake has both file dependencies and target dependencies, and
>> both have to be set up correctly in order for parallel builds to work
>> properly.  CMake often does the right thing here, but there are also
>> cases where you have to explicitly tell it about dependencies. For
>> more detail on the two kinds of dependencies look extremely carefully
>> at the documentation of add_custom_command, add_custom_target, and
>> add_dependencies.  I tend to carefully re-read that documentation
>> every time I implement a new test target.
>
> Ok, so let's say I have a custom command that generates code from
> .idl file for instance. If I give all the targets that require that
> generated code a dependency on a target that depends on the generated
> files - is that enough ?

Yes.  add_custom_command generates OUTPUT files.  What I tend to do in
my own build systems is immediately after any add_custom command I use
add_custom_target to create a target that _file_ depends on those
OUTPUT files.  Then I use add_dependencies to establish what other
targets depend on that custom_target.  You will see that pattern often
for ephcom-3.0.0 and te_gen-2.0.0, and as a result parallel builds
just work for those projects without issues.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________