[CMake] Parallel make & custom command

[Bill Hoffman]

On 6/30/2011 5:23 PM, Alan W. Irwin wrote:
> On 2011-06-30 22:15+0200 Alexander Neundorf wrote:
>
>> On Wednesday 29 June 2011, Marcel Loose wrote:
>> ...
>>> After I had posted my question I realized that this issue has come up
>>> quite recently on the mailing list in a thread that I started -- see
>>> http://www.mail-archive.com/cmake@cmake.org/msg36362.html. Although the
>>> original question was related to building mulitple targets in parallel,
>>> Michael Hertling showed that CMake inherently has problems with parallel
>>> builds when custom targets/commands come into play.
>>
>> Really ?
>> We use a lot of custom commands in KDE, all the time, everywhere, and
>> people
>> build KDE on big build clusters, i.e. very parallel, without problems.
>>
>> Maybe there are problems when multiple targets depend on the same
>> generated
>> file, but I think the "cmake inherently has problems with parallel
>> builds when
>> custom targets/commands come into play." is wrong in this very generic
>> way.
>
> I agree with Alex here. It is always possible, of course, to set up
> custom commands and custom targets properly so that parallel builds
> work correctly. The trick is to avoid two or more targets depending
> on the same custom command. So bug 0012311 should just be closed
> since that is based on a misunderstanding of how to properly set up
> custom commands and targets.
>
> The above rule of thumb is easy to implement for simple build systems,
> but for complex build systems it is easy to make file or target
> dependency mistakes so that parallel builds do not work correctly.
> This is the number-one issue I have had difficulty with for our PLplot
> CMake-based build system because that system has lots of custom
> commands and a complex web of file and target dependencies that
> changes each time we add new build or test features.
>
> To help with the general difficulty of getting dependencies right,
> would it be possible to implement a CMake change so that for a given
> flag (say --check-dependencies) the cmake application issued warnings
> for file and target dependency issues that would cause problems for
> parallel builds for a given CMake-based build system? That would help
> a lot to solve this topic that keeps coming up again and again on this
> list and also make existing CMake-based build systems much easier to
> maintain.
>
> Assuming such a change was implemented, then the standard FAQ response
> to this topic would then boil down to "run cmake with the
> --check-dependencies flag to find out the dependency issues you have
> to solve before you can reliably run a parallel build."
>

If we had a magic flag that did --check-dependencies we would just auto 
add the depends.  The problem is these depends can be hard to find. 
Some of them can not even be found until everything has been built or at 
least all files have been generated, and all source files have been 
scanned for depend information.  CMake saves the depend generation of 
source files to build time.  So, at CMake time we don't even know which 
source files include which other source files.   This stuff can be 
nested as well.  You could generated foo.c which includes bar.h which is 
also generated right after car.h which is included by car.c all of which 
might be in different targets and directories.  So, although all things 
are possible, this one is very hard to do....

-Bill