View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0010082CMakeCMakepublic2009-12-26 04:462016-06-10 14:31
ReporterJoerg Faschingbauer 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionmoved 
PlatformOSOS Version
Product VersionCMake-2-8 
Target VersionFixed in Version 
Summary0010082: ADD_CUSTOM_COMMAND and ADD_CUSTOM_TARGET: parallel make executes command multiple times
DescriptionCustom targets are obviously processed using sub-make invocations. This leads to multiple builds of their dependencies when doing a parallel build.

The following example is a bit contrived - it's the minimum reproducer I can come up with. In real life I use multiple generator steps which are cascaded on top of each other, with the output of each step hooked into the 'all' target using ADD_CUSTOM_TARGET(name ALL ...).

Using the 'Unix Makefiles' generator and make -j2, you'll see that the file 'generated' is generated twice ('custom command running' is printed twice).

PROJECT(test)
CMAKE_MINIMUM_REQUIRED(VERSION 2.6)

ADD_CUSTOM_COMMAND(
    OUTPUT generated
    COMMAND sleep 1
    COMMAND echo custom command running
    COMMAND touch generated
    )

ADD_CUSTOM_TARGET(
    my-all-1
    ALL
    DEPENDS generated
    )

ADD_CUSTOM_TARGET(
    my-all-2
    ALL
    DEPENDS generated
    )
TagsNo tags attached.
Attached Files

 Relationships

  Notes
(0018995)
Bill Hoffman (manager)
2009-12-26 11:39

 Yup, sometimes you need to tell CMake if two targets depend on each other. So, in your example, if you add add_dependencies(my-all-1 my-all-2) it should work.
(0019006)
Joerg Faschingbauer (reporter)
2009-12-29 05:18

 Ok, this works, thanks.
Sorry, I cannot refrain from asking if this is a workaround or the way I'm expected to write the rules.
(0019306)
Joerg Faschingbauer (reporter)
2010-01-26 06:37

 I see that a sub-make is started for *every* top-level target, so this one will fail too with a parallel build, in the sense that the generator is called in parallel.

PROJECT(test)
CMAKE_MINIMUM_REQUIRED(VERSION 2.6)

ADD_CUSTOM_COMMAND(
    OUTPUT generated-main.c
    COMMAND echo custom command enter
    COMMAND sleep 1
    COMMAND echo 'int main() {}' > generated-main.c
    COMMAND echo custom command leave
    )

ADD_EXECUTABLE(executable1 generated-main.c)
ADD_EXECUTABLE(executable2 generated-main.c)

Here the solution is, in analogy to your suggestion upon the initial report, to add an explicit dependency executable2 -> executable1. I do this because I *know* that otherwise there's going to be a conflict on the generator.

The effect of this is that, in 'make' terms, the sub-make that is called for executable2 has nothing to do because generated-main.c is already there.

Is this a general advice for projects that have code generators? In fact, the real situation is more complex as the project is large and there are quite a few generators involved. Generally speaking, I'd have to artificially chain together all top-level targets that nobody else depends on, in order to prevent such clashes.

Without sub-make invocations, make would handle the situation more gently by its nature, and **not even think of generating twice**.

I believe that the way how CMake uses recursive makes is the root cause of many problems. See for example the thread that was started with http://www.cmake.org/pipermail/cmake/2007-December/018392.html. [^] I read the discussion thoroughly and I am quite sure that the original poster is bitten by the same issue as I am.

I wonder if somebody followed Brad King's advice in http://www.cmake.org/pipermail/cmake/2007-December/018516.html [^] and filed a dedicated bug report. If not, I strongly advocate to take the current issue for this, or (better yet) to file another one with a title that is more to the point of the problem's root - recursive make.
(0019311)
Brad King (manager)
2010-01-26 09:17

 General advice when a generated source is used in multiple targets: generate it with another custom target, and make all the targets that use the source depend on the custom target.

As for recursive make, I challenge anyone to come up with non-recursive Makefiles that can handle build-time dependency scanning. Using GNU-make extensions is not allowed because we have to work with ancient UNIX make.
(0026844)
Andreas Mohr (reporter)
2011-06-12 13:52

 Brad, about recursive make: perhaps the discussion at http://stackoverflow.com/questions/559216/what-is-your-experience-with-non-recursive-make [^] contains a few insights? (albeit at least some of these postings talk about requiring GNU make features, which one - and you ;) - would want to avoid as a restriction, for obvious reasons).

Benchmark item: on my build tree (>> 50 projects, tons of additional non-executable custom targets, perhaps 500 targets all in all) an empty build at -j3 takes around 15 seconds on dual core (and that's with many things streamlined as much as possible).

Hmm, just realized that I'm perhaps a bit off-topic...

Thanks!
(0030552)
Brad King (manager)
2012-08-13 10:44

 Sending issues I'm not actively working on to the backlog to await someone with time for them.

If an issue you care about is sent to the backlog when you feel it should have been addressed in a different manner, please bring it up on the CMake mailing list for discussion. Sign up for the mailing list here, if you're not already on it:

 http://www.cmake.org/mailman/listinfo/cmake [^]

It's easy to re-activate a bug here if you can find a CMake developer or contributor who has the bandwidth to take it on.
(0041636)
Kitware Robot (administrator)
2016-06-10 14:27

 Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2009-12-26 04:46 Joerg Faschingbauer New Issue
2009-12-26 11:38 Bill Hoffman Status new => assigned
2009-12-26 11:38 Bill Hoffman Assigned To => Bill Hoffman
2009-12-26 11:39 Bill Hoffman Note Added: 0018995
2009-12-29 05:18 Joerg Faschingbauer Note Added: 0019006
2010-01-26 06:37 Joerg Faschingbauer Note Added: 0019306
2010-01-26 09:03 Bill Hoffman Assigned To Bill Hoffman => Brad King
2010-01-26 09:17 Brad King Note Added: 0019311
2011-06-12 13:52 Andreas Mohr Note Added: 0026844
2012-08-13 10:44 Brad King Status assigned => backlog
2012-08-13 10:44 Brad King Note Added: 0030552
2016-06-10 14:27 Kitware Robot Note Added: 0041636
2016-06-10 14:27 Kitware Robot Status backlog => resolved
2016-06-10 14:27 Kitware Robot Resolution open => moved
2016-06-10 14:31 Kitware Robot Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team