MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0010082CMakeCMakepublic2009-12-26 04:462016-06-10 14:31
ReporterJoerg Faschingbauer 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionmoved 
PlatformOSOS Version
Product VersionCMake-2-8 
Target VersionFixed in Version 
Summary0010082: ADD_CUSTOM_COMMAND and ADD_CUSTOM_TARGET: parallel make executes command multiple times
DescriptionCustom targets are obviously processed using sub-make invocations. This leads to multiple builds of their dependencies when doing a parallel build.

The following example is a bit contrived - it's the minimum reproducer I can come up with. In real life I use multiple generator steps which are cascaded on top of each other, with the output of each step hooked into the 'all' target using ADD_CUSTOM_TARGET(name ALL ...).

Using the 'Unix Makefiles' generator and make -j2, you'll see that the file 'generated' is generated twice ('custom command running' is printed twice).

PROJECT(test)
CMAKE_MINIMUM_REQUIRED(VERSION 2.6)

ADD_CUSTOM_COMMAND(
    OUTPUT generated
    COMMAND sleep 1
    COMMAND echo custom command running
    COMMAND touch generated
    )

ADD_CUSTOM_TARGET(
    my-all-1
    ALL
    DEPENDS generated
    )

ADD_CUSTOM_TARGET(
    my-all-2
    ALL
    DEPENDS generated
    )
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2009-12-26 04:46Joerg FaschingbauerNew Issue
2009-12-26 11:38Bill HoffmanStatusnew => assigned
2009-12-26 11:38Bill HoffmanAssigned To => Bill Hoffman
2009-12-26 11:39Bill HoffmanNote Added: 0018995
2009-12-29 05:18Joerg FaschingbauerNote Added: 0019006
2010-01-26 06:37Joerg FaschingbauerNote Added: 0019306
2010-01-26 09:03Bill HoffmanAssigned ToBill Hoffman => Brad King
2010-01-26 09:17Brad KingNote Added: 0019311
2011-06-12 13:52Andreas MohrNote Added: 0026844
2012-08-13 10:44Brad KingStatusassigned => backlog
2012-08-13 10:44Brad KingNote Added: 0030552
2016-06-10 14:27Kitware RobotNote Added: 0041636
2016-06-10 14:27Kitware RobotStatusbacklog => resolved
2016-06-10 14:27Kitware RobotResolutionopen => moved
2016-06-10 14:31Kitware RobotStatusresolved => closed

Notes
(0018995)
Bill Hoffman   
2009-12-26 11:39   
Yup, sometimes you need to tell CMake if two targets depend on each other. So, in your example, if you add add_dependencies(my-all-1 my-all-2) it should work.
(0019006)
Joerg Faschingbauer   
2009-12-29 05:18   
Ok, this works, thanks.
Sorry, I cannot refrain from asking if this is a workaround or the way I'm expected to write the rules.
(0019306)
Joerg Faschingbauer   
2010-01-26 06:37   
I see that a sub-make is started for *every* top-level target, so this one will fail too with a parallel build, in the sense that the generator is called in parallel.

PROJECT(test)
CMAKE_MINIMUM_REQUIRED(VERSION 2.6)

ADD_CUSTOM_COMMAND(
    OUTPUT generated-main.c
    COMMAND echo custom command enter
    COMMAND sleep 1
    COMMAND echo 'int main() {}' > generated-main.c
    COMMAND echo custom command leave
    )

ADD_EXECUTABLE(executable1 generated-main.c)
ADD_EXECUTABLE(executable2 generated-main.c)

Here the solution is, in analogy to your suggestion upon the initial report, to add an explicit dependency executable2 -> executable1. I do this because I *know* that otherwise there's going to be a conflict on the generator.

The effect of this is that, in 'make' terms, the sub-make that is called for executable2 has nothing to do because generated-main.c is already there.

Is this a general advice for projects that have code generators? In fact, the real situation is more complex as the project is large and there are quite a few generators involved. Generally speaking, I'd have to artificially chain together all top-level targets that nobody else depends on, in order to prevent such clashes.

Without sub-make invocations, make would handle the situation more gently by its nature, and **not even think of generating twice**.

I believe that the way how CMake uses recursive makes is the root cause of many problems. See for example the thread that was started with http://www.cmake.org/pipermail/cmake/2007-December/018392.html. [^] I read the discussion thoroughly and I am quite sure that the original poster is bitten by the same issue as I am.

I wonder if somebody followed Brad King's advice in http://www.cmake.org/pipermail/cmake/2007-December/018516.html [^] and filed a dedicated bug report. If not, I strongly advocate to take the current issue for this, or (better yet) to file another one with a title that is more to the point of the problem's root - recursive make.
(0019311)
Brad King   
2010-01-26 09:17   
General advice when a generated source is used in multiple targets: generate it with another custom target, and make all the targets that use the source depend on the custom target.

As for recursive make, I challenge anyone to come up with non-recursive Makefiles that can handle build-time dependency scanning. Using GNU-make extensions is not allowed because we have to work with ancient UNIX make.
(0026844)
Andreas Mohr   
2011-06-12 13:52   
Brad, about recursive make: perhaps the discussion at http://stackoverflow.com/questions/559216/what-is-your-experience-with-non-recursive-make [^] contains a few insights? (albeit at least some of these postings talk about requiring GNU make features, which one - and you ;) - would want to avoid as a restriction, for obvious reasons).

Benchmark item: on my build tree (>> 50 projects, tons of additional non-executable custom targets, perhaps 500 targets all in all) an empty build at -j3 takes around 15 seconds on dual core (and that's with many things streamlined as much as possible).

Hmm, just realized that I'm perhaps a bit off-topic...

Thanks!
(0030552)
Brad King   
2012-08-13 10:44   
Sending issues I'm not actively working on to the backlog to await someone with time for them.

If an issue you care about is sent to the backlog when you feel it should have been addressed in a different manner, please bring it up on the CMake mailing list for discussion. Sign up for the mailing list here, if you're not already on it:

 http://www.cmake.org/mailman/listinfo/cmake [^]

It's easy to re-activate a bug here if you can find a CMake developer or contributor who has the bandwidth to take it on.
(0041636)
Kitware Robot   
2016-06-10 14:27   
Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.