MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0014729CMakeCMakepublic2014-02-03 00:442016-06-10 14:31
ReporterNAKAMURA Takumi 
Assigned ToKitware Robot 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionmoved 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0014729: Add hook for project-defined check-build-system actions
DescriptionExperimentally I am writing "Custom generator". This is intended for auto dependency scanner.

It doesn't work, at least with Ninja Generator.
As far as I see build.ninja, a few issues...

1) "ninja -t clean" removes foo.cmake.

  It's Ninja's behavior. To avoid this, another rule should be added for custom_command that touches generated cmakefiles (included by CMakeLists).
  For example,

  rule CUSTOM_GENERATOR
    command = $COMMAND
    description = $DESC
    restat = 1
    generator = 1

2) foo.cmake is added to phony target list, "A missing CMake input file is not an error.", although foo.cmake has a rule.

  Output of custom_command should be excluded from phony targets even if it were included from CMakeLists.

3) Also generator-dependent file, for example foobar.c might be put into phony list, or Ninja could not rebuild build.ninja when foobar.c were disappeared (in revision walking).
Steps To Reproducecmake_minimum_required(VERSION 2.8)

# It's supposed:
# - Scan dependent file(s).
# - Update when something happened. ;)
# For now, manually update tmpl.txt.

# "touch foobar.c" doesn't trigger regeneration.
# "echo >> tmpl.txt; touch foobar.c" triggers regeneration.

add_custom_command(OUTPUT foo.cmake
  COMMAND ${CMAKE_COMMAND} -E copy_if_different
  ${CMAKE_CURRENT_SOURCE_DIR}/tmpl.txt
  ${CMAKE_CURRENT_BINARY_DIR}/foo.cmake
  DEPENDS foobar.c
  COMMENT "Updating foo.cmake"
  )
add_custom_target(theGenerator DEPENDS foo.cmake)

execute_process(
  COMMAND ${CMAKE_COMMAND} -E copy
  ${CMAKE_CURRENT_SOURCE_DIR}/tmpl.txt
  ${CMAKE_CURRENT_BINARY_DIR}/foo.cmake
  )

include(${CMAKE_CURRENT_BINARY_DIR}/foo.cmake)

add_executable(foobar foobar.c)

# Almost all targets may depend on theGenerator.
add_dependencies(foobar theGenerator)
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2014-02-03 00:44NAKAMURA TakumiNew Issue
2014-02-03 13:38Brad KingNote Added: 0035024
2014-02-03 13:40Brad KingNote Added: 0035025
2014-02-25 05:05NAKAMURA TakumiNote Added: 0035180
2014-02-25 08:02Brad KingNote Added: 0035182
2014-02-25 08:05Brad KingNote Added: 0035183
2014-02-25 08:21NAKAMURA TakumiNote Added: 0035184
2014-02-25 08:31Brad KingNote Added: 0035185
2014-02-25 08:32NAKAMURA TakumiNote Added: 0035186
2014-02-25 08:38Brad KingNote Added: 0035187
2014-02-25 08:38NAKAMURA TakumiNote Added: 0035188
2014-02-25 09:09Brad KingNote Added: 0035189
2014-02-25 09:25NAKAMURA TakumiNote Added: 0035190
2014-02-25 10:21Brad KingNote Added: 0035193
2014-02-25 11:27NAKAMURA TakumiNote Added: 0035194
2014-02-25 11:37Brad KingStatusnew => backlog
2014-02-25 11:37Brad KingProduct VersionCMake 2.8.12.1 =>
2014-02-25 11:37Brad KingSummarySupport "custom cmake generator" => Add hook for project-defined check-build-system actions
2014-02-25 11:39Brad KingNote Added: 0035195
2014-02-25 12:21NAKAMURA TakumiNote Added: 0035196
2016-06-10 14:29Kitware RobotNote Added: 0042475
2016-06-10 14:29Kitware RobotStatusbacklog => resolved
2016-06-10 14:29Kitware RobotResolutionopen => moved
2016-06-10 14:29Kitware RobotAssigned To => Kitware Robot
2016-06-10 14:31Kitware RobotStatusresolved => closed

Notes
(0035024)
Brad King   
2014-02-03 13:38   
FYI, after seeing the configure_file hack that LLVM uses to add dependencies to re-run CMake, I added the CMAKE_CONFIGURE_DEPENDS directory property:

 Allow projects to specify extra inputs to CMake
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=82d43175 [^]
(0035025)
Brad King   
2014-02-03 13:40   
Please try the Ninja generator in current CMake 'master' as of commit adf309cd0db0db51913b2054400e595cb60c0eba or later. The rules to re-run CMake have been updated since 2.8.12.
(0035180)
NAKAMURA Takumi   
2014-02-25 05:05   
Sorry for the delay. I tried: v2.8.12.2-1497-gcd8c797
It seems it doesn't resolve any of my issues.

FYI, this is my practical tweaks.
https://github.com/chapuni/llvm-project/commits/cmake/tddeps [^]
(0035182)
Brad King   
2014-02-25 08:02   
The line

  DEPENDS foobar.c

should be

  DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/foobar.c

or some other full path.

Using copy-if-different to produce a custom command OUTPUT will mean that when the dependencies are out of date but the output happens to not change the rule will run every time. You should use a separate timestamp file as the OUTPUT that is always updated while producing the copy-if-different output as a side-effect.
(0035183)
Brad King   
2014-02-25 08:05   
When testing dependency rules please use the Makefile generator first. The Ninja generator is known to not properly enforce CMake dependency rules in all cases: see issue 0014728 and issue 0014747.
(0035184)
NAKAMURA Takumi   
2014-02-25 08:21   
I assume foobar.c woud be in ${CMAKE_SOURCE_DIR}.
Regardles of ${CMAKE_SOURCE_DIR}, CMake can find foobar.c in ${CMAKE_SOURCE_DIR} for me.

About "copy_if_different", please suppose it "to emulate a script, to update foo.cmake conditionally".
I am using it since it can help to reproduce my issues.
(0035185)
Brad King   
2014-02-25 08:31   
Re 0014729:0035184: Use "cmake -E copy" to update foo.cmake unconditionally in your test cases.

I do not understand exactly what you mean by "Custom generator" or "intended for auto dependency scanner". Of what is it scanning dependencies? What needs to process those dependencies? In other words, what is the purpose of tddeps?
(0035186)
NAKAMURA Takumi   
2014-02-25 08:32   
I prefer Ninja Generator because generated build.ninja is easy for me to read and understand.

I knew my testcase doesn't work as expected with Makefile.
As far as I investigated, "cmake_check_build_system" won't have any precedent targets like "theGenerator".

With my testcase, "echo >> ${src}/tmpl.txt;touch ${src}/foobar.c;make" doesn't reconfigure with 1st run.

$ echo >> ../src/tmpl.txt ;touch ../src/foobar.c
# 1st run
$ make
[ 50%] Updating foo.cmake
[ 50%] Built target theGenerator
Scanning dependencies of target foobar
[100%] Building C object CMakeFiles/foobar.dir/foobar.c.o
Linking C executable foobar
[100%] Built target foobar

# 2nd run
$ make
-- Configuring done
-- Generating done
-- Build files have been written to: /home/chapuni/BUILD/cmake/issue14729/make
[ 50%] Built target theGenerator
[100%] Built target foobar
$
(0035187)
Brad King   
2014-02-25 08:38   
Re 0014729:0035186:

> I prefer Ninja Generator because generated build.ninja is easy for me to read and understand.

What I'm saying is that the Makefile generator's behavior is more representative of the intended dependency rules for CMake. The Ninja generator is an approximation of them that is not yet fully mature. Do not try to interpret CMake's intentions by reading build.ninja files.
(0035188)
NAKAMURA Takumi   
2014-02-25 08:38   
"theGenerator (aka tddeps)" is expected;

- scan all headers and all source files
- if dependency graph is changed, update foo.cmake (aka tddeps.cmake).
  (Then it should trigger regeneration)

I don't expect;

- Regeneration would run every time (woot!)
- User(developer) is required to run make|ninja twice.

Then, I'd like to put pre-cmake script before "cmake --check-build-system".
(0035189)
Brad King   
2014-02-25 09:09   
Re 0014729:0035188: so the goal is to generate add_dependencies calls between libraries based on dependencies scanned from the source content? That is not in the scope of what CMake is intended to support.
(0035190)
NAKAMURA Takumi   
2014-02-25 09:25   
I could improve my script to traverse sources and emit inter-libs deps, but I have no plan to do.

For now, (in tddeps), I am adding appropriate dependencies between tblgen'd headers and objects, both libraries and executables.

What is "out of the cmake's scope"? I'd just like a hook point to let CMake under control.

FYI, I wrote ad-hoc patches for cmake to do.
https://github.com/chapuni/CMake/commits/chapuni/trunk [^]

Expected behavior is:
$ touch ../src/foobar.c # Suppose not to update foo.cmake.
$ ninja
[1/2] Updating foo.cmake
[1/2] Building C object CMakeFiles/foobar.dir/foobar.c.o
[2/2] Linking C executable foobar

$ echo >> ../src/tmpl.txt ;touch ../src/foobar.c # Suppose foo.cmake is updated.
$ ninja
[1/2] Updating foo.cmake
[2/2] Re-running CMake...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/chapuni/BUILD/cmake/issue14729/build
[1/2] Building C object CMakeFiles/foobar.dir/foobar.c.o
[2/2] Linking C executable foobar

# Iterative run does nothing.
$ ninja
ninja: no work to do.
(0035193)
Brad King   
2014-02-25 10:21   
The canonical form of using generated headers across multiple libraries is:

 add_executable(generator ...)
 add_custom_command(OUTPUT generated.h COMMAND generator ... DEPENDS generator)
 add_custom_target(run_generator DEPENDS generated.h)
 add_library(somelib ...) # uses generated.h
 add_dependencies(somelib run_generator)

where the "somelib" lines can be repeated for every library. The way CMake intends this to be expressed is by explicit add_dependencies calls in the CMakeLists.txt files. That way at build time CMake does not have to worry about generated headers because it will not scan dependencies of somelib until the generated header is known to be up to date.

It looks like you're trying to scan sources to detect which libraries actually use the generated headers and automatically construct add_dependencies calls accordingly. Of course when sources change they must be re-scanned. The feature mentioned in 0014729:0035024 would allow you to make CMake re-run every time any source file changes so it can execute_process something to re-scan, but that is less than optimal. Instead you need hooks in the cmake --check-build-system (or equivalent for each generator) step to do the scanning efficiently at build time and re-run CMake fully only when necessary (e.g. foo.cmake changes in your examples). Currently this is not supported by CMake.

Is this an accurate summary of the situation?
(0035194)
NAKAMURA Takumi   
2014-02-25 11:27   
Yes, sure. I wish I wouldn't hear "CMake would never support it". :)

And I don't think it urgently. This is the "feature request".
(0035195)
Brad King   
2014-02-25 11:39   
Re 0014729:0035194: This is a reasonable request now that I understand it, thanks. I've updated the issue summary description accordingly.
(0035196)
NAKAMURA Takumi   
2014-02-25 12:21   
Makes sense, thanks.

A part of this issue might be resolved with "custom dependency scanner in add_custom_command", I think.
(0042475)
Kitware Robot   
2016-06-10 14:29   
Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.