[CMake] Dependencies scanning for non-c/c++ files

[Michael Hertling]

On 04/03/2011 02:25 AM, Aaron_Wright at selinc.com wrote:
> I have some m4 files in my build that include other m4 files, so there's a 
> dependency between m4 files that can change at any time. I can calculate 
> the dependency at configure time, but what can I do when the files change 
> and I need to recalculate the dependencies? This is obviously handled for 
> c/c++ files by CMake. Any ideas?
> 
> ---
> Aaron Wright

Build-time dependency scanning without resorting to CMake's built-in
dependency scanner is possible but ugly, in a sort. First of all, you
need a custom target which suitably sets up the dependencies, and you
need to make the CMakeLists.txt file depend on them so that a change
of the dependencies will trigger a reconfiguration-before-rebuild in
order to recognize these new dependencies. In the following examples,
this gets done by a CMake script that gathers all files in a denoted
directory and writes their names to a file containing a CMake SET()
command, and this file gets included in the CMakeLists.txt. In that
way, changing dependencies invalidate the CMakeLists.txt file, and
the filenames - assigned to a variable by the SET() command - are
used to populate the OBJECT_DEPENDS property of a target, so the
latter gets also rebuilt when those files are touched.

Now, the key consideration is that you need two independent Make runs
for the home-brewed build-time dependency scanning to work: The first
run triggers the above-mentioned custom target to update dependencies,
and the second run builds your project with up-to-date dependencies -
maybe after a reconfiguration. AFAICS, this cannot be done in one go
because of the possibly needed intermediate CMake run to reconfigure.

In order to perform these two independent Make runs, you might:

1) Do it manually, e.g. "make dependencies; make". The downside is
   that you'll miss changed dependencies if you forget to perform
   the "dependencies" run.

2) For each affected target, introduce a helper target which triggers
   the dependencies target and subsequently builds the actual target
   via "cmake --build". Look at the following:

# CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(DEPENDENCIES C)
ADD_CUSTOM_TARGET(dependencies
    ${CMAKE_COMMAND}
    -DTEMPLATE=${CMAKE_SOURCE_DIR}/filelist.cmake.in
    -DVARIABLE=DEPENDENCIES
    -DDIRECTORY=${CMAKE_SOURCE_DIR}/dependencies
    -DOUTPUT=${CMAKE_BINARY_DIR}/dependencies.txt
    -P ${CMAKE_SOURCE_DIR}/dependencies.cmake)
IF(NOT EXISTS ${CMAKE_BINARY_DIR}/dependencies.txt)
    EXECUTE_PROCESS(
        COMMAND ${CMAKE_COMMAND}
        -DTEMPLATE=${CMAKE_SOURCE_DIR}/filelist.cmake.in
        -DVARIABLE=DEPENDENCIES
        -DDIRECTORY=${CMAKE_SOURCE_DIR}/dependencies
        -DOUTPUT=${CMAKE_BINARY_DIR}/dependencies.txt
        -P ${CMAKE_SOURCE_DIR}/dependencies.cmake)
ENDIF()
FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(void){return 0;}\n")
INCLUDE(${CMAKE_BINARY_DIR}/dependencies.txt)
SET_SOURCE_FILES_PROPERTIES(${CMAKE_BINARY_DIR}/main.c
    PROPERTIES OBJECT_DEPENDS "${DEPENDENCIES}")
ADD_EXECUTABLE(main0 EXCLUDE_FROM_ALL main.c)
ADD_CUSTOM_TARGET(main ALL
    COMMAND ${CMAKE_COMMAND}
    --build ${CMAKE_BINARY_DIR}
    --target dependencies
    COMMAND ${CMAKE_COMMAND}
    --build ${CMAKE_BINARY_DIR}
    --target main0)

# dependencies.cmake:
FILE(GLOB FILES ${DIRECTORY}/*)
CONFIGURE_FILE(${TEMPLATE} ${OUTPUT} @ONLY)

# filelist.cmake.in:
SET(@VARIABLE@ @FILES@)

The dependencies target runs the dependencies.cmake script which
generates the dependencies.txt file with the filenames from the
dependencies directory; the dependencies.txt file must be created
once initially and it contains the above-mentioned SET() command,
i.e. SET(DEPENDENCIES <dependencies-directory-files>). The actual
target is main0, but the target which is triggered when building
the project is main, and it's this main target that performs the
two independent Make runs using the CMake "--build" option. Note
that due to the use of CONFIGULE_FILE(), the dependencies target
can run each time without causing the project's reconfiguration,
provided the dependencies have not changed. However, if a file
in the dependencies directory is just touched, the main target
gets rebuilt due to the OBJECT_DEPENDS property.

3) Use a helper project which allows unmodified target names:

# CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(DEPENDENCIES C)
EXECUTE_PROCESS(
    COMMAND ${CMAKE_COMMAND}
    -E make_directory ${CMAKE_BINARY_DIR}/helper)
EXECUTE_PROCESS(
    COMMAND ${CMAKE_COMMAND}
    -DTOPLEVEL=${CMAKE_BINARY_DIR}
    ${CMAKE_SOURCE_DIR}/helper
    WORKING_DIRECTORY ${CMAKE_BINARY_DIR}/helper)
ADD_CUSTOM_TARGET(dependencies
    ${CMAKE_COMMAND}
    -DTEMPLATE=${CMAKE_SOURCE_DIR}/filelist.cmake.in
    -DVARIABLE=DEPENDENCIES
    -DDIRECTORY=${CMAKE_SOURCE_DIR}/dependencies
    -DOUTPUT=${CMAKE_BINARY_DIR}/dependencies.txt
    -P ${CMAKE_SOURCE_DIR}/dependencies.cmake)
IF(NOT EXISTS ${CMAKE_BINARY_DIR}/dependencies.txt)
    EXECUTE_PROCESS(
        COMMAND ${CMAKE_COMMAND}
        -DTEMPLATE=${CMAKE_SOURCE_DIR}/filelist.cmake.in
        -DVARIABLE=DEPENDENCIES
        -DDIRECTORY=${CMAKE_SOURCE_DIR}/dependencies
        -DOUTPUT=${CMAKE_BINARY_DIR}/dependencies.txt
        -P ${CMAKE_SOURCE_DIR}/dependencies.cmake)
ENDIF()
FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(void){return 0;}\n")
INCLUDE(${CMAKE_BINARY_DIR}/dependencies.txt)
SET_SOURCE_FILES_PROPERTIES(${CMAKE_BINARY_DIR}/main.c
    PROPERTIES OBJECT_DEPENDS "${DEPENDENCIES}")
ADD_EXECUTABLE(main main.c)

# helper/CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(HELPER NONE)
ADD_CUSTOM_TARGET(helper ALL
    COMMAND ${CMAKE_COMMAND} --build ${TOPLEVEL} --target dependencies
    COMMAND ${CMAKE_COMMAND} --build ${TOPLEVEL})
ADD_CUSTOM_TARGET(main
    COMMAND ${CMAKE_COMMAND} --build ${TOPLEVEL} --target dependencies
    COMMAND ${CMAKE_COMMAND} --build ${TOPLEVEL} --target main)

The dependencies.cmake script and filelist.cmake.in template are the
same as in 2). Effectively, this means you'll have the helper targets
in the helper project so there's no need to have supplementary target
names like main0. Nevertheless, it also means that you need to build
your project or denoted targets from the helper project's build tree,
e.g. "make -C helper" or "make -C helper main" if you reside in your
actual project's build tree as usual.

Besides, the dependencies.txt file does not need to contain a SET()
command; it's just handy to set the OBJECT_DEPENDS property so you
can handle time-stamp dependencies along with file-level ones. In
fact, the dependencies.txt file just needs to be capable of being
included in CMakeLists.txt and get touched when the dependencies
you want to track have changed. If you do not use OBJECT_DEPENDS
in the above-noted manner, the filelist.cmake.in template could
even comprise solely a simple comment, e.g. "# @FILES@".

Currently, because of the two Make runs, I don't see a possibility
to set up things in a way so that this kind of dependency scanning
works as it does with CMake's built-in scanner, but perhaps, you
can adapt one of these approaches to suit your needs.

Regards,

Michael