[cmake-developers] CMake is slow for project with hundred target and one command with large number of dependencies

[Nicolas Desprès]

Hi all,

I am using CMake again since a couple of weeks for a project involving the
build of recognition system. This system is made of hundred sub recognition
system that must be all trained with about ~10k files as input.

I ended up writing something like that:
====== BEGIN
# A project with hundred targets depending on one command with 10k
dependencies.
cmake_minimum_required(VERSION 2.8)
project(HundredTargetsDepOnOneCommandWithManyDeps)

include(CTest)

# Define LARGE_LIST with 10000 input files.
include(large_list.cmake)

foreach(i ${LARGE_LIST})
  file(WRITE ${i} "input")
endforeach(i)

add_custom_command(
  OUTPUT bottle_neck
  COMMAND ${CMAKE_COMMAND} -E touch bottle_neck
  DEPENDS ${LARGE_LIST}
  COMMENT "Executing command with large number of dependencies..."
  )

set(OUTPUTS )
foreach(i RANGE 100)
  set(_output output_${i})
  add_custom_target(${_output}-target ALL
    DEPENDS bottle_neck
    )
  list(APPEND OUTPUTS ${_output})
endforeach(i)

add_executable(AlwaysTrue AlwaysTrue.c)
====== END

Note that the hundred targets have only one dependency. Running CMake on
this project is slow. The configuration is fast but the generation is very
slow. Here some examples:

With Unix Makefiles

$ time make rebuild_cache
Running CMake to regenerate build system...
-- Configuration done
(this is long)
-- Generation done
(this is long)
-- Build files have been written to: ...
make rebuild_cache  120.54s user 6.08s system 97% cpu 2:10.07 total

Memory used is normal (htop indicates 82488 VIRT) and CPU is at 100%.

$ time make
make  129.46s user 3.76s system 99% cpu 2:14.28 total

With Ninja:

$ time ninja -v rebuild_cache
ninja rebuild_cache  98.89s user 0.10s system 99% cpu 1:39.02 total
$ time ninja
ninja  0.01s user 0.04s system 86% cpu 0.055 total

So here my two questions:
1/ Do you think it is possible to improve the performance? (I have not
profile it yet).
2/ Is there a way to set a timeout to ctest so that my test fail. I have
that in Tests/CMakeLists.txt but it does not work:

  set(_test_name HundredTargetsDepOnOneCommandWithManyDeps)
  add_test(${_test_name} ${CMAKE_CTEST_COMMAND}
    --build-and-test
    "${CMake_SOURCE_DIR}/Tests/${_test_name}"
    "${CMake_BINARY_DIR}/Tests/${_test_name}"
    --build-two-config
    ${build_generator_args}
    --build-project ${_test_name}
    --timeout 10
    --test-command AlwaysTrue)
  list(APPEND TEST_BUILD_DIRS
    "${CMake_BINARY_DIR}/Tests/${_test_name}")
  unset(_test_name)

I have pushed my work so far on my github clone of CMake.
https://github.com/nicolasdespres/CMake/tree/topic/large-deps-perf

I am willing to do the work on this topic but I wanted to have your insight
first.

Thanks,

-- 
Nicolas Desprès
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20130712/a25acf6e/attachment.html>