CTest Dashboard Client Guide

Introduction

A CDash dashboard collects the results of configuring, building, and testing a project on many machines. The ctest(1) program acts as a Dashboard Client: it runs those steps and submits the results to a CDash server.

This guide shows how to prepare a project for dashboard submissions, how to submit results from an existing build tree, and how to write a script that runs a dashboard on a schedule. See the ctest(1) manual for a complete reference.

Preparing a Project

A project enables dashboard submissions by including the CTest module in its top-level CMakeLists.txt file:

cmake_minimum_required(VERSION 4.0)
project(MyProject)

include(CTest)

if(BUILD_TESTING)
  add_executable(my_test my_test.cxx)
  add_test(NAME my_test COMMAND my_test)
endif()

The module calls enable_testing() and adds the BUILD_TESTING option. It also writes the dashboard settings to a configuration file in the build tree.

The project also needs a CTestConfig.cmake file next to its top-level CMakeLists.txt file to say where to submit results. A CDash server offers this file for download on the settings page of each project. It contains settings such as:

set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
set(CTEST_SUBMIT_URL "https://my.cdash.org/submit.php?project=MyProject")

The CTEST_NIGHTLY_START_TIME variable must match the nightly start time configured in CDash, so that all Nightly submissions from one night appear on the same day.

Submitting from a Build Tree

The quickest way to submit results is to run ctest from an existing build tree. Configure and build the project as usual, then run an Experimental dashboard:

cmake -S MyProject -B build
cmake --build build
cd build
ctest -D Experimental

This runs the Start, Configure, Build, Test, Coverage, and Submit steps. The results are recorded in the Testing directory of the build tree before they are submitted.

The CTest module also adds an Experimental target to the build system that does the same thing:

cmake --build build --target Experimental

Running Individual Steps

A dashboard may also be run one step at a time. The ctest -M option selects the mode and each ctest -T option adds a step:

ctest -M Experimental -T Start -T Configure -T Build -T Test -T Submit

Results are recorded under the tag created by the most recent Start step, and later invocations add to it. By default, the Submit step sends every result recorded under that tag, including results from earlier invocations. Set CTEST_SUBMIT_PARTS to submit only some of them. For example, to rerun the tests and submit only the new test results:

ctest -M Experimental -T Test -T Submit -D CTEST_SUBMIT_PARTS=Test

Overriding Settings

Settings in the configuration file can be overridden for one run with the -D <var>=<value> option. For example, to submit with a different build name:

ctest -D Experimental -D CTEST_BUILD_NAME=linux-gcc-debug

Writing a Dashboard Script

Machines that submit dashboards on a schedule usually run a script with ctest -S instead. A script can check out the source tree, start from an empty build tree, and choose which steps to run. It does not depend on a build tree that was configured beforehand.

The script is written in the cmake-language(7). It sets variables that hold the dashboard settings and calls the CTest Commands to run each step. The following script runs a Nightly dashboard for a project hosted in a Git repository:

# Describe this machine and build.
set(CTEST_SITE "my-machine")
set(CTEST_BUILD_NAME "linux-gcc-release")

# Choose where to put the source and build trees.
set(CTEST_DASHBOARD_ROOT "$ENV{HOME}/dashboards")
set(CTEST_SOURCE_DIRECTORY "${CTEST_DASHBOARD_ROOT}/MyProject")
set(CTEST_BINARY_DIRECTORY "${CTEST_DASHBOARD_ROOT}/MyProject-build")

# Choose how to configure and build.
set(CTEST_CMAKE_GENERATOR "Ninja")
set(CTEST_CONFIGURATION_TYPE "Release")

# Clone the repository on the first run, and update it later.
find_program(CTEST_GIT_COMMAND NAMES git)
set(repo "https://example.com/MyProject.git")
if(NOT EXISTS "${CTEST_SOURCE_DIRECTORY}")
  set(CTEST_CHECKOUT_COMMAND
    "\"${CTEST_GIT_COMMAND}\" clone ${repo} \"${CTEST_SOURCE_DIRECTORY}\"")
endif()

# Start each night with an empty build tree.
ctest_empty_binary_directory("${CTEST_BINARY_DIRECTORY}")

ctest_start(Nightly)
ctest_update()
ctest_configure()
ctest_build()
ctest_test()
ctest_submit()

Run the script from any directory outside the build tree:

ctest -S nightly.cmake -V

The ctest -V option shows the progress of each step.

Notes on the script:

Passing Values to a Script

One script can serve several builds by reading values from the command line. Each -D <var>=<value> option sets a variable before the script runs:

if(NOT DEFINED CTEST_BUILD_NAME)
  set(CTEST_BUILD_NAME "linux-gcc-release")
endif()

ctest -S nightly.cmake -D CTEST_BUILD_NAME=linux-gcc-debug

Variables whose names do not begin with CTEST_ must be given with a type, such as -D dashboard_model:STRING=Nightly.

Continuous Dashboards

A Continuous dashboard runs repeatedly during the day and submits results only when the source tree has changed. In a script, use the RETURN_VALUE option of ctest_update() to stop early when nothing changed:

ctest_start(Continuous)
ctest_update(RETURN_VALUE count)
if(count EQUAL 0)
  return()
endif()
ctest_configure()
ctest_build()
ctest_test()
ctest_submit()

A Continuous script usually keeps its build tree between runs, so it does not call ctest_empty_binary_directory(). This keeps each build incremental.

Checking for Failures

Each command can store its result in a variable so the script can react to failures. For example, to skip the tests when the build fails:

ctest_build(NUMBER_ERRORS errors)
if(errors EQUAL 0)
  ctest_test()
endif()
ctest_submit()

See the documentation of each command for the values it reports.

Running on a Schedule

A dashboard script can be run by the system scheduler (such as crontab on Linux and macOS), or via a continuous integration platform (such as GitHub Actions). Schedule Nightly scripts to start shortly after the CTEST_NIGHTLY_START_TIME so that each submission is associated with the intended day.

Using CMake Presets

Added in version 4.4.

A project that provides cmake-presets(7) can use them in a dashboard. In a script, set CTEST_PRESET to use one preset name for the configure, build, and test steps, or set CTEST_CONFIGURE_PRESET, CTEST_BUILD_PRESET, and CTEST_TEST_PRESET separately:

set(CTEST_SOURCE_DIRECTORY "${CTEST_DASHBOARD_ROOT}/MyProject")
set(CTEST_BINARY_DIRECTORY "${CTEST_DASHBOARD_ROOT}/MyProject-build")
set(CTEST_PRESET "ci-release")

ctest_start(Nightly)
ctest_configure()
ctest_build()
ctest_test()
ctest_submit()

Settings from the script, such as CTEST_BINARY_DIRECTORY, take priority over the corresponding settings from the preset.

From the command line, the --source-dir option lets the Configure step create a new build tree from a preset. If the ci-release preset sets binaryDir to ${sourceDir}/build/${presetName}, then:

ctest -M Experimental -T Configure --source-dir MyProject \
  -D CTEST_CONFIGURE_PRESET=ci-release
ctest -M Experimental -T Build -T Test -T Submit \
  --test-dir MyProject/build/ci-release

Adding More Steps

Coverage

To record test coverage, build the project with coverage enabled, such as with the --coverage flag of GCC, and call ctest_coverage() after ctest_test():

find_program(CTEST_COVERAGE_COMMAND NAMES gcov)

ctest_start(Nightly)
ctest_configure(OPTIONS "-DCMAKE_CXX_FLAGS=--coverage")
ctest_build()
ctest_test()
ctest_coverage()
ctest_submit()

See the CTest Coverage Step for the supported tools.

Memory Checking

To run the tests under a memory checker, call ctest_memcheck() instead of, or in addition to, ctest_test(). The CTEST_MEMORYCHECK_TYPE variable selects the tool:

set(CTEST_MEMORYCHECK_TYPE "Valgrind")
find_program(CTEST_MEMORYCHECK_COMMAND NAMES valgrind)

ctest_start(Nightly)
ctest_configure()
ctest_build()
ctest_memcheck()
ctest_submit()

See the CTest MemCheck Step for the supported tools, including the sanitizers supported by Clang and GCC.

Detailed Build Reports

By default, CTest finds build errors and warnings by scanning the build output. Set CTEST_USE_LAUNCHERS in CTestConfig.cmake to instead report the command line and output of each compile or link step that produces a warning or error:

set(CTEST_USE_LAUNCHERS 1)

This works with the Makefile Generators and Ninja Generators.

Notes and Extra Files

A submission can include notes files, such as a copy of the dashboard script, to show on CDash. Set CTEST_NOTES_FILES before calling ctest_submit():

set(CTEST_NOTES_FILES "${CMAKE_CURRENT_LIST_FILE}")

From the command line, use the --add-notes option.

Next Steps