[cmake-developers] Adding compile and build type tests to CMake/CTest

Brad King brad.king at kitware.com
Fri Sep 15 08:22:27 EDT 2017


On 09/14/2017 11:33 PM, Edward Diener wrote:
> Boost Build has tests for running an application successfully or not, 
> for compiling one or more source files successfully or not, and for 
> building one or more source files into an exe or not. These tests in 
> Boost Build are the run/run-fail, compile/compile-fail, and 
> link/link-fail rules.

Good summary.  One major difference is that Boost Build directly controls
the entire build graph and is natively aware of expected failure cases.
CMake needs to generate an external build system e.g. as Visual Studio
`.vcxproj` files with no good way to represent expected failure directly.
This is a technical reason CMake has never gained this functionality,
but perhaps it could be overcome with a careful design.  See below.

> CMake/CTest has the exact equivalent to the run/run-fail rule in its 
> add_test framework

While `add_test` can do run/run-fail it doesn't do so given just a
source file like Boost Build does.  `add_test` expects to be given a
command referring to some executable that already exists or is built
elsewhere in the project via `add_executable`.  Therefore some work
would be needed even to reproduce Boost's run/run-fail infrastructure.

> Hopefully CMake developers will get the message and make the necessary 
> update to CMake/CTest.

We understand the value of the tests but simply don't have the need for
them in our own use cases.  This is a non-technical reason CMake does
not have this functionality, but it doesn't stand in the way of others
contributing a solution.

> Is there any consensus that these other two types of tests might be 
> valuable for CMake/CTest, or any way to make this happen ?

I agree they would be useful.

>From some quick brainstorming, one idea to deal with the technical
challenge raised above is to add a new "BUILD_TEST" target property
to tell CMake that a target is meant only to be built during testing.
We allow dependencies among BUILD_TEST targets on each other and on
normal targets, but not from normal targets to BUILD_TEST targets.
That will give a separable part of the build graph to generate only
for testing.  Then add_test can create tests that try to build one
of the BUILD_TEST targets and expect one of the compile/compile-fail,
link/link-fail, or run/run-fail cases.

For generators like Visual Studio we could lave the BUILD_TEST
targets out of the main `.sln` file so they wouldn't show up when
users load the projects.  Instead they would be placed in a second
`.sln` file used only for testing.

Further design work is likely needed but this is a start.

---------------------------------------------------------------------

IMO it will be better to first prototype this externally to gain some
experience with it and work with existing CMake releases.  One can do
this by using the `export()` command to make all targets in the build
tree available to externally-built projects (which is useful on its own).
With that a few approaches can be used to drive testing as an external
build.

Then have a custom target that configures the test-only part of the
project externally but doesn't build it.  The test-only project is
externally-built and can load the targets file created by `export()`
above.  Then use `add_test` to create tests that drive the build on
parts of the external test-only project and check for appropriate
success/failure and output matching.

-Brad


More information about the cmake-developers mailing list