[cmake-developers] Adding compile and build type tests to CMake/CTest

[Edward Diener]

On 9/15/2017 8:22 AM, Brad King wrote:
> On 09/14/2017 11:33 PM, Edward Diener wrote:
>> Boost Build has tests for running an application successfully or not,
>> for compiling one or more source files successfully or not, and for
>> building one or more source files into an exe or not. These tests in
>> Boost Build are the run/run-fail, compile/compile-fail, and
>> link/link-fail rules.
> 
> Good summary.  One major difference is that Boost Build directly controls
> the entire build graph and is natively aware of expected failure cases.
> CMake needs to generate an external build system e.g. as Visual Studio
> `.vcxproj` files with no good way to represent expected failure directly.
> This is a technical reason CMake has never gained this functionality,
> but perhaps it could be overcome with a careful design.  See below.

I am a relative beginner with CMake although pretty knowledgeable about 
Boost Build and Visual Studio, so please take my comments based on those 
criteria. A gui environment like Visual Studio does provide 
functionality to just compile one or more source files, and it also 
provides functionality to just build an executable from one or more 
source files without actually running that executable. I do not know how 
CMake "picks up" the results of its add_test when the build engine is 
Visual Studio, but however it does so it seems it could also "pick up" 
the result of a compile or of a compile/link in Visual Studio also, 
although I do not know how difficult this would be to do.

> 
>> CMake/CTest has the exact equivalent to the run/run-fail rule in its
>> add_test framework
> 
> While `add_test` can do run/run-fail it doesn't do so given just a
> source file like Boost Build does.  `add_test` expects to be given a
> command referring to some executable that already exists or is built
> elsewhere in the project via `add_executable`.  Therefore some work
> would be needed even to reproduce Boost's run/run-fail infrastructure.

Boost Build's version of 'run/run-fail' does build an executable from 
the source file(s) that are part of its 'run/run-fail' rule. Giving a 
target name for that executable is optional, else the target name is 
taken from the first source file mentioned. I am not asking that CMake's 
'add_test' completely duplicate Boost Build's 'run/run-fail' rules and I 
do not mind the way that CMake works using an 'add_executable' as long 
as the 'add_executable' for testing can be turned off as a normal build 
target. In general I am not asking that CMake completely duplicate 
exactly Boost Build in these testing scenarios, but that it offer 
similar functionality so that the basic functionality of the Boost Build 
'run', 'compile', and 'link' rules have a CMake equivalent. I think the 
current 'add_executable', 'add_test' is a good enough equivalent to 
Boost Build's 'run/run-fail' rule. I would just like to see a built-in 
equivalent to testing under CMake which involves only a successful 
compilation or not, equivalent to Boost Build's 'compile/compile-fail' 
rule and a built-in equivalent to testing which involves only a 
successful build or not of a non-build target equivalent to Boost 
Build's 'link/link-fail' rule. I think both of the latter are valid 
testing methodologies which do not involve an actual run-time test.

> 
>> Hopefully CMake developers will get the message and make the necessary
>> update to CMake/CTest.
> 
> We understand the value of the tests but simply don't have the need for
> them in our own use cases.  This is a non-technical reason CMake does
> not have this functionality, but it doesn't stand in the way of others
> contributing a solution.

Who are "we" in your previous paragraph ? I believe that you as a 
developer understand that compile-time testing is a reality in the C++ 
world of programming, although I am not sure what other computer 
languages CMake supports and if compile-time testing exists in those 
other languages. Outside of Boost, which pushes the bounds of C++ 
library development and for whom, due to Boost's basis of heavily 
relying on template programming, and where compile-time testing has long 
been a reality, other C++ programmers have occasionally mentioned a need 
for compile-time testing under CMake to be added to CMake's run-time 
testing infrastructure. I am just suggesting that this should not be a 
popularity contest but just a recognition that compile-time testing 
should be supported by CMake within the same generally testing framework 
in which CMake supports run-time testing.

I freely admit that I am a basically a CMake beginner even as an 
end-user, and certainly not a CMake developer and do not know what 
adding compile-time testing would entail at the CMake developer level. 
But as a Boost library contributor, who has designed compile-time tests 
for my own supported libraries in Boost, and who has seen compile-time 
tests for many other Boost libraries, the fact that Boost Build contains 
support for compile-time testing and CMake does not is disconcerting to 
me if Boost is going to successfully move from Boost Build to CMake.

> 
>> Is there any consensus that these other two types of tests might be
>> valuable for CMake/CTest, or any way to make this happen ?
> 
> I agree they would be useful.
> 
>  From some quick brainstorming, one idea to deal with the technical
> challenge raised above is to add a new "BUILD_TEST" target property
> to tell CMake that a target is meant only to be built during testing.
> We allow dependencies among BUILD_TEST targets on each other and on
> normal targets, but not from normal targets to BUILD_TEST targets.
> That will give a separable part of the build graph to generate only
> for testing.  Then add_test can create tests that try to build one
> of the BUILD_TEST targets and expect one of the compile/compile-fail,
> link/link-fail, or run/run-fail cases.
> 
> For generators like Visual Studio we could lave the BUILD_TEST
> targets out of the main `.sln` file so they wouldn't show up when
> users load the projects.  Instead they would be placed in a second
> `.sln` file used only for testing.
> 
> Further design work is likely needed but this is a start.
> 
> ---------------------------------------------------------------------
> 
> IMO it will be better to first prototype this externally to gain some
> experience with it and work with existing CMake releases.  One can do
> this by using the `export()` command to make all targets in the build
> tree available to externally-built projects (which is useful on its own).
> With that a few approaches can be used to drive testing as an external
> build.
> 
> Then have a custom target that configures the test-only part of the
> project externally but doesn't build it.  The test-only project is
> externally-built and can load the targets file created by `export()`
> above.  Then use `add_test` to create tests that drive the build on
> parts of the external test-only project and check for appropriate
> success/failure and output matching.
> 
> -Brad
>