[CMake] CTest examples

[Chris Hillery]

For CTest, running a test means running an executable and seeing whether the
result of that process is 0 (success) or not 0 (failure). It is also
possible to scan the output of the executable for certain regular
expressions to determine success or failure.

It's a very simple and somewhat limited mechanism. It does depend on you
writing your own test driver, which makes sense because CMake can't possibly
know how to test your code.

As an example, Zorba is an XQuery (xml query language) processor. The
majority of our tests are actual XQueries stored in text files with the
extension .xq, along with corresponding expected results in .xml files. We
have a testdriver program which executes the query, checks the results, and
returns 0 or 1.

Our CMakeLists looks vaguely like this:

add_executable(testdriver ${TESTDRIVER_SRCS})
file(GLOB_RECURSE TESTFILES "${CMAKE_CURRENT_SOURCE_DIR}" "*.xq")
foreach(TESTFILE ${TESTFILES})
  add_test(${TESTFILE} testdriver ${TESTFILE})
endforeach()

This creates one test for each .xq file. The name of the test is the name of
the .xq file, and that same name is passed as an argument to testdriver.

This works well, and lets us use ctest's functionality like -R to run
certain tests by name. It also results in nice easy-to-read listings on
CDash because there's one line for each test, making it easy to examine
failures. (Although given that we have over 20,000 tests, some pages on
CDash take a looooong time to load...) The main downside is speed, since
every single test has to start up a whole Zorba process.

When you get to unit testing, however, IMHO things don't work as well. CMake
does offer the create_test_sourcelist() command, and you should look into
it. We use that in a couple places to create a single test driver from
several separate small unit test drivers. However, you still need to write a
loop in CMakeLists to add multiple tests using that test driver.

In another project I work on, I've been using CxxTest for a unit testing
framework. This is a really nice setup as it makes it very trivial to write
many unit tests. However, the integration with ctest is poor. There's a
reasonably good FindCxxTest script shipped with CMake that makes it easy to
get CxxTest suites running; in a nutshell:

find_package(CxxTest)
cxx_add_test(my_unit_test my_unit_test.cpp
"${CMAKE_CURRENT_SOURCE_DIR}/my_unit_test.h")

Unit tests in CxxTest are written as classes in a .h file; cxx_add_test()
runs CxxTest to generate the named .cpp file from that, then builds it into
the named test driver and adds a CTest test for it. So far so good. However,
the main benefit of CxxTest is grouping similar test cases into a single .h
file, probably with similar setup and so forth. Unfortunately cxx_add_test()
only adds one test to CTest, and that one test runs all the test cases in
the .h file. So you lose the ability to run certain tests with ctest -R, and
the reports on CDash will just show whether ALL the tests in my_unit_test
succeeded or not.

The fundamental problem is that ctest requires a test case to be a program
execution, and has no facilities to allow a program to represent multiple
tests. So either you pay the cost in speed, or you lose some of the
flexibility ctest and cdash offer. (To be fair, even if this were fixed in
ctest, CxxTest would also need some modifications as it currently doesn't
let you run subsets of tests either.)

Ceej
aka Chris Hillery

On Fri, Jul 9, 2010 at 4:37 AM, Bo Thorsen <bo at askmonty.org> wrote:

> Hi people,
>
> I have converted a set of applications to cmake and cpack, and now have my
> eyes set on ctest.
>
> I'd like to hear if someone here has some good advice, or links to good
> advice, on how to structure tests. I'm searching for help on how to put
> different tests into what executables. On how to handle multiple tests on
> each classes, on how to best structure the test of the static libraries (all
> of those are part of the source tree) that are linked in to the application.
> And on how to test classes from the main application itself.
>
> I have read the ctest FAQ, documentation etc. and still don't know anything
> that help me write the actual test code.
>
> From the looks of it, ctest only provides the framework to run a test, no
> help is given to write the code of the tests themselves, is this right? I
> have previously been using cppunit, and it looks like this will still be
> useful.
>
> To sum it up, I'm looking for real life advice on what you guys have done
> with ctest. This information seem almost completely missing on the net,
> where all searches on ctest leads to useless presentation on ctest features.
>
> Cheers,
>
> Bo Thorsen.
> Monty Program AB.
>
> --
>
> MariaDB: MySQL replacement
> Community developed. Feature enhanced. Backward compatible.
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20100709/c3c85865/attachment-0001.htm>