[CMake] CTest examples

[David Cole]

On Fri, Jul 9, 2010 at 5:54 PM, Chris Hillery <chillery-cmake at lambda.nu>wrote:

> For CTest, running a test means running an executable and seeing whether
> the result of that process is 0 (success) or not 0 (failure). It is also
> possible to scan the output of the executable for certain regular
> expressions to determine success or failure.
>
> It's a very simple and somewhat limited mechanism. It does depend on you
> writing your own test driver, which makes sense because CMake can't possibly
> know how to test your code.
>
> As an example, Zorba is an XQuery (xml query language) processor. The
> majority of our tests are actual XQueries stored in text files with the
> extension .xq, along with corresponding expected results in .xml files. We
> have a testdriver program which executes the query, checks the results, and
> returns 0 or 1.
>
> Our CMakeLists looks vaguely like this:
>
> add_executable(testdriver ${TESTDRIVER_SRCS})
> file(GLOB_RECURSE TESTFILES "${CMAKE_CURRENT_SOURCE_DIR}" "*.xq")
> foreach(TESTFILE ${TESTFILES})
>   add_test(${TESTFILE} testdriver ${TESTFILE})
> endforeach()
>
> This creates one test for each .xq file. The name of the test is the name
> of the .xq file, and that same name is passed as an argument to testdriver.
>
> This works well, and lets us use ctest's functionality like -R to run
> certain tests by name. It also results in nice easy-to-read listings on
> CDash because there's one line for each test, making it easy to examine
> failures. (Although given that we have over 20,000 tests, some pages on
> CDash take a looooong time to load...) The main downside is speed, since
> every single test has to start up a whole Zorba process.
>
> When you get to unit testing, however, IMHO things don't work as well.
> CMake does offer the create_test_sourcelist() command, and you should look
> into it. We use that in a couple places to create a single test driver from
> several separate small unit test drivers. However, you still need to write a
> loop in CMakeLists to add multiple tests using that test driver.
>
> In another project I work on, I've been using CxxTest for a unit testing
> framework. This is a really nice setup as it makes it very trivial to write
> many unit tests. However, the integration with ctest is poor. There's a
> reasonably good FindCxxTest script shipped with CMake that makes it easy to
> get CxxTest suites running; in a nutshell:
>
> find_package(CxxTest)
> cxx_add_test(my_unit_test my_unit_test.cpp
> "${CMAKE_CURRENT_SOURCE_DIR}/my_unit_test.h")
>
> Unit tests in CxxTest are written as classes in a .h file; cxx_add_test()
> runs CxxTest to generate the named .cpp file from that, then builds it into
> the named test driver and adds a CTest test for it. So far so good. However,
> the main benefit of CxxTest is grouping similar test cases into a single .h
> file, probably with similar setup and so forth. Unfortunately cxx_add_test()
> only adds one test to CTest, and that one test runs all the test cases in
> the .h file. So you lose the ability to run certain tests with ctest -R, and
> the reports on CDash will just show whether ALL the tests in my_unit_test
> succeeded or not.
>
> The fundamental problem is that ctest requires a test case to be a program
> execution, and has no facilities to allow a program to represent multiple
> tests. So either you pay the cost in speed, or you lose some of the
> flexibility ctest and cdash offer. (To be fair, even if this were fixed in
> ctest, CxxTest would also need some modifications as it currently doesn't
> let you run subsets of tests either.)
>
> Ceej
> aka Chris Hillery
>
>
> On Fri, Jul 9, 2010 at 4:37 AM, Bo Thorsen <bo at askmonty.org> wrote:
>
>> Hi people,
>>
>> I have converted a set of applications to cmake and cpack, and now have my
>> eyes set on ctest.
>>
>> I'd like to hear if someone here has some good advice, or links to good
>> advice, on how to structure tests. I'm searching for help on how to put
>> different tests into what executables. On how to handle multiple tests on
>> each classes, on how to best structure the test of the static libraries (all
>> of those are part of the source tree) that are linked in to the application.
>> And on how to test classes from the main application itself.
>>
>> I have read the ctest FAQ, documentation etc. and still don't know
>> anything that help me write the actual test code.
>>
>> From the looks of it, ctest only provides the framework to run a test, no
>> help is given to write the code of the tests themselves, is this right? I
>> have previously been using cppunit, and it looks like this will still be
>> useful.
>>
>> To sum it up, I'm looking for real life advice on what you guys have done
>> with ctest. This information seem almost completely missing on the net,
>> where all searches on ctest leads to useless presentation on ctest features.
>>
>> Cheers,
>>
>> Bo Thorsen.
>> Monty Program AB.
>>
>> --
>>
>> MariaDB: MySQL replacement
>> Community developed. Feature enhanced. Backward compatible.
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
>>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>


Nice analysis!

Thanks for taking the time to write this up and send it along to the mailing
list.

Have a good weekend,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20100709/26981942/attachment.htm>