[CMake] ctest // fixtures and --repeat-until-fail <n>

[Craig Scott]

On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <sergey.nikulov at gmail.com>
wrote:

> Hello All,
>
> Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
> --repeat-until-fail <n> option?
>
> My expectation, if I requesting a test run for example 10 times, the
> sequence should be as follows:
>
> fixture_setup, then test, then fixture_cleanup repeats 10 times
>
> But in practice, I've got following sequence
>
> 1. fixture_setup runs 10 times
> 2. test runs 10 times
> 3. fixture_cleanup runs 10 times
>
> Is it expected behavior?
> I'm using ctest version 3.14.2
>

The documentation for the --repeat-until-fail option says "Require each
test to run <n> times without failing in order to pass". A fixture setup or
cleanup test is still a test, so they will also be run <n> times. The logic
that implements repeating tests sets up a counter on each test and it runs
that test <n> times before marking that test as complete. This is why you
see fixture_setup run 10 times, then test runs 10 times and lastly
fixture_cleanup runs 10 times.

Whenever I've wanted to use --repeat-until-fail (admittedly not all that
common for me), I'm generally only interested in repeatedly running one
particular test. For that case, I would typically run the test once with
fixture cleanups disabled (i.e. use a command line "ctest -FC . -R
myFlakeyTest"). I can then run ctest again for just my flaky test with all
fixtures disabled (e.g. "ctest -FA . -R myFlakeyTest --repeat-until-fail
10"). This strategy won't work for everyone's situation, but it has been
effective for me.

-- 
Craig Scott
Melbourne, Australia
https://crascit.com

Get the hand-book for every CMake user: Professional CMake: A Practical
Guide <https://crascit.com/professional-cmake/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20190416/b1ea0c15/attachment.html>