[CMake] ctest // fixtures and --repeat-until-fail <n>

[Craig Scott]

On Tue, Apr 16, 2019 at 11:36 PM Sergei Nikulov <sergey.nikulov at gmail.com>
wrote:

> >
> > On Tue, Apr 16, 2019 at 10:07 PM Sergei Nikulov <
> sergey.nikulov at gmail.com> wrote:
> >>
> >> Hello All,
> >>
> >> Has anybody knows how FIXTURES_SETUP/FIXTURES_CLEANUP should work with
> >> --repeat-until-fail <n> option?
> >>
> >> My expectation, if I requesting a test run for example 10 times, the
> >> sequence should be as follows:
> >>
> >> fixture_setup, then test, then fixture_cleanup repeats 10 times
> >>
> >> But in practice, I've got following sequence
> >>
> >> 1. fixture_setup runs 10 times
> >> 2. test runs 10 times
> >> 3. fixture_cleanup runs 10 times
> >>
> >> Is it expected behavior?
> >> I'm using ctest version 3.14.2
> >
> >
> > The documentation for the --repeat-until-fail option says "Require each
> test to run <n> times without failing in order to pass". A fixture setup or
> cleanup test is still a test, so they will also be run <n> times. The logic
> that implements repeating tests sets up a counter on each test and it runs
> that test <n> times before marking that test as complete. This is why you
> see fixture_setup run 10 times, then test runs 10 times and lastly
> fixture_cleanup runs 10 times.
> >
>
> I understand that fixture is still a test.
> To achieve repeatable behavior -R "test_continues_*" is enough.
>
> But if it calls FIXTURE why not apply the specific behavior as
> expected from fixture (init/teardown)?
>

It's mostly related to the way it is implemented internally. Because a
fixture setup/cleanup is still just another test, it has all the features
of a regular test, including honouring the option that tells ctest to run
it multiple times. For some projects, it may be desirable for the fixture
setup/cleanup to run only once but re-run the fixture-requiring test(s) to
run multiple times. For other projects, it might be desirable for the
fixture setup/cleanup tests to also run multiple times (a setup test might
simply be polling a service to confirm it is up, for example). Since ctest
cannot know which of the two scenarios the project wants, it simply treats
all tests the same, whether they are fixture setup/cleanup tests or not.

Perhaps one way of dealing with this might be to add a new boolean test
property which, when set, has the meaning of "only run once" when the
--repeat-until-fail option is used.

-- 
Craig Scott
Melbourne, Australia
https://crascit.com

Get the hand-book for every CMake user: Professional CMake: A Practical
Guide <https://crascit.com/professional-cmake/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20190427/91b6d77f/attachment.html>