[cmake-developers] [CMake] Setup/tear down steps for CTest

[Ben Boeckel]

On Tue, Aug 23, 2016 at 08:00:09 +0200, Rolf Eike Beer wrote:
> Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott:
> > So how would you want the feature to work? I'd suggest an initial set of
> > requirements something like the following:
> > 
> >    - Need to support the ability to define multiple setup and/or tear down
> >    tasks.
> >    - It should be possible to specify dependencies between setup tasks and
> >    between tear down tasks.
> >    - Individual tests need to be able to indicate which setup and/or tear
> >    down tasks they require, similar to the way DEPENDS is used to specify
> >    dependencies between test cases.
> >    - When using ctest --rerun-failed, ctest should automatically invoke any
> >    setup or tear down tasks required by the test cases that will be re-run.
> >    - Setup or tear down tasks which reference executable targets should
> >    substitute the actual built executable just like how add_custom_command()
> > does.
> 
> -need a way to mark if 2 tests with the same setup/teardown can share those or 
> if they need to run for every of them

Proposal:

    add_test(NAME setup-foo ...)
    set_tests_properties(setup-foo PROPERTIES
      SETUP_GROUP foo
      SETUP_STEP SETUP_PER_TEST) # Also SETUP_ONCE.
    add_test(NAME use-foo ...)
    set_tests_properties(use-foo PROPERTIES
      SETUP_GROUP foo) # implicit depends on all SETUP_GROUP foo / SETUP_STEP SETUP_* tests.
    add_test(NAME use-foo2 ...)
    set_tests_properties(use-foo2 PROPERTIES
      SETUP_GROUP foo)
    add_test(NAME teardown-foo2 ...)
    set_tests_properties(teardown-foo2 PROPERTIES
      SETUP_GROUP foo
      SETUP_STEP TEARDOWN) # implicit depends on all non-TEARDOWN steps

Multiple setup/teardown steps could be done with DEPENDS between them.

> -the default for each test is "no s/t", which means it can't be run with any 
> of the above in parallel (especially for compatibillity)[1]
> -need a way to tell if a test doesn't care about those

Making RESOURCE_LOCK a rwlock rather than a mutex might make sense here.
SETUP_STEP bits have a RESOURCE_LOCK_WRITE group_${group}, otherwise it
is RESOURCE_LOCK_READ group_${group}.

> 1) think of a database connector test: the test that will check what happens 
> if no DB is present will fail if the setup step "start DB" was run, but not 
> the teardown

RESOURCE_LOCK on that group_${group} can fix that I think.

> > Some open questions:
<snip>

I agree with what Eike said.

--Ben