[cmake-developers] [CMake] Setup/tear down steps for CTest

[Rolf Eike Beer]

Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott:
> Cheeky way to get me more involved in contributing, but okay, I'll bite. ;)
> Switching discussion to the dev list.
> 
> So how would you want the feature to work? I'd suggest an initial set of
> requirements something like the following:
> 
>    - Need to support the ability to define multiple setup and/or tear down
>    tasks.
>    - It should be possible to specify dependencies between setup tasks and
>    between tear down tasks.
>    - Individual tests need to be able to indicate which setup and/or tear
>    down tasks they require, similar to the way DEPENDS is used to specify
>    dependencies between test cases.
>    - When using ctest --rerun-failed, ctest should automatically invoke any
>    setup or tear down tasks required by the test cases that will be re-run.
>    - Setup or tear down tasks which reference executable targets should
>    substitute the actual built executable just like how add_custom_command()
> does.

-need a way to mark if 2 tests with the same setup/teardown can share those or 
if they need to run for every of them
-the default for each test is "no s/t", which means it can't be run with any 
of the above in parallel (especially for compatibillity)[1]
-need a way to tell if a test doesn't care about those

1) think of a database connector test: the test that will check what happens 
if no DB is present will fail if the setup step "start DB" was run, but not 
the teardown

> Some open questions:
> 
>    - Should setup and tear down tasks be defined in pairs, or should they
>    completely independent (this would still require the ability to specify a
> dependency of a tear down task on a setup task)?

The test could be "shutdown daemon" or "delete files" so I would keep them 
separated. They may be created by the same command, so they could be batched 
anyway.

>    - Should the setup and tear down tasks be defined by a new CTest/CMake
>    command or extend an existing mechanism (e.g. add_custom_command())?

Don't clutter existing commands more than needed. If it's something new, then 
create a new command (they could still share C++ code). If it's basically the 
same as anything existing at the end then use that.

>    - If no test case has a dependency on a setup or tear down task, should
>    that task be skipped? Perhaps tasks need to have a flag which indicates
>    whether they always run or only if a test case depends on it.

Keep backward compatibility. I.e. if I now add a new test with s/t, then every 
other test should still run (and succeed) as before.

>    - What terminology to use? Things like GoogleTest use terms like test
>    *fixtures* for this sort of thing. The terms setup and tear down are a
>    bit imprecise and cumbersome, so we would probably need something better
>    than those.
>    - Would it make sense for the ctest command line to support disabling
>    setup and/or tear down steps? I can see some potential scenarios where
> this may be desirable, but maybe this is getting too ambitious for a
> starting set of requirements.

IMHO that doesn't make sense. One could think about an option to disable the 
s/t merging, i.e. that they are really called alone for every test.

>    - What should happen if a setup or tear down task fails? How would
>    failure be detected? How would such failures impact things like a CDash
>    test report, etc.?

Then the test fails, just the same as it now does when it can't find the 
executable.

Eike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20160823/50190d4c/attachment.sig>