[cmake-developers] [CMake] Setup/tear down steps for CTest

[Craig Scott]

On Tue, Aug 23, 2016 at 4:00 PM, Rolf Eike Beer <eike at sf-mail.de> wrote:

> Am Dienstag, 23. August 2016, 10:06:01 schrieb Craig Scott:
> > Cheeky way to get me more involved in contributing, but okay, I'll bite.
> ;)
> > Switching discussion to the dev list.
> >
> > So how would you want the feature to work? I'd suggest an initial set of
> > requirements something like the following:
> >
> >    - Need to support the ability to define multiple setup and/or tear
> down
> >    tasks.
> >    - It should be possible to specify dependencies between setup tasks
> and
> >    between tear down tasks.
> >    - Individual tests need to be able to indicate which setup and/or tear
> >    down tasks they require, similar to the way DEPENDS is used to specify
> >    dependencies between test cases.
> >    - When using ctest --rerun-failed, ctest should automatically invoke
> any
> >    setup or tear down tasks required by the test cases that will be
> re-run.
> >    - Setup or tear down tasks which reference executable targets should
> >    substitute the actual built executable just like how
> add_custom_command()
> > does.
>
> -need a way to mark if 2 tests with the same setup/teardown can share
> those or
> if they need to run for every of them
> -the default for each test is "no s/t", which means it can't be run with
> any
> of the above in parallel (especially for compatibillity)[1]
> -need a way to tell if a test doesn't care about those
>
> 1) think of a database connector test: the test that will check what
> happens
> if no DB is present will fail if the setup step "start DB" was run, but not
> the teardown
>

So maybe that requires being able to specify that tests for resource XXX
and resource YYY cannot be executed concurrently. Maybe that's a separate
change that could be made independent of this proposed improvement, since
it would apply even for existing CMake functionality. I see the value, I'm
just trying to sort out what is really needed from what is nice-to-have but
could be done as a subsequent improvement later.



>
> > Some open questions:
> >
> >    - Should setup and tear down tasks be defined in pairs, or should they
> >    completely independent (this would still require the ability to
> specify a
> > dependency of a tear down task on a setup task)?
>
> The test could be "shutdown daemon" or "delete files" so I would keep them
> separated. They may be created by the same command, so they could be
> batched
> anyway.
>

Agreed, it seems clear now that keeping them separate is preferable.




>
> >    - Should the setup and tear down tasks be defined by a new CTest/CMake
> >    command or extend an existing mechanism (e.g. add_custom_command())?
>
> Don't clutter existing commands more than needed. If it's something new,
> then
> create a new command (they could still share C++ code). If it's basically
> the
> same as anything existing at the end then use that.
>

See my other email reply just now. I think re-using the existing commands
and concepts and adding the RESOURCE_SETUP and RESOURCE_CLEANUP test
properties might be the most seamless from an end user perspective. I might
change my mind once I dig into the CMake source code though. ;)




>
> >    - If no test case has a dependency on a setup or tear down task,
> should
> >    that task be skipped? Perhaps tasks need to have a flag which
> indicates
> >    whether they always run or only if a test case depends on it.
>
> Keep backward compatibility. I.e. if I now add a new test with s/t, then
> every
> other test should still run (and succeed) as before.
>

Definitely. Existing projects should receive zero impact from any changes
made. New functionality should be opt-in.



>
> >    - What terminology to use? Things like GoogleTest use terms like test
> >    *fixtures* for this sort of thing. The terms setup and tear down are a
> >    bit imprecise and cumbersome, so we would probably need something
> better
> >    than those.
> >    - Would it make sense for the ctest command line to support disabling
> >    setup and/or tear down steps? I can see some potential scenarios where
> > this may be desirable, but maybe this is getting too ambitious for a
> > starting set of requirements.
>
> IMHO that doesn't make sense. One could think about an option to disable
> the
> s/t merging, i.e. that they are really called alone for every test.
>

To reduce complexity, I'm gravitating that way too. If you define a
setup/cleanup task, then why allow disabling it? If developers really want
that, they could wrap defining the setup/cleanup definitions inside an if()
block controlled by a CMake option or something similar.



>
> >    - What should happen if a setup or tear down task fails? How would
> >    failure be detected? How would such failures impact things like a
> CDash
> >    test report, etc.?
>
> Then the test fails, just the same as it now does when it can't find the
> executable.
>

Seems sensible.


-- 
Craig Scott
Melbourne, Australia
http://crascit.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20160901/b2b3bc62/attachment-0001.html>