[cmake-developers] [CMake] Setup/tear down steps for CTest

[Brad King]

On 09/08/2016 10:15 AM, Craig Scott wrote:
> the current behaviour of DEPENDS. At the moment, if test B depends
> on test A, test B still executes if test A fails.
> It is unexpected because I'd initially have thought of DEPENDS as
> meaning I can't run test B if test A fails, after all, B depends
> on A which I'd interpret to mean if A fails, then something B requires
> isn't working. Conversely, this is also useful because until now,
> DEPENDS was the only way to get cleanup functionality to run after
> other tests, and if those other tests fail, we still want the
> cleanup to occur.

At one time we only had serial testing so the order of tests was
fully controllable and based on the order of addition.  There were
never any conditions for whether a test would run based on results
of other tests.  Then when parallel testing was added we need a
way to restore *order* dependencies, so DEPENDS was added just
for that.  Maybe a better name would have been RUN_AFTER.

> adding a DEPENDS_ON_SUCCESS test property or something similar
> which would implement the perhaps more intuitive behaviour of not
> running dependent tests when a dependee fails. If that was done,
> then implementing the "don't run fixture tests if any fixture
> setup fails" logic would be trivial.

The semantics of this will have to be carefully though out, in
particular with respect to enabling test dependencies.  Right now
ctest arguments like -E can exclude tests.  What if those are
dependencies of included tests?

I think if we introduce the notion of tests requiring other tests
then a new model of test selection and enablement needs to be
designed.  Some kind of test DAG could be defined with various
roots and subgraphs being selectable an causing all reachable
tests to be included.

-Brad