[cmake-developers] [CMake] Setup/tear down steps for CTest

[Daniel Pfeifer]

On Thu, Sep 8, 2016 at 5:52 PM, Brad King <brad.king at kitware.com> wrote:
> On 09/08/2016 10:15 AM, Craig Scott wrote:
>> the current behaviour of DEPENDS. At the moment, if test B depends
>> on test A, test B still executes if test A fails.
>> It is unexpected because I'd initially have thought of DEPENDS as
>> meaning I can't run test B if test A fails, after all, B depends
>> on A which I'd interpret to mean if A fails, then something B requires
>> isn't working. Conversely, this is also useful because until now,
>> DEPENDS was the only way to get cleanup functionality to run after
>> other tests, and if those other tests fail, we still want the
>> cleanup to occur.
>
> At one time we only had serial testing so the order of tests was
> fully controllable and based on the order of addition.  There were
> never any conditions for whether a test would run based on results
> of other tests.  Then when parallel testing was added we need a
> way to restore *order* dependencies, so DEPENDS was added just
> for that.  Maybe a better name would have been RUN_AFTER.
>
>> adding a DEPENDS_ON_SUCCESS test property or something similar
>> which would implement the perhaps more intuitive behaviour of not
>> running dependent tests when a dependee fails. If that was done,
>> then implementing the "don't run fixture tests if any fixture
>> setup fails" logic would be trivial.
>
> The semantics of this will have to be carefully though out, in
> particular with respect to enabling test dependencies.  Right now
> ctest arguments like -E can exclude tests.  What if those are
> dependencies of included tests?
>
> I think if we introduce the notion of tests requiring other tests
> then a new model of test selection and enablement needs to be
> designed.  Some kind of test DAG could be defined with various
> roots and subgraphs being selectable an causing all reachable
> tests to be included.

This could be expanded even further. If "tests requiring other tests"
is generalized to "tests requiring X", wouldn't this allow incremental
testing?  Say you change one file in your project. You rebuild only
the parts of the project that are affected by this change. Then you
rerun only the tests that are affected by the change. This really has
to be carefully thought out.

Cheers, Daniel