[cmake-developers] [CMake] Setup/tear down steps for CTest

[Craig Scott]

On Fri, Sep 9, 2016 at 5:14 AM, Daniel Pfeifer <daniel at pfeifer-mail.de>
wrote:

> On Thu, Sep 8, 2016 at 5:52 PM, Brad King <brad.king at kitware.com> wrote:
> > I think if we introduce the notion of tests requiring other tests
> > then a new model of test selection and enablement needs to be
> > designed.  Some kind of test DAG could be defined with various
> > roots and subgraphs being selectable an causing all reachable
> > tests to be included.
>
> This could be expanded even further. If "tests requiring other tests"
> is generalized to "tests requiring X", wouldn't this allow incremental
> testing?  Say you change one file in your project. You rebuild only
> the parts of the project that are affected by this change. Then you
> rerun only the tests that are affected by the change. This really has
> to be carefully thought out.
>

Interesting idea, but yes potentially a big body of work. That said, the
fixtures implementation up for review already gets you most of the way
there I think. One could potentially define a fixture that covers the tests
for functionality a developer is working on. A test is able to require
multiple fixtures, so developers could define fixtures to whatever
granularity they wanted. We could add option(s) to the ctest command line
which allow selection of tests based on fixture names in the same way we
currently do with labels and test names. This would allow developers to run
just the set of tests related to a particular functionality. One could
argue that labels already get you most of the way there now too, but since
fixtures would come with clean handing of setup/cleanup dependencies, they
may be more useful to developers. They also feel a bit closer to the
implementation than what labels are likely to cover and so fixtures might
be a better fit for the scenario you mentioned. I'd also be skeptical
whether we could make it easy/convenient/robust for CMake to work out which
tests to re-run based on which files had to be rebuilt. Asking developers
to pick which fixture(s) to retest instead might be a good compromise due
to the simplicity of both use and implementation.

Even aside from whether the above would satisfy the proposed scenario,
maybe it's worth considering adding test selection based on fixture names
regardless. I think this probably should be a separate branch that follows
from the current one though so it doesn't turn into a monster of a merge.
;) I'd be happy to look at that idea once the current branch is done and
in, since I have a concrete use case driving me to get that one completed
first.

-- 
Craig Scott
Melbourne, Australia
http://crascit.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20160910/4121905f/attachment.html>