[CMake] test property COST not working in cmake 2.8.3?

Zach Mullen zach.mullen at kitware.com
Tue Nov 30 13:29:37 EST 2010


Hm, yours was a use case we didn't really consider when we were making
changes to cost behavior.  The middle ground here would be to respect costs
in the non-parallel case when they are expressed explicitly, but not to
cost-order them automatically based on their previous run times.

-Zach

On Tue, Nov 30, 2010 at 12:43 PM, Tyler Roscoe <tyler at cryptio.net> wrote:

> On Fri, Nov 26, 2010 at 10:38:44AM -0500, Zach Mullen wrote:
> > I just realized why this isn't working -- it's actually not a regression.
>
> Maybe we have different definitions of "regression". I see a feature
> that used to do one thing but which now does something else.
>
> Here is what the docs say about the COST property:
>
>    # COST: Set this to a floating point value. Tests in a test set will be
>    # run in descending order of cost.
>
>    This property describes the cost of a test. You can explicitly set this
>    value; tests with higher COST values will run first.
>
> I don't see anything there about parallel or non-parallel runs. It seems
> to me that if I set the COST property, I should be able to control the
> order in which tests run, period. So at the very least, the docs should
> be updated if you intend to change the behavior.
>
> >  In this release we decided that the costs should only be taken into
> account
> > in a parallel case (ctest -j N).  Many users have implicit dependencies
> > based on the order of their add_test calls, so we didn't want to break
> > backward compatibility for those not using parallel ctest.
>
> It looks like ctest -j2 is respecting COST. Currently I have several
> tests that cannot run at the same time as others (they touch the same
> resources and/or running two of them at once would crush the machine).
> If I could get the old COST behavior by running ctest -j1, that might be
> an acceptable workaround, but it does not appear to work today.
>
> > The non-parallel way to specify a test to run last is simply to make it
> the
> > last add_test call.
>
> My CMake projects are modular (I imagine this is true for many CMake
> users). Each module is responsible for adding its own unit tests and
> code quality checks. As I said in my initial email, the code quality
> checks must run after the unit tests so that accurate code coverage
> values can be calculated. I can try to insure that my add_unittest()
> functions all run before my add_code_quality() functions, but that seems
> brittle and error-prone. It was much nicer when I could just tell
> add_code_quality() to add all its tests with COST -1000 to guarantee
> they run after everything else.
>
> I can imagine ways to work around this problem, but they all seem rather
> clunky, especially when COST used to solve the problem so simply and
> elegantly.
>
> I hope we can reach a useful middle ground about the future of the COST
> property. In its current state, it is of no use to me.
>
> Thanks,
> tyler
>
> > On Fri, Nov 26, 2010 at 10:20 AM, Zach Mullen <zach.mullen at kitware.com
> >wrote:
> > > On Tue, Nov 23, 2010 at 6:02 PM, David Cole <david.cole at kitware.com
> >wrote:
> > >
> > >> It might be due to this commit:
> > >>
> > >>
> http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=142edf8ad4baccd991a6a8a3e5283d0b575acca2
> > >> (first released in 2.8.3)
> > >>
> > >> Or this one:
> > >>
> > >>
> http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=b4d27dc041c9164d6f3ad39e192f4b7d116ca3b3
> > >> (first released in 2.8.2)
> > >>
> > >> Either way, seems like a bug to me. If you explicitly specify a COST
> > >> property value, especially a negative one to induce "last run" status,
> > >> then it should be honored over either historical average measurement
> > >> or "failed last time, so run it first this time" behavior.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20101130/26124298/attachment.htm>


More information about the CMake mailing list