[cmake-developers] [CMake] How to set _default_ timeout for the ctest command? (fwd)

[Alan W. Irwin]

On 2015-12-15 11:20-0500 Ben Boeckel wrote:

> On Mon, Dec 14, 2015 at 23:01:31 -0800, Alan W. Irwin wrote:
>> On the cmake general list, Brad recently answered my original query on this
>> subject and appears to agree with me that that ctest --timeout
>> option should always have the highest priority, i.e., override any
>> timeout set by the project such as the above TIMEOUT property.  If the
>> consensus continues to be that is the desired behaviour it appears
>> some CMake/CTest code changes will be necessary to change to that
>> behaviour.
>
> Hmm. I don't know. Some tests have timeouts set because they either *do*
> take that long or *shouldn't* take as long as the timeout (e.g., those
> that might deadlock).

That's fine.  Projects should be able to set _default_ timeouts on
individual tests or their entire set of tests in the most intelligent
way they can think of.  But ultimately the user should have the
ability to override whatever value was set in case they have an
extraordinarily slow or fast machine or there is a configuration
possible on the project (the case I ran into with lapack where I
promoted double precision to quadruple precision on an Intel box which
must fall back to software emulation of quadruple precision) where all
tests average much slower than normal.  (The average slowdown factor
is roughly 60 for the quadruple precision lapack tests.)

> I think, instead, that --min-timeout and --max-timeout options might be
> better which allow you to say "this machine is slow; tests may take
> longer (max(property, option))" or "this machine is fast, clamp down the
> timeout (min(property, option))". Alternatively, a --timeout-scale
> option to multiply all timeouts in properties might be better.

I think we are stuck with --timeout since it has been available as a
ctest option for a long time.  So the first priority is that option
should override anything set by the project, i.e., anything set by
the project should be considered to be a default value.

That said, most cases including the ones I discussed above are scaling
ones so I agree it would be worthwhile to implement an additional
overriding --timeout-scale option for ctest which multiplies all
default timeouts set by the project.  If the user specifies both
--timeout and --timeout-scale then one of them should be ignored in a
well-documented way (probably --timeout since --timeout-scale is a bit
more sophisticated).

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________