[cmake-developers] [CMake] How to set _default_ timeout for the ctest command? (fwd)

[Alan W. Irwin]

On 2015-12-15 13:53-0500 Ben Boeckel wrote:

> On Tue, Dec 15, 2015 at 10:33:38 -0800, Alan W. Irwin wrote:
>> On 2015-12-15 11:20-0500 Ben Boeckel wrote:
>>> I think, instead, that --min-timeout and --max-timeout options might be
>>> better which allow you to say "this machine is slow; tests may take
>>> longer (max(property, option))" or "this machine is fast, clamp down the
>>> timeout (min(property, option))". Alternatively, a --timeout-scale
>>> option to multiply all timeouts in properties might be better.
>>
>> I think we are stuck with --timeout since it has been available as a
>> ctest option for a long time.  So the first priority is that option
>> should override anything set by the project, i.e., anything set by
>> the project should be considered to be a default value.
>
> Changing --timeout's behavior isn't going to help you since old CMake
> versions won't work as intended. I don't think changing its behavior is
> best. For example, in our Buildbot infrastructure, we set the default
> timeout to 3 minutes because the default timeout for CTest causes tests
> to take *way* too long (I think it seems to be 25 minutes?). Some tests,
> however, *do* take longer than that on even reasonable hardware, so
> those should ignore the lower default timeout we set. Breaking this gets
> a *big* -1 from me.

There are a number of issues you have lumped together here.

1. Old versus new.  We should plan the ideal timeout experience for
the user here, and if we get that right once the modified version of
--timeout and the proposed new --timeout-scale lands, then those
having trouble with timeouts for old CMake/CTest versions can be
referred to the new CMake/CTest version.

2. Global versus individual test timeouts set by the project.  I
believe we are all agreed here; individual project timeouts supersedes global
project timeouts.

The above considerations are orthogonal to my principal concern which
is the user should have command-line freedom to supersede any timeout
set by a project since it is usually impossible for the project to
anticipate the user's timeout needs 100 per cent of the time.  I think
the --timeout-scale option you proposed which would multiply both a
project's global and individual test timeouts would be a very nice way
to implement this desired user freedom.

So our disagreement really boils down to what to do with --timeout. 
It is obvious that is an extremely brute force option, but sometimes
that is useful so I think it should be changed to take precedence
(just like the proposed --timeout-scale) over both global and
individual test timeouts set by the project.  However, if you feel
strongly for some reason that this option should have precedence over
the global default set by the project but not over individual test
timeouts set by the project, then I would be willing to go along with
that so long as --timeout-scale is there to provide the desired user
freedom and that departure in behaviour for --timeout for what happens
with --timeout-scale was clearly documented.

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________