[cmake-developers] [CMake] How to set _default_ timeout for the ctest command? (fwd)

[Ben Boeckel]

On Tue, Dec 15, 2015 at 10:33:38 -0800, Alan W. Irwin wrote:
> On 2015-12-15 11:20-0500 Ben Boeckel wrote:
> > I think, instead, that --min-timeout and --max-timeout options might be
> > better which allow you to say "this machine is slow; tests may take
> > longer (max(property, option))" or "this machine is fast, clamp down the
> > timeout (min(property, option))". Alternatively, a --timeout-scale
> > option to multiply all timeouts in properties might be better.
> 
> I think we are stuck with --timeout since it has been available as a
> ctest option for a long time.  So the first priority is that option
> should override anything set by the project, i.e., anything set by
> the project should be considered to be a default value.

Changing --timeout's behavior isn't going to help you since old CMake
versions won't work as intended. I don't think changing its behavior is
best. For example, in our Buildbot infrastructure, we set the default
timeout to 3 minutes because the default timeout for CTest causes tests
to take *way* too long (I think it seems to be 25 minutes?). Some tests,
however, *do* take longer than that on even reasonable hardware, so
those should ignore the lower default timeout we set. Breaking this gets
a *big* -1 from me.

If you want compatibility between ctest versions, write a script to add
arguments to a list based on the version that is running the script (any
new command line arguments should get corresponding ctest_test()
keywords).

> That said, most cases including the ones I discussed above are scaling
> ones so I agree it would be worthwhile to implement an additional
> overriding --timeout-scale option for ctest which multiplies all
> default timeouts set by the project.  If the user specifies both
> --timeout and --timeout-scale then one of them should be ignored in a
> well-documented way (probably --timeout since --timeout-scale is a bit
> more sophisticated).

I think it would be "apply the scaling, but if there is no value, use
the --timeout value". Whether that is pre- or post-scaling is arguable
(one makes more sense reading, the other makes it easier to script).

--Ben