[cmake-developers] [CMake] How to set _default_ timeout for the ctest command? (fwd)

[Alan W. Irwin]

I would appreciate it if a CMake developer who is familiar with the CTest timeout
logic would answer my question below that was addressed to 
"@CMake developers:".

Without such an answer (even if the conclusion is there is no way for
an individual software package such as lapack to set a different
_default_ timeout for CTest other than 1500 seconds), I will not be
able to figure out what to advise the lapack developers.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________

---------- Forwarded message ----------
Date: Wed, 9 Dec 2015 19:06:00 -0800 (PST)
From: Alan W. Irwin <irwin at beluga.phys.uvic.ca>
To: Marcel Loose <loose at astron.nl>
Cc: cmake at cmake.org
Subject: Re: [CMake] How to set _default_ timeout for the ctest command?

On 2015-12-09 09:23+0100 Marcel Loose wrote:

> Hi Alan,
> 
> Just by experimenting, I discovered that you also need to set
> DART_TESTING_TIMEOUT. I'm not sure what the relationship (if any) is
> between CTEST_TEST_TIMEOUT and DART_TESTING_TIMEOUT. I set them both in
> the file CTestConfig.cmake. I get the feeling that DART_TESTING_TIMEOUT
> is there for less than historic reasons. Maybe someone from Kitware can
> shed a light on this.

Thanks, Marcel, for trying those experiments.

I tried some additional experiments and found that all that is needed is
to set the CTEST_TEST_TIMEOUT value before CTest is included in my
top-level CMakeLists.txt file.  (My previous experiment set that value
after the include, but before the first add_test command.) Note that
I do not use CTestConfig.cmake at all.

So the current experimental logic is
# Control the default timeout for all ctests.
set(CTEST_TEST_TIMEOUT 1)
include(CTest)

Those commands generate the following timeout results in the build tree.

software at raven> find . -type f |xargs grep -i timeout
./CMakeCache.txt:DART_TESTING_TIMEOUT:STRING=1
./CMakeCache.txt://ADVANCED property for variable: DART_TESTING_TIMEOUT
./CMakeCache.txt:DART_TESTING_TIMEOUT-ADVANCED:INTERNAL=1
./DartConfiguration.tcl:# TimeOut is the amount of time in seconds to wait for 
processes
./DartConfiguration.tcl:# to complete during testing.  After TimeOut seconds, 
the
./DartConfiguration.tcl:TimeOut: 1
./DartConfiguration.tcl:# For CTest submissions that timeout, these options

So for this case, DART_TESTING_TIMEOUT is automatically set to the
specified CTEST_TEST_TIMEOUT value.  (If CTEST_TEST_TIMEOUT is not set,
the cached value of DART_TESTING_TIMEOUT is 1500, and the value of the
timeout in DartConfiguration.tcl is also set to 1500).

Afterwards, ctest timed out at 1 second for all tests that exceeded
that value, but it turns out that is a hard-coded value which cannot
be superseded by the ctest --timeout option.  So this approach is not
really ideal for the lapack developers who will want to increase the
default limit by only an order of magnitude or less to substantially
reduce the probability someone with a slow computer or special
circumstances will have to override using the ctest --timeout option,
but that change likely will not reduce that probability to zero so
keeping the ability to override the lapack timeout default by the
ctest --timeout option is still likely imporatant to them.

@CMake developers:

The documentation of CTEST_TEST_TIMEOUT (and DART_TESTING_TIMEOUT)
includes the following language:

"The default timeout for each test if
not specified by the `TIMEOUT` test property.

Also, the documentation of the ctest --timeout option says:

`--timeout <seconds>`
  Set a global timeout on all tests.

This option will set a global timeout on all tests that do not
already have a timeout set on them.

So I guess it could be argued that CTEST_TEST_TIMEOUT does impose a
specific timeout on all tests, and that is why the --timeout option is
being ignored for this case.  But that is not particularly useful
behaviour, and it would be far better in my opinion for the --timeout
option to work for all tests where the TIMEOUT test property has
not been specifically set by the project's build system.

I would appreciate some further investigation of these matters by a
knowledgable CMake developer, and if it turns out this issue is
actually caused by a cmake or ctest bug, then I would be willing to
write up the appropriate bug report if it turns out an instantaneous
fix could not be made.

I also tried the experiment of dropping the setting of
CTEST_TEST_TIMEOUT and using set_tests_properties to set the TIMEOUT
for individual commands, but consistent with the above wording of the
documentation, the ctest --timeout option was ignored for this case. I
am OK with that limitation so long (see above) as some change is made
so CTEST_TEST_TIMEOUT is interpreted exactly the same as the 1500
default timeout, i.e., the CTEST_TEST_TIMEOUT value can be overridden
by the ctest --timeout option.

Alan

__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake