[cmake-developers] [CMake] How to set _default_ timeout for the ctest command? (fwd)

Tue Dec 15 01:09:39 EST 2015

Hi Alan,

Are you just looking for the TIMEOUT property on the tests?

https://cmake.org/cmake/help/v3.0/command/set_tests_properties.html

This is something that I managed to find with the most basic google-ing myself. :-/ And I use it like:

set_tests_properties( ${testName}Test PROPERTIES TIMEOUT 120 )

Or would you like to be able to set some global property/variable that would set the default timeout for all the created tests without you having to set these things individually? (I just wrote a wrapper function to set up all my tests in the way that I wanted to.)

Best,
           Attila

> On Dec 14, 2015, at 9:36 PM, Alan W. Irwin <irwin at beluga.phys.uvic.ca> wrote:
> 
> I would appreciate it if a CMake developer who is familiar with the CTest timeout
> logic would answer my question below that was addressed to "@CMake developers:".
> 
> Without such an answer (even if the conclusion is there is no way for
> an individual software package such as lapack to set a different
> _default_ timeout for CTest other than 1500 seconds), I will not be
> able to figure out what to advise the lapack developers.
> 
> Alan
> 
> __________________________
> Alan W. Irwin
> 
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
> 
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
> 
> Linux-powered Science
> __________________________
> 
> ---------- Forwarded message ----------
> Date: Wed, 9 Dec 2015 19:06:00 -0800 (PST)
> From: Alan W. Irwin <irwin at beluga.phys.uvic.ca>
> To: Marcel Loose <loose at astron.nl>
> Cc: cmake at cmake.org
> Subject: Re: [CMake] How to set _default_ timeout for the ctest command?
> 
> On 2015-12-09 09:23+0100 Marcel Loose wrote:
> 
>> Hi Alan,
>> Just by experimenting, I discovered that you also need to set
>> DART_TESTING_TIMEOUT. I'm not sure what the relationship (if any) is
>> between CTEST_TEST_TIMEOUT and DART_TESTING_TIMEOUT. I set them both in
>> the file CTestConfig.cmake. I get the feeling that DART_TESTING_TIMEOUT
>> is there for less than historic reasons. Maybe someone from Kitware can
>> shed a light on this.
> 
> Thanks, Marcel, for trying those experiments.
> 
> I tried some additional experiments and found that all that is needed is
> to set the CTEST_TEST_TIMEOUT value before CTest is included in my
> top-level CMakeLists.txt file.  (My previous experiment set that value
> after the include, but before the first add_test command.) Note that
> I do not use CTestConfig.cmake at all.
> 
> So the current experimental logic is
> # Control the default timeout for all ctests.
> set(CTEST_TEST_TIMEOUT 1)
> include(CTest)
> 
> Those commands generate the following timeout results in the build tree.
> 
> software at raven> find . -type f |xargs grep -i timeout
> ./CMakeCache.txt:DART_TESTING_TIMEOUT:STRING=1
> ./CMakeCache.txt://ADVANCED property for variable: DART_TESTING_TIMEOUT
> ./CMakeCache.txt:DART_TESTING_TIMEOUT-ADVANCED:INTERNAL=1
> ./DartConfiguration.tcl:# TimeOut is the amount of time in seconds to wait for processes
> ./DartConfiguration.tcl:# to complete during testing.  After TimeOut seconds, the
> ./DartConfiguration.tcl:TimeOut: 1
> ./DartConfiguration.tcl:# For CTest submissions that timeout, these options
> 
> So for this case, DART_TESTING_TIMEOUT is automatically set to the
> specified CTEST_TEST_TIMEOUT value.  (If CTEST_TEST_TIMEOUT is not set,
> the cached value of DART_TESTING_TIMEOUT is 1500, and the value of the
> timeout in DartConfiguration.tcl is also set to 1500).
> 
> Afterwards, ctest timed out at 1 second for all tests that exceeded
> that value, but it turns out that is a hard-coded value which cannot
> be superseded by the ctest --timeout option.  So this approach is not
> really ideal for the lapack developers who will want to increase the
> default limit by only an order of magnitude or less to substantially
> reduce the probability someone with a slow computer or special
> circumstances will have to override using the ctest --timeout option,
> but that change likely will not reduce that probability to zero so
> keeping the ability to override the lapack timeout default by the
> ctest --timeout option is still likely imporatant to them.
> 
> @CMake developers:
> 
> The documentation of CTEST_TEST_TIMEOUT (and DART_TESTING_TIMEOUT)
> includes the following language:
> 
> "The default timeout for each test if
> not specified by the `TIMEOUT` test property.
> 
> Also, the documentation of the ctest --timeout option says:
> 
> `--timeout <seconds>`
> Set a global timeout on all tests.
> 
> This option will set a global timeout on all tests that do not
> already have a timeout set on them.
> 
> So I guess it could be argued that CTEST_TEST_TIMEOUT does impose a
> specific timeout on all tests, and that is why the --timeout option is
> being ignored for this case.  But that is not particularly useful
> behaviour, and it would be far better in my opinion for the --timeout
> option to work for all tests where the TIMEOUT test property has
> not been specifically set by the project's build system.
> 
> I would appreciate some further investigation of these matters by a
> knowledgable CMake developer, and if it turns out this issue is
> actually caused by a cmake or ctest bug, then I would be willing to
> write up the appropriate bug report if it turns out an instantaneous
> fix could not be made.
> 
> I also tried the experiment of dropping the setting of
> CTEST_TEST_TIMEOUT and using set_tests_properties to set the TIMEOUT
> for individual commands, but consistent with the above wording of the
> documentation, the ctest --timeout option was ignored for this case. I
> am OK with that limitation so long (see above) as some change is made
> so CTEST_TEST_TIMEOUT is interpreted exactly the same as the 1500
> default timeout, i.e., the CTEST_TEST_TIMEOUT value can be overridden
> by the ctest --timeout option.
> 
> Alan
> 
> __________________________
> Alan W. Irwin
> 
> Astronomical research affiliation with Department of Physics and Astronomy,
> University of Victoria (astrowww.phys.uvic.ca).
> 
> Programming affiliations with the FreeEOS equation-of-state
> implementation for stellar interiors (freeeos.sf.net); the Time
> Ephemerides project (timeephem.sf.net); PLplot scientific plotting
> software package (plplot.sf.net); the libLASi project
> (unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
> and the Linux Brochure Project (lbproject.sf.net).
> __________________________
> 
> Linux-powered Science
> __________________________
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake-developers