[cmake-developers] ctest-fix-run-serial topic regressions

[David Cole]

Serial tests must run in the order of the add_test calls. The CDash test suite depends on that for its dashboards.

CDash should be explicit about its dependencies, but it is not, and at present, it requires serial running of its tests in the order listed in the CMakeLists file.

Please make sure that "ctest" with no -j arguments runs the CDash test suite in the same order as ctest 2.8.12

Thanks,
David


> On Dec 20, 2013, at 1:28 PM, Brad King <brad.king at kitware.com> wrote:
> 
> Nils,
> 
> With regard to the topic discussed here:
> 
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/8290
> 
> Since this topic was merged to master by commit 38fc334f
> (Merge topic 'ctest-fix-run-serial', 2013-10-26) I've noticed
> two regressions in CTest behavior.
> 
> The first one I noticed is that tests that have dependencies are
> now run before all other tests even without "-j".  Previously
> serial testing would run tests in the order declared.  I would
> prefer that this continue to be the case except for enforcing
> dependencies.  The dependency-based ordering should be a stable
> sort of the original order.  This is minor so I didn't think it
> was worth bringing up on its own.  However, I think it is worth
> addressing after the following issue is addressed.
> 
> The second one is a new bug that is responsible for some of the
> missing dashboard submissions recently.  Somehow CTest stops
> running tests and loops forever if there are problems running
> some tests.  It can be reproduced on Linux in a CMake build
> tree as follows:
> 
> $ cd CMake-build
> $ make -j 8
> $ rm -f bin/cmake
> $ bin/ctest -M Experimental -T Start -T Test -j 8 \
>             -E '^(CTestTestTimeout|BootstrapTest)$'
> # (The -E option is not actually needed but saves time
> # while reproducing the problem.)
> 
> Most tests immediately fail due to the missing cmake binary.
> However, when CTest finishes running tests it loops forever.
> 
> According to "git bisect" this problem was introduced by the
> ctest-fix-run-serial topic.  I discovered it because one of the
> dashboard submissions I run uses ctest from 'next' each night
> to drive the dashboard script and stopped submitting.  The
> host machine had accumulated one busy-loop process per day.
> After killing them I ran the dashboard script by hand and found
> that there was a compile error that caused bin/cmake to be
> missing.  Since CTest got stuck during ctest_test() it never
> submitted the build failure to the dashboard either.
> 
> Anyway, please take a look at the above issues.
> 
> Thanks,
> -Brad
> --
> 
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/cgi-bin/mailman/listinfo/cmake-developers