[CMake] random BAD_COMMAND error on ctest

[Eric Noulard]

2013/1/8 BILODEAU Andre <andre.bilodeau at safran-engineering.com>:
> I just began to make ctest replace autotools for the program I develop, but
> I still have some problems while running ctest.
>
> Computer:
> Intel(R) Core(TM)2 Duo CPU
> E7500 @ 2.93GHz
> RAM 3,37 GO
>
> System:
> Microsoft Windows XP
> Version 2002
> Service Pack 3
>
> Environment:
> Cygwin 1.7.17(0.262/5/3)
>
> cmake version:
> 2.8.9
>
> Context:
> Application made using cmake. Testing by ctest.
> Tests are run either by the command:
> make test
> or
> ctest
> Example:
> ------------------------------------------------------------------------
> $ ctest
> Test project /cygdrive/d/F053022/Programs/middle-1.4/build
>       Start  1: t_proto_child
>  1/30 Test  #1: t_proto_child ....................   Passed    0.09 sec
>       Start  2: t_CsvReader
> ...
>
> 18/30 Test #18: test_05 ..........................   Passed    0.28 sec
>       Start 19: test_06
> 19/30 Test #19: test_06 ..........................   Passed    0.23 sec
>       Start 20: test_07
> 20/30 Test #20: test_07 ..........................***Not Run   0.25 sec
>       Start 21: test_08
> 21/30 Test #21: test_08 ..........................***Not Run   0.27 sec
>       Start 22: test_09
> 22/30 Test #22: test_09 ..........................   Passed    0.33 sec
>       Start 23: test_10
> ...
>
> 25/30 Test #25: test_12 ..........................   Passed    0.33 sec
>       Start 26: test_13
> 26/30 Test #26: test_13 ..........................   Passed    0.31 sec
>       Start 27: test_14
> 27/30 Test #27: test_14 ..........................***Not Run   0.37 sec
>       Start 28: test_15
> 28/30 Test #28: test_15 ..........................   Passed    0.44 sec
>       Start 29: test_16
> 29/30 Test #29: test_16 ..........................   Passed    0.53 sec
>       Start 30: test_17
> 30/30 Test #30: test_17 ..........................***Not Run   0.39 sec
> 87% tests passed, 4 tests failed out of 30
> Total Test time (real) =   7.86 sec
> The following tests FAILED:
>          20 - test_07 (BAD_COMMAND)
>          21 - test_08 (BAD_COMMAND)
>          27 - test_14 (BAD_COMMAND)
>          30 - test_17 (BAD_COMMAND)
> Errors while running CTest
> ------------------------------------------------------------------------
>
> The tests that fail make it randomly. That is, for instance, if I run,
> for the test test_07 that failed:
>
> ctest -R test_07
>
> the test may succeed. Or if I run make test again, the set of tests that
> fail may be different.
>
> All test_nn test are built on the same model. for instance:
>
> add_test (test_07 ${PROJECT_SOURCE_DIR}/test/test_driver
>     -N 07 -S 0 -P test_parser)
>
> As the tests were run originally from automake, an environment variable
> is set, at end of CMakeLists.txt file by the command:
>
> set_tests_properties (
>     test_01 test_02 test_03 test_04 test_05
>     test_06 test_07 test_08 test_09 test_10
>     test_11 test_12 test_13 test_14 test_15
>     test_16 test_17
>     PROPERTIES ENVIRONMENT srcdir=${PROJECT_SOURCE_DIR}/test)
>
> A random failure of the tested program is discarded, because the tested
> program is activated through a shell, "test_driver".
>
> This shell outputs a trace upon entry and exit.
> The final trace on exit is produced by the following script lines (korn
> shell). If
> the trace produced contains "PASS", I don't see how the exit status could
> differ from 0:

Unless the run of the test timed-out just before exiting ?

Did you try setting
PASS_REGULAR_EXPRESSION
or
FAIL_REGULAR_EXPRESSION

(with set_tests_properties) in order to use output of the test to evaluate
FAIL/PASS status instead of returned exit code?



>
> if [[ ${errors} -gt 0 ]] ; then
>     MSG=FAIL
> else
>     MSG=PASS
> fi
> echo "${MSG}" >> trc_${NUM}.txt
> exit ${errors}
>
> All tests mentioned as "Not Run" are actually run, since a corresponding
> trace output file is found for each test.
>
> In all cases, the trace output is normal, indicating correct entry and
> a normal status on exit, just before the exit command, whith the 0
> parameter that would normally correspond to success.


Did you try to run

ctest -VV

i.e. in extra verbose mode and try to track down the difference between failing
and non-failing case?

Does this happen in alternate environment, i.e. under Linux?

>
> Thus, the problem seems to come from ctest itself.
> ------------------------------------------------------------------------
> On the internet, typing "cmake random BAD_COMMAND error" gets some stories,
> but with no interesting answer.
>
> Is this king of behavior already known ?
> Does that correspond to a bug ?
> Is there a known workaround ?
>
> andre
>
>
>
>
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake



-- 
Erk
Le gouvernement représentatif n'est pas la démocratie --
http://www.le-message.org