[CMake] FW: CTest question
T.Sariyski
tzsariysk at gmail.com
Thu May 18 13:32:39 EDT 2017
Hi Eric,
I noticed something that may relates to the problem. Like I said in my previous mail, add_test is a call to cmake:
ADD_TEST(NAME ${TARGET_NAME}
…
COMMAND ${CMAKE_COMMAND}
-DPROGRAM1
-DPROGRAM2
…
Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically? Here is what I noticed. If a test is executed from its own binary directory, ‘ctest -VV’, it always pass. If the same test is executed from the main binary directory, ‘ctest -VV -R zamod’, it fails because PROGRAM2 is executed before PROGRAM1 is finished. Attached are the screen outputs for both runs.
Thanks,
Ted
================================================================
cd CTEST/zamod
s1002807:zamod tesari$ ctest -VV
UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl
UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl
Test project /Users/tesari/build/gnu.dbg/CTEST/zamod
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: zamod_serial
1: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"
1: Test timeout computed to be: 9.99988e+06
1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod"
1: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
1: /Users/tesari/kodiak/trunk/benchmark/Scripts
1: Number of arguments: 4 arguments.
1: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
1: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3 arguments:
1: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
1: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
1: 3 0
1: zamod Q0.4 PASS
1: zamod Q0.5 PASS
1: Kodiak ctest: SUCCESS
1/1 Test #1: zamod_serial ..................... Passed 5.73 sec
================================================================
s1002807:gnu.dbg tesari$ ctest -VV -R zamod
UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
Test project /Users/tesari/build/gnu.dbg
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 113
Start 113: zamod_serial
113: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"
113: Test timeout computed to be: 1500
113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod && /Users/tesari/build/gnu.dbg/bin/kodiak zamod"
113: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
113: /Users/tesari/kodiak/trunk/benchmark/Scripts
113: Number of arguments: 4 arguments.
113: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
113: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with 3 arguments:
113: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
113: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
113: 3 0
113: /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk does not exist!
From: Eric Noulard [mailto:eric.noulard at gmail.com]
Sent: Wednesday, May 17, 2017 10:54 PM
To: T.Sariyski <tzsariysk at gmail.com <mailto:tzsariysk at gmail.com> >
Cc: CMake ML <cmake at cmake.org <mailto:cmake at cmake.org> >
Subject: Re: [CMake] CTest question
We lack information on how your test are failing. If they all timeout then may be you should setup a proper timeout for the test. The actual execution speed of a test may depend on the load of your machine or the network (depending on what the test are actually doing).
This may be even more sensible if your test are run in parallel. Do you run ctest with --parallel option?
Do your tests depends on each other? If so did you setup test dependencies?
Le 17 mai 2017 10:59 PM, "T.Sariyski" < <mailto:tzsariysk at gmail.com> tzsariysk at gmail.com> a écrit :
Hi,
I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the number of these false failures is about one third. I guess that the problem is in the way I set my tests, but I have no clue where to look, or how to ‘debug’ the ctest run.
Any advice is highly appreciated.
Thanks,
Ted
--
Powered by www.kitware.com <http://www.kitware.com>
Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
Kitware offers various services to support the CMake community. For more information on each offering, please visit:
CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html
Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170518/9d217fbf/attachment-0001.html>
More information about the CMake
mailing list