[CMake] FW: CTest question

Eric Noulard eric.noulard at gmail.com
Thu May 18 13:38:58 EDT 2017 

Hi,

I don't that 2 execute_process in sequence in a cmake script could be
executed out-of-order but I let CMake developer answer more firmly about
that.
The order of each test created by add_test may vary though.

2017-05-18 19:32 GMT+02:00 T.Sariyski <tzsariysk at gmail.com>:

> Hi Eric,
>
>
>
> I noticed something that may relates to the problem. Like I said in my
> previous mail, add_test is a call to cmake:
>
>
>
>     ADD_TEST(NAME ${TARGET_NAME}
>
>        …
>
>        COMMAND ${CMAKE_COMMAND}
>
> -DPROGRAM1
>
> -DPROGRAM2
>
> …
>
>
>
> Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically?
> Here is what I noticed. If a test is executed from its own binary
> directory, ‘ctest -VV’, it always pass. If the same test is executed from
> the main binary directory, ‘ctest -VV -R zamod’,  it fails because PROGRAM2
> is executed before PROGRAM1 is finished. Attached are the screen outputs
> for both runs.
>
>
>
> Thanks,
>
> Ted
>
>
>
>
>
> ================================================================
>
> cd CTEST/zamod
>
> s1002807:zamod tesari$ ctest -VV
>
> UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/
> CTEST/zamod/DartConfiguration.tcl
>
> UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/
> CTEST/zamod/DartConfiguration.tcl
>
> Test project /Users/tesari/build/gnu.dbg/CTEST/zamod
>
> Constructing a list of tests
>
> Done constructing a list of tests
>
> Updating test list for fixtures
>
> Added 0 tests to meet fixture requirements
>
> Checking test dependency graph...
>
> Checking test dependency graph end
>
> test 1
>
>     Start 1: zamod_serial
>
>
>
> 1: Test command: /Applications/CMake.app/Contents/bin/cmake "-D"
> "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~"
> "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod
> &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D"
> "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/
> benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.
> dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
> "-P" "run_and_assert.cmake"
>
> 1: Test timeout computed to be: 9.99988e+06
>
> 1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod
> &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod"
>
> 1: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/
> assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/
> hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
>
> 1: /Users/tesari/kodiak/trunk/benchmark/Scripts
>
> 1: Number of arguments: 4 arguments.
>
> 1: Argument List: ['/Users/tesari/kodiak/trunk/
> benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk',
> '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
>
> 1: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called
> with  3  arguments:
>
> 1: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
>
> 1: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
>
> 1: 3 0
>
> 1:            zamod             Q0.4 PASS
>
> 1:            zamod             Q0.5 PASS
>
> 1: Kodiak ctest: SUCCESS
>
> 1/1 Test #1: zamod_serial .....................   Passed    5.73 sec
>
>
>
>
>
> ================================================================
>
> s1002807:gnu.dbg tesari$ ctest -VV -R zamod
>
> UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/
> DartConfiguration.tcl
>
> Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
>
> UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/
> DartConfiguration.tcl
>
> Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl
>
> Test project /Users/tesari/build/gnu.dbg
>
> Constructing a list of tests
>
> Done constructing a list of tests
>
> Updating test list for fixtures
>
> Added 0 tests to meet fixture requirements
>
> Checking test dependency graph...
>
> Checking test dependency graph end
>
> test 113
>
>     Start 113: zamod_serial
>
>
>
> 113: Test command: /Applications/CMake.app/Contents/bin/cmake "-D"
> "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~"
> "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod
> &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D"
> "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/
> benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.
> dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
> "-P" "run_and_assert.cmake"
>
> 113: Test timeout computed to be: 1500
>
> 113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod
> &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod"
>
> 113: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/
> assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/
> hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"
>
> 113: /Users/tesari/kodiak/trunk/benchmark/Scripts
>
> 113: Number of arguments: 4 arguments.
>
> 113: Argument List: ['/Users/tesari/kodiak/trunk/
> benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk',
> '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']
>
> 113: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called
> with  3  arguments:
>
> 113: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk
>
> 113: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk
>
> 113: 3 0
>
> 113: /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk  does not exist!
>
>
>
>
>
>
>
> *From:* Eric Noulard [mailto:eric.noulard at gmail.com
> <eric.noulard at gmail.com>]
> *Sent:* Wednesday, May 17, 2017 10:54 PM
> *To:* T.Sariyski <tzsariysk at gmail.com>
> *Cc:* CMake ML <cmake at cmake.org>
> *Subject:* Re: [CMake] CTest question
>
>
>
> We lack information on how your test are failing. If they all timeout then
> may be you should setup a proper timeout for the test. The actual execution
> speed of a test may depend on the load of your machine or the network
> (depending on what the test are actually doing).
>
> This may be even more sensible if your test are run in parallel. Do you
> run ctest with --parallel option?
>
>
>
> Do your tests depends on each other? If so did you setup test dependencies?
>
>
>
> Le 17 mai 2017 10:59 PM, "T.Sariyski" <tzsariysk at gmail.com> a écrit :
>
> Hi,
>
>
>
> I wonder what is the reason that sometimes ctest ends with numerous
> ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At
> times the number of these false failures is about one third. I guess that
> the problem is in the way I set my tests, but I have no clue where to look,
> or how to ‘debug’ the ctest run.
>
> Any advice is highly appreciated.
>
>
>
> Thanks,
>
> Ted
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at http://www.kitware.com/
> opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>



-- 
Eric
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170518/51c5947f/attachment-0001.html>

More information about the CMake mailing list