[CMake] FW: CTest question

Thu May 18 13:32:39 EDT 2017

Hi Eric, 

I noticed something that may relates to the problem. Like I said in my previous mail, add_test is a call to cmake:

    ADD_TEST(NAME ${TARGET_NAME}

       …

       COMMAND ${CMAKE_COMMAND} 

-DPROGRAM1 

-DPROGRAM2  

…

Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically? Here is what I noticed. If a test is executed from its own binary directory, ‘ctest -VV’, it always pass. If the same test is executed from the main binary directory, ‘ctest -VV -R zamod’,  it fails because PROGRAM2 is executed before PROGRAM1 is finished. Attached are the screen outputs for both runs. 

Thanks,

Ted

================================================================

cd CTEST/zamod

s1002807:zamod tesari$ ctest -VV

UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl

UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/CTEST/zamod/DartConfiguration.tcl

Test project /Users/tesari/build/gnu.dbg/CTEST/zamod

Constructing a list of tests

Done constructing a list of tests

Updating test list for fixtures

Added 0 tests to meet fixture requirements

Checking test dependency graph...

Checking test dependency graph end

test 1

    Start 1: zamod_serial

1: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"

1: Test timeout computed to be: 9.99988e+06

1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod"

1: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"

1: /Users/tesari/kodiak/trunk/benchmark/Scripts

1: Number of arguments: 4 arguments.

1: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']

1: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with  3  arguments:

1: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk

1: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk

1: 3 0

1:            zamod             Q0.4 PASS

1:            zamod             Q0.5 PASS

1: Kodiak ctest: SUCCESS

1/1 Test #1: zamod_serial .....................   Passed    5.73 sec

================================================================

s1002807:gnu.dbg tesari$ ctest -VV -R zamod

UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl

Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl

UpdateCTestConfiguration  from :/Users/tesari/build/gnu.dbg/DartConfiguration.tcl

Parse Config file:/Users/tesari/build/gnu.dbg/DartConfiguration.tcl

Test project /Users/tesari/build/gnu.dbg

Constructing a list of tests

Done constructing a list of tests

Updating test list for fixtures

Added 0 tests to meet fixture requirements

Checking test dependency graph...

Checking test dependency graph end

test 113

    Start 113: zamod_serial

113: Test command: /Applications/CMake.app/Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"

113: Test timeout computed to be: 1500

113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/CTEST/zamod &&  /Users/tesari/build/gnu.dbg/bin/kodiak zamod"

113: -- Executing "python;/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py;/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk;/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk;0"

113: /Users/tesari/kodiak/trunk/benchmark/Scripts

113: Number of arguments: 4 arguments.

113: Argument List: ['/Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py', '/Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk', '0']

113: /Users/tesari/kodiak/trunk/benchmark/Scripts/assert_case.py called with  3  arguments:

113: 1 /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk

113: 2 /Users/tesari/kodiak/trunk/CTEST/zamod/../GOLD/zamod/hskdk

113: 3 0

113: /Users/tesari/build/gnu.dbg/CTEST/zamod/hskdk  does not exist!

From: Eric Noulard [mailto:eric.noulard at gmail.com] 
Sent: Wednesday, May 17, 2017 10:54 PM
To: T.Sariyski <tzsariysk at gmail.com <mailto:tzsariysk at gmail.com> >
Cc: CMake ML <cmake at cmake.org <mailto:cmake at cmake.org> >
Subject: Re: [CMake] CTest question

We lack information on how your test are failing. If they all timeout then may be you should setup a proper timeout for the test. The actual execution speed of a test may depend on the load of your machine or the network (depending on what the test are actually doing).

This may be even more sensible if your test are run in parallel. Do you run ctest with --parallel option? 

Do your tests depends on each other? If so did you setup test dependencies?

Le 17 mai 2017 10:59 PM, "T.Sariyski" < <mailto:tzsariysk at gmail.com> tzsariysk at gmail.com> a écrit :

Hi,

I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the number of these false failures is about one third. I guess that the problem is in the way I set my tests, but I have no clue where to look, or how to ‘debug’ the ctest run.

Any advice is highly appreciated.

Thanks,

Ted

--

Powered by www.kitware.com <http://www.kitware.com> 

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170518/9d217fbf/attachment-0001.html>