[CMake] CTest question

[T.Sariyski]

Hi Eric,

 

I am sorry I did not state the question clear enough. Let me first say that tests does not depend on each other, and  running with or without –parallel does not make much of difference.

 

Here is how I set the tests . This is a Fortran code, it does not return. ADD_TEST command is a call to CMAKE, which (i) executes the Fortran code, and (ii) executes a python script, a metrics of how close the solution is to the reference data. I have a PASS_REGULAR_EXPRESSION property,  passRegex, associated with the test. When a test fails, the output is that the required regular expression ${passRegex} is not found. A test and the python script are attached. Please let me know if more information is needed. 

 

Thanks in advance,

Ted

 

 

CMAKE_MINIMUM_REQUIRED(VERSION 2.8)

project(project_name Fortran ) 

enable_testing()

 

set (SRUN_COMMAND /bin/sh -c "cd ${ CMAKE_CURRENT_BINARY_DIR } &&  $<TARGET_FILE:my_exec> ${PROJECT_NAME}")

    add_custom_target(${PROJECT_NAME}_srun COMMAND ${SRUN_COMMAND})

    set(TARGET_NAME ${PROJECT_NAME}_serial)

    add_test(NAME ${TARGET_NAME}

       WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}

       COMMAND ${CMAKE_COMMAND} 

         -D "PRE_DELETE_OUTPUT_FILES:BOOL=true" 

         -D "OUTPUT_FILES:STRING=${files_to_delete}" 

         -D "PROGRAM1:STRING=${SRUN_COMMAND}" 

         -D "PROGRAM2:STRING=python;${assert_case};${hs_work_file};${hs_gold_file};0"         

         -P "run_and_assert.cmake")

 

    set(passRegex "CALCULATION TERMINATED BECAUSE STOP TIME ACHIEVED" "ctest: SUCCESS")

    set_property(TEST ${TARGET_NAME}

                 PROPERTY PASS_REGULAR_EXPRESSION "${passRegex}")

    

 

/> cat run_and_assert.cmake

…

EXECUTE_PROCESS(COMMAND ${PROGRAM1}

                RESULT_VARIABLE error

                ERROR_FILE err.log

                OUTPUT_FILE out.log

                )

IF(error)

    MESSAGE(FATAL_ERROR "Executing \"${PROGRAM1}\" FAIED! error=${error}")

ENDIF()

 

EXECUTE_PROCESS(COMMAND ${PROGRAM2}

                RESULT_VARIABLE status)

IF(status)

    MESSAGE(SEND_ERROR "Executing \"${PROGRAM2}\" FAILED! status=${status}")

ENDIF()

 

 

 

From: Eric Noulard [mailto:eric.noulard at gmail.com] 
Sent: Wednesday, May 17, 2017 10:54 PM
To: T.Sariyski <tzsariysk at gmail.com>
Cc: CMake ML <cmake at cmake.org>
Subject: Re: [CMake] CTest question

 

We lack information on how your test are failing. If they all timeout then may be you should setup a proper timeout for the test. The actual execution speed of a test may depend on the load of your machine or the network (depending on what the test are actually doing).

This may be even more sensible if your test are run in parallel. Do you run ctest with --parallel option? 

 

Do your tests depends on each other? If so did you setup test dependencies?

 

Le 17 mai 2017 10:59 PM, "T.Sariyski" <tzsariysk at gmail.com <mailto:tzsariysk at gmail.com> > a écrit :

Hi,

 

I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the number of these false failures is about one third. I guess that the problem is in the way I set my tests, but I have no clue where to look, or how to ‘debug’ the ctest run.

Any advice is highly appreciated.

 

Thanks,

Ted


--

Powered by www.kitware.com <http://www.kitware.com> 

Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170518/9901242b/attachment.html>