[cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[Nils Gladitz]

Hello,

I'm trying to fix: http://public.kitware.com/Bug/view.php?id=14484
which I reported earlier.

I pushed my topic "ctest-fix-run-serial" to stage.

Could some of you take a look at my changes?
They seem to fix the issue for me and don't seem to break any of the 
existing tests but
the code is foreign to me and I am not aware of the history and 
potential pitfalls that go with it.

I use an updated test case that I attached to this mail (I hope it gets 
preserved in the mailing list) otherwise I hope at least this paste link 
survives: http://pastebin.com/3zXsTrem

It sets up 4 Tests:
     i_like_company
     i_like_company_too
     i_have_dependencies (depends on "i_want_to_be_alone)
     i_want_to_be_alone (required to run serial)

Each test consumes 3 seconds.

With vanilla 2.8.12 a parallel test run (e.g. ctest -j16) gives me:

    Test project /home/ngladitz/src/test/build
         Start 1: i_like_company
         Start 4: i_want_to_be_alone
         Start 2: i_like_company_too
    1/4 Test #1: i_like_company ...................   Passed    3.01 sec
    2/4 Test #4: i_want_to_be_alone ...............   Passed    3.01 sec
    3/4 Test #2: i_like_company_too ...............   Passed    3.01 sec
         Start 3: i_have_dependencies
    4/4 Test #3: i_have_dependencies ..............   Passed    3.02 sec

    100% tests passed, 0 tests failed out of 4

    Total Test time (real) =   6.05 sec

Note how the serial test "i_want_to_be_alone" starts and ends roughly at 
the same time as "i_like_company" and "i_like_comany_too".

With my branch I get:

    Test project /home/ngladitz/src/test/build
         Start 1: i_like_company
         Start 2: i_like_company_too
    1/4 Test #1: i_like_company ...................   Passed    3.02 sec
    2/4 Test #2: i_like_company_too ...............   Passed    3.02 sec
         Start 4: i_want_to_be_alone
    3/4 Test #4: i_want_to_be_alone ...............   Passed    3.03 sec
         Start 3: i_have_dependencies
    4/4 Test #3: i_have_dependencies ..............   Passed    3.02 sec

    100% tests passed, 0 tests failed out of 4

    Total Test time (real) =   9.08 sec

Note how the whole test run takes 9 rather than 6 seconds.

The test runs with vanilla are sporadic and sometimes give the correct 
results consistently for a while which makes this difficult to test 
consistently.

I am grateful for any guidance and insight you may provide.

Thanks in advance.

Nils
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20131017/5f0c8a1e/attachment.html>
-------------- next part --------------
if(TEST_NAME)
	set(CTEST_RUN_CURRENT_SCRIPT 0)
	ctest_sleep(3)
	return()
endif()

cmake_minimum_required(VERSION 2.8.12)

enable_testing()

function(my_add_test NAME COST)
	add_test(${NAME} 
		${CMAKE_CTEST_COMMAND} -DTEST_NAME=${NAME} 
		-S ${CMAKE_CURRENT_LIST_FILE})
	set_tests_properties(${NAME} PROPERTIES COST ${COST})
endfunction()

my_add_test(i_like_company 1000)
my_add_test(i_like_company_too 0)

my_add_test(i_have_dependencies 1000)
set_tests_properties(i_have_dependencies PROPERTIES 
	DEPENDS "i_want_to_be_alone")

my_add_test(i_want_to_be_alone 100)
set_tests_properties(i_want_to_be_alone PROPERTIES RUN_SERIAL 1)