[CMake] parallel ctest spawning too many threads

[Kelly Thompson]

Fellow CMake users:

I'm having a problem with one of my projects when I run 'ctest -j<N>'.  The
problem is that after 70-90 tests, ctest decides to submit all of the
remaining tests at once (~200 tests worth).  For example (actual test names
replaced to protect the innocent :-)

% ctest -j16
Test project /var/tmp/foo
        Start 192: testA_2
        Start 191: testB_4
        Start 190: testC_3
        Start 189: testD_2
        Start 188: testE_4
  1/279 Test #189: testD_2 .................................   Passed
1.81 sec
        Start 187: testF_3
  2/279 Test #192: testA_2 .................................   Passed
2.98 sec
  3/279 Test #191: testB_4 .................................   Passed
2.81 sec
        Start 186: testG_2
        Start 185: testH_4
  4/279 Test #188: testE_4 .................................   Passed
2.81 sec
  5/279 Test #190: testC_3 .................................   Passed
2.92 sec
        Start 184: testI_3
        Start 183: testJ_2
  6/279 Test #187: testF_3 .................................   Passed
2.02 sec
        Start 182: testK_4
  7/279 Test #186: testG_2 .................................   Passed
2.44 sec

...

 82/279 Test #249: testZA_2 ................................   Passed
8.46 sec
 83/279 Test #245: testZB_2 ................................   Passed
3.12 sec
        Start 278: testZC_4
 84/279 Test #278: testZC_4 ................................   Passed
4.27 sec
        Start 266: testZD_4
        Start 263: testZE_4
        Start 275: testZF_4
        Start 270: testZG_4
        Start 269: testZH_4

# ...        about 190 tests started all at the same time.

        Start  89: testZZA_1
        Start  88: testZZB_1
^C

These tests are running under mpirun (trailing number indicates number of
cores needed for the test) and so the total load on my machine jumps to
about 400!

My tests are registered using commands similar to this:

            add_test( 
               NAME    testA_4
               COMMAND ${MPIEXEC} ${MPIEXEC_NUMPROC_FLAG} 4
                       ${MPIEXEC_POSTFLAGS}
                       $<TARGET_FILE:Ut_testA_4_exe> 
               )
            set_tests_properties( testA_4
               PROPERTIES PROCESSORS "4" )

Has anyone seen this 'many threads spawning' behavior for ctest (I couldn't
find anything in the bug tracker).  Is there a known resolution?

Thank you,

-kt
-----
Kelly (KT) Thompson
kgt at lanl.gov
505.665.8090

Los Alamos National Laboratory
CCS-2, MS D409, TA-3/B-422/R-101