[CMake] CTest outside of CMake

[David Cole]

On Mon, Oct 17, 2011 at 11:04 AM, Tim Gallagher
<tim.gallagher at gatech.edu> wrote:
> Hi,
>
> Sorry I wasn't clearer. That section was from a script that we call with ctest. The entire section that runs things looks like
>
> # Load our custom settings
> ctest_read_custom_files("${CTEST_REPO_DIRECTORY}")
> ctest_start("Nightly")
> ctest_update()
> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES})
>  set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_CONFIGURE_DICT}")
>  ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>  set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}")
>  ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
> endforeach()
> ctest_test(BUILD "${CTEST_REPO_DIRECTORY}")
> ctest_submit()
>
> And we call ctest with
>
> ctest -S CTestScript.cmake
>
> My question is, if I only want to run a few tests, ie only run tests 2, 3, 4:
>
> ctest -S CTestScript.cmake -I 2,4
>
> This only run the ctest_test() on tests 2, 3, and 4, but I don't know how to determine inside the script which tests will be run, so it configures/builds all of the available tests since I have that loop.
>
> So, is there a variable that gets set or some way to determine which tests are called from the command line inside the script in the event something like -I or -R flags are used? Is there a variable that contains the list of tests that CTest will run?
>

No, there is no such variable. The list of tests is not knowable until
after the configure step in a CMake-controlled project. The list of
tests is only read and known internally during the execution of the
ctest_test command. How does ctest even know the list of tests in your
case, when you are running a "setup.py" script as the configure
step...? Are you writing the CTestTestfile.txt files yourself?


> Hope that's clearer. Thanks,
>
> Tim
>
> ----- Original Message -----
> From: "Eric Noulard" <eric.noulard at gmail.com>
> To: gtg085x at mail.gatech.edu
> Cc: cmake at cmake.org
> Sent: Monday, October 17, 2011 3:06:48 AM
> Subject: Re: [CMake] CTest outside of CMake
>
> 2011/10/17 Tim Gallagher <tim.gallagher at gatech.edu>:
>> Hi,
>>
>> We have our project set up to run a series of test cases, but we are treating it as if we are not using CMake. I found a tutorial online on how to set it up, and we have it running just fine when we want to run all of the tests. But, I haven't figured out how to do the configure/build steps for selected tests only.
>>
>> For example, each test has it's own configure command and build command. If I do
>>
>> ctest -I 2,2
>>
>> then it configures and builds all the tests, but will only run the second test.
>>
>> The driver script contains a section like:
>>
>> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES})
>>  set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_CONFIGURE_DICT}")
>>  ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>>  set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}")
>>  ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>> endforeach()
>>
>> and the LESLIE_AVAILABLE_TESTCASES is a list of test cases names that match the add_test() test names. Is it possible when running CTest with -I or -R (or even without -I or -R) to figure out which tests it selects so we can build LESLIE_AVAILABLE_TESTCASES from that?
>
> I'm not sure to understand your whole testing process flow,
> Is the script above part of a ctest -S script?
>
> Filtering some tests may be done with test NAMES or LABELS
> see EXCLUDE / INCLUDE or
> EXCLUDE_LABEL / INCLUDE_LABEL
>  arguments of ctest_test (ctest --help-command ctest_test)
>
> However you does not seem to use ctest_test ?
> How do you run your ctest command (which arguments/options are you using)?
>
> Concerning labels
> see http://www.kitware.com/blog/home/post/11 and reference therein.
>
>
>
> --
> Erk
> Membre de l'April - « promouvoir et défendre le logiciel libre » -
> http://www.april.org
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake