[CMake] CTest outside of CMake

[David Cole]

You could approach it in a different way:

Rather than having the end user running ctest control what tests are
run with -I or -R, you could have them provide a file that lists the
tests to run. Then you could construct the list from that file, run
only the configure/build steps that are necessary, and then use
ctest_test along with its INCLUDE argument to limit the tests to the
set specified.

Just an idea.


HTH,
David


On Mon, Oct 17, 2011 at 12:12 PM, Tim Gallagher
<tim.gallagher at gatech.edu> wrote:
> Yes, we're writing the CTestTestfile.cmake with all of the available tests using add_test(). So it sounds like there is no way to only configure/build selected tests when using CTest outside of CMake, short of hand-writing the CTestTestfile with only the desired tests.
>
> I was going to suggest having a BUILD_COMMAND and CONFIGURE_COMMAND option to the add_test() call, but that sounds like that also won't fix the problem because the add_test is after the configure/build step. Obviously those would not be useful for projects using CMake, but it would be good for ones not using CMake.
>
> The way the entire process is set up -- we have our source code built with CMake. All of our test cases were created prior to CMake'ing things and are fairly involved to set up, so we have that setup.py script. That script creates the bin, input, and output directories, copies over the needed input files, etc.. Then it modifies the input files for the specific test case and writes out a restart file with the initial conditions. Then, in the bin directory, it creates a build directory where it configures CMake for the main code, and another build directory where it configures CMake for the utilities. Then it builds them all and runs the test cases on our cluster. The tests can be run external to ctest with:
>
> ./setup.py -t <test category> <test name> <python dictionary of options>
>
> and we were trying to wrap all of that process up with CTest and use CDash to track the cases. But as users write more test cases, they asked if it was possible to make sure their case worked with CTest without having to build/run all of the other cases.
>
> It is like pulling teeth to just get them to add the add_test() line in the CTestTestfile even though it's just copy-pasting what's already there, so I was hoping there would be a way to do this without making them write custom files because I don't think that would actually happen. But, if that's not possible, then we'll work with what we have.
>
> Thanks,
>
> Tim
>
>
> ----- Original Message -----
> From: "David Cole" <david.cole at kitware.com>
> To: gtg085x at mail.gatech.edu
> Cc: cmake at cmake.org
> Sent: Monday, October 17, 2011 11:54:57 AM
> Subject: Re: [CMake] CTest outside of CMake
>
> On Mon, Oct 17, 2011 at 11:04 AM, Tim Gallagher
> <tim.gallagher at gatech.edu> wrote:
>> Hi,
>>
>> Sorry I wasn't clearer. That section was from a script that we call with ctest. The entire section that runs things looks like
>>
>> # Load our custom settings
>> ctest_read_custom_files("${CTEST_REPO_DIRECTORY}")
>> ctest_start("Nightly")
>> ctest_update()
>> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES})
>>  set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_CONFIGURE_DICT}")
>>  ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>>  set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}")
>>  ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>> endforeach()
>> ctest_test(BUILD "${CTEST_REPO_DIRECTORY}")
>> ctest_submit()
>>
>> And we call ctest with
>>
>> ctest -S CTestScript.cmake
>>
>> My question is, if I only want to run a few tests, ie only run tests 2, 3, 4:
>>
>> ctest -S CTestScript.cmake -I 2,4
>>
>> This only run the ctest_test() on tests 2, 3, and 4, but I don't know how to determine inside the script which tests will be run, so it configures/builds all of the available tests since I have that loop.
>>
>> So, is there a variable that gets set or some way to determine which tests are called from the command line inside the script in the event something like -I or -R flags are used? Is there a variable that contains the list of tests that CTest will run?
>>
>
> No, there is no such variable. The list of tests is not knowable until
> after the configure step in a CMake-controlled project. The list of
> tests is only read and known internally during the execution of the
> ctest_test command. How does ctest even know the list of tests in your
> case, when you are running a "setup.py" script as the configure
> step...? Are you writing the CTestTestfile.txt files yourself?
>
>
>> Hope that's clearer. Thanks,
>>
>> Tim
>>
>> ----- Original Message -----
>> From: "Eric Noulard" <eric.noulard at gmail.com>
>> To: gtg085x at mail.gatech.edu
>> Cc: cmake at cmake.org
>> Sent: Monday, October 17, 2011 3:06:48 AM
>> Subject: Re: [CMake] CTest outside of CMake
>>
>> 2011/10/17 Tim Gallagher <tim.gallagher at gatech.edu>:
>>> Hi,
>>>
>>> We have our project set up to run a series of test cases, but we are treating it as if we are not using CMake. I found a tutorial online on how to set it up, and we have it running just fine when we want to run all of the tests. But, I haven't figured out how to do the configure/build steps for selected tests only.
>>>
>>> For example, each test has it's own configure command and build command. If I do
>>>
>>> ctest -I 2,2
>>>
>>> then it configures and builds all the tests, but will only run the second test.
>>>
>>> The driver script contains a section like:
>>>
>>> foreach(TESTCASE ${LESLIE_AVAILABLE_TESTCASES})
>>>  set(CTEST_CONFIGURE_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_CONFIGURE_DICT}")
>>>  ctest_configure(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>>>  set(CTEST_BUILD_COMMAND "./setup.py -t ${TESTCASE} ${LESLIE_BUILD_DICT}")
>>>  ctest_build(BUILD "${CTEST_REPO_DIRECTORY}" APPEND)
>>> endforeach()
>>>
>>> and the LESLIE_AVAILABLE_TESTCASES is a list of test cases names that match the add_test() test names. Is it possible when running CTest with -I or -R (or even without -I or -R) to figure out which tests it selects so we can build LESLIE_AVAILABLE_TESTCASES from that?
>>
>> I'm not sure to understand your whole testing process flow,
>> Is the script above part of a ctest -S script?
>>
>> Filtering some tests may be done with test NAMES or LABELS
>> see EXCLUDE / INCLUDE or
>> EXCLUDE_LABEL / INCLUDE_LABEL
>>  arguments of ctest_test (ctest --help-command ctest_test)
>>
>> However you does not seem to use ctest_test ?
>> How do you run your ctest command (which arguments/options are you using)?
>>
>> Concerning labels
>> see http://www.kitware.com/blog/home/post/11 and reference therein.
>>
>>
>>
>> --
>> Erk
>> Membre de l'April - « promouvoir et défendre le logiciel libre » -
>> http://www.april.org
>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake