[CMake] Building and executing tests when building updated libraries

[Robert Dailey]

On Tue, Feb 21, 2012 at 12:37 PM, David Cole <david.cole at kitware.com> wrote:

> On Tue, Feb 21, 2012 at 1:27 PM, Robert Dailey <rcdailey at gmail.com> wrote:
>
>> Hi,
>>
>> I'm using Visual Studio as my generator for my CMake projects. As of
>> right now, I make my tests depend on the libraries they test. So for
>> example, tests named:
>>
>> test_thingA
>> test_thingB
>>
>> will all depend on library:
>>
>> libfoo.lib
>>
>> When I build target "libfoo" in visual studio, it would be nice to have
>> all dependent tests build as well, and have them each execute.
>>
>> The goal for all of this is to make it as convenient as possible for
>> developers on my team to RUN TESTS on their code before they submit to
>> version control. I want to make it automated, so when they rebuild the
>> library, the testing automatically happens. I'd also obviously create an
>> option in cmake cache to turn this automation off should it become too
>> annoying.
>>
>> If this isn't a good idea, can someone recommend a good workflow for
>> running tests locally prior to checking in source code?
>>
>> ---------
>> Robert Dailey
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
>>
>
>
> If you're using add_test in your CMakeLists files, then the perfect way to
> prove that the tests all work on a developer's machine is for him or her to
> run:
>
>   ctest -D Experimental
>
> after making local mods, and before pushing the changes to your source
> control system.
>
> That will configure, build all and run all the tests. And submit the
> results to your CDash server so that is public evidence that he actually
> did run the tests, and hopefully that they all passed on his machine at
> least.
>
> You can also restrict the set of tests that run using -R or -I or -L on
> the ctest command line, although, you should strive to have your test suite
> be brief enough that it's not painful for folks to run the full test suite
> prior to checkin.
>

I think this is a reasonable idea for small projects, but in general I
disagree with running all tests.

There are hundreds of projects (probably 150) and hundreds more of tests
(probably 10 tests per project). In general agile methodology, it only
makes sense to unit test those components which have changed. Unit testing
a dependent component that did not have a source code change will not be
needed or beneficial.

All of these tests can take hours to run, which isn't unacceptable because
it's a full test suite. Only the build server kicks off a build and runs
the FULL test suite (thus running ctest -D Experimental as you have
suggested). Developers just do an intermediate check by unit testing only
the parts of the code base that have changed. This is essential for
practices like continuous integration.

Ideally the pipeline goes like this:

   1. Programmer makes a change to a certain number of libraries
   2. Programmer runs the relevant tests (or all) for each of the libraries
   that were changed.
   3. Once those tests have passed, the developer submits the source code
   to version control
   4. The build server is then instructed to run a full build and test of
   the entire code base for each checkin.
   5. The build server can then run any integration tests that are
   configured (not sure how these would be setup in CMake - probably again as
   tests, but not specific to only a single project)
   6. Build is considered "complete" at this point.

Seems like there would be no choice but to run them individually in this
case, since CMake really shines only in steps after #3
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20120221/1658e4a4/attachment.htm>