[CMake] ctest meant to be re-entrant?

[Chris Hillery]

>
> Thanks very much.  It's maybe more than can/should be done using "ctest".
>  That is, some shell scripting might be in order as well...
>

Well, I can tell you that all of what you want and more are completely
possible using CMake/CTest alone, because I'm doing everything you list
below already. :)

I'm going to slightly re-order some of your requests below, just FYI.

I have a CDash server set up.  Every night, shortly after midnight, I want a
> bunch of build machines, running various operating systems, to build and
> test the software, and report their results to CDash.
>
>
Yep, I'm doing this, with the added excitement that the various OS and
test-suite configurations are brought up on Amazon Elastic Compute Cloud
virtual machines. (Automating that was not as easy as I would have liked, I
can tell you.)


If possible, I'd like to ensure that when ctest invokes cmake, the CMake
> variables mentioned above end up with the same values as they would have if
> the shell scripts mentioned above had been used to kick of cmake.  This is a
> relatively low-priority requirement, but it would be nice.
>

What I think you need to accomplish all of what you want is to create a
"new-style" ctest script. When used in this way, ctest really is a full
scripting language, with the added benefits of being fully cross-platform
and integrated with cmake.

IMHO, this is amongst the very best features of cmake/ctest. It's
unfortunately also one of the least well-documented. The best documentation
I'm aware of at the moment is this page:

  http://www.cmake.org/Wiki/CTest:Using_CTEST_and_CDASH_without_CMAKE

which is not somewhere it would ever occur to you to look because you ARE
using CMake, but there it is. If you ignore the stuff about
CTestTestfile.cmake (which cmake will generate for you) and the callouts to
execute_process(), it covers most of what you need to know.

To start with, create a baseline ctest script, which explicitly does the
same steps that ctest -D does implicitly. It will look something like this:

set (CTEST_SOURCE_DIRECTORY "/path/to/sourcecode")
set (CTEST_BINARY_DIRECTORY "${CTEST_SOURCE_DIRECTORY}/build")
ctest_start (Nightly)
ctest_update (SOURCE "${CTEST_SOURCE_DIRECTORY}")
ctest_configure (BUILD "${CTEST_BINARY_DIRECTORY}")
ctest_build(BUILD "${CTEST_BINARY_DIRECTORY}")
ctest_test(BUILD "${CTEST_BINARY_DIRECTORY}")
ctest_submit ()

(Note: this depends on the source code already having been checked out from
svn. The URL I listed above has some instructions about configuring it to do
the initial checkout automatically, but that's something to add a bit later
when the basics are working.)


My project consists of two, somewhat-related bodies of source code: MOOS and
> IVP.  Each has its own CMake-based build system.  Each comes with a shell
> script that invokes "cmake" with some options, and then "make".  The options
> supplied to cmake like CMAKE_BUILD_TYPE, CMAKE_CXX_FLAGS_RELEASE, and some
> application-specific CMake variables.
>
>
You should be able to specify those variables in your ctest script, before
any of the ctest_*() commands are used. If the flags and variables are
different in various configurations, then I'd suggest the following: First,
create simple .cmake files with a complete set of variables in it. For
instance, create Debug-linux.cmake with

 set (CMAKE_BUILD_TYPE "Debug")
 set (CMAKE_CXX_FLAGS_DEBUG "...whatever...")

Put all of those .cmake files in a subdirectory next to your ctest script
called "conf", and then put the following at the top of your ctest script:

include ("${CTEST_SCRIPT_DIRECTORY}/conf/${CTEST_SCRIPT_ARG}")

Now, to run the "Debug-linux" configuration, you can invoke your script
(from cron or whatever) as

  ctest -S /path/to/script.cmake,Debug-linux

(Aside: this is the only way I know to pass variables to a ctest script; -D
doesn't work because it's used for Dashboards. Basically, whatever you put
after a comma in the -S argument to ctest will be made available to the
ctest script as CTEST_SCRIPT_ARG. It's ugly, but it gets the job done. Oh,
and CTEST_SCRIPT_DIRECTORY is set to the directory of the ctest script
itself.)


>
> To date I've been ignoring MOOS with regards to CTest.  I do have to build
> MOOS prior to building IVP, but I wanted to keep things simple for now and
> just focus on ctest'ing IVP.
>
> Since MOOS and IVP are in separate source trees and are built at separate
> times, I have no expectation that their building / testing results can end
> up in the same CDash dashboard.  But it would be awfully nice if they did,
> because we really see them as two parts of the same product.  It's only an
> accident of nature that they're not part of the very same CMake build tree.
>

There's no reason that two different products can't submit to the same CDash
project. Just specify the same CDash URL in your CTestConfig.cmake.

When you've gotten things working the way you want, you may want to read
this:

  CDash subprojects... <http://www.kitware.com/blog/home/post/11>

and the Kitware Quarterly newsletter that is mentioned therein. That will
explain how to make CDash display things more nicely for multiple projects
by designating them as subprojects of a main project. But that is an
optional extra.


> Furthermore, I want a shell command that I've designed to decide, for each
> build, what appears in CDash's "Build Name" column.  I want this because
> "Linux-c++" doesn't sufficiently differentiate the various build platforms
> I'm using.
>

This is easily done via the CTEST_BUILD_NAME variable, which I see has been
discussed in another thread. Hopefully once you get a functional ctest
script going, this will Just Work(tm) for you now. Just put it as another
set() line in your ctest script or the included configuration file.


> If possible, I'd like the CDash report to include a mention of how the
> sourcecode checkout went.  If the sourcecode checkout fails, I'd like CDash
> to show that, rather than just having no report from the client that had the
> problem.
>


> I want a fresh checkout of the source code each time I do this.  Merely
> updating it isn't clean enough.  Also, I'm using svn.
>

The URL I listed above had some information about how to get ctest to do an
initial svn checkout for you. I personally am explicitly invoking "svn co"
with execute_process(), which means that any failures there don't end up on
cdash as you say. I think possibly if you do what the URL lists, though, you
might get what you want.


> I'd like to continue specifying my tests by having "ADD_TEST(...)" calls
> sprinkled throughout my project's CMakeLists.txt files.
>

Yes, please; don't ever stop doing that. It's just the right approach.

Hope this helps! Good luck... as I said, you can make ctest do some
wonderful things. My current script is much more involved than this, and it
handles a remote build/test server for our code as well as nightly testing
on several platforms with different build and test-suite configurations.

Ceej
aka Chris Hillery
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20100621/87b584e0/attachment-0001.htm>