[cmake-developers] Unit testing CMake modules

[Wouter Klouwen]

Hi all,

We have a rather large amount of CMake code (4k+ lines) in various
modules/functions. They contain the common logic for many of our
projects. This makes it quite an important base for everything our team
does.

Rather shamefully at present these modules are rather poorly tested. I'd
like to improve this, but the current way of testing CMake code
is typically to run a trimmed project, and to verify whether certain
invocations produce a certain output or file hierarchy.
This involves a bit of infrastructure and can be a bit cumbersome
maintain, to diagnose when tests fail, and it requires a separate run of
the tests.
The overall cumbersomeness of the setup in turn discourages in our team,
including myself, from adding tests.

I'd like a more integrated approach that makes running at least some
basic tests part of the build progress, and a more direct way of
reporting failures.

In other programming environments, testing often involves some kind of
mocking environment, and CMake helpfully allows the overriding of CMake
built in functions, though this is typically discouraged.

In my ideal world it would be possible to save the state of the current
function set up, then call a function with a certain given number of
parameters, and expect a certain sequence of events, such as functions
to be called, environment variables to be set, etc.
Something akin to CMakePushCheckState, except for built in functions.

Then a module could provide for some functions that would set up
expectations and verify them, within a run of cmake, or possibly some
other commands could be added, to give some syntactic glossy coat to it.

As it wouldn't actually trigger any of the expensive generating
functions, it would be lightweight, quick to run and give pretty direct
errors in terms of failed expectations, reducing debug time.

If it was done with CMake commands, I might imagine it to look something
like:

function(foobar)
   # function to test, does something non trivial
   if ("FOO" IN_LIST ARGV)
      install(FILES foo DESTINATION foo_dir)
   else("BAR" IN_LIST ARGV)
      message(FATAL_ERROR "Some error")
   else("BAZ" IN_LIST ARGV)
      set(BAZ True PARENT_SCOPE)
   endif()
endfunction()

test(foobar)
   expect(WITH "FOO" CALL install FILES foo DESTINATION foo_dir)
   expect(WITH "BAR" CALL message FATAL_ERROR "Some error")
   expect(WITH "BAZ" ENVIRONMENT BAZ True)
endtest(foobar)

What do people think? Is this crazy? Is there a quicker way to get
somewhere close? Should I put some effort into making this into an
actual proposal/working code?

Thanks in advance,
    W


This transmission contains information that may be confidential and contain personal views which are not necessarily those of YouView TV Ltd. YouView TV Ltd (Co No:7308805) is a limited liability company registered in England and Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower Thames Street, London, EC3R 6YT. For details see our web site at http://www.youview.com