[cmake-developers] Unit testing CMake modules

[Stuermer, Michael SP/HZA-ZSEP]

Hello Wouter,

testing CMake code is indeed very important. This is why kitware does it as well. Did you check the CMake code testing infrastructure in "Tests/RunCMake" in the sources? It is a very flexible concept which makes adding tests easy enough for everyone contribute (IMO). At the beginning it might be a bit confusing how expected results etc. are handled, but once you understand how it works it's really nice.

Another (maybe the main) advantage: you have tons of examples and the RunCMake infrastructure is maintained so you don't have to it all on your own.

best regards,
Michael

> -----Ursprüngliche Nachricht-----
> Von: cmake-developers [mailto:cmake-developers-bounces at cmake.org] Im
> Auftrag von Wouter Klouwen
> Gesendet: Dienstag, 29. Mai 2018 18:36
> An: CMake Developers
> Betreff: [cmake-developers] Unit testing CMake modules
> 
> Hi all,
> 
> We have a rather large amount of CMake code (4k+ lines) in various
> modules/functions. They contain the common logic for many of our projects.
> This makes it quite an important base for everything our team does.
> 
> Rather shamefully at present these modules are rather poorly tested. I'd like
> to improve this, but the current way of testing CMake code is typically to run
> a trimmed project, and to verify whether certain invocations produce a
> certain output or file hierarchy.
> This involves a bit of infrastructure and can be a bit cumbersome maintain, to
> diagnose when tests fail, and it requires a separate run of the tests.
> The overall cumbersomeness of the setup in turn discourages in our team,
> including myself, from adding tests.
> 
> I'd like a more integrated approach that makes running at least some basic
> tests part of the build progress, and a more direct way of reporting failures.
> 
> In other programming environments, testing often involves some kind of
> mocking environment, and CMake helpfully allows the overriding of CMake
> built in functions, though this is typically discouraged.
> 
> In my ideal world it would be possible to save the state of the current
> function set up, then call a function with a certain given number of
> parameters, and expect a certain sequence of events, such as functions to
> be called, environment variables to be set, etc.
> Something akin to CMakePushCheckState, except for built in functions.
> 
> Then a module could provide for some functions that would set up
> expectations and verify them, within a run of cmake, or possibly some other
> commands could be added, to give some syntactic glossy coat to it.
> 
> As it wouldn't actually trigger any of the expensive generating functions, it
> would be lightweight, quick to run and give pretty direct errors in terms of
> failed expectations, reducing debug time.
> 
> If it was done with CMake commands, I might imagine it to look something
> like:
> 
> function(foobar)
>    # function to test, does something non trivial
>    if ("FOO" IN_LIST ARGV)
>       install(FILES foo DESTINATION foo_dir)
>    else("BAR" IN_LIST ARGV)
>       message(FATAL_ERROR "Some error")
>    else("BAZ" IN_LIST ARGV)
>       set(BAZ True PARENT_SCOPE)
>    endif()
> endfunction()
> 
> test(foobar)
>    expect(WITH "FOO" CALL install FILES foo DESTINATION foo_dir)
>    expect(WITH "BAR" CALL message FATAL_ERROR "Some error")
>    expect(WITH "BAZ" ENVIRONMENT BAZ True)
> endtest(foobar)
> 
> What do people think? Is this crazy? Is there a quicker way to get somewhere
> close? Should I put some effort into making this into an actual
> proposal/working code?
> 
> Thanks in advance,
>     W
> 
> 
> This transmission contains information that may be confidential and contain
> personal views which are not necessarily those of YouView TV Ltd. YouView
> TV Ltd (Co No:7308805) is a limited liability company registered in England and
> Wales with its registered address at YouView TV Ltd, 3rd Floor, 10 Lower
> Thames Street, London, EC3R 6YT. For details see our web site at
> http://www.youview.com
> --
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers