[CMake] How to add a test and compare its output to a reference file ?

[Michael Wild]

On 13. Sep, 2008, at 10:57, Judicaël Bedouet wrote:

> Thanks to both of you.
>
> I appreciate your example Michael but it can't work under Visual  
> Studio. Command should be something like
> ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/hello$ 
> {CMAKE_EXECUTABLE_SUFFIX}
> but apparently, at this stage, CMake has no way of knowing if build  
> type is Debug or Release.
> Tell me if I'm wrong.


I don't think you completely understood my code... I essentially  
replaced Bill's two lines by a single line which calls a CMake script.  
So if you manage to find out what configuration you are running in the  
calling script (probably a CMakeLists.txt), then you are able to pass  
that information on to my runTest.cmake script. runTest.cmake is just  
a "wrapper" script which makes your test program behave as it should:  
output a "pass/fail" message and exit with either 0 or non-zero status  
by running the test program, redirecting the output into a file, and  
then comparing the contents of the output against a reference. The  
script is NOT run at configure-time or build-time, but when ctest is  
run.

>
> Moreover, I need at last a test whose command begins by the test  
> program to be able to do coverage and dynamic analysis.
>

Huh? I don't understand that sentence, sorry...

> The minute I get back to work, I will try Bill's solution.
>
> ADD_TEST(MyTestCreateTest test --arg_test --output output)
> ADD_TEST(MyTestCreateCompare ${CMAKE_COMMAND} -E compare_files  
> output outputTest)
>
> With first line, coverage and dynamic analysis will be done and  
> there is no problem with Visual Studio.

As I said above, with my "wrapper" script, this is also the case.

> I must change source code of test programs to output a file but it's  
> only two lines to redirect std::cout (and maybe std::cerr) to a file.

Whether you want to do this depends on how many test-programs you have  
to adapt... 2-3, OK, but if there are hundreds???

> With second line, I would be able to compare output of the test  
> program to a reference file.

My script does that too.

However, I have to concede that my solution (and also Bill's) is a bit  
of a quick hack. Actually your test program should be the one  
determining whether the test passed or failed, not cmake/ctest. But as  
I said, if you have hundreds of test-programs it might not be  
practical to adapt them all...

HTH

Michael