[CMake] How to add a test and compare its output to a reference file ?

[Judicaël Bedouet]

On Sun, Sep 14, 2008 at 10:14 AM, Michael Wild <themiwi at gmail.com> wrote:

>
> On 13. Sep, 2008, at 10:57, Judicaël Bedouet wrote:
>
>  Thanks to both of you.
>>
>> I appreciate your example Michael but it can't work under Visual Studio.
>> Command should be something like
>> ${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/hello${CMAKE_EXECUTABLE_SUFFIX}
>> but apparently, at this stage, CMake has no way of knowing if build type
>> is Debug or Release.
>> Tell me if I'm wrong.
>>
>
>
> I don't think you completely understood my code... I essentially replaced
> Bill's two lines by a single line which calls a CMake script.


I think I understood it. I tested it with Makefiles and there is no problem.
But it doesn't work with Visual Studio. I try to update your code to make it
work but I think it's not possible. Read this.
http://www.cmake.org/cmake/help/cmake-2.6.html#variable:CMAKE_CFG_INTDIR



> So if you manage to find out what configuration you are running in the
> calling script (probably a CMakeLists.txt), then you are able to pass that
> information on to my runTest.cmake script.


This is exactly the problem. Under Visual Studio, it's impossible to know
the configuration.



> runTest.cmake is just a "wrapper" script which makes your test program
> behave as it should: output a "pass/fail" message and exit with either 0 or
> non-zero status by running the test program, redirecting the output into a
> file, and then comparing the contents of the output against a reference. The
> script is NOT run at configure-time or build-time, but when ctest is run.


Fine.



>
>
>> Moreover, I need at last a test whose command begins by the test program
>> to be able to do coverage and dynamic analysis.
>>
>>
> Huh? I don't understand that sentence, sorry...


Oh, sorry. I use valgrind and gcov to perform coverage and memory checking
tests. If ADD_TEST command begins by ${CMAKE_COMMAND}, I will perform
memcheck tests on CMake and not on my test program. If I choose your method,
I must therefore replace your line with two and execute each test twice.

add_test( hello_test_pass ${CMAKE_COMMAND} -D
TEST_PROGRAM=${CMAKE_BINARY_DIR}/hello${CMAKE_EXECUTABLE_SUFFIX} -D
TEST_ARGS=1 -D TEST_OUTPUT=tmpfile.txt -D
TEST_REFERENCE=${CMAKE_SOURCE_DIR}/testref.txt -P
${CMAKE_SOURCE_DIR}/runTest.cmake ) # To compare output to reference file
add_test( hello 1) # To perform memcheck tests



>
>  The minute I get back to work, I will try Bill's solution.
>>
>> ADD_TEST(MyTestCreateTest test --arg_test --output output)
>> ADD_TEST(MyTestCreateCompare ${CMAKE_COMMAND} -E compare_files output
>> outputTest)
>>
>> With first line, coverage and dynamic analysis will be done and there is
>> no problem with Visual Studio.
>>
>
> As I said above, with my "wrapper" script, this is also the case.


"Coverage and dynamic analysis will be done and there is no problem with
Visual Studio"
I can't see how (Have a look at CTest documentation).



>
>
>  I must change source code of test programs to output a file but it's only
>> two lines to redirect std::cout (and maybe std::cerr) to a file.
>>
>
> Whether you want to do this depends on how many test-programs you have to
> adapt... 2-3, OK, but if there are hundreds???
>

Not so many.



>
>
>  With second line, I would be able to compare output of the test program to
>> a reference file.
>>
>
> My script does that too.


>
> However, I have to concede that my solution (and also Bill's) is a bit of a
> quick hack. Actually your test program should be the one determining whether
> the test passed or failed, not cmake/ctest. But as I said, if you have
> hundreds of test-programs it might not be practical to adapt them all...
>
> HTH
>
> Michael


Thanks for your time,

J. Bedouet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20080914/179c636b/attachment.htm>