[CMake] How to add a test and compare its output to a reference file ?

[Judicaël Bedouet]

Thanks to both of you.

I appreciate your example Michael but it can't work under Visual Studio.
Command should be something like
${CMAKE_BINARY_DIR}/${CMAKE_CFG_INTDIR}/hello${CMAKE_EXECUTABLE_SUFFIX}
but apparently, at this stage, CMake has no way of knowing if build type is
Debug or Release.
Tell me if I'm wrong.

Moreover, I need at last a test whose command begins by the test program to
be able to do coverage and dynamic analysis.


The minute I get back to work, I will try Bill's solution.

ADD_TEST(MyTestCreateTest test --arg_test --output output)
ADD_TEST(MyTestCreateCompare ${CMAKE_COMMAND} -E compare_files output
outputTest)

With first line, coverage and dynamic analysis will be done and there is no
problem with Visual Studio. I must change source code of test programs to
output a file but it's only two lines to redirect std::cout (and maybe
std::cerr) to a file.

With second line, I would be able to compare output of the test program to a
reference file.

Thanks,

J. Bedouet


On Fri, Sep 12, 2008 at 4:22 PM, Michael Wild <themiwi at gmail.com> wrote:

>
> On 12. Sep, 2008, at 16:14, Bill Hoffman wrote:
>
>  Judicaël Bedouet wrote:
>>
>>> Hi,
>>> I 'm converting a project to CMake. Tests in this project print messages
>>> to cout and these output traces are compared to reference files. If output
>>> and its reference are different, the corresponding test is considered as
>>> failed. I can't see how to make these tests work with CMake.
>>> I try to run a complex command in ADD_TEST like
>>> ADD_TEST (Test test --arg_test > output && ${DIFF_COMMAND} output
>>> reference_file).
>>> DIFF_COMMAND is equal to diff under Unix and fc under Windows. But CMake
>>> interprets test commands and '>' is passed as an argument to test ("test"
>>> "--arg_test" ">" "output" "&&" "diff" "output" "reference_file").
>>> I try to set property PASS_REGULAR_EXPRESSION to contents of reference
>>> file. It works for little files but it's difficult for large files :
>>>  - All characters * . ( ) ^ $ ... must be backslashed.
>>>  - In this project, generated files are encoded in UTF-8 and I'm not sure
>>> that CMake regular expressions support UTF-8.
>>>  - When there is a difference, I can't see where it is. I understand that
>>> PASS_REGULAR_EXPRESSION wasn't thought in terms of matching whole file.
>>> I could modify source of program tests by redirecting std::cout to a file
>>> and comparing this file to the reference file. But before doing this, I
>>> would like to know if there is a way to make a test with command like `test
>>> args > file && diff` or if it's possible to add a test property which
>>> compare output of the program to a no regular expression.
>>>
>>
>> You can do something like this:
>>
>> ADD_TEST(MyTestCreate Test test --arg_test --output output)
>> ADD_TEST(MyTestCreateCompare ${CMAKE_COMMAND} -E compare_files output
>> outputTest)
>>
>>
>> -Bill
>>
>
>
> Or, if you don't want to change your test-runners, you can create a
> runTest.cmake script which EXECUTE_PROCESSes the test-runner, puts the
> output into a file, captures exit status and stderr and then does a
> compare_files as Bill has shown above.
>
> Find attached a not-so elegant proof of concept.
>
>
> HTH
>
> Michael
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20080913/bf8100c9/attachment-0001.htm>