[CMake] CxxTest failure output running unit tests

[Philip Lowman]

On Mon, Dec 8, 2008 at 8:14 PM, Bill Hoffman <bill.hoffman at kitware.com>wrote:

>
>  Furthermore, the possibility remains that CxxTest  may be able to be
>> customized to simplify the output when running within another testing
>> system.
>>
>>  Does the  Testing/Temporary/LastTest.log file help at all?   So, if you
>>> run tests and there are failures, you can look at that file to see the
>>> detail.  Maybe if ctest did something like this:
>>> The following tests FAILED:
>>>       2 - unittest_addition_fail (Failed)
>>> (See Testing/Temporary/LastTest.log for details.)
>>>
>>
>> LastTest.log is a nice thing to have but if the user has 800 unit tests I
>> think he'd prefer not to look through the entire logfile.
>>
>>  If you had 800 tests and 10 or so lines for each failed tests, I would
> think having them scroll by on the screen would be bad.  I would rather see
> that there are some failures, and if I want to find them I can look in the
> file for them.


Yes, it would definitely be a bad default for this reason (especially if
they all fail) =)


>
>  How about giving the user the command to run to get information for the
>> tests that failed?
>>
>> (Run "ctest --output-on-failure -N 2,5,9-11" for details about the failed
>> tests.)
>> Obviously -N would also need to be implemented for this to work right (-I
>> is kinda bloated and doesn't support multiple ranges).
>>
>>  We already have a -N and it does something different.  Maybe -I can be
> extended.  I would hate to bloat even more with an alternate command line
> option that did the same thing....


Bill,

I'm afraid that extending -I any further would yield a command that is even
more complicated and confusing that it is already.  Sometimes it's best to
just start over, I'm afraid.  Users understand what "2,5,9-11" mean in the
context of running tests.  Obviously -I is useful when you're doing striding
but other than that it is awfully complicated and looks like it has been
extended to death.  Witness the support for -I <x,y,z,[a],[b],...> | Test
file]  where "a" and "b" are "additional tests" and "Test file" is...
actually, I have no idea what "Test file" is?  Three command line options,
one for specifying individual tests by number or range, one for striding,
and one for the "Test file" would be so much more clear than one option that
tries to do everything.

Regarding "-N" currently being used, how about "-n" for this?  CTest seems
to not have any lowercase individual character arguments.

However, all that said, I suppose an option to output on failure would not
> be so bad as long as it is not the default.


All things being equal, I actually think output on failure probably would
have been a better default for most users with the current behavior of CTest
being achieved through some kind of "--summarize" flag.  Of course my
opinion is coming as a C++ developer where if a "unit test" fails within
CxxTest we're dealing with perhaps 15 lines of output.  Obviously CTest does
more than just unit tests.  When you test CMake within CTest I can imagine
some of the more complicated tests like bootstrap could be closer to 1500
lines of output (wouldn't want that one to fail) =)

-- 
Philip Lowman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20081208/ca31c272/attachment-0001.htm>