[cmake-developers] RFC: pretty/HTML output in test logs in ctest/cdash

[Alexander Neundorf]

Hi,

attached are two patches for ctest and cdash from Volkan (on CC).
They are not ready for inclusion, but before we continue to work in this, we'd 
like to get some comments whether this is a good idea, the right approach, 
etc.
So, here we go.

We are using cdash a lot, and for many of our tests the log is quite verbose 
and somewhat hard to read. When looking for the reason why a test failed, it 
can be somewhat hard to find the line which contains "ERROR".
Also, a lot of stuff is repeated every line, which also makes the log harder 
to read, e.g. the directory part of the file where the log came from:

<timestamp> /some/where/deep/in/a/subdir/test.py: Something...
<timestamp> /some/where/deep/in/a/subdir/test.py: happend...
<timestamp> /some/where/deep/in/a/subdir/test.py: here...
<timestamp> /some/where/deep/in/a/subdir/test.py: ...

The directory part is actually so wide that we have to scroll to see actual 
text.


So, to help with this, we came up with the idea to use HTML to format the log 
so that it becomes easier to read.

The attached patch for ctest adds a new test property GENERATE_HTML, and if 
this is enabled, ctest generates HTML. It searches the lines which made the 
test fail and colors them red, and the lines which make a test pass are 
colored green.
This makes it easier to find the fail/succeed lines.

(In the patch the fail strings are still hardcoded, this would have to use the 
respective test property.)

On the cdash side, if "</span>" is found in the log, it is considered HTML and 
not escaped. I guess this should be done in a different way, which does not 
involve parsing the log text.

Comments so far ?


For making our log lines shorter, we are not sure yet how to do that.
We could add support for squish XML to ctest, so that ctest reads the squish 
XML and converts it to HTML which is in some way shorter, maybe a HTML table, 
or something.
Or we could add a test property with a regular expression, if that regexp 
matches, ctest replaces it with a shorter version or something.
Or we could modify our test scripts so that they already generate the pretty 
HTML, which ctest then simply sends to cdash.
That's the ideas we have so far, but it doesn't feel quite right yet.

So, what do you think ?

Alex
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-added-html-generation-capability-to-colorize-plain-o.patch
Type: text/x-patch
Size: 6062 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20130418/942fdc4b/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cdash.patch
Type: text/x-patch
Size: 1100 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20130418/942fdc4b/attachment-0005.bin>