[cmake-developers] CTest vs. gcov

[Rolf Eike Beer]

Hi all,

I have been wrestling with uncovered files for a while. The current state 
doesn't satisfy me, so I thought to share my findings and we can hopefully 
come up with something better. If you want to look at some numbers you can 
compare the 2 nightly results here: 
http://my.cdash.org/index.php?project=Qsmtp&date=2012-02-05 [1]

voyager runs with a more or less plain 2.8.7 (default gentoo build), while eto 
runs current next with the patches described below.

I'll describe what happened and what I did now. I got aware of the nice but 
undocumented CTEST_EXTRA_COVERAGE_GLOB that allows uncovered files to show up 
on the dashboard. I enabled this and basically got the results you see for 
voyager. A small hints for those that want to try: if you place your build 
directory inside the source directory (i.e. the usual "mkdir build && cd build 
&& cmake ..") than things like the compiler test files may show up in the 
coverage. I added "/CMakeFiles/" to CTEST_CUSTOM_COVERAGE_EXCLUDE to get rid 
of them.

Ok, now I have these uncovered files in the dashboard, but compared to what 
gcov would report the numbers of uncovered lines are insanely big. This comes 
because simply _every_ line is counted as uncovered, may it be code, comment, 
whitespace, #include or whatever. I then added 
3ea6be30819e78ddf3d97378a519dc50836049a3 to it, which simply ignored empty 
lines. This was the best compromise between effort and effect for me: it was 
only a very few lines of code, it would reduce the numbers by ~10% for my 
coding style and I had to know nothing about the language used. If you want to 
test it out you can get that commit from the next branch. David didn't like 
it, so I had to revert.

Instead he told me to just add proper coverage. Well, let's see ;) I then 
wrote some test cases calling a binary consisting of multiple files. Only a 
few of them were actually used, the others were uncovered. To my surprise 
those other files which had been reported to have no coverage and a high 
number of uncovered lines now were reported as uncovered with 0 lines in 
CDash. First I thought of a bug in gcov, but by examination of the files that 
gcov and ctest generated I found out that everything was fine there.

The problem here is that ctest does this code when generating Coverage.xml:

     covSumFile << "\t<File Name=\"" << cmXMLSafe(fileName)
       << "\" FullPath=\"" << cmXMLSafe(
         this->CTest->GetShortPathToFile(fullFileName.c_str()))
      << "\" Covered=\"" << (tested > 0 ? "true":"false") << "\">\n"
       << "\t\t<LOCTested>" << tested << "</LOCTested>\n"
       << "\t\t<LOCUnTested>" << untested << "</LOCUnTested>\n"
       << "\t\t<PercentCoverage>";

So if no lines are covered then it reports 'Covered="false"' which leads CDash 
to entirely ignoring the line count and not showing a link to the file. In 
contrast it does that when collecting files using CTEST_EXTRA_COVERAGE_GLOB:

    covSumFile << "\t<File Name=\"" << cmXMLSafe(fileName)
      << "\" FullPath=\"" << cmXMLSafe(i->c_str())
      << "\" Covered=\"true\">\n"
      << "\t\t<LOCTested>0</LOCTested>\n"
      << "\t\t<LOCUnTested>" << untested << "</LOCUnTested>\n"

Covered is always set to true here. So the quick and dirty fix I used was to 
change the one line to

      << "\" Covered=\"" << (tested+untested > 0 ? "true":"false") << "\">\n"

which at the end gave me the results shown by eto in the CDash link above.

Ok, so much to the history behind everything, now the questions: was that the 
right approach? What does 'Covered="false"' actually means? Should we 
unconditionally send true for gcov files, too? This would at least give 
consistent results. Is it at the end a CDash bug? Or should CTest be fixed 
similar to the way I did it for my local build?

Opinions? Ideas? Patches? Money?

Greetings,

Eike

1) You can also see the shiny new CDash version the admins brought to us at 
the weekend. Nice work, guys!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20120206/8c5253bd/attachment.sig>