MantisBT - CDash
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0008670CDashpublic2009-03-04 09:172009-03-30 08:18
ReporterRoscoe A. Bartlett 
Assigned ToJulien Jomier 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version1.4 
Summary0008670: CDash Dev: Failed tests shown in orange instead of red
Description
Why is it that failed tests are being shown in orange (like a warning)
instead of red (as an error) on the dashboard. For example, at:

  http://trilinos-dev.sandia.gov/cdash/index.php?project=Trilinos&display=project&date=2009-03-05 [^]

the row:

  godel.sandia.gov Linux-Nightly-SERIAL_DEBUG_ICPC 0 0 0 0 0.2 0 46 0.7 0 1 49 0.1 2009-03-04T03:45:20 MST Teuchos

colors the '1' failed test for Teuchos in orange instead of red. I
assume this is not intended and is a defect since even one failed test
on other web pages is shown in read. For exmaple, on the page:


   http://trilinos-dev.sandia.gov/cdash/index.php?project=Trilinos&date=2009-03-05 [^]

the row:

   Triutils 0 0 6 0 6 0 0 1 5 2009-03-04 10:58:52

shows the '1' failed test in red instead of orange.

This is not a high priority issue for Trilinos but it would help to
have all failed tests shown in red when scaning a large number of
builds as shown, for example, at:

  http://trilinos-dev.sandia.gov/cdash/index.php?project=Trilinos&display=project&date=2009-03-05 [^]
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2009-03-04 09:17Roscoe A. BartlettNew Issue
2009-03-04 09:30Bill HoffmanStatusnew => assigned
2009-03-04 09:30Bill HoffmanAssigned To => Julien Jomier
2009-03-04 09:31Bill HoffmanNote Added: 0015531
2009-03-06 09:31Roscoe A. BartlettNote Added: 0015589
2009-03-27 23:29Julien JomierNote Added: 0015827
2009-03-27 23:29Julien JomierStatusassigned => resolved
2009-03-27 23:29Julien JomierFixed in Version => 1.4
2009-03-27 23:29Julien JomierResolutionopen => fixed
2009-03-30 08:18Julien JomierStatusresolved => closed

Notes
(0015531)
Bill Hoffman   
2009-03-04 09:31   
This is by design. I wonder if it can be made a project option.
(0015589)
Roscoe A. Bartlett   
2009-03-06 09:31   
I would be helpful if CDash would allow us to mark all errors in red.

Just and an observation, it seems that CDash is set up by default to encourage less than strong software engineering practices (e.g. emails only when an error first appears, coloring tests failures in orange). This is a system that encourages ignoring recurring errors.

In my opinion, CDash should be set up by default to encourage strong software engineering best practices that include, for example:

1) Most all failures are intolerable and 'stop the line' should be used to fix them ASAP (See "Implementing Lean Software Development" and "Continuous Integration" at http://www.cs.sandia.gov/~rabartl/readingList.html [^]).

2) If some error is 'tolerable', then that test and/or feature should disabled or removed and an appropriate bug item so some other event should occur to mark this. If the feature is still useful, then the issue should be pushed back onto the project's Sprint or Product Backlog to be addressed at a later time.

If other projects want to relax strong software engineering practices, then CDash should allow this through project options but these should not be the default, in my opinion.

Anyway, for Trilinos, all errors on supported builds are interoperable and they need to be dealt with in a timely manner.

Note that right now there are several errors on the Trilinos CDash website which I am allowing because it is useful to test and demonstrate the CTest/CDash software. Once we switch over to CTest/CDash as our official testing system, these will be dealt with and we will be all 'green' most days.
(0015827)
Julien Jomier   
2009-03-27 23:29   
Test 'not run' are now showing up in yellow and test 'failed' showing up in red.