[cmake-developers] CTest: Request review of topic "ctest-fix-run-serial"

[David Cole]

> I just realized that CTest only prioritized failed tests in parallel
> test runs since it only modified COSTs in that case. So Casey was 
quite
> right noticing that there is a behavior change here.
>
> I created a topic "restore-serial-failed-test-order" which limits 
that
> behavior to parallel test runs.
> Should anyone prefer that behavior I could merge it to next.

I think that would be wise for backwards compatibility. The recent 
email discussions about CMake's /STACK argument change without a policy 
highlight the importance of this.


> Pro/con the current behavior in next:
> + Consistent sorting for serial and parallel test runs
> - There may be users who expect a fixed order of tests in serial runs
> - There may be tests in CMake that expect a fixed order of tests in
> serial runs(?)
> + When bug fixing or implementing (in case of TDD) the previously 
failed
> tests are those that a user is waiting for the most so it may make 
sense
> to have them run first (I assume that is why it was implemented for
> parallel testing)

It's not the tests in CMake that we have to worry about. We can always 
adapt the test suite of CMake to account for new behavior if necessary.

It's the tests in all the other projects in the world, that have not 
taken advantage of parallel testing yet that you will have to be 
concerned about. (For a notable example, the CDash project cannot use 
parallel testing. It's tests are highly dependent on order, and can 
only be done serially as the tests in its suite do not use the DEPENDS 
property to specify order, they rely on the default ordering. You can 
verify this by running a CDash dashboard with any proposed build of 
CMake/ctest...)

In serial testing, the default ordering of the tests should match the 
order of the add_test calls encountered during CMakeLists processing.


HTH,
David C.