[cmake-developers] Tests with custom launchers and "Not Run"

[Mathias Gaunard]

On 09/18/2012 11:37 PM, David Cole wrote:
> I don't like it. Existing tests that run and return, for example, a
> number of errors that occurred, will magically appear as "not run" when
> that number just so happens to be 77.
>
> If there are enough people who think this is "simple and works" and are
> not concerned about the accidental matching of an intentional return
> value of 77 that does NOT mean "not run" ... then I will relent and say,
> so be it, and allow it in.
>
> But only if there are some people who speak up here or add notes to the bug.
>
> It just seems wrong to me to treat 77 as some special number here.

Well, you need to interpret a return value as a special value to be able 
to tell executables could not be tested.

Without this, a whole category status is entirely unusable to whoever 
wants to use custom launchers! That's a pretty serious issue to me.

77 already exists in the autotools world, so it seems like a good idea 
to re-use it.

Making the executable return the number of failing tests is a very bad 
idea: it doesn't even scale beyond 255 for the average operating system. 
Certain operating systems also have funny behaviour with exit status as 
well: with OpenVMS, all even values are considered failures and all odd 
values are considered successes.