[cmake-developers] [ANNOUNCE] cmake-unit - A new unit testing framework for CMake

Eric Wing ewmailing at gmail.com
Fri Jan 29 16:47:47 EST 2016


On 1/29/16, Brad King <brad.king at kitware.com> wrote:
> On 01/25/2016 11:01 AM, Sam Spilsbury wrote:
>> Over the last two years I've been working on a new unit testing
>> framework for CMake. I started this project because I've been writing
>> some other fairly complex CMake modules and I kept making mistakes
>> which would take me hours to debug weeks down the line.
>>
>> The project is, rather unimaginatively, called cmake-unit. The key
>> difference between this and the RunCMake framework that's used by
>> CMake internally is that cmake-unit more closely resembles an xUnit
>> architecture. It has support for a generic assert_that function and
>> matchers, test-case autodiscovery, coverage report generation and
>> allows you to keep all your tests together in a single file. The only
>> downside so far is that it runs a fair bit slower than RunCMake does.
>
> Nice.  I think if something like this were to be provided by upstream
> CMake it would be better to implement it directly in C++, at least for
> the test execution and output matching parts.  That would be much faster
> than a pure CMake language implementation and also would not need hacks
> to execute CMake language functions.
>
> One of the weaknesses of RunCMake infrastructure is that the CMake regex
> engine is not meant for large expressions.  For example, the number of
> ()-groups is limited to 10 or so.  It is also not as expressive as
> modern regex implementations offer.  There has been discussion about
> replacing the regex engine previously:
>
>  https://cmake.org/pipermail/cmake-developers/2011-November/014249.html
>
> but there are major compatibility challenges:
>
>  https://cmake.org/pipermail/cmake-developers/2011-November/014376.html
>
> OTOH in a brand new interface we could use an alternate regex engine
> from the beginning.
>
> -Brad
>

Just a random thought on the regex issue since we were talking about
Lua in the other thread. Not saying this is the way to go, but
something worth knowing about.

Lua has a library called LPeg, which was created by the Lua authors
themselves. It is basically academically rigorous rethink of the regex
problem.

There is a good presentation of it here:
https://vimeo.com/1485123

But in a nutshell, "RegEx" (distinct for Computer Science notion of
regular expressions) are a hack because they leave the domain of CS
and what can be understood/proven about computability. This results in
implementation dependent, unpredictable performance costs for RegEx
with no mathematical grounding. And a lot of the features bolted on to
RegEx have odd behaviors and limitations that can be
surprising/frustrating. Lua thought about this problem not only
because the lack of elegance, but also because PCRE implementations
are huge...much larger than Lua itself.

So they turned to old research on something called PEGs (Parseable
Expression Grammars). They are akin to Context Free Grammars...drawing
the analogy that CFGs are one up the computability ladder from regular
expressions. This allows PEGs to capture more complicated/expressive
things that RegEx has been abused to try to handle. (Some examples are
in the video.)

I'm by no means an LPeg expert, but I had one project where I had to
do Android package name validation, which follows the rules of Java
language package names. LPeg comes with a convenience layer called
"re" (which might stand for regular expression...haven't thought if
they mean in the 'pure' sense), which allows you to write your
patterns in something that resembles BNF grammars. It was a real
delight because suddenly everything was really clear and readable. (I
have a Perl background so I'm very used to crazy RegEx.)

Anyway, I figure one reason CMake doesn't already have a full blown
PCRE engine is because of the size and complexity. (I know SWIG
doesn't include PCRE for some reason and encourages you to download it
separately and conditionally compiles (disabling features) based on
whether you have it or not.) So this might be another be another angle
to look at if a Lua effort is serious. (Timing will be a problem
because I suspect the Lua thing is pretty far off. But this is another
angle to consider about what we would want a Lua engine to do for
CMake.) Last I checked, my built LPeg binary was under 50k.


More information about the cmake-developers mailing list