[cmake-developers] CMake alternative language

[Eric Wing]

Just to answer some of your questions on this...



>
> I think the first step is design brainstorming and discussion to see if
> we can find something acceptable.
>
> IIRC one problem with earlier attempts at introducing Lua was that CMake's
> dynamic scoping is difficult to reconcile with Lua's lexical scoping.  Lua
> cannot be given direct access to CMake variable scopes because Lua data
> structures may outlive the CMake dynamic scopes, leading to dangling
> references.

I don't have a good solution for this. I think the original CMake/Lua
bridge was basically a 1-to-1 API mapping with the CMake API. So the
idea was anything you do in native Lua lives in your Lua state
following the standard Lua lifecycles of things, but communicating
with CMake required going through the CMake public API (via
functions/macros via Lua bindings to the API). For global variables, I
think there was a special Lua table for CMake globals which had a
metamethod which did something similar to set the CMake values.

Thus, I can't remember if there were any dangling data structures. I
think about ADD_LIBRARY or ADD_EXECUTABLE which essentially returns a
TARGET object which has a lifecycle. When returned to Lua, we wrap it
in a userdata. The wrapper could lose its scope and the wrapper could
get garbage collected. But the backend doesn't need to actually
collect or destroy the object so CMake can still manage the resource
itself.

An aside though, I do find the scoping and lifetime rules of CMake
very difficult to work with. I feel like I still don't have a good
intuition or understanding of when many things happen in CMake. Since
we use CMake to compile C/C++/etc which are lexical scoping, I find
this to be challenging, especially since I don't really want to be
focusing on details like this when I'm just trying to get my program
compiled. So figuring out how to better embrace (Lua's) lexical
scoping moving forward might be a win.


> I'd also prefer an approach that allows us to offer newer versions of Lua
> as they come out while still processing old scripts unchanged.  This means
> that all Lua code must lie in a context where CMake knows the version of
> Lua for which it was written.  I'd like to retain the possibility of
> supporting multiple Lua versions to allow future such version transitions.
> (Implementation symbols can be mangled to support multiple versions linked
> in one program.)

This should be possible, though it's not a topic I've carefully
followed myself. I think simply mangling all the API functions
differently should prevent any conflicts. (I don't think Lua has any
global/static shared state internally which avoids those problems.)

I should also point out though that if you are going to support
multiple simultaneous versions of Lua, things could get messy if the
user needs Lua state preserved across multiple calls (i.e. not a
purely functional design). So what happens if the user starts in Lua
5.3 and creates some data, but then calls a script that only works in
Lua 5.1. The VMs can't be mixed.


>
>   cmake_lua(<version> [LOCAL <local>...] CODE <code>...)
>
> The <version> must be "5.2", the Lua language version.

One thing to keep in mind is that for at least Lua 5, most of the
scripting side changes have been pretty minor. Most of the time,
script authors figure out how to write a script that runs across all
versions of Lua 5. So specifying a list of supported versions may be
desirable.




>   number: value is interpreted as a decimal representation of
>           a floating point value to be stored in a Lua number.

One interesting introduction to Lua 5.3 (already out) is integer
subtypes in Lua, so it now has separate float and integer types if
useful.


>
> set(deg 90)
> cmake_lua(5.2 LOCAL number{deg} CODE [[
>   return { sin = math.sin(deg*math.pi) }]])
> message("sin(${deg}') = ${sin}')
>

So I personally imagined writing much bigger chunks in Lua directly
and directly invoking CMake APIs (via bindings, sort of like the
prototype). Not that I disagree with anything here. But I think for my
use case, the time I need Lua the most is for complicated code that is
really unwieldily in the native CMake language, not for little things.

For example, one of the worst parts of CMake code I have right now in
my current project is I need to track all the resource files
(separated into types like images, scripts, dynamic libraries,
plugins) in my project and retain the relative directory hierarchy for
everything so I can reconstruct them in the application
packaging/bundling process (a custom step I have as part of the build)
preserving the layout (but with platform aware adjustments as needed).
Basically, I wanted multi-dimensioned tables, but I don't get those in
CMake, so I'm doing really painful things with global strings and
appending key names to strings to make new unique string lists. It is
a beast of a system right now because so many things have to use it
and the key names are often done with variables there are a lot of
nested variable expansions which make the code hard to read. And this
data has to persist so I can set and get at different parts of the
build cycle.



> I'm raising the declarative approach here because it is an important
> consideration.  See the "CMake daemon for user tools" thread for
> discussion of how a declarative spec would greatly improve integration
> with IDEs and other tools:
>
>  https://cmake.org/pipermail/cmake-developers/2016-January/027413.html
>
> A declarative spec allows tools to load and work with it easily.
> A JSON document with a schema is easy to load, edit, validate, and
> save, for example.  A procedural specification must be executed to be
> interpreted, and only humans can really edit it.
>

Yes, I agree that IDE interoperability would be great and declarative
data is much nicer.
I have no problem with JSON, though consider the pure Lua table too.
It is basically the same thing (discovered before JSON), but if Lua is
going to be the native language, then certain things become very
trivial, fast, and efficient if you use Lua tables (since you can just
call dostring/dofile).


My historic difficulty with declarative build systems has been that
they break down for things that are conceptually very simple, like
your very first conditional. For example, I need to include FileA for
Platforms X, Y, but FileB for Z. I remember in Ant, just trying to
write the XML for this was really painful. Then the conditionals
change over time and get fancier because you introduce Platform W
which might need FileC+FileA or B or have some other precondition.
Assuming you could have some kind of control flow, you are still
ultimately explicitly writing every permutation which becomes
unwieldily for other reasons (maybe Lua nested tables and references
could help not make this as bad).


Maybe the answer is that the user never writes the declarative parts,
but CMake then generates the flattened/evaluated declarative spec. But
that prevents the two-way IDE interoperability which sucks and we're
back to where we started.