[cmake-developers] CMake alternative language (was: Using CMake as a library from Python)

[Brad King]

Hi Charles,

Thanks for your efforts in exploring this topic.  CMake's current language
grew incrementally out of something that was not originally intended as a
programming language.  The cmState refactoring Stephen Kelly has started
is a huge step toward enabling alternative languages, but there is a long
way to go.

A few general thoughts:

* One rule we have for CMake is to never expose any public SDK of the C++
  implementation structures.  We want to be able to rewrite them arbitrarily
  at any time.  Therefore any solution that needs to access the C++
  structures must be integrated into CMake upstream and expose functionality
  only through other languages or file formats.

* The cmState infrastructure builds on a "snapshot" design with a goal of
  being able to "fork" configuration/generation temporarily and then revert
  back, and to be able to re-start configuration from the middle.  These
  goals may be incompatible with any language whose implementation we do
  not fully control unless it is allowed to execute only in isolated and
  independent snippets.  These are not hard goals, but it is a trade-off
  to keep in mind.  Stephen may be able to elaborate more on the snapshot
  approach if needed.

* A problem with the current design is that the entire configuration process
  is logically serial making parallel evaluation hard or impossible.  In
  many cases each add_subdirectory can be processed independently, but this
  will require semantic changes to allow.

On 01/04/2016 02:41 AM, Charles Huet wrote:
> I'm trying to be as declarative as possible, because really like how readable
> simple QML programs are, and I think it would be perfect for a buildsystem.

Ideally most of the specification (sources, libraries, executables, etc.)
should be in a pure format that can be evaluated without side effects (e.g.
declarative or functional).  This rules out both Python and Lua, but the
specification format does not have to be the main entry point.  There could
be some imperative configuration step that does system introspection and
then loads the pure specification and evaluates it as needed for the specific
environment.

If we're going to go through the effort to provide an alternative input format,
I think we should strive for this approach because it will be more flexible in
the long run.  A pure specification format will allow easy loading/saving by
other tools, IDEs, etc., without having to process any imperative logic.

> Actually, I'm directly using the cmMakefile, because I did not want to wrap all
> the commands, and it seemed backwards to me to wrap them.

Yes.  Any alternative format should be processed directly into the structures
used by the generators.  The cmState work has separated the generate-time
representation quite a bit from the configuration-time (cmake-language-specific)
representation, but I think there is still further work needed to finish that.

>> Having said all that, Brad favors Lua I believe, and he favors a different
>> approach (which no one is working on as far as I know) to adding a new
>> language. So wait to hear from him to know whether it is something that
>> would be upstreamable.
> 
> Have any details on the approach in question?

See above.  Lua has come up several times in the past in particular because
its implementation is meant to be small and embeddable.  I've thought a few
times about how to make Lua scripting available from within the CMake language
in a clean way, but that will not be as valuable as the above pure-spec approach.

> Here is what my test POC looked like for generating a simple shared library:
> 
>     #!/usr/bin/env python
>     # -*- coding: utf-8 -*-
>     importcmake
>     cmake.init("Ninja","/media/dev/src/cmaketest","/media/dev/build/cmaketest")
>     myProject=cmake.Project("MyTestProject")
>     myProject.targets=[cmake.SharedLibrary("testLibrary",["lib.cxx"])]
>     cmake.generate()

I do not think we should have the build specification depend on processing
code like this.  It is not compatible with cmake-gui where the configuration
and generation steps are triggered by the user.  However, this does serve
as a good POC that we can populate the generator structures with another
language.

In summary, I think work in this direction should first focus on designing
a declarative (or functional) specification format where most of the project
information can be specified.  Then a cmake-language command can be written
to load and evaluate a specification file (as a transition).  Finally we
could look at replacing the entry-point language with something else.  At
that point we could have closures passed as parameters to the evaluation of
the pure spec in order to get custom generate-time logic.

Thanks,
-Brad