[cmake-developers] Generating buildsystem metadata from CMake

Stephen Kelly steveire at gmail.com
Wed Mar 11 06:10:30 EDT 2015


Hi,

Following from the thread here:

 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12394
 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12650

I'm starting to gather requirements and make sure the feature is
well designed to satisfy the needs we're already aware of, and fits with
the features CMake currently has.

The aim is to generate a structured file containing metadata relating the
buildsystem.

To help with completing the design of this feature, I've written
documentation (documentation driven design), and a unit test
containing a CMakeLists.txt file which exercises many modern CMake
features in the "generate-metadata" branch in my clone.

Mostly the design I propose can be read in the documentation I wrote. I'm
interested in any feedback.

 https://gitorious.org/cmake/steveires-cmake/source/generate-metadata:Tests/Metadata/CMakeLists.txt
 http://www.steveire.com/cmake-future/manual/cmake-metadata-generation.7.html

I expect to require a few iterations to figure out what the metadata files
should contain in the end.  Note that there are already some differences
between my design and Aleix's implementation, such as that my design
proposes one metadata file per config. There are also some things
missing like location, because it is not yet clear to me whether build
or install locations are needed etc.

The content of the metadata file is determined by the build properties, and
is necessarily similar to the compile-related content created when
generating the actual buildsystem.  It additionally contains information
about the output locations of build artifacts and information relating to
the cmake description itself.

Goals include:

* Make it possible for IDEs to access the compile-related information for
  autocompletion and code navigation etc purposes.

* Remove the need for IDEs to parse generated Makefiles or Ninja files to
  access compile-related information.  The structure of those files is not
  'stable', while the content of the metadata file is stable.
    http://thread.gmane.org/gmane.comp.programming.tools.cmake.user/48412

http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11081

* Remove the need for users to create a new build directory and new build
  in order to use or switch IDEs.  QtCreator requires that
  the C::B 'extra generator is used as it parses compile information from
  that.  Other 'extra generators' such as for eclipse, sublime, kate etc
  also require fresh/new build directories, although the actual buildsystem
  they create is identical (assuming using all Makefile based or
  all Ninja based 'extra generators')

* Make it possible to write a plugin for the editors/IDEs such as sublime
  which consumes the metadata file and invokes the build using whatever
  buildsystem the user already has a build directory for, instead of
  writing an 'extra generator' and maintaining it in the cmake repo.
    http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/9004

* Make it possible for editors/IDEs to allow specifying the configuration
  at build-time, where the IDE has that feature, and where a multi-config
  generator is available.  That is, QtCreator provides user-interface for
  choosing debug/release config to build.  Currently it can't offer that
  when using cmake, because it only allows the use of Makefile or Ninja
  generators, in order to make use of the C::B file.  QtCreator should be
  able to use the Xcode or Visual Studio generators, generate the metadata
  file(s), and invoke `cmake --build . --config ${CONFIG}` as
  appropriate.  Eclipse, Sublime and other editors have similar abilities
  to invoke config-specific builds after generation.

* Provide a list of 'build targets', which can be listed and invoked in
  IDE/editor user interface.  Build targets for all linked binaries
  and utilties are provided.  The tooling is expected to perform filtering
  on the target types to show only executables and utilities for
  execution, for example.

* Provide a list of source files per target per type of source file, eg
  object sources, header files, generated files, files excluded from the
  active configuration/platform/compiler, non-compiled files.

* Make it more easy for an IDE to support actions such as 'remove file
  from the project', which requires removing it from the CMakeLists.txt
  at the appropriate place, and 'add new file/class to target', which
  involves adding code to the CMakeLists.txt file at the appropriate
  place.  Most likely the easiest way to do the latter is using the
  target_sources() command, and to support the former, the location of
  the declaration of the target, and all target_sources() calls would
  need to be recorded.  Even that is not enough because of transitive
  consumption of source files through the link interface, but that is
  likely irrelevant.

* Provide information about the entire build graph of link-dependencies
  for visulization and dependency analysis
    http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/4042

* Provide enough information about runtime link dependencies for IDEs to
  be able to properly invoke targets with debuggers, profilers and other
  tools.

http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11011


The CMAKE_EXPORT_COMPILE_COMMANDS variable already exists to generate
compile command lines, but that is a file whose format is defined by
Clang, not by CMake, and other tools in the Clang ecosystem also support
it.  It is not extensible or versioned in the way that a CMake-defined
file format is, so I don't propose using it as a starting point.

Notes:

* I deliberately didn't push an implementation to my repo, so that this
  discussion can focus on the design of the feature and contents of the
  file first.

* In the documentation and the test I didn't mention that the metadata file
  is JSON.  I'm not opposed to it, but I think there should be some kind of
  schema defining how to read the metadata file.  If we go with JSON
(instead of xml, say), then we'd have to define our own schema format, which
  is not impossible.  My proposal is to generate the schema beside the metadata
  file instead of writing a metadata version into the metadata file
itself.  Does
  anyone feel strongly about the file being JSON?  I just want to record the
  reason for everything in this design phase - we don't have to spend a lot
  of time on it.

* I didn't document the location or directory.  I'm not clear on whether
  it is supposed to be the build location, or the install location(s!),
  or all of those.

* I don't generate 'dependencies' (actually the list of files which the
  buildsystem re-generation depends on) as Aleix did, because there is no
  well-defined usefulness for that list yet.

* I documented the 'name' for the targets, and the TARGET_FILE_NAME
  etc information, and can push an implementation which generates them.

http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12619

* If it is useful to preprocess/compile/assemble individual files from
  IDEs, as made possible by the Makefiles and Ninja generators, we'll need
  to design that.

http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=12429

* Some more information from project() may be relevant, but it's not clear
  yet. We will likely know more when we have decided the file format and
  generated some 'interesting' metadata files.

* We might need backtraces for not only all add_library and add_executable
  calls, but also all target_sources() calls.

* We list the Generator used to configure the build.  That way IDEs know
  which build output parser to use.

* I propose to generate one metadata file per configuration active in the
  buildsystem.  That means that the Makefile and Ninja generators would
  generate only one file, and the multi-config generators would generate
  multiple files.  When changing the 'active configuration', IDEs would
  then read whatever metadata file is relevant to the active configuration.

  This also means that conditions don't need to be added inside the metadata
  file for configurations in order to show files 'excluded' by being part of
  a non-active configuration.  The code implementing discovery of excluded
  files is in the generate-metadata branch in my clone.


http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10711/focus=11323
    "At least Qt Creator does not need information on which conditions to
     be met for a file to become part of the current build."

* I propose generating language-specific source lists, because CMake can
  build files with compilers which do not match their file extension.

* I propose generating language-specific compile properties in the metadata
  file if there a language specific variant groups.  For example if the
  compile options themselves are language-specific (the feature enabling that
  was merged to CMake master this week), then a property for each language
  is specified.  Otherwise only a language-agnostic property is generated. This
  can vary per-configuration for a multi-config generator, so each metadata
  file for a multi-config generator could have different properties present.
  Consumers read the language specific property if it is present, and read
  the language agnostic version otherwise.

* Generating metadata only (without generating buildsystem files) is not
  currently in scope.  This was requested several times, but it is not
  clear why.

* How much information does tooling need about installation?  Targets
  can use different include directories and compile definitions in their
  install locations compared to their build locations.  If IDEs want to
  provide some user interface related to the project files in their
  install location, perhaps a separate solution based on cmExportFile*
  is needed.  For future investigation.


Thanks,

Steve.


More information about the cmake-developers mailing list