[cmake-developers] Generating buildsystem metadata from CMake

[Stephen Kelly]

Tobias Hunger wrote:

> Hi Stephen,
> 
> sorry for being late to the party again:-)
> 
> I am just reading your documentation.

Great, thanks for all this. Did you also read the unit test and generated 
file?

I'm working on updating my branch based on all of the feedback in this 
thread so far.

> 
> Some nits:
> 
> * In the "Introduction" you say CMAKE_GENERATE_METADATA needs to be
> ON, in "Generating Metadata" it needs to be set to a version number.

Thanks, fixed locally.

> * In "Target Properties/Conditional Properties", "backtrace": Which
> frame is the current one? The top one or the bottom one? I've seen
> both approaches in the wild:-)

Documented as 'most recent first' locally.

> * In "Target Properties/Conditional Properties and optional
> properties": Would it make sense to have a list of target_files, each
> with a filepath and a type? That would be more similar to the targets
> which also are a list with name/type. 

What semantics would the 'type' have? I don't really understand what you 
propose here.

> Are these paths in the build or in the install location?

These paths are all absolute - they are most likely either in the source or 
build location. The user is free to specify any local file though of course.

> * In "Metadata Contents/Optional Properties" you propose to have
> several language specific properties. Would it make sense to group
> those together into one "language" property?

> * In "Target Properties/Conditional Optional Properties": Again,
> wouldn't a set of sources arrays with a list of filepaths and all the
> related meta information for this set of files be nicer? That way all
> the "language fallback properties" and "source file properties" would
> be in one place.
> 
>   Somewhat like this (going with JSON-ish syntax here for ease of
>   writing):
> 
>    sources: [ {
>        language: "C++",
>        type: sources,
>        defines: "FOO=1",
>        include_paths: "/usr/include",
>        files: [ "/full/path/a.cpp", "/full/path/b.cpp" ]
>    },
>    {
>        language: "C++",
>        type: headers,
>        files: [ "/full/path/a.h" ]
>    },
>    {
>        language: "C++",
>        type: generated,
>        files: [ "/full/path/config.h" ]
>    }]

>> * I propose generating language-specific source lists, because CMake can
>>   build files with compilers which do not match their file extension.
> 
> Makes a lot of sense, even though I am not too happy with the syntax.
> See the nits above. 

Yes, there are clearly many options for how to organize this information. 
I'll see about gathering them and working what code using each different 
option would look like.

>   This can express defines set for individual files, which I think you
> can have in CMake, can't you?

Yep.

> I do not see how to express that within your proposal.

See the 

 Tests/Metadata/cmake-metadata-Linux-GNU-4.9.json 

file I checked into the repo, which contains a sources property for 
specifying those source-specific properties.

> The rest of my replies are inline:
> 
> On Wed, Mar 11, 2015 at 11:10 AM, Stephen Kelly
> <steveire at gmail.com> wrote: <snip>
>> I expect to require a few iterations to figure out what the metadata
>> files
>> should contain in the end.  Note that there are already some differences
>> between my design and Aleix's implementation, such as that my design
>> proposes one metadata file per config. There are also some things
>> missing like location, because it is not yet clear to me whether build
>> or install locations are needed etc.
> 
> These configurations are only relevant to generators that can support
> e.g. having debug and release builds in the same build directory or
> did I misunderstand this?

Yes, your understanding is correct.

> How are we supposed to handle a set of files for these configurations?

Yes, that's how I see it working...

> How can we notice when one gets added and removed?

I'm not experienced enough with those generators to know. I'm not even 
certain whether it is possible to add to the list of configurations for 
those generators, or if the list is hardcoded.

Something to look into or get information from someone else on this list if 
they know.

I think in the worst case, you'd watch the CMakeCache.txt and read the 
CMAKE_CONFIGURATION_TYPES from there. I know you want to care about only one 
file, but maybe two is not so bad :), and there is potentially other stuff 
in the cache which might be useful, and which might not lend itself to the 
json metadata file. We'll have to see.

> Having everything in one file would probably be easier, as that is
> just one file what needs watching:-) Could we maybe just tag the
> targets with the configurations they apply to and maybe also add the
> same tag to the source file groups proposed above?

That may be an option too. I wrote the solution with multiple files because 
we need to specify different 'object sources', 'other sources', 'header 
sources' and 'excluded sources' for each config, and only one config at a 
time is relevant.

>> The content of the metadata file is determined by the build properties,
>> and is necessarily similar to the compile-related content created when
>> generating the actual buildsystem.  It additionally contains information
>> about the output locations of build artifacts and information relating to
>> the cmake description itself.
> 
> Output location of build artifacts: That is the location inside the
> build directory, isn't it? What about install locations?

I deliberately used a vague term 'output location', because it's not clear 
to me which locations we should generate in the metadata file (and why).

>> * Make it more easy for an IDE to support actions such as 'remove file
>>   from the project', which requires removing it from the CMakeLists.txt
>>   at the appropriate place, and 'add new file/class to target', which
>>   involves adding code to the CMakeLists.txt file at the appropriate
>>   place.  Most likely the easiest way to do the latter is using the
>>   target_sources() command, and to support the former, the location of
>>   the declaration of the target, and all target_sources() calls would
>>   need to be recorded.  Even that is not enough because of transitive
>>   consumption of source files through the link interface, but that is
>>   likely irrelevant.
> 
> That would be the icing on the cake, but getting a full description of
> the project is what I care for at this point.

Can you expand on what 'a full description of the project' means to you? Do 
you mean something different than 'how each object is compiled'?

> I do not care too much on the format, but I am strongly in favor of
> anything that I can conveniently parse with Qt. 

Yes, that needs to be a requirement, given the clients we're talking about 
at this point.

> So XML and JSON are thus both fine with me:-)
> 
> I would prefer JSON, simply because I find that way more human-readable.

I think it's fine too. I've always been apprehensive about it because it 
seems uncommon for people to use a schema with it because it's 'easier' that 
way. But jsonschema seems to be everything we need.

>> * I didn't document the location or directory.  I'm not clear on whether
>>   it is supposed to be the build location, or the install location(s!),
>>   or all of those.
> 
> I see four options to place this file:

This is a misunderstanding. The 'location or directory' I referred to was 
'the location or directory of the linked binary target'. The metadata file 
goes in the build directory.

>> * I don't generate 'dependencies' (actually the list of files which the
>>   buildsystem re-generation depends on) as Aleix did, because there is no
>>   well-defined usefulness for that list yet.
> 
> How is the IDE supposed to know that it needs to re-run cmake then? If
> a file changes that may very well change the list of files that the
> IDE needs to display, so re-running cmake only at the next build is
> not going to work.

Ok, I didn't realize that's the scenario we're talking about for that 
content. Are you saying that you would 

  Set up a filesystem watcher for each of the (very many) files.
  If one of the files changes:
    Run `cmake .` in the build dir
  If the metadata file changes:
    Reload the data it contains

Is it a problem if the value of 'very many' above is in the hundreds?

>> * Some more information from project() may be relevant, but it's not
>> clear
>>   yet. We will likely know more when we have decided the file format and
>>   generated some 'interesting' metadata files.
> 
> That is definitely needed! What is this file about(full path to top
> level CMakeLists.txt that was used to generate this build directory as
> well as the configuration that  was used!) is essential to answer:-)

As it happens, the project() command doesn't really concern either of these 
pieces of information, but that's just implementation detail.

> I would love to have something like ccmake built into Qt Creator at
> some point, and having this information would spare me poking around
> CMakeCache.txt and whatnot:-)

If you mean you would like a fully-featured cache editor in Creator, then 
reading CMakeCache.txt is unavoidable, but you'd probably fork some of the 
Source/QtDialog code as a starting point.

> For that to work information on project this build directory is for as
> well as which Qt/compiler/whatnot was used in the project is essential
> for Creator to properly configure the code model and the rest of
> Creator.

For CMake, there's nothing particularly special about Qt - we'd probably 
also have to provide information about which Boost etc, but actually that 
information is in the CMakeCache.txt. We might be better off creating a 
stable interface to reading that file somehow. Eg

 $ cmake -E read_cache Qt5Core_DIR Boost_ROOT KF5Archive_DIR ZLIB_LIBRARY
 {
   "Qt5Core_DIR": "/opt/qt5/...",
   "BOOST_ROOT": "/opt/boost/1.58/...",
   "KF5Archive_DIR": "/opt/kf5/karchive/...",
   "ZLIB_LIBRARY": "/opt/zlib/..."
 }


As for 'when do I re-run it?' - the answer is whenever the CMakeCache.txt is 
changed.

> Having the full configuration available would also remove the need to
> have special flags in the "Metadata Contents" section for the compiler
> used.

I don't know what you mean. You mean '-g', '-O3' etc? Or are you referring 
to what I wrote about the pattern of the compile command line, and the order 
of where includes definitions and compile flags appear?

> 
>> * We might need backtraces for not only all add_library and
>> add_executable
>>   calls, but also all target_sources() calls.
> 
> I do not see any obvious use for a full backtrace, but then it is
> probably interesting to have to generate the build system structure we
> display in the projects view in Qt Creator...

In the unit test, I show that things like loop controls and functions/macros 
appear in the backtraces. Those don't seem useful to the idea you describe 
here. In fact they'd probably get in the way.

> Please keep it:-)
> 
> Apropos project view: IIRC cmake allows to define lists of files to be
> displayed together in visual studio, doesn't it? Would it be possible
> to make this data available to other IDEs, too?

Yes, probably. I don't have any first hand experience with what people use 
that for. The command is source_group() I think.

> 
>> * I propose to generate one metadata file per configuration active in the
>>   buildsystem.  That means that the Makefile and Ninja generators would
>>   generate only one file, and the multi-config generators would generate
>>   multiple files.  When changing the 'active configuration', IDEs would
>>   then read whatever metadata file is relevant to the active
>>   configuration.
> 
> How can I get the list of files that are relevant? 

If I understand you correctly, the answer is the union of the source groups 
I defined in the file for a particular config (eg object sources, header 
sources, extra sources, excluded sources).

All files (listed declaratively as in the unit test I wrote) appear in all 
config-specific metadata files, but might be specified in a different group 
(ie, usually either in 'object sources' or in 'excluded sources'). So, at 
least for the purpose of listing source files, there is no need for a master 
file.

> Will there be one
> master file that lists all the available configurations and their
> configuration description file?

I don't think there needs to be, unless you count reading the CMakeCache.txt 
CMAKE_CONFIGURATION_TYPES cache value as that... 

>> * I propose generating language-specific compile properties in the
>> metadata
>>   file if there a language specific variant groups.  For example if the
>>   compile options themselves are language-specific (the feature enabling
>>   that was merged to CMake master this week), then a property for each
>>   language
>>   is specified.  Otherwise only a language-agnostic property is
>>   generated. This can vary per-configuration for a multi-config
>>   generator, so each metadata file for a multi-config generator could
>>   have different properties present. Consumers read the language specific
>>   property if it is present, and read the language agnostic version
>>   otherwise.
> 
> I do not fully get where you are going here.

If you are dealing with a cxx file and there is a "cxx_compile_options" 
property, then you read that and you're done. Otherwise, you read 
"compile_options".

> 
>> * Generating metadata only (without generating buildsystem files) is not
>>   currently in scope.  This was requested several times, but it is not
>>   clear why.
> 
> Not sure... When opening a CMakeLists.txt, ideally I would ask the
> user for a build directory and a cmake generator to be used and run
> cmake once.

Yes, so far this is possible, and doesn't carry a requirement to not 
generate an actual buildsystem...

> Afterwards I would have a project description that I can generate a
> ccmake-like config UI from. If the user changes that I run cmake again
> and afterwards the project is fully set up and documented in the --
> now updated -- project description file.
> 
> Afterwards I will need to regenerate the configuration description
> each time something changed. That should be as fast as possible, as
> the data I display in the UI is stale at that point. If not generating
> the complete build-system makes this step faster, then I am all for
> that:-) OTOH, it probably does not matter too much if the description
> is created first and the build system is updated only afterwards in
> the background.

Yes, if speed is the concern, then we'll have to see what the gains are of 
not generating the buildsystem (probably much more significant on Windows 
actually).

> Unrelated question: Can I query the available generators in a
> machine-readable format with user friendly names and their name from
> cmake somehow?

Not yet. We can add that though with something like 

 $ cmake -E list_generators
 {
   "generators": ["Unix Makefiles", "Ninja", "Xcode"]
 }

etc.

> 
>> * How much information does tooling need about installation?  Targets
>>   can use different include directories and compile definitions in their
>>   install locations compared to their build locations.  If IDEs want to
>>   provide some user interface related to the project files in their
>>   install location, perhaps a separate solution based on cmExportFile*
>>   is needed.  For future investigation.
> 
> Why is that necessary?

I have no idea. The topic of installed location (of the targets being 
developed! Not of dependencies!) was raised in the other thread, but the 
actual requirements were not clear to me, so I added 'find out what is 
needed and why' to the todo list :).

> My goal is to have solid support for stand-alone cmake-based projects
> in Creator. If such projects reference installed libraries I would
> expect that to be reflected in the compiler and linker flags.

Yes, it is.

> Why
> would that need special support for include directories and compile
> definitions?
> 
> Creator needs to know where all files will end up on install since we
> might need to copy them to a remote machine (e.g. a phone, or a Linux
> box) to run/debug there. Installing everything with some temporary
> directory prefixed to all paths does work, though. We can then rsync
> the temporary directory to a device. But even then we will still need
> to know where the executables end up on the device to run/debug.

The CMAKE_STAGING_PREFIX is designed for this purpose. You would specify it 
when running cmake and copy the files from there.

 http://www.cmake.org/cmake/help/git-next/variable/CMAKE_STAGING_PREFIX.html
 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/8363/focus=8629
 
It's basically the same as extprefix in the Qt configure script.

> PS: I hope there is not too much redundant stuff in here that is
> already discussed in the follow up mails I only skimmed over so far:-)

It's all good thanks!

Steve.