[cmake-developers] Generating buildsystem metadata from CMake

[Stephen Kelly]

Tobias Hunger wrote:

>>>    sources: [ {
>>>        language: "C++",
>>>        type: sources,
>>>        defines: "FOO=1",
>>>        include_paths: "/usr/include",
>>>        files: [ "/full/path/a.cpp", "/full/path/b.cpp" ]
>>>    },
>>>    {
>>>        language: "C++",
>>>        type: headers,
>>>        files: [ "/full/path/a.h" ]
>>>    },
>>>    {
>>>        language: "C++",
>>>        type: generated,
>>>        files: [ "/full/path/config.h" ]
>>>    }]
>
> The type is used to distinguish between the different kinds of files.
> Your proposal uses different keys for all the different kinds of
> files, my is to use the same key for all sources and distinguish
> between them by giving a type.

Ok, I understand the proposal.

>> I think in the worst case, you'd watch the CMakeCache.txt and read the
>> CMAKE_CONFIGURATION_TYPES from there. I know you want to care about only
>> one file, but maybe two is not so bad :), and there is potentially other
>> stuff in the cache which might be useful, and which might not lend itself
>> to the json metadata file. We'll have to see.
>
> So we are back to parsing random files in internal formats again:-)

Well, or getting the information you need into the metadata.json or make 
cmake output it with a special command, as I suggested below.

> Could you add have a "configuration list" key in the top level of each
> file that lists all available configurations (including itself)? That
> might be the lowest overhead solution that does not require parsing
> extra files that I can think of right now.

For Ninja and Makefile generators, CMake does not necessarily know 'all' of 
the configurations. It only knows what the user specified as the 
CMAKE_BUILD_TYPE. I could have a CMakeLists.txt with a condition like:

 if (CMAKE_BUILD_TYPE STREQUAL Coverage)
   # Foo
 endif()

and I could document to users/developers that they use it for coverage.

CMake would not treat the above any differently to any other condition on 
any other variable. CMake can probably list its own built-in configurations, 
like Debug/Release/RelWithDebInfo/MinSizeRel, but that's not a generic 
solution. If you want though we can do that.

> I definitely will not hardcode cmake generator-A has configurations X
> and Y while generator-A supports configurations X, Z and D into the
> cmake plugin. I do prefer not to support configurations at all to
> doing that.

I think your position is reasonable. It is likely that you should consider 
Makefile and Ninja generators to be single-configuration, and we provide a 
way to give you the CMAKE_CONFIGURATION_TYPES for the multi-config 
generators.

> What we do there is to check the default project build location for
> folders that contain a build of the current sources.
>
> To do so we need to know which code was built in that directory

I get this: The JSON file in the build dir would contain the path to the 
source and that part of the problem is solved, right?

> and
> which kit (think set of compiler, Qt, some other settings) were used
> to in the built in that directory.

Is this hypothetical? Currently I can add a compiler location and a Qt 
location in QtCreator, and I can create a kit and give it a name and 
partcular compiler and Qt. However, the information I configure there and 
the kit that I choose is not passed to cmake when creator invokes that, 
right? CMake will do its own determination of the compiler to use, and will 
find Qt on its own, right? Unless the user manually sets those as arguments 
in the 'Run CMake' dialog page.

My point is that you have to pass CMake this information before it can pass 
it back to you :). I might be looking at an old creator which doesn't have a 
relevant feature here though?

Or are you thinking of scenarios such as 'the user creates a build outside 
of Creator and then tries to open it in creator'. Given that 'a kit' is 
something internal to Creator, there is no way CMake could tell you the kit 
in that scenario. Is that obvious or am I missing something? I think the 
best thing you can do there is find out which compiler was used, and what 
the value of Qt5Core_DIR is, and map that to things you know about available 
kits to find the match if there is one.

In other words, this again collapses to 'we need a way to read cache 
values'.

> We also need to figure out the
> exact configuration (parameters passed to the "configure" step of the
> build system) that was built there.

My first question is 'Why?'. Running 'cmake .' in the build dir re-uses the 
options which were passed to it before.

Aside from that, particular arguments can be retrieved from the cache. It 
seems that some stable interface to the cache is needed anyway. At most, 
this is just another reason to add a stable interface to it, right?

> Since this is a feature our qmake users love I would also like to make
> that possible with cmake. It would be great if I could get most
> answers from the json file.

I see. How do you determine which kit to use if you are imported an 
externally generated qmake build? Do you use heuristics like I described 
above?

>>   Set up a filesystem watcher for each of the (very many) files.
>>   If one of the files changes:
>>     Run `cmake .` in the build dir
>>   If the metadata file changes:
>>     Reload the data it contains
>
> Yes, that's the scenario.

In the 'one of the files changes' scenario, how do you imagine the change 
happened? Something the user did, either in creator, or a git checkout, or 
anything else?

>> Is it a problem if the value of 'very many' above is in the hundreds?
>
> I would probably trim down the list to those actually in the project
> or build directories, but I would prefer having too much information
> than too little:-)

I don't see any reason to watch anything in the build dir. It is 'owned' by 
cmake and the user has no business messing with any files there. 
Additionally, any messing around they do there would immediately be 
overwritten by Creator running in the background.

It seems that the relevant locations are 'within the source tree' and 'some 
external site of cmake files', such as the cmake installation itself, or 
cmake helper files shipped by a 'platform' like KDE.

What do you think? Are changes to files outside of the source tree relevant 
at all? Is the 'you update your system cmake' scenario important enough that 
Creator should immediately update the content it shows without user 
interaction?

FWIW I think the IDE should not do this 'regeneration in the background' for 
me when I edit files outside of the IDE. If I have Creator open in the 
background and then I go to the console and edit some files (possibly 
outside of the source dir of the project which is open in Creator), and save 
multiple times over a few seconds, I don't want Creator to be constantly re-
running cmake for me. Or I'm doing a 'rebase -i' or anything else which is 
not for Creator to worry about. I also wonder how cmake would react if I 
edited those files and I ran 'cmake .' myself, and Creator did the same 
thing at the same time. I don't know that cmake currently defends against 
that.

I only want the IDE to re-generate when I'm actually working in the IDE. If 
I edit any file and hit Ctrl+B, the IDE runs 'cmake --build .' and cmake 
regenerates if needed at that point. It's 'ok' if the project tree is 
technically 'stale' after I edit a file and before I hit Ctrl+B. It's 'ok' 
in the sense that the alternative is 'worse' as far as my opinion goes. You 
also have the option of running 'cmake .' any time any file is saved in the 
IDE, if that improves the experience, but I strongly believe you shouldn't 
be watching the files for outside changes and re-generating in response.

That's not to say we won't 'give you the rope'. But before we would do that, 
cmake would have to be defensive against concurrent runs in the same build 
dir.

> In addition I will probably have a "Run cmake" option in the menu to
> force the metadata to regenerate.

Yes, that's already there in the 'build' menu anyway.

>>> I would love to have something like ccmake built into Qt Creator at
>>> some point, and having this information would spare me poking around
>>> CMakeCache.txt and whatnot:-)
>>
>> If you mean you would like a fully-featured cache editor in Creator, then
>> reading CMakeCache.txt is unavoidable, but you'd probably fork some of
>> the Source/QtDialog code as a starting point.
>
> Is CMakeCache.txt considered to be a implementation detail of cmake,
> or is it expected that external applications will parse it?

As far as I know the file is an implementation detail. So, I guess if we add 
some stable interface for reading it, you wouldn't really have to read the 
CMakeCache.txt file directly and could 'write' it by invoking cmake with -D 
arguments.

>>  $ cmake -E read_cache Qt5Core_DIR Boost_ROOT KF5Archive_DIR ZLIB_LIBRARY
>>  {
>>    "Qt5Core_DIR": "/opt/qt5/...",
>>    "BOOST_ROOT": "/opt/boost/1.58/...",
>>    "KF5Archive_DIR": "/opt/kf5/karchive/...",
>>    "ZLIB_LIBRARY": "/opt/zlib/..."
>>  }
>
> Yes, that would be nice, but once you add "cmake -E list_cache" you
> could just as well just add the whole config into the json file and
> let its users worry whether or not to actually read the section.

When you wrote 'config' you meant 'cache key value pairs' or simply 'cache'.

The key difference is that my proposal is only requested values, so is 
minimal.

As you suggest, we could write all of the cache data into the generated 
cmake-metadata.json file. Alternatively, we could design the interface such 
that you specify not only what version of the metadata you want, but also 
which cache values

 $ cmake . -DCMAKE_GENERATE_METADATA=3.3 -DCMAKE_METADATA_CACHE="Foo;Bar"

to generate a cache section in the metadata file with the 'Foo' and 'Bar' 
key-value pairs.

Is there any particular reason you want "everything" instead of specifying 
what you want? You don't know how to understand 'everything'. Is it just 
because of the idea you wrote above about a cache editor within Creator? Why 
not just run cmake-gui as an external process?

Note also that nothing in the cache tells you the path to qmake for Qt 5 or 
anything like that. The Qt5Core_DIR is a path to 
`qt_installation/lib/cmake/Qt5Core`. Depending on how Qt was built, qmake 
may or may not be in `../../bin`. I don't know if that's relevant to 
Creator. At any rate, the content of the Qt5Core_DIR variable is what 
identifies Qt to CMake. qmake does not identify Qt 5 for CMake. However, if 
you run `qmake -query QT_INSTALL_LIBS` for each of your available Qts, you 
can map to the right one easily enough.

>>> Having the full configuration available would also remove the need to
>>> have special flags in the "Metadata Contents" section for the compiler
>>> used.
>>
>> I don't know what you mean. You mean '-g', '-O3' etc? Or are you
>> referring to what I wrote about the pattern of the compile command line,
>> and the order of where includes definitions and compile flags appear?
>
> You documented a couple of keys in the top level that deal with the
> compiler (See "Metadata Contents/Optional Properties", e.g.
> "<lang>_compiler").
>
> I will need more information than just the compiler used: The Qt
> version used is obviously important to us,  as are all settings
> related to cross-compilation.

By 'important to us', do you mean 'we provide interface allowing the user to 
edit the values, and we pass those values to CMake on behalf of the user'? 
Is that currently the case? If it is not currently the case, does the fact 
that the metadata file does not currently exist somehow block that? If so, 
how? That feature seems completely orthogonal to the metadata file to me. It 
could be added to Creator at any time if it is not there already.

> Ideally I would get away with just the json file (which would require
> copying all the configuration into it), but if I am required to parse
> CMakeCache.txt anyway, then I do not see the need to copy settings
> that are readily available there.

Right.

> Well, I am mostly interested in the tree of CMakeLists.txt files and I
> think I can re-generate that from the backtrace:-)

Right.

> Maybe we could have a tree of "source objects" in the targets stead of a
> list?

Maybe. I think it makes sense to leave behind discussion of the exact format 
and structure of the json file. It feels like we need to take a step back 
and gather more requirements.

>> Yes, so far this is possible, and doesn't carry a requirement to not
>> generate an actual buildsystem...
>
> I agree with Anton that it would be nice not to have to ask for a
> build directory and generator first thing.

Adding a way to make cmake dump the metadata to stdout would be possible, 
but it wouldn't help...

> Many users just want to
> browse some project and an IDE should support that use case as well as
> possible. Nagging those users about a build directory or generator is
> not the best user experience.

... CMake needs a directory to run in. It needs to generate files to compile 
to test the compiler features, determine the platform etc. The content of 
the project depends on the outcome of those configure-time tests. You need 
to give cmake a temp directory in this scenario *anyway*. A temp directory 
from your OS is fine, and you can read the metadata file from there.

This doesn't change the scope or requirements afaics.

>>  $ cmake -E list_generators
>>  {
>>    "generators": ["Unix Makefiles", "Ninja", "Xcode"]
>>  }
>>
>> etc.
>
> That would be *really* nice to have once this metadata file is in place:-)

It is also orthogonal to the metadata of the build itself and can be 
designed separately.

I filed

 http://public.kitware.com/Bug/view.php?id=15462

if you want to engage in the design or implementation of that.


>>> Creator needs to know where all files will end up on install since we
>>> might need to copy them to a remote machine (e.g. a phone, or a Linux
>>> box) to run/debug there. Installing everything with some temporary
>>> directory prefixed to all paths does work, though. We can then rsync
>>> the temporary directory to a device. But even then we will still need
>>> to know where the executables end up on the device to run/debug.
>>
>> The CMAKE_STAGING_PREFIX is designed for this purpose. You would specify
>> it when running cmake and copy the files from there.
>>
>>  http://www.cmake.org/cmake/help/git-next/variable/CMAKE_STAGING_PREFIX.html
>>  
http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/8363/focus=8629
>>
>> It's basically the same as extprefix in the Qt configure script.
>
> I am aware of that:-)
>
> Creator would still need to know where the files went that the user
> wants to run/debug though. So at least for the binaries a way to map
> them from the build to the install location is needed for our
> use-case.

Ok.

So, requirements gathered in this email:

 * Being able to read cache values is of primary importance
 * * Open question whether values should be 'on demand' or 'always all'
 * Knowing which CMake files in the source directory need to be watched for 
changes is very important.
 * Knowing which CMake files outside of the source directory need to be 
watched for changes is very important.

Major open question: How do we handle configs. Let's get to that next.

Thanks,

Steve.