[cmake-developers] Generating buildsystem metadata from CMake

[Stephen Kelly]

Anton Makeev wrote:

> 
>> On 01 Apr 2015, at 00:41, Stephen Kelly
>> <steveire at gmail.com> wrote:
>> 
>> Anton Makeev wrote:
>> 
>>> The problem with the following format:
>>> ---
>>> "include_directories" : ["/foo", "/opt"]
>>> "compile_definitions" : ["DEF=\"Foo\"", "OTHER_DEF=1"]
>>> "compile_flags": [ "-c" ]
>>> —
>>> Is that it’s incomplete and cannot be used directly:
>>> * include directories list misses compiler-defined search paths
>> 
>> CMake gathers those during configure time, and we can add them to the
>> metadata too.
> 
> However appealing it is, I don’t think it’s a good option,

Ok, I understand now, because the used built-in defines are different more 
often than the built-in include directories (-iquote and -I- are rarely used 
in my experience, but -std=, -ansi, -fPIC which all cause defines to be 
added, are common).

> We’d rather do it lazily when times come to process the given
> file having specialized caches on IDE side.

Ok. Do you currently use compile-commands.json for this?

>> Are you asking for that as an option? That would be an independent
>> feature request. I'd suggest filing it in the tracker.
>> Otherwise, the only way CMake currently would pass such options to the
>> compiler is if the user specified them in CMAKE_CXX_FLAGS or
>> add_compile_options() or similar.
> 
> Yes, I mean that user may easily have -iquote directly passed as command
> line option and the IDE need to know that.

> (e.g. include_directories may some day have a flag to indicate whether it
> “” or <>)

Ok, I filed 

 http://public.kitware.com/Bug/view.php?id=15491

Please add any additional information you have, or any design ideas for how 
the command interface should be.



>> CMake does not attempt to parse CMAKE_CXX_FLAGS for things which look
>> like include directories. That variable is just an opaque string of data
>> from the point of view of CMake. It is not clear to me whether you are
>> suggesting that CMake should do that. If you do want to suggest that,
>> please file an issue in the tracker.
>> Similarly, CMake does not attempt to analyse the meaning of options
>> specified in add_compile_options(). Again, it's not clear to me whether
>> you are suggesting a new feature for CMake. If you are, please file it in
>> the tracker.
> 
> No, I don’t suggest CMake should do that, sorry for the confusion.
> I was just explaining why we need the whole compiler-specific command
> line, instead of
> a list of separate flags and include directories

Ok, thanks. I get it now :).

> The current proposal (with having separate include_directories,
> compile_definitions, compile_flags) will force the IDE to assemble the
> compiler-specific command line itself that is quite prone to errors and
> will require constant maintenance since additional options may appear
> (e.g. include_directories may some day have a flag to indicate whether it
> “” or <>)

I agree. However, there are two separate use-cases to address.

1) Get information from CMake so that the IDE can understand the C++
2) Get information to understand the CMake code.
 
You are talking about the first, but I think the second is useful too and an 
IDE could offer features related only to the CMake code (ie unrelated to the 
c++ code) based on that information (such as 'Add include directory' 
action).

So, in the end, we probably need tooling to be able to get the information 
in both forms. Ie, as an array of directories, and as a string of compiler 
command line options.

We already have the latter in the form of compile-commands.json. One of the 
reasons I wonder if you are using that already is to know whether 

1) That is already a good enough solution 
or 
2) The new metadata file should contain similar information

I'm wondering if we can learn anything from your experience with the 
existing file.

>>> * compiler definition do not include compiler-defined(built-in)
>>> definitions etc.
>> 
>> CMake does not currently gather those during configure, as far as I know.
>> That could be added though.
> 
> It'd rather not, since the set may be different for the specific files.

Yes, you're right I think.

> Given the details above, please let me know, if you still need them,
> I’ll try to collect the specific cases that we support from our test
> suite.

I think the details you provided above are enough for understanding, but 
getting some cases from your test suite relating to this stuff would indeed 
be interesting, I expect.

> We currently reconstruct the command line from the
> '<build_dir>/CMakeFiles/Target1.dir/flags.make’ file for each target,

I see. There are several disadvantages to that, among them being that it 
ties you to makefiles (until you duplicate your work to support Ninja), and 
the structure of those files should not be considered 'stable'. CMake should 
be able to change the way it generates the buildsystem-specific files in any 
release.


> We then add additional flags like -dD -M etc. to get the necessary
> information from the compiler and call (compiler is actually called only
> once for the unique set of options with cashing etc.). And we have the
> most accurate defines and include paths for the code model, that we can
> possibly get (afaik).

Cool. Thanks for the details.

> Here is possible options to get this info from metadata.json as I see it:
> 
> 1) every file has a full associated '<compiler_path> <target_defines>
> <file_defines> <target_flags> <file_flags>' command line. This will
> produce lot of duplication, though; but it’s the best option for IDE
> integration, since we don’t need any additional logic in the IDE.

It also seems similar to (but not exactly the same as) the existing optional 
JSON file compile-commands.json.

What do you think of the option of generating both something like

"targets": {
  "foo": {
    "include_directories": ["/opt/foo", "/opt/bar"],
    "quote_include_directories": ["/opt/quote1", "/opt/quote2"],
    "all_compile_flags": "-I/opt/foo -I/opt/bar -iquote /opt/quote1 ...",
    "sources": [
      "file1.cpp",
      "file2.cpp",
      { 
        "file3.cpp": 
        {
          "all_compile_flags":
                "-I/opt/foo (etc, as above) -DFILE3_ONLY -Wall"
        } 
      },
      "file4.cpp",
    ]
  }
}

for a CMakeList.txt file like

 add_executable(foo file1.cpp file2.cpp file3.cpp file4.cpp)
 target_include_directories(foo PRIVATE "/opt/foo" "/opt/bar")
 target_include_directories(foo QUOTE PRIVATE "/opt/quote1" "/opt/quote2")
 set_source_files_properties(file3.cpp PROPERTIES 
   COMPILE_FLAGS "-DFILE3_ONLY -Wall"
 )


That is, the "all_compile_flags" in the target JSON-object contains the same 
as what CMake actually generates (need to check what it does for IDEs). 
Those flags apply to all sources specified in the target object, except for 
those source files which specify exceptions. Exceptions are specified by a 
JSON-object instead of a string in the "sources" array. If it's an object 
with a "all_compile_flags" property, then that overrides the property of the 
same name on the target JSON-object, and again, it is what CMake actually 
generates for the source file in the buildsystem.

Does that make sense? That means you would read the "all_compile_flags" for 
the specific source file if it is present, and use the "all_compile_flags" 
for the target otherwise (which you already have cached).

I know the "include_directories" array is redundant/duplicated information, 
but as it is provided for a different purpose, I think that is ok.

> I hope I haven’t confuse you even more with this explanation.

Thanks, your explanation is great.

Steve.