[cmake-developers] Generating buildsystem metadata from CMake

[Anton Makeev]

> On 21 Mar 2015, at 09:41, Stephen Kelly <steveire at gmail.com> wrote:
> 
> Anton Makeev wrote:
> 
>> The other thing that seems troubling to me is that since file, target,
>> language compiler options are split into different parts of metadata, the
>> IDE need to know exactly how to assemble them back into the compiler’s
>> command line (e.g. what flags go first file’s or language’s), duplicating
>> CMake's logic that may be different from version to version and from
>> compiler to compiler. The exact command line is needed to get the actual
>> and precise defines, include search paths etc. from the compiler.
> 
> Yes. I previously proposed the <lang>_compile_command to contain information 
> about how to built it:
> 
> http://www.steveire.com/cmake-future/manual/cmake-metadata-generation.7.html#optional-properties

Thanks for pointing, I missed the purpose of that property.
If we are after a compact form, this should work, I guess.

> However, I think it might be better to generate something similar to what is 
> currently generated in compile-commands.json into cmake-metadata.json. That 
> is, we would generate (in some context)
> 
> "include_directories" : ["/foo", "/opt"]
> "compile_definitions" : ["DEF=\"Foo\"", "OTHER_DEF=1"]
> "compile_command": "-c -DDEF=\"Foo\" -DOTHER_DEF=1 -I/foo -I/opt"
> 
> So, "compile_command" contains approximately what you can currently get from 
> compile-commands.json.
> 
> The other properties contain things which are specifically known to be 
> include directories or compile definitions, as javascript arrays. These 
> properties are obviously redundant information, so I wonder if they should 
> be generated at all? Is the compile command I wrote above easy to parse? Or 
> is it sufficiently difficult to parse that this redundant information should 
> be provided?

Agree, I don’t see any point in having this redundant info. I’d better not have it to avoid confusion:
the include_directories lists is incomplete (it doesn’t contain compiler-defined search paths) and should not be used for any purpose.

>> This would be really helpful indeed, currently, we have to introspect
>> CMakeLists.txt files in order to find the most probably place where new
>> files should be placed (works only in basic cases now). And being able to
>> do so correctly is also crucial for refactoring (e.g. extract class).
> 
> Given the backtrace, you can navigate up the scope from the most recent 
> frame to get out of any functions, macros or loops. You can then add a 
> target_sources() line directly after that. 
> 
> That algorithm will work for every case (not just basic cases) as far as I 
> can tell and is available with CMake 3.1.

OK, it’s definitely better than nothing, so we can start with the target backtraces.

>>> * I didn't document the location or directory.  I'm not clear on whether
>>> it is supposed to be the build location, or the install location(s!),
>>> or all of those.
>> 
>> It would be useful, though, to have a location of generated files for each
>> target: in case metadata misses some information (and I think it won’t
>> cover every possible need anytime soon), IDE will be able to get if from
>> generated makefiles.
> 
> Yes, we can at least provide the build location in an obvious way. We can 
> discuss install locations eventually.

OK

>>> * I don't generate 'dependencies' (actually the list of files which the
>>> buildsystem re-generation depends on) as Aleix did, because there is no
>>> well-defined usefulness for that list yet.
>> 
>> As Tobias pointed, we at least need to know what files are the part of
>> CMake project, that is, the list of all CMakeLists.txt and *.cmake files,
>> used for generation (ideally, including missing ones, since in that case
>> IDE could be able to tell when missing file is created and refresh the
>> project)
> 
> As I wrote to Tobias, I'm apprehensive about this, and it would require 
> other work to make cmake parallel safe first. 
> 
> I think if the IDE does not have focus it should not be running 'cmake .' on 
> my behalf. I think if the IDE newly gets focus you can maybe run 'cmake .' 
> at *that* point (after the user is done with their rebase or whatever). That 
> doesn't require giving you a list of files to watch. Maybe I'm missing 
> something though.

Agree, the IDE should not do any automatic stuff, when user doesn’t expect it.
Though, the use-case here is quite different:
Consider a project that have 'add_directory(missing_dir)’ in one of the CMakeLists files.
If IDE were aware that missing_dir/CMakeLists.txt is a required file, it might then automatically 
trigger update, when the user creates this file inside the IDE (not externally). 
Or it may even have a quick-fix ‘Create missing CMakeList.txt’.

Also, files updated and created during VCS (called from the IDE) should be automatically recognized, of course.


>>> * Some more information from project() may be relevant, but it's not
>>> clear
>>> yet. We will likely know more when we have decided the file format and
>>> generated some 'interesting' metadata files.
>> 
>> Project name, list of the configurations are most needed ones.
>> We also use CMAKE_<lang>_SOURCEFILE_EXTENSIONS to determine if a given
>> file is potentially source file or not.
> 
> As CMake already knows which files are 'object sources', the metadata will 
> provide that. Also, the <lang> extensions is not enough. See the unit test I 
> created and in particular the compiled_as_cxx.c file.

For the existing files we indeed have all the necessary info in the proposed format.
Though, it’s about newly-created files: when a user creates a new file ‘foo.zxy', IDE should decide,
wether to add it  (or suggest to) to a target or not. The list of source files extensions is what 
may help here.

>> This has already been discussed but I give our usage scenario:
>> 
>> in CLion we retrieve the list of all build types (aka configurations,
>> Debug, Release etc) 
> 
> From where do you currently retrieve this list? I guess you look at all 
> cache keys named 
> 
> CMAKE_.*_FLAGS_(.*)
> 
> and list the matches?
> and list the matches?

It comes from CMAKE_CONFIGURATION_TYPES, if project defined such a variable.
With default of Debug/Release/RelWithDebInfo/MinSizeRel

>> and then generate project using Makefiles generator
>> for each of them. This is necessary because of several reasons: 1) To be
>> able to correctly build language model, we need to know, when a file is
>> used in several configurations, which means, it's compiler settings and
>> macros are different.
>>    E.g. some branches of code may not be available in Debug or Release
>>    and we give user an option to quickly switch between them in the
>>    editor.
> 
> This seems similar to what Tobias talked about.
> 
>> I don’t know if it’s possible at all, but it would be great if we could
>> have info for all configurations generated in one go (not only for
>> multi-config, but for single-config generators as well like Ninja and
>> Makefiles).
> 
> I can think of two ways to make that possible:
> 
> 1) Create new mulit-config generators, or add options for the existing ones.
> 2) Add a generic multi-configuration mode to cmake:
> http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/10873/focus=10912
> http://public.kitware.com/Bug/view.php?id=14539
> 
> I consider both out of scope for this thread though.

Agree, and thanks for opening the discussion.

>>> * Generating metadata only (without generating buildsystem files) is not
>>> currently in scope.  This was requested several times, but it is not
>>> clear why.
>> 
>> It’s simply to be able to get this the information as quickly as possible.
>> I’m not sure which part is most slow, but, say, InsightToolKit 4.5
>> (http://www.itk.org/Wiki/ITK/Source <http://www.itk.org/Wiki/ITK/Source>),
>> generates in couple of minutes. 
> 
> CMake has a 'configure' step followed by a 'generate step'. Your count of 
> minutes must be the sum of both. The information determined during those 
> steps is exactly what the metadata file should contain. Avoiding all of it 
> would leave you with no metadata.

I see, you’re most likely right.
Anyway, optimization is out of scope, I just wanted to clarify why this could be desirable.

>> The regeneration, even when nothing was
>> changes, a few dozens of seconds.
> 
> This cost is mostly the 'generation step', as everything from the 'configure 
> step' was cached and available for re-generation.
> 
>> Plus, we’d prefer being able to open the project without any questions to
>> user, e.g. not asking, which generator he/she prefers. If we generate
>> using ‘wrong’ default generator we’ll need to regenerate everything again
>> when user decides to change it.
> 
> It seems like you can use a throwaway temp directory until the user chooses 
> a generator. I am sympathetic to the idea of 'not breaking the users flow', 
> but I don't currently have any idea how to avoid it.
> 
> Everything does indeed have to be re-generated if the generator is changed, 
> and cmake currently issues an error if you use a different -G option than 
> was originally used in a build dir. If you really want to change that 
> behavior, I suggest a separate bug report to track the idea. As I said I am 
> sympathetic to the idea, but I don't see a way. If you file a bug, maybe 
> Brad will have an idea or can say it's fundamentally out of scope.
> 

>> Another benefit of skipping actual generation is possibly better error
>> recoverability. That is, some generators may fail here and there if the
>> project is incorrectly configured (e.g. source files are missing).
>> Skipping the generation phase will (probably) help getting the project
>> metadata even in that case.
> 
> I don't think that's the case. The 'generate step' is the point where the 
> metadata is generated, and that step begins strictly after the 'configure 
> step' ends. The scenario you describe is errors during the 'configure step'. 
> That means no metadata for you.
> 
> If this is possible to change, it's out of scope of this current design 
> work. I'd suggest a separate bug report.

OK, let’s put aside this part for now.

> 
>> But anyway, it seems a little outside of the scope of the discussion.
> 
> Yep :).
> 
> Thanks,
> 
> Steve.
> 


Regards,
Anton Makeev
JetBrains
http://www.jetbrains.com
"Develop with pleasure!"

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2856 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20150330/1a883900/attachment.bin>