[cmake-developers] Experiments in CMake support for Clang (header & standard) modules

[David Blaikie]

On Tue, Jul 24, 2018 at 3:20 PM Stephen Kelly <steveire at gmail.com> wrote:

> David Blaikie wrote:
>
> > (just CC'ing you Richard in case you want to read my ramblings/spot any
> > inaccuracies, etc)
> >
> > Excuse the delay - coming back to this a bit now. Though the varying
> > opinions on what modules will take to integrate with build system still
> > weighs on me a bit
>
> Can you explain what you mean by 'weighs on you'? Does that mean you see
> it
> as tricky now?


Yes, to some extent. If the build system is going to require the
compiler-callsback-to-buildsystem that it sounds like (from discussions
with Richard & Nathan, etc) is reasonable - yeah, I'd say that's a bigger
change to the way C++ is compiled than I was expecting/thinking of going
into this.

Some folks see this as not "tricky" but just "hey, C++ has been getting
away with a vastly simplified build model until now - now that it's getting
modular separation like other languages, it's going to have to have a build
system like other languages" (ala Java/C#, Haskell/Go maybe, etc) - but I'm
not especially familiar with any other languages enterprise-level build
systems (done a bit of C# at Microsoft but wasn't looking closely at the
build system - insulated from it by either Visual Studio or the OS release
processes, etc - and the last time I did Java I was working on small enough
things & without many cores, so throwing all the .java files in a package
at the compiler in one go was totally reasonable)


> I've kind of been assuming that people generally think it is not tricky,
> and
> I'm just wrong in thinking it is and I'll eventually see how it is all
> manageable.
>

I think it's manageable - the thing that weighs on me, I suppose, is
whether or not the community at large will "buy" it, as such. And part of
that is on the work we're doing to figure out the integration with build
systems, etc, so that there's at least the first few pieces of support that
might help gain user adoption to justify/encourage/provide work on further
support, etc...

> To that end, Stephen, I've made a fork of your example repository & a very
> > simple/direct change to use C++ modules as currently implemented in
> Clang.
>
> Interesting, thanks for doing that! Here's the link for anyone else
> interested:
>
>  https://github.com/dwblaikie/ModulesExperiments


Oh, thanks - totally didn't realize I failed to link that!


> > The build.sh script shows the commands required to build it (though I
> > haven't checked the exact fmodule-file dependencies to check that they're
> > all necessary, etc) - and with current Clang top-of-tree it does build
> and
> > run the example dinnerparty program.
>
> Ok. I tried with my several-weeks-old checkout and it failed on the first
> command with -modules-ts in it (for AbstractFruit.cppm - the simplest one).
>
> I'll update my build and try again, but that will take some time.
>

Huh - I mean it's certainly a moving target - I had to file/workaround a
few bugs to get it working as much as it is, so not /too/ surprising. Did
you get it working in the end? If not, could you specify the exact revision
your compiler's at and show the complete output?


> > If you happen to try experimenting with any ways the commands in the
> > build.sh file could be run from CMake it a sensible way - even if you
> > hypothesize what -MM support (or other compiler hooks like the dependency
> > server I alluded to above, etc) for modules might look like to do so, I'd
> > love to chat about it/throw ideas around/try mocking up/prototyping the
> > sort of compiler support (I don't think there's -MM support yet, but I
> > could see about adding it, for example) that seems like it might be most
> > useful.
>
> Yes, that would be useful I think.
>
> > One thing I'm vaguely concerned about is actually the dependency of
> > building modules within a single library (as in this project/example - at
> > least the way I've built it for now - I didn't try building it as
> separate
> > .so/.a files). At least across-library we can work at the library
> > granularity and provide on the command line (or via a file as GCC does)
> > the module files for all the modules from dependent libraries. But I'm
> not
> > sure how best to determine the order in which to build files within a
> > library
>
> Exactly, that's one of the reasons my repo has libraries of multiple files
> with dependencies between them, but building everything into one
> executable
> also exposes that issue.
>

Ah, when I say "library" I mean static (.lib/.a) or dynamic (.so/.dll/etc)
- same situation in either. But yeah - any situation where there are
multiple modules within a single library, getting those dependencies will
involve some discovery (either a pre-parsing check (same way a build system
might check which headers are included in a file) or the compiler callback
system, where the compiler asks the build system for the modules it needs)
- and even in the one-module-per-library


> I guess it would work in a similar way to determining the order to link
> libraries though, so probably not a major problem.
>

The dependency between libraries is usually explicit in the build system,
right? I don't think that would be the case for modules - there'd be a
reasonable expectation that


> > - that's where the sort of -MM-esque stuff, etc, would be
> > necessary.
>
> Would it? I thought the -MM stuff would mostly be necessary for
> determining
> when to rebuild? Don't we need to determine the build order before the
> first
> build of anything? The -MM stuff doesn't help that.
>

-MM produces output separate from the compilation (so far as I can tell -
clang++ -MM x.cpp doesn't produce anything other than the makefile fragment
on stdout) & finds all the headers, etc. So that's basically the same as
what we'd need here - but -MM only has to preprocess, whereas this would
have to do some C++ parsing to find imports, etc. But it would only have to
look for direct dependencies (unlike -MM which walks through includes to
find other includes) - and then the build system could re-invoke it for the
next module and so on...

>> It seems quite common to have one PCH file per shared library (that's
> >> what Qt does for example). What makes you so sure that won't be the case
> >> with modules?
> >>
> >
> > Can't say I've worked with code using existing PCH - if that seems common
> > enough, it might be a good analogy/guidance people might follow with C++
> > modules.
>
> Perhaps. I wonder how the C++ code/build.sh code would change in that
> scenario? If there were only four modules - one for each of the libraries
> (as delimited in the CMakeLists). Would the C++ code change then too
> (something about building partial modules from each of the multiple cppm
> files?), or how would the build.sh code change?
>

One module per library would be fairly clear, I think.

Looking at your example - if you have a library for all the fruits and
libabstractfruit, libfruitsalad, libnotfruitsalad, and libbowls - then
you'd have one module interface for each of those (AbstractFruit.cppm,
FruitSalad.cppm, NotFruitSalad.cppm, Bowls.cppm) that would be imported (so
replace "import Apple", "import Grape" with "import FruitSalad", etc... ) &
the implementations could be in multiple files if desired (Apple.cpp,
Grape.cpp, etc).


> >> Ok. That's not much better though. It still means editing/generating the
> >> buildsystem each time you add an import.
> >
> >
> > Isn't that true today with headers, though?
>
> No. Imagine you implemented FruitBowl.cpp in revision 1 such that it did
> not
> #include Grape.h and it did not add the Grape to the bowl.
>
> Then you edit FruitBowl.cpp to #include Grape.h and add the Grape to the
> bowl. Because Grape.h and Apple.h are in the same directory (which you
> already have a -Ipath/to/headers for in your buildsystem), in this (today)
> scenario, you don't have to edit the buildsystem.
>

Well, you don't have to do it manually, but your build system ideally
should reflect this new dependency so it knows to rebuild FruitBowl.cpp if
Grape.h changes.


> In your port, you would have to add an import of Grape (fine, equivalent),
> add the Grape to the bowl (the same as today), but additionally, you have
> to
>
>  * add -fmodule-file=Grape.pcm to the compile line or run your buildsystem
> generator such as CMake to cause that compile line to be generated with
> the
> argument added.
>  * Generate Grape.pcm (because the library has 1000 fruit classes in it
> and
> your buildsystem is smart enough to lazily generate pcm files as needed)
>
> Partly that is a result of the totally-granular approach you took to
> creating modules (one module per class). If you used the approach of one
> module per library, then you would not need to touch the buildsystem.


Right - much like if you have coarse grained headers then picking up a new
dependency doesn't involve adding a new #include nor modifying the build
system.

Though in both header and module cases, there's some room for optimization
from the coarse grained library dependencies you might have - if libA
depends on libB, not every C++ source file in libA needs to be rebuilt if
any header in libB is modified. That's why build systems use -MM-like
behavior to create more fine-grained dependencies, both to ensure things
are rebuilt when needed, but also to ensure they aren't rebuilt more often
than needed too.


> I hope
> that's clear now, as I've mentioned it multiple times, but I wonder if
> that
> part of what I say is being understood.
>
> > But today the build system
> > does this under the covers with header scanning, -MM modes, etc?
>
> Perhaps. I notice that running CMake on my llvm/clang/clang-tools-extra
> checkout takes a non-zero amount of time, and for other buildsystems takes
> a
> significantly non-zero amount of time.
>
> Many buildsystem generators already avoid the time/complexity of
> automatically regenerating the buildsystem when needed. Users have to
> leave
> their IDE and run a script on the command line.
>

That surprises me a bit - if I were using an IDE, I'd expect it to update
the build system based on changes I made there - add a source file,
#include, etc, and it should be reflected in the build system files. That's
sort of meant to be one of the perks of using an IDE, I thought - that it
kept all that in sync because it knows about what I'm doing when I say
"create a new C++ source file" - rather than working on the command line
where the build system isn't notified when I open vim and save a new .cpp
file (unless the build system is using globs to consider anything with the
.cpp file extension to be part of the project).


> I wonder if people will use C++ modules if CMake/their generator has to be
> re-run (automatically or through explicit user action) every time they add
> 'import foo;' to their C++ code... What do you think?
>

If it's automatic & efficient (I hope it doesn't redo all the work of
discovery for all files - just the ones that have changed) it seems
plausible to me.

Sorry for the rather long delay on this - hopefully it helps us converge a
little.

I'll try to find some time to get back to my original prototype & your
replies to do with that to see if I can flesh out the simpler "one module
per library (with some of the inefficiency of just assuming strong
dependencies between libraries, rather than the fine grained stuff we could
do with -MM-esque support), no external modules" scenario (& maybe the
retro/"header modules" style, rather than/in addition to the new C++
modules TS/atom style) - would be great to have a reasonable prototype of
that as a place to work from, I think.

- Dave

> What would be different about having a similar requirement for modules
> > (but a somewhat different discovery process)?
>
> I hope the above is clear. Please let me know if not. Maybe I'm missing
> something still.
>
> Thanks,
>
> Stephen.
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake-developers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake-developers/attachments/20180823/5ac11242/attachment-0001.html>