[CMake] Experiments in CMake support for Clang (header & standard) modules

[Stephen Kelly]

On 04/20/2018 01:39 AM, David Blaikie wrote:
> Hi there,
>
> I'm experimenting with creating examples (& potential changes to CMake
> itself, if needed/useful) of building clang modules (currently using
> the semi-backwards compatible "header modules", with the intent of
> also/moving towards supporting pre-standard C++ modules in development
> in Clang).

Great! Thanks for reaching out. Sorry it has taken me a while to
respond. Have you had other response off-list?

> The basic commands required are:
>
>   clang++ -fmodules -xc++ -Xclang -emit-module -Xclang
> -fmodules-codegen -fmodule-name=foo foo.modulemap -o foo.pcm
>   clang++ -fmodules -c -fmodule-file=foo.pcm use.cpp
>   clang++ -c foo.pcm
>   clang++ foo.o use.o -o a.out

Ok. Fundamentally, I am suspicious of having to have a
-fmodule-file=foo.pcm for every 'import foo' in each cpp file. I
shouldn't have to manually add that each time I add a new import to my
cpp file. Even if it can be automated (eg by CMake), I shouldn't have to
have my buildsystem be regenerated each time I add an import to my cpp
file either.

That's something I mentioned in the google groups post I made which you
linked to. How will that work when using Qt or any other library?

Today, a beginner can find a random C++ book, type in a code example
from chapter one and put `g++ -I/opt/book_examples prog1.cpp` into a
terminal and get something compiling and running. With modules, they'll
potentially have to pass a whole list of module files too.

Lots of people manually maintain Makefile-based buildsystems today, and
most other companies I've been inside of have their own custom tool or
bunch of python scripts, or both. Manually changing such buildsystems to
add -fmodule-file or -fmodule-map-file each time an import is added is a
significant barrier.

Will my project have to compile the modules files for all of my
dependencies? Even more complication for my buildsystem. Do I have to
wait for my dependencies to modularize bottom-up before I can benefit
from modules? If my dependency does add 'module foo' to their header
files, or whatever the current syntax is, can I continue to #include
<foo> or is that a source incompatible change?

I raised some of these issues a few years ago regarding the clang
implementation with files named exactly module.modulemap:

http://clang-developers.42468.n3.nabble.com/How-do-I-try-out-C-modules-with-clang-td4041946.html

http://clang-developers.42468.n3.nabble.com/How-do-I-try-out-C-modules-with-clang-td4041946i20.html

Interestingly, GCC is taking a directory-centric approach in the driver
(-fmodule-path=<dir>) as opposed to the 'add a file to your compile line
for each import' that Clang and MSVC are taking:

 http://gcc.gnu.org/wiki/cxx-modules

Why is Clang not doing a directory-centric driver-interface? It seems to
obviously solve problems. I wonder if modules can be a success without
coordination between major compiler and buildsystem developers. That's
why I made the git repo - to help work on something more concrete to see
how things scale.

Having just read all of my old posts again, I still worry things like
this will hinder modules 'too much' to be successful. The more (small)
barriers exist, the less chance of success. If modules aren't
successful, then they'll become a poisoned chalice and no one will be
able to work on fixing them. That's actually exactly what I expect to
happen, but I also still hope I'm just missing something :). I really
want to see a committee document from the people working on modules
which actually explores the problems and barriers to adoption and
concludes with 'none of those things matter'. I think it's fixable, but
I haven't seen anyone interested enough to fix the problems (or even to
find out what they are).

Anyway, you are not here for my rants.

> My current very simplistic prototype, to build a module file, its
> respective module object file, and include those in the library/link
> for anything that depends on this library:
>
>   add_custom_command(
>           COMMAND ${CMAKE_CXX_COMPILER} ${CMAKE_CXX_FLAGS} -xc++ -c
> -Xclang -emit-module -fmodules -fmodule-name=Hello
> ${CMAKE_CURRENT_SOURCE_DIR}/module.modulemap -o
> ${CMAKE_CURRENT_BINARY_DIR}/hello_module.pcm -Xclang -fmodules-codegen
>           DEPENDS module.modulemap hello.h

Why does this command depend on hello.h? If that is changed and
module.modulemap is not, what will happen?

>           OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/hello_module.pcm
>           COMMENT "Generating hello_module.pcm"
>   )
>   add_library (Hello hello.cxx
> ${CMAKE_CURRENT_BINARY_DIR}/hello_module.pcm)
>   target_include_directories(Hello PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
>   target_compile_options(Hello PUBLIC -fmodules -Xclang
> -fmodule-file=${CMAKE_CURRENT_BINARY_DIR}/hello_module.pcm)
>
> (this is based on the example in the CMake docs using Hello/Demo)

Good that you got something working.

> This also required one modification to CMake itself to classify a pcm
> file as a C++ file that needs to be compiled (to execute the 3rd line
> in the basic command list shown above).

An alternative to patching CMake might be

 set_source_files_properties(${CMAKE_CURRENT_BINARY_DIR}/hello_module.pcm
PROPERTIES LANGUAGE CXX)

though hopefully that is also temporary.

> But this isn't ideal - I don't /think/ I've got the dependencies quite
> right & things might not be rebuilding at the right times.
> Also it involves hardcoding a bunch of things like the pcm file names,
> header files, etc.

Indeed. I think part of that comes from the way modules have been
designed. The TS has similar issues.

> Ideally, at least for a simplistic build, I wouldn't mind generating a
> modulemap from all the .h files (& have those headers listed in the
> add_library command - perhaps splitting public and private headers in
> some way, only including the public headers in the module file,
> likely). Eventually for the standards-proposal version, it's expected
> that there won't be any modulemap file, but maybe all headers are
> included in the module compilation (just pass the headers directly to
> the compiler).

In a design based on passing directories instead of files, would those
directories be redundant with the include directories?

One of the problems modules adoption will hit is that all the compilers
are designing fundamentally different command line interfaces for them.
Buildsystems will have to be rewritten to take advantage of modules, and
they will be annoying to use and adopt.

> This also doesn't start to approach the issue of how to build modules
> for external libraries - which I'm happy to discuss/prototype too,
> though interested in working to streamline the inter-library but
> intra-project (not inter-project) case first.

Yes, there are many aspects to consider.

Are you interested in design of a CMake abstraction for this stuff? I
have thoughts on that, but I don't know if your level of interest
stretches that far.

> Stephen - I saw you were asking some questions about this here
> ( https://groups.google.com/a/isocpp.org/forum/#!topic/modules/sDIYoU8Uljw
> <https://groups.google.com/a/isocpp.org/forum/#%21topic/modules/sDIYoU8Uljw> &
> https://github.com/steveire/ModulesExperiments - didn't really
> understand how this example applied/worked, though - I guess maybe
> it's a prototype syntax proposal?)

It is a set of pre-modules libraries, some of which depend on one
another and with some transitive dependencies in the headers.

I made it to be 'a few steps above trivial' in the hope that someone
would show me how to port it to modules-ts (even if the result does not
build). So far, no one has.

Can you help? It would really help my understanding of where things
currently stand with modules. For example, is there only one way to port
the contents of the cpp files?

After that, is there one importable module per class or one per shared
library (which I think would make more sense for Qt)? And is transitive
dependency expressed in the header files after porting? I think that
last one is dealt with by the 'export import' syntax

The git repo is an attempt to make the discussion concrete because it
would show how multiple classes and multiple libraries with dependencies
could interact in a modules world. I'm interested in what it would look
like ported to modules-ts, because as far as I know, clang-modules and
module maps would not need porting of the cpp files at all.

> Basically: What do folks think about supporting these sort of features
> in CMake C++ Builds? Any pointers on how I might best implement this
> with or without changes to CMake?

I think some design is needed up front. I expect CMake would want to
have a first-class (on equal footing with include directories or compile
definitions and with particular handling) concept for modules, extending
the install(TARGET) command to install module binary files etc.

To do that kind of design, I at least would need to be able to
experiment or conceptualize examples which are not totally trivial, such
as the starting point in my repo.

On the CMake side, I think something like this should be the target
(note that no compiler command line interface works like this today,
which I think is a barrier to adoption):

 add_library(foo foo.cpp)

 # Target property to enable modules for the target
 set_property(TARGET foo PROPERTY USE_CXX_MODULES ON)

 # Note: Use target_include_directories to specify module search
 # paths (how GCC and MSVC work)
 # Also note: compilers should use the -I paths as a module path search
list so
 # that CMake does not have to pass the same list as both -I and as
-fmodule-path=
 # or similar entries.
 # Also note: This is source compatible with the cmake code that already
exists!
 # The existance of /opt/bar/bing.<ext> makes 'import bing;' work.
 target_include_directories(foo PRIVATE /opt/bar)

 # Possibly need a new command to specify headers (there is
 # other motivation for this in CMake, so use a generic name without
'modules' in it)
 # Because foo has USE_CXX_MODULES ON, foo.h is processed as a module
 # and a binary representation is created for import. Other properties can
 # be set on foo.h with set_source_files_properties() to pass other
command line
 # options when generating the module.
 target_headers(foo PRIVATE foo.h)

Also - in the right design, CMake does not have to regenerate
-fmodule-file or whatever into the compile line any time the user adds
'import something;', which is the case with clang now afaik. Please
correct me if that is not correct.

I know some people at Kitware have been thinking about modules though,
so I'd be interested in any other insights from there. Brad, can you
comment?

Here's some other reading material for anyone else following along:

https://izzys.casa/posts/millennials-are-killing-the-modules-ts.html
https://build2.org/article/cxx-modules-misconceptions.xhtml


Thanks,

Stephen.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20180430/8527c9da/attachment-0001.html>