<div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Fri, Aug 24, 2018 at 2:35 AM Stephen Kelly <<a href="mailto:steveire@gmail.com" target="_blank">steveire@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="m_1612126189405771706m_-6139492096525901871m_2333793003776550079moz-cite-prefix">On 24/08/18 02:32, David Blaikie wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">On Tue, Jul 24, 2018 at 3:20 PM Stephen Kelly
<<a href="mailto:steveire@gmail.com" target="_blank">steveire@gmail.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">David
Blaikie wrote:<br>
<br>
> (just CC'ing you Richard in case you want to read my
ramblings/spot any<br>
> inaccuracies, etc)<br>
> <br>
> Excuse the delay - coming back to this a bit now.
Though the varying<br>
> opinions on what modules will take to integrate with
build system still<br>
> weighs on me a bit<br>
<br>
Can you explain what you mean by 'weighs on you'? Does that
mean you see it <br>
as tricky now?</blockquote>
</div>
<div dir="ltr">
<div class="gmail_quote">
<div><br>
Yes, to some extent. If the build system is going to
require the compiler-callsback-to-buildsystem that it
sounds like (from discussions with Richard & Nathan,
etc) is reasonable - yeah, I'd say that's a bigger change
to the way C++ is compiled than I was expecting/thinking
of going into this.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
Yes.</div><div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div> </div>
</div>
</div>
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I've
kind of been assuming that people generally think it is
not tricky, and <br>
I'm just wrong in thinking it is and I'll eventually see
how it is all <br>
manageable.<br>
</blockquote>
</div>
</div>
<div dir="ltr">
<div class="gmail_quote">
<div><br>
I think it's manageable - the thing that weighs on me, I
suppose, is whether or not the community at large will
"buy" it, as such. </div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
Yes, that has been my point since I first started talking about
modules. I don't think modules will gain a critical mass of adoption
as currently designed (and as currently designed to work with
buildsystems).</div></blockquote></div><div dir="ltr"><div class="gmail_quote"><div><br>For myself, I don't think I'd go that far (I think the current design might be feasible) - and I'm mostly trying to set aside those concerns to get to more concrete things - to work through the build system ramifications, prototype things, etc, to get more concrete experience/demonstration of the possibilities and problems.<br></div></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div>And part of that is on the work we're doing to figure
out the integration with build systems, etc, so that
there's at least the first few pieces of support that
might help gain user adoption to justify/encourage/provide
work on further support, etc... </div></div></div></div></blockquote></div><div text="#000000" bgcolor="#FFFFFF">
Yes, reading the document Nathan sent us on June 12th this year, it
seems that CMake would have to implement a server mode so that the
compiler will invoke it with RPC. That server will also need to
consume some data generated by CMake during buildsystem generation
(eg user specified flags) and put that together with information
sent by the compiler (eg ) in order to formulate a response. It's
complex. Maybe CMake and other buildsystem generators can do it, but
there are many bespoke systems out there which would have to have
some way to justify the cost of developing such a thing.</div></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>Yeah, certainly a possibility - maybe it'd be enough of a wedge to cause people to collapse the large build system space into fewer options - many other languages require this sort of level of coupling between compiler/build system/language.<br><br>But getting that momentum started would be getting the main build systems supporting it & starting at the leaves (independent projects starting to write modular code for themselves - even if all their external dependencies aren't) - then, with enough users using it downstream, might be some libraries providing a modular option (or maybe even only provide it as modules & that'd be the wedge I'm talking about - then teams/projects that don't support modules would be left out a bit & provide an incentive for them to move over to have modules support to use these dependencies).</div></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite">
<div dir="ltr"><div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> The build.sh script shows the commands required to
build it (though I<br>
> haven't checked the exact fmodule-file dependencies
to check that they're<br>
> all necessary, etc) - and with current Clang
top-of-tree it does build and<br>
> run the example dinnerparty program.<br>
<br>
Ok. I tried with my several-weeks-old checkout and it
failed on the first <br>
command with -modules-ts in it (for AbstractFruit.cppm -
the simplest one).<br>
<br>
I'll update my build and try again, but that will take
some time.<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div dir="ltr">
<div class="gmail_quote">
<div>Huh - I mean it's certainly a moving target - I had to
file/workaround a few bugs to get it working as much as it
is, so not /too/ surprising. Did you get it working in the
end? If not, could you specify the exact revision your
compiler's at and show the complete output?</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
Yes, I got it working. See <br>
<br>
<a class="m_1612126189405771706m_-6139492096525901871m_2333793003776550079moz-txt-link-freetext" href="https://www.mail-archive.com/cmake-developers@cmake.org/msg18623.html" target="_blank">https://www.mail-archive.com/cmake-developers@cmake.org/msg18623.html</a><br>
<br>
<div> <br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> But
I'm not sure how best to determine the order in which to
build files within a library - that's where the sort of
-MM-esque stuff, etc, would be</blockquote></div></div></div></blockquote></div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
> necessary.<br>
<br>
Would it? I thought the -MM stuff would mostly be
necessary for determining <br>
when to rebuild? Don't we need to determine the build
order before the first <br>
build of anything? The -MM stuff doesn't help that.<br>
</blockquote></div></div></div></blockquote></div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_quote">
<div><br>
-MM produces output separate from the compilation (so far
as I can tell - clang++ -MM x.cpp doesn't produce anything
other than the makefile fragment on stdout) & finds
all the headers, etc. So that's basically the same as what
we'd need here</div>
</div></div></div></blockquote></div><div text="#000000" bgcolor="#FFFFFF"><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"></div>
</div>
</div>
</blockquote>
<br>
Are you sure? I thought compiling with -MM gives us information that
we need before we compile the first time. Sorry if that was not
clear from what I wrote above. I see a chicken-egg problem. However,
I assume I'm just misunderstanding you (you said that -MM would be
used to determine build order for the initial build) so let's just
drop this.</div></blockquote></div></div><div dir="ltr"><div class="gmail_quote"><div><br>Yeah, still a bit confused - not sure ignoring this tangent is useful, maybe there's something in the misunderstanding here.<br><br>clang -MM doesn't compile the source file, and prints out something like:<br><br>f1.o: f1.cpp foo.h<br><br>So we know that f1.cpp needs foo.h - now, in this case it would error if foo.h didn't exist, because -MM has to look through all the inclusions to provide all the transitive inclusions. The same isn't true of modules - if, instead of foo.h this was the foo module (foo.cppm) and that module depends on bar.cppm - then the output for f1.cpp would just be "f1.o: f1.cpp foo.pcm" - without needing to compile foo.cppm, then the build system could ask "what are the dependencies for foo.cppm (knowing it can generate foo.pcm)" and get "foo.pcm.o: foo.cppm bar.pcm", etc. Very rough/hand-wavy, but the general idea I think is reasonable.<br><br>(there are some bonus wrinkles for legacy imports, but there are some ways to address that too)</div></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div>Looking at your example - if you have a library for all
the fruits and libabstractfruit, libfruitsalad,
libnotfruitsalad, and libbowls - then you'd have one
module interface for each of those (AbstractFruit.cppm,
FruitSalad.cppm, NotFruitSalad.cppm, Bowls.cppm) that
would be imported (so replace "import Apple", "import
Grape" with "import FruitSalad", etc... ) & the
implementations could be in multiple files if desired
(Apple.cpp, Grape.cpp, etc).<br>
</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
Could you show me what that would look like for the repo? I am
interested to know if this approach means concatenating the content
of multiple files (eg Grape.h and Apple.h) and porting that result
to a module. My instinct says that won't gain adoption.</div></blockquote></div></div><div dir="ltr"><div class="gmail_quote"><br>Sure, let's see... <br><br><a href="https://github.com/dwblaikie/ModulesExperiments/commit/4438a017c422c37106741253a78e2bd7ee99c43e">https://github.com/dwblaikie/ModulesExperiments/commit/4438a017c422c37106741253a78e2bd7ee99c43e</a><br><br>I mean it could be done in other ways - you could #include Grape.h and Apple.h into Fruit.cppm, I suppose. Could allow you to keep the old headers for non-modular users & just wrap them up in a module (same way I did for the "std" module in this example).<br></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">>>
Ok. That's not much better though. It still means
editing/generating the<br>
>> buildsystem each time you add an import.<br>
> <br>
> <br>
> Isn't that true today with headers, though? <br>
<br>
No. Imagine you implemented FruitBowl.cpp in revision 1
such that it did not <br>
#include Grape.h and it did not add the Grape to the bowl.<br>
<br>
Then you edit FruitBowl.cpp to #include Grape.h and add
the Grape to the <br>
bowl. Because Grape.h and Apple.h are in the same
directory (which you <br>
already have a -Ipath/to/headers for in your buildsystem),
in this (today) <br>
scenario, you don't have to edit the buildsystem.<br>
</blockquote>
<div><br>
Well, you don't have to do it manually, but your build
system ideally should reflect this new dependency so it
knows to rebuild FruitBowl.cpp if Grape.h changes.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
I never said it had to be done manually in the real world. I
mentioned that in the context of your script. The point I keep
making is that the buildsystem has to be regenerated.</div></blockquote><div><br></div><div>*nod* Though the same seems to be true today when #includes change - the build system has to become aware of the new dependencies that have been introduced (either discovered during compilation with -MD (generating the .o file and the .d Makefile fragment at the same time) or separately/ahead of time with -MM (generating the Makefile fragment on stdout and not performing compilation)).</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I wonder
if people will use C++ modules if CMake/their generator
has to be <br>
re-run (automatically or through explicit user action)
every time they add <br>
'import foo;' to their C++ code... What do you think?<br>
</blockquote>
<div><br>
</div>
<div>If it's automatic & efficient (I hope it doesn't
redo all the work of discovery for all files - just the
ones that have changed) it seems plausible to me.</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
At least in the CMake case, the logic is currently coarse - if the
buildsystem needs to be regenerated, the entire configure and
generate steps are invoked. </div></blockquote><div><br></div><div>When I add a new include to a file currently, it doesn't look like cmake re-runs. Any idea why that is? I guess ninja knows enough to update for the new include dependency - perhaps it'd need to rerun cmake if my new #include was of a generated file (even if there were already rules for generating that file?)?<br><br>If modules worked similarly to the way things seem to work here - importing an external module that hadn't been imported anywhere by your project before might rerun cmake, but importing that module into a second source file would be akin to #including a file that already had a rule for generating it - so it wouldn't rerun cmake, I don't think.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">Maybe that can be changed, but that's
just more effort required on the part of all buildsystem generators,
including bespoke ones. I think the level of effort being pushed on
buildsystems is not well appreciated by the modules proposal. <br></div></blockquote><div><br></div><div>Perhaps - and that's what I'm trying to work through & write up.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">What I see as a worst-case scenario is:<br>
<br>
* Modules gets added to the standard to much applause<br>
* User realize that they have to rename all of their .h files to
cppm and carefully change those files to use imports. There are new
requirements regarding where imports can appear, and things don't
work at first because of various reasons.<br>
* Maybe some users think that creating a module per library is a
better idea, so they concat those new cppm files, sorting all the
imports to the top.<br>
* Porting to Modules is hard anyway, because dependencies also need
to be updated etc. Developers don't get benefits until everything is
'just right'.<br>
* Some popular buildsystems develop the features to satisfy the new
requirements<br>
* Most buildsystems, which are bespoke, don't implement the GCC
oracle-type stuff and just fudge things with parsing headers using a
simple script which looks for imports. It kind of works, but is
fragile.<br>
* Lots of time is spent on buildsystems being regenerated, because
the bespoke systems don't get optimized in this new way.<br>
* After a trial run, most organizations that try modules reverse
course and stop using them.<br>
* Modules deemed to have failed.<br>
<br>
Maybe I'm being too negative, but this seems to be the likely result
to me. I think there are more problems lurking that we don't know
about yet. But, I've said this before, and I still hope I'm wrong
and just missing something.</div></blockquote><div><br></div><div>I think that's certainly a possibility - but I'm approaching this optimistically - rather than trying to prove it can't work, I'm trying to figure out how it would work & if those solutions all turn out to be unworkable/untenable by the community, then that's some useful data to feed back into the committee process.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_quote">
<div>Sorry for the rather long delay on this - hopefully it
helps us converge a little.<br>
<br>
I'll try to find some time to get back to my original
prototype & your replies to do with that to see if I
can flesh out the simpler "one module per library (with
some of the inefficiency of just assuming strong
dependencies between libraries, rather than the fine
grained stuff we could do with -MM-esque support), no
external modules" scenario (& maybe the retro/"header
modules" style, rather than/in addition to the new C++
modules TS/atom style) - would be great to have a
reasonable prototype of that as a place to work from, I
think.</div>
</div>
</div>
</div>
</blockquote>
<br></div><div text="#000000" bgcolor="#FFFFFF">
Yes, sounds interesting.<br>
<br>
There are other things we would want to explore then too. In
particular, in my repo, all of the examples are part of the same
buildsystem. We should model external dependencies too - ie, pretend
each library has a standalone/hermetic buildsystem. That would mean
that AbstractFruit would generate its own pcm files to build itself,
but each dependency would also have to generate the AbstractFruit
pcm files in order to compile against it as an external library
(because pcm files will never be part of an install step, or a linux
package or anything - they are not distribution artifacts).<br></div></blockquote><div><br></div><div>Yep, eventually I'll get to that - trying to focus on small, incremental steps to start with. The first adoption of modules will likely be on leaf projectns in part because of this complexity - and they can always wrap any external dependencies in modules ala the 'std' module in my example. Getting leaf projects adopting this functionality would be a good gateway towards pressure on build systems to support that and eventually to support modularized libraries.<br><br>- Dave</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
<br>
Thanks,<br>
<br>
Stephen.<br>
<br>
<br>
</div>
</blockquote></div></div></div>