[cmake-developers] daemon-mode meeting last Tuesday

Tobias Hunger tobias.hunger at gmail.com
Wed Jun 29 02:58:44 EDT 2016


Hello everyone,

On Wed, Jun 29, 2016 at 12:41 AM, Stephen Kelly <steveire at gmail.com> wrote:
> Tobias Hunger wrote:
> Name
> ----
>
> Last year before publication I was calling this feature a 'metadata server'
> (class called cmMetadataServer etc). I probably made a mistake renaming it
> to a daemon. Now, 'metadata server' might be better and more specific than
> 'server' alone.

I do not care what we end up calling the baby, but considering that I
do not expect more server modes to pop up in cmake (and considering
that we do not yet fully know where we will end up with our little
server), I think server is a fine name.

> Change Notification
> -------------------
>
> I wrote previously that I think change notification is an essential feature
> of the first version of the protocol - otherwise the output is no better
> than a static generated file.
>
>  http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16589/focus=16648
>
> Tobias was concerned that implementing filesystem watching for all necessary
> platforms is an unreasonable burden and he would prefer something like
> QFileSystemWatcher.
>
> libuv has a portable abstraction for filesystem change notification similar
> in a way to QFileSystemWatcher. I implemented a basic use of the libuv API
> fs notification API last year.
>
> Action item for me is to dig out that work or do it again. Then the metadata
> server can emit a message on the protocol when the source directory content
> changes.

Creator will watch files itself. It has rather nice logic to save file
descriptors when doing that. And since file descriptors are a scarce
system resource I will need to turn file watching in server mode off.

I do see a use-case where having the cmake server do the watching for
its users. I am in no way opposed to doing that, but for me it is
definitely not a feature required for the first version.

> Feature Flags
> -------------
>
>> 1.2 Protocol versions
>>
>> * We need the daemon-mode to support different protocol versions.
>>
>> * We want to have the option to mark protocol versions as experimental.
>
> As far as I know, Chrome has a development model whereby unfinished features
> are checked into master immediately and developed there, instead of being
> developed in a branch. The new/experimental features are disabled by default
> with 'feature flags' so that people/CI can enable and test features which
> are candidates to be enabled by default to everyone.
>
> We discussed whether this would make sense for the server instead of
> 'experimental' protocol versions. We decided that such feature flags would
> make sense if there would be many experimental features in flight from
> multiple contributors over multiple cmake releases. We don't expect this to
> be the case, so we consider feature flags to be over-engineered for the
> server for now.

Thanks for documenting that:-) I remembered that I had forgot to
mention something we agreed on to not do, but I totally forgot what
that was.

> InReplyTo
> ---------
>
> Tobias told me that the only reason for this is to make it easier for people
> developing the actual protocol to paste blocks of text which contain
> multiple server messages. For example a developer could paste a 'configure'
> and a 'compute' and a 'codeModel' message over and over while implementing
> the 'codeModel' message. If the 'configure' or 'compute' fail for some
> reason with an error Tobias thinks it is confusing to figure out which one
> had the error after the fact. I don't agree that that is hard or confusing.
>
> To be clear: The inReplyTo will never be used by client tools. It is only
> for the above use-case.
>
> My concern is that if we add something only for a use-case like that, client
> tools will start to use it for something it is not designed for. Then we
> will regret that we did not design something better for the clients actual
> need (because inReplyTo isn't exactly what they actually need).
>
> So, I am strongly in favor of keeping the protocol content to the absolute
> minimum - only include things which we know client use-cases for.
> Downstreams abusing CMake script APIs for things they were not intended for
> is a long term problem for CMake. We should avoid introducing that problem
> in the server protocol.
>
> Especially as inReplyTo, even in its current form is 'fuzzy' to me: Is it in
> spontaneous messages which are not in response to anything (message()
> output, change notifications etc)? Does it have 'special' values?

"inReplyTo" is set in all messages generated by the server in reply to
a request received by the client. If the server sends a signal that is
not in reply to a request by the client (e.g. filesystem
notification), then that will not have inReplyTo set.

Considering that some requests trigger a whole lot of messages (e.g.
"configure" triggers messages and progress notifications to be sent
before the final reply), I find this field highly useful.

> Initialization Granularity
> --------------------------
>
> We talked about what should happen when a client starts the metadata server
> and what sequence of protocol commands the client has to send to set things
> up properly.
>
> In my implementation, the initial handshake causes the configure step and
> the compute step to be executed. The client can't cause only the configure
> step to be executed. This means that an out-of-process cmake-gui or other
> cache editor:
>
>  http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16550/focus=16556
>
> using the metadata server would have to do a full handshake instead of
> 'just' the configure part for example, making it not analogous to the
> current cmake-gui.
>
> So, a granular approach allows the use-case of partial initialization while
> also making it possible for a client to do 'full initialization' by
> requesting the 'configure', 'compute' and 'generate' steps explicitly.
>
> As with the cmake-gui, it might make sense to specify the 'generate' command
> as implying the dependee two. We might benefit from some state-machine
> diagram telling users (and us) what protocol commands are possible in which
> state.

I would prefer to have simple building blocks in the server that
applications can combine in useful ways to bigger blocks that can be
used in exactly one way. We need the individual steps anyway, and the
target audience for the server interface is technically savvy enough
to figure out that they need to do "configure" and "compute" to get to
a state where the required information is available.

Personally I would find figuring out what the difference between
"initializeCMakeFully" and the "configure"/"compute" sequence is more
annoying than just having the latter.

> Capabilities
> ------------
>
>> * Move generators and other static information into a separate cmake
>> commandline invocation (http://public.kitware.com/Bug/view.php?id=15462):
>
> Previously Tobias wanted this only as part of the protocol, not as something
> that can be emitted with a regular command line request in the same manner
> as --help for example.
>
> I think such capabilities information should be available easily to anyone
> writing scripts which invoke cmake.
>
> Tobias needs that information for Qt Creator because it needs to know what
> generators are available for example. That is determined when Creator is
> started each time.
>
> Tobias mentioned that such capability queries should be fast - executing
> lots of tools to find out their versions and capabilities is something that
> slows down the start up of QtCreator. He reasoned that he should only have
> to run one cmake - start the server only - not run cmake first to get the
> capabilities, then run it a second time in server mode.
>
> I noted that starting the server just for a query like this is very much
> likely to be slower than just getting the information on the command line.
> Additionally, the server can be started later when loading a project instead
> of when starting QtCreator itself. Tobias was convinced by that.

We did agree here, but I was not convinced by this argument:-)

Starting up applications is expensive, especially in the case where
something goes wrong and locks up. I would greatly prefer just
starting the server mode and extract all information from there. That
is one application startup and I have a way to retrieve everything I
need from there.

But this approach requires kind of implies that I can also configure
the daemon *after* it was started. You convinced me that would not
work well. With having to give generators at startup, a
pre-daemon-mode way to retrieve the generator data is necessary.

> Use-case Driven Design
> ----------------------
>
>> 2.6) project (Return project structure)
>>
>> * Rename to "codemodel" to make the use-case for the data more clear.
>
> One of the differences between the protocol proposed by Tobias is that he
> has one protocol command which returns a structure of all targets, their
> files and the includes, defines and flags of the files.

... plus an interface for clients to filter the output (e.g. only
return certain configurations, targets or even things like target
types).

> In contrast, I had
> implemented an approach which provides multiple protocol commands returning
> more-granular structures:
>
>  http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/16589/focus=16640
>
> In the discussion we realized that the two approaches satisfy two different
> use-cases. Tobias needs to get all C++ source files and the includes,
> defines and flags in order to populate a C++ code model for the project.
> That allows things like 'find usages of this symbol in the project' etc.
>
> A granular way of returning this information to the client would be
> inconvenient and ineffective.
>
> Whereas the use-case for the structure Tobias wants has a use-case of
> populating a C++ code model, my proposed protocol commands for target_info,
> file_info etc are for the use-case of understanding the CMake code.
>
> That is why the 'codeModel' in Tobias' proposal does not list include
> directories, compile definitions etc of targets, but only of files. The code
> model cares about compilations. It is not concerned with targets. The
> protocol commands I proposed are for helping the user to understand what the
> effect of CMake code like
>
>  target_include_directories(MyTarget ${SOME_VAR})
>  target_compile_definitions(MyTarget ${SOME_OTHER_VAR})
>  target_link_libraries(MyTarget ${YET_ANOTHER_VAR})
>
> means. The variables (and transitive usage requirements) make the impact of
> those lines invisible in the code, as my video demonstrates.
>
> Those kinds of protocol commands and features are in scope for the protocol,
> and perhaps also for QtCreator some day, but they are out of scope for now.
>
> As a result of that part of the discussion, I think it might make sense to
> try to use 'use-case driven design' when trying to decide what should be in
> the protocol.
>
> There may be some element of 'redundancy' occurring in the protocol as a
> result. For example, both the 'codeModel' protocol command and the
> 'target_info' would return 'stuff (including object sources in both cases)
> associated with a target'. We don't yet know if that is a problem.

I still think that having one command to query everything cmake knows
about the project structure with ways to limit the information
returned is way better than having five different commands that each
return a subset of that information. Any such subset-based interface I
ever ended up using return what the developers of the interface
thought would make sense for me to know, not what I actually need.

This discussion is for later though. For now I just removed all the
code to limit the selection and just return everything creator needs,
so for the time being this is very use-case driven design. We can
descent into this particular rabbits hole whenever we want *after* the
first version is merged:-)

> Other Movers
> ------------

This section is actually new information we did not discuss in person:-)

> Microsoft just published a JSON protocol for language servers to get
> information into VSCode, and for other user tools to do the same.
>
>  https://www.reddit.com/r/programming/comments/4qad14
>
> This could be interesting to learn from. Although they say the protocol is
> JSON, I can only seem to find TypeScript protocol stuff. Presumably that
> gets translated to JSON for wire communication, but that's not clear from
> their docs.
>
>  https://github.com/google/xi-editor

The protocols defined by microsoft seem to be more interesting later,
when we actually get to the language handling parts.

> C++11
> -----
>
> One central difference between the cmake metadata server code and the rest
> of CMake is that I used cxx_auto_type and cxx_range_for extensively for my
> sanity and productivity.

I just followed Stephen's style and used those, too. Actually I did
not even realize that they were not used anywhere else in cmake till
Stephen mentioned this:-).

These are so easy to get used to:-)

Best Regards,
Tobias


More information about the cmake-developers mailing list