[cmake-developers] daemon-mode meeting last Tuesday

[Tobias Hunger]

Hello CMake Developers,

Stephen Kelly and me met last Tuesday to talk about the daemon-mode patch we
both have been working on. It was a very productive meeting: We managed to
resolve almost all the differences of opinion we had.

Here is a short summary of what we talked about:

0) General

  * We both want a daemon mode:-)
  
  * Some names could be improved. Suggestions welcome.
  
  * Having cmMessanger would be nice and will make things easier.
  
  * We agreed that we should get the basic infrastructure part reviewed first,
    followed by the functionality up to the project structure (codemodel).
    
  * We both think it only makes sense to merge the infrastructure part into
    cmake (if it passes review first of course) once we have some functionality
    that is genuinely useful. So we want to aim at having the infrastructure
    and the codemodel merged in one go.



1) Infrastructure:

  1.1 Communication:
  
  * Clients should talk to daemon mode via a socket (libuv supports that), 
    but we also want to support stdin/stdout at least for a "debug mode"
     * There should be a separate command line switch to start stdin/out mode
       and for socket mode
     * This is needed before it makes sense to merge the patch set.


  1.2 Protocol versions

  * We need the daemon-mode to support different protocol versions. These
    need to be versioned independently of the cmake binary itself and we will
    try to keep the protocol versions working as long as that is reasonable,
    but would try to support them for at least a year or two (if there
    are no overriding concerns).
    
  * We want to have the option to mark protocol versions as experimental.
    Experimental protocol versions come with *NO* guarantees and can change
    *AT ANY TIME*. If a experimental protocol ends up in a cmake release there
    still are no guarantees. I would like to see experimental protocol versions
    in cmake releases, since that makes it really easy for other parties to
    evaluate changes. In my experience hardly anybody builds custom branches:-/
    
    All this would need to be communicated by cmake e.g. in the release notes.


  1.3 Basic Protocol
  
  * Client/daemon talk via JSON messages.
  
  * All messages have a "type" with a string specifying the type of message.
  
  * We (mostly) have a request/reply model. The client sends a request message
    and will receive a reply or an error.
  
  * Requests contain a type, and an optional cookie (which will be returned
    together with any message sent by the daemon in response to the request).
    A cookie can be any JSON object.
    
  * A reply is a JSON message of type "reply". It contains the cookie of its
    request and any data that the daemon wants to send its client.
    
  * An error is a JSON message of type "error". It contains the cookie of its
    request and an "errorMessage".
    
  * There are more JSON messages for progress information and messages, etc. and
    we can add more as needed.
  
  TO DECIDE:
  
  * Currently a reply (and error and all other messages in response to a
    request) contain a "inReplyTo" with the type string from the request
    triggering the reply.
    
    Stephen thinks that is not necessary and argues anything that is not
    strictly necessary should not be in the first version of the protocol.
    
    I like having this as it makes it helps to identify what a reply/error is
    related to. I often paste a whole string of requests into the daemon and
    if there is an error it is really easy to see how for things got.
    
  * Should we have a split between header (cookie/type/inReplyTo) and data?
  
    Stephen suggested passing all data that is from/for the client in a
    "body" attribute to make messages more uniform.
    
    I think that is just adding one more level into the JSON file for no
    real benefit.


    
2) Basic Command Set


  2.1) Reset (Reset to a state right after daemon start)
  
  * Remove this. This is error prone as data structures may not be cleaned out
    properly.

    
  2.2) globalSettings (Query generators, debug flags, etc.)
  
  * Move generators and other static information into a separate cmake
    commandline invocation (http://public.kitware.com/Bug/view.php?id=15462):
    
    * Generators with information on them
    * CMake version information
    * Module path
    * Daemon mode support
    
    all should be available in a machine readable way from that command.
    
    TODO: Figure out how to get all the generator information, e.g. which values
    the flags some generator taken can have (architecture, etc.). Stephen
    volunteered to check into this.
    
  * We can report the selected generator/extragenerator here.
  
  * Undecided about whether it makes sense to report source/builddirectory here

    
  2.3) setGlobalSettings (set the information from globalSettings)
  
  * This is currently used to set sourcedirectory, builddirectory and
    generator.
    
    These three should be passed in via the command line instead, again
    to avoid trouble with users switching to different build directories/etc.
    while the daemon is running, which might trigger problems with state
    cleanup, etc.
    
  * The rest (trigger developer warnings, etc.) seems fine.
  
  * Generator/build- and sourcedirectory may not be set (even if reported
    via globalSettings). Extragenerator can be set.

  
  2.4) configure (Trigger cmake to configure a builddirectory)
  
  * Fine, handles "-DSOMETHING=other", etc. via normal cmake commandline
    parsing.
    
  TODO: Validate that it handles broken input with decent error reporting.

  
  2.5) generate
  
  * Rename this to "compute", which is what this actually triggers in cmake.

  
  2.6) project (Return project structure)
  
  * add backtrace information to targets
  
  * Rename to "codemodel" to make the use-case for the data more clear.
  
  * Focus on providing data for the use-case of feeding a code model.
  
  * Make the configuration the top-level, then report projects, targets and
    files (in groups) beneath that. Currently projects are the top level, then
    there is configurations, targets and files.
    
    Switching this around should offer more flexibility to change cmake
    internals going forward.
    
  * We do not want cmake internals to leak into the output. So do not report
    cmake features, etc.
    
  * We try to minimize the assumptions of cmake internals in the implemention
    to allow cmake room to innovate without breaking changes to this.
    
  * Policies should not be visible at this level and do not need to be exposed
    to clients.
    
  FUTURE WORK:
  
  Stephen and I did not agree on how to handle other but similar use-cases that
  need similar information as to what is in this view and we will need to
  revisit this topic as new use-cases will get added in the future:
  
  Stephen prefers having special commands for each use-case, returning output
  tailored to that use-case.
  
  I would prefer having one command to gather the complete state of the
  project structure and then allow the user to limit what is returned from this
  command (e.g. only show one configuration/target/types of targets/etc. and
  options to filter out e.g. backtraces from the data that is returned).
  
  Stephens approach has the advantage that it is easier to use and more
  predictable what is returned. My approach allows client authors more
  flexibility as to which information they need without having to ask
  cmake developers to add a new command/change an existing one.
  
  But this is for the future: For the first iteration we want to concentrate on
  data relevant to feeding a code model.

  TODO:
  
  * Figure out whether a PDB file is actually going to be build. Currently
    the PDB file is added to the list of artifacts whenever there is a .lib
    file, which is probably wrong. Brad: Do you have an idea how to
    get this information?


  2.7 cache (report contents of CMakeCache.txt file)
  
  * Review by other potential users would be appreciated, but no obvious
    problems seen.

  
  2.8 buildsystem (return cmake files)
  
  * Return a list of cmake files (CMakeLists.txt, etc.) in different categories
    (in source directory/in build directory/in the system).
    
  * Stephen suggested to rename this to "cmake_rerun_depends". I am not sure
    that name is much better than what is currently used.


    
3.0 Features from Stephen's branch that need porting

   We only skimmed this, as many of his highlights need refactoring in cmake
   to land first.


   3.1 parse
   
   * This duplicates the parser and will need some refactoring.
   
   
Status Information:

I started to add some tasks to github to track what needs to be done before
proposing the daemon-mode branch for release. This can be viewed here:

https://github.com/hunger/CMake/issues

Best Regards,
Tobias

-- 
Tobias Hunger, Senior Software Engineer | The Qt Company
The Qt Company GmbH, Rudower Chaussee 13, D-12489 Berlin
Geschäftsführer: Mika Pälsi, Juha Varelius, Mika Harjuaho. Sitz der
Gesellschaft: Berlin, Registergericht: Amtsgericht Charlottenburg, HRB 144331 B