[cmake-developers] cmake daemon mode protocol

[Stephen Kelly]

Tobias Hunger wrote:

> Hi Stephen,
> 
> I have successfully build and run your cmake server mode changes and
> the python client script does work as advertised.

Thanks for doing that!

> I do have a couple of remarks about it. This is more intended as a
> starting point for discussion as a real proposal. Would something
> along these lines be possible:


> * Start daemon-mode without any additional parameters
> 
> * Daemon responds with "
>   {
>     "type": "handShake",
>     "version": "3.5.x",
>     "supportedProtocols": "3.5"
>   }

As you discovered, the cmake binary might not even support the daemon mode 
at all.

I think what you are really asking for is 'the capabilities of the cmake 
binary'. In that sense, you might also want to know what generators are 
available, whether daemon mode is available (and with which protocol 
versions) etc.

The issue at

 http://public.kitware.com/Bug/view.php?id=15462

proposes to dump that information to stdout with a different -E command.

> 
> * Client Handshake:
>   {
>     "type": "handshake",
>     "requestProtocolVersion": "3.5"
>   }
> 
> * Daemon:
>   {
>     "type:"handshake",
>     "protocolVersion": "3.5"
>   }

Something like this is possible too. It's not far off what is already done 
in the branch.

> 
> At this point the normal operation should be possible.
> 
> * Client:
>   {
>      "type": "setProjectInformation",
>      "sourceDirectory": "/some/path",
>      "cookie": "1"
>   }
> 
>   Being able to set the source directory is important so that a fresh
> checkout can be configured. There is no build directory available at
> that time...

I understand the desire to inspect a cmake project without asking the user 
to specify a build dir, but cmake has to write build files somewhere in 
order to test the compiler etc. 

For this case, I suggest that if the user tries to 'open the source 
directory', you would use QTemporaryDir to build in a temporary location and 
run the daemon there. I believe clion does something equivalent to that. 

Is that viable? I suppose you are suggesting that cmake do that instead of 
leaving it for clients to do?

> * Daemon should do progress handling while reading the CMakeLists.txt
> or whatever it does:
>   { "type": "progress", "state": "busy", "task": "someId",
> "taskDescription": "Reading CMakeLists.txt files", "cookie": "1" }
> 
> * Daemon sends occasional:
>   { "type": "progress", "task": "someId", progressTotal: 25,
> progressCurrent: 12, "cookie": "1" }

Yes. CMake can provide approximate values of how complete the configure step 
is. See cmGlobalGenerator::AddMakefile. That could be exposed to the daemon 
'somehow', such as by defining some virtual callback interface. 

Richer information about some semantics like 'task' and 'busy state' could 
also be provided in a similar way I expect, assuming those could be defined.

> * Finally Daemon responds sends data as that is available now.
>   {
>     "type": "projectInformation",
>     "sourceDirectory": "/some/path",
>     "buildDirectory": "",
>     "projectName": "SomeProject",
>     "cookie": "1"
>   }

At this point in your proposal, cmake has run the configure step in a 
temporary directory, right? So it would be 

  "buildDirectory": "/tmp/foo/bar"

right?

> * Client:
>   {
>     "type": "setProjectInformation",
>     "buildDirectory": "/some/other/path",
>     "cookie": "2"
>   }

As this sets a new build directory, cmake will have to run the configure 
step from scratch in this new location.

> * Daemon does progress indication as above...
> 
> * Daemon responds with project info:
>   {
>     "type": "progressInformation",
>     "sourceDirectory": "/some/path",
>     "buildDirectory": "/some/other/path",
>     "projectName": "SomeProject",
>     "cookie": "2"
>   }

I must admit I'm not seduced by this idea as it seems to be that it should 
be easy for an IDE to build in a temporary dir itself. 

However, I think it makes sense to design the protocol to handle specifying 
either a sourceDir or a buildDir (or both if from scratch). Starting daemon 
mode without arguments as you suggest could work (though we would have to 
see how that affects the protocol). Or with a different argument (eg

 cmake -E daemon buildDir
 cmake -E daemon_source sourceDir

) could work. I am also just brainstorming ideas, not proscribing anything.

> We might also want a "ping"/"pong" to detect if the daemon is still
> responding. 

Yes, I had the same thought. I also considered self-terminating the daemon 
if there is no message on the wire for N seconds, effectively requiring the 
IDE to ping every N seconds if it is not doing anything else.  I considered 
this because otherwise the IDE might crash and leave zombie cmake daemons 
running, but I didn't investigate that possibility.

> That would require the daemon to be able to handle
> multiple requests to be in flight... not sure that this is worth it.

Yes, adding a multi-threading requirement to the daemon might make some 
implementation more difficult. I've seen some videos on designs for things 
like that though, and I am interested in trying them out.

> But if we can not do that, then we need to provide constant progress
> information, which is also not great:-)

The good news is that after the initial configure and compute steps are 
finished (where progress makes sense anyway I think), all other requests are 
'very fast'.

> Not sure about the need for cookies (text set by the client and
> repeated by the server in all its replies to that request). I do think
> having them makes sense, even when only one request can be active at
> any time: They make it so much easier to sanity-check communication.

Yes, I considered something like that in the protocol too. I didn't add any 
because I didn't encounter the need yet in the parts of the protocol I 
implemented and the proof-of-concept tooling I made, so I deferred that part 
of the design space until a time when more requirements are known.

> Consistently having a "type" in all messages going back and forth is
> probably a good idea for the same reason as having cookies.

Yep.

Thanks for this useful feedback!

Steve.