[cmake-developers] Converting CMake documentation to reStructuredText and Sphinx

[Brad King]

Hi Folks,

The need for a more flexible documentation system has come up a few
times and proposals for new general-purpose documentation sections
were made at least in two threads:

 Adding some generic section to the docs ?
 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/5924

 cmake --help-concept
 http://thread.gmane.org/gmane.comp.programming.tools.cmake.devel/7799

CMake's builtin documentation generation using C++ string-literals
with simple "indented => preformatted" markup has served us well for
many years but it is time to move to something better.  Lightweight
markup languages have a similar feel to our existing system but are
much more widely used and mature.  After considering markup languages
such as Markdown and AsciiDoc I settled on reStructuredText:

 http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html

It has a great extension mechanism and is supported by fantastic
tools like Sphinx to handle suites of multiple document files with
cross-referencing links among them:

 http://sphinx-doc.org/

---------------------------------------------------------------------

I've prototyped a topic branch to convert CMake's documentation over
to .rst source files and build them using Sphinx.  It is published
in my Github repository:

 https://github.com/bradking/CMake/tree/doc-reStructuredText

The last commit adds the generated man and html files for purposes
of demonstration.  Read the commit message to see how to view them.
I also temporarily published the generated html documentation here:

 http://www.cmake.org/cmake/help/v2.8.12.20131009-g9a890/
 (Try out the Index and Search pages!)

Take a look at the converted documentation sources here:

 https://github.com/bradking/CMake/tree/doc-reStructuredText/Help

Note that the conversion process is automated so the document files
can be made better with manual tuning over time after the permanent
conversion is done.  We can also add arbitrary new manual pages
pretty easily as demonstrated by the cmake-generator-expressions
page.

The topic includes a cmRST class that does basic reStructuredText
processing so that most of the --help-* options can continue to
work.

---------------------------------------------------------------------

The topic also defines Sphinx extensions to add reStructuredText
directives for cmake "domain" objects like commands and variables.
This can be used to add indexed and x-refable variable documentation
inside .cmake module files.  Module files can contain rst markup
inside comments.  For example, here is part of Modules/CPack.cmake:

 #.rst:
 # CPack
 # -----
 #
 # Build binary and source package installers.
 #
 # Variables common to all CPack generators
 # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 #
 [snip]
 #
 # .. variable:: CPACK_PACKAGE_NAME
 #
 #    The name of the package (or application). If
 #    not specified, defaults to the project name.

It gets loaded from a Help/module/CPack.rst document containing

 .. cmake-module:: ../../Modules/CPack.cmake

where "cmake-module" is a directive added in our Sphinx extension
that knows how to scan .cmake comments to extract "#.rst:" blocks
(which can now be anywhere in the file e.g. next to macro defs).

Variables and commands defined with directives like the above can
be cross-referenced with cmake domain roles e.g.

 See the :variable:`CPACK_PACKAGE_NAME` variable...

Our Sphinx extension also automatically transforms all documents
in Help/<type>/*.rst files to cmake domain objects of <type> so
they are indexed and can be cross-referenced e.g.

 See command :command:`add_dependencies` to add target-level deps.
 See the :prop_tgt:`OUTPUT_NAME` target property to set a file name.
 See the :variable:`CMAKE_<LANG>_FLAGS` to add flags.

---------------------------------------------------------------------

Since this topic makes sweeping changes throughout the source code
we'll have to schedule a "flag day" on which I will freeze the
repository and apply the permanent conversion.  Currently I plan
for this to be just after 2.8.13 is released.

-Brad