[CMake] Finding Python3

[Michael Hertling]

On 07/18/2010 09:07 PM, Branan Purvine-Riley wrote:
> On Sunday 18 July 2010 09:15:17 Michael Hertling wrote:
>> On 07/18/2010 06:50 AM, Branan Riley wrote:
>>> I've mad a very cursory effort to add Python 3 support to CMake. All
>>> I've done so far is take FindPythonLibs and FindPythonInterp, and
>>> change the variable names and the python versions. It doesn't yet have
>>> framework support on Mac, as I have no idea how to specify a Python 3
>>> framework. Besides that, the code should probably be adjusted so it's
>>> not duplicated across two modules - a FindPythonLibsCommon.cmake file
>>> or something that does all the "hard" work.
>>>
>>> I'm making this available now so people can play with it and give
>>> feedback. I'll clean up the code and get another message out when I
>>> think it's in a state it can actually be merged into the main CMake
>>> git tree.
>>>
>>> For now, my work is in a clone on github at
>>> http://github.com/branan/CMake/tree/python3 . You can add my work as a
>>> tracking branch to a checkout of the main CMake git tree with the
>>> following commands:
>>>
>>> git remote add -t python3 branan git://github.com/branan/CMake.git
>>> git fetch branan
>>> git checkout -b python3 branan/python3
>>
>> After having taken a look at FindPython3{Interp,Libs}.cmake, I would
>> recommend strongly against hardcoding version numbers in find modules.
>> There has already been discussions revealing serious drawbacks of this
>> habit, e.g. <http://www.mail-archive.com/cmake@cmake.org/msg28878.html>.
>> Moreover, FIND_PACKAGE() supports the specification of a version, and I
>> wonder why this feature is hardly used in such cases, in particular for
>> new modules. Furthermore, your approach probably suffers from the same
>> shortcoming as the current FindPython{Interp,Libs}.cmake: They can not
>> guarantee to deliver consistent interpreter/libraries combinations, cf.
>> <http://www.mail-archive.com/cmake@cmake.org/msg27324.html>.
>>
>> What would be really needed is a comprehensive FindPython.cmake, see
>> <http://www.mail-archive.com/cmake@cmake.org/msg27382.html>, which
>> addresses these issues entirely, i.e. version support and version
>> consistency and the question if the interpreter may be invoked to
>> gather information about the specific python installation, cf.
>> <http://public.kitware.com/Bug/view.php?id=2257#c11766>. See
>> <http://www.mail-archive.com/cmake@cmake.org/msg29701.html>
>> for my basic suggestions w.r.t. FindPython.cmake, in short:
>>
>> - A version specification through FIND_PACKAGE() is respected.
>> - At least, two components "interpreter" and "libraries", and if both
>>   are requested they must be consistent with regard to their versions.
>> - An external flag PYTHON_NATIVE indicates whether the interpreter may
>>   be invoked as this is the most secure and most elegant way to learn
>>   the location of the interpreter's belonging libraries but can't be
>>   done unconditionally, OTOH, for known reasons. So, it's up to the
>>   user if the interpreter may be run within FindPython.cmake.
>>
>> IMO, the transition from Python 2 to 3 is a great occasion to tackle
>> the problems of the appropriate find modules and for the emergence
>> of FindPython.cmake.
>>
>> Regards,
>>
>> Michael

[...]

> Unfortunately, I can't think of a way to get out of using hardcoded version 
> lists, since python is often in specific locations (or has a specific executable 
> name) based on its version number. I did make the version lists more robust, 
> basing them off of FindBoost.cmake. Hopefully this, combined with using the 
> interpreter to find the libraries, will help mitigate the problems seen with 
> the earlier FindPython* modules.

IMO, avoiding hardcoded versions is quite easy w.r.t. the interpreter:
Usually, it is installed as "python" and "python2.6", e.g. So, if you
request the interpreter without specifying a version you will get the
unversioned executable, and if the latter is not found, i.e. each one
is versioned, the search simply fails, and you must mention a version
with FIND_PACKAGE() to succeed; different locations can be handled as
customary with CMAKE_PREFIX_PATH et al.

The actual challenge is to find the python libraries with no version
at hand and without querying the interpreter as there is no symlink
"libpython.so" and even the static library incorporates the version
in its name. Perhaps, one can use a workaround like the following
to get the python version with a little help from outside:

CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)

PROJECT(PYTHONVERSION NONE)

FUNCTION(GET_PYTHON_VERSION VERSION)
    SET(v)
    FOREACH(i IN LISTS PYTHON_ROOT ARGN)
        FILE(GLOB j "${i}/lib/python*")
        FOREACH(k IN LISTS j)
            IF(k MATCHES "(^|/)python[0-9].*\$")
                STRING(
                    REGEX REPLACE "^.*python([0-9].*)\$" "\\1" l "${k}"
                )
                LIST(APPEND v "${l}")
            ENDIF()
        ENDFOREACH()
    ENDFOREACH()
    LIST(GET v 0 w)
    SET(${VERSION} ${w} PARENT_SCOPE)
ENDFUNCTION()

GET_PYTHON_VERSION(PYTHONVERSION "/usr" "/usr/local")
MESSAGE("PYTHONVERSION: ${PYTHONVERSION}")

The function GET_PYTHON_VERSION() looks for typical python library
directories and extracts the version from the first one found; the
search starts at PYTHON_ROOT which can be set by the CMakeLists.txt
or via the cache; further directories can be specified as fallbacks.
The result returned by the function may be used in subsequent calls
to FIND_LIBRARY(), so it's not necessary to hardcode versions in a
FindPython.cmake, and using variables like PYTHON_ROOT is rather
common. Of course, GET_PYTHON_VERSION() shouldn't be used if a
version is explicitly specified via FIND_PACKAGE().

In summary, I'm somewhat confident that it is possible to get rid of
hardcoded versions, at least in a FindPython.cmake, and it should be
definitely aimed for as this habit requires continuous maintenance
and means asking for trouble - sooner or later.

Btw, <http://www.mail-archive.com/cmake@cmake.org/msg28114.html> could
ease the situation dramatically, so we should possibly keep an eye on.

Regards,

Michael