[CMake] Installing Python Modules with CMake

[Michael Hertling]

On 01/04/2011 08:50 AM, Michael Wild wrote:
> On 01/03/2011 06:23 PM, Todd Gamblin wrote:
>> On Jan 3, 2011, at 1:34 AM, Michael Wild wrote:
>>
>>> On 01/03/2011 09:11 AM, Todd Gamblin wrote:
>>>> Is there some standard way in CMake to get a project-local install destination for python libraries, e.g.:
>>>>
>>>> 	${CMAKE_INSTALL_PREFIX}/lib/python2.6/site-packages
>>>>
>>>> Is this something find modules typically provide?  I can construct this name easily enough myself, but it involves querying the python executable for its version then regexing that string, and I feel like FindPythonLibs should do stuff like that for me.  It would be nice to have something called, say, "CMAKE_PYTHON_INSTALL_PREFIX" set to this path in advance, so that I can just install python modules there.
>>>>
>>>> Or am I missing something?  I couldn't fund much on installing python files in the docs.
>>>>
>>>> -Todd
>>>>
>>>
>>> I think you'll have to query Python yourself. Of course, this becomes
>>> difficult if you're cross-compiling, in which case you would need to
>>> parse the <prefix>/lib/pythonX.Y/config/Makefile file yourself and fetch
>>> all the relevant variables (datarootdir, BINDIR, DESTSHARED, LIBDEST,
>>> etc.) and then ask the user what prefix and exec_prefix he wants
>>> (presumably CMAKE_INSTALL_PREFIX maps to prefix, and usually exec_prefix
>>> is the same as prefix, but that's not always the case). Not sure how
>>> that works on Windows, though...
>>
>> That's kind of difficult because if I have just 'python', I don't have the version yet, so how am I supposed to find the Makefile?
>>
>> Off the top of my head, the most portable way to do this would be to find the python-config (which you can get by appending -config to the exe path, even if it's just python and not pythonX.Y) and then:
>>
>> 	head -1 `which ${PYTHON_CONFIG}` | sed 's/.*\([0-9].[0-9]\)$/\1/'
>>
>> Or the equivalent in CMake using string(REGEXP ...).
>>
>> That will get you the version without running anything, and then you can go and find the Makefile.
> 
> 
> That only works on *NIX, on Windows there's AFAIK no python-config. So
> if you plan to cross-compile for Windows you might run into trouble.
> Probably it's best to not try being über-smart, and just leave it to the
> user to provide the information.
> 
> 
>> In the non cross-compiled case, though, it would be nice to just have a module that gives you all the information from python-config.
> 
> Better run
> 
> ${PYTHON_EXECUTABLE} -c "import sys; sys.stdout.write(sys.prefix+'\n')"
> 
> and similar, which also works on WIN32.
> 
>>
>> -Todd
>>
>>
>>> I wouldn't rely too much on FindPythonLibs. The whole FindPython* is a
>>> sorry mess (IMHO) and needs some serious overhaul, but that is some
>>> major undertaking an apparently the suffering wasn't big enough so far
>>> for somebody to take action...
>>
>> I noticed you guys had a long discussion about finding Python 3.  Did you come to any conclusions on this?  It would be nice to have a python detection Find module that works if you *just* give it your python executable.  I don't mind writing something like this up if no one else is up for it, but if I'm going to write it I'd like to make something that people will use.
>>
>> Thoughts?
>>
> 
> AFAIK, the issue hasn't been resolved. One of the big problems IMHO is,
> that FindPythonInterp and FindPythonLibs can report different python
> installations. Also, what should happen if the user manually changes the
> cache entry for the interpreter, but doesn't update the libraries? Also,
> some users of the module will just want the "most recent version" of
> Python, while others will specifically want the most recent 2.x version
> because their code isn't ready for python3. If
> you can come up with something that works well on all important
> platforms and is reasonably backwards compatible, you probably would
> make a lot of people happy ;-)
> 
> Michael

On Todd's request for commentary. Please note that the following
remarks are merely my personal opinion; of course, one can take
different positions:

A major obstacle for the development of smart/robust/comprehensive
find modules is the current incapability of the four find functions
FIND_{PATH,FILE,PROGRAM,LIBRARY}() to process filename patterns, e.g.
FIND_PROGRAM(PYTHON_EXECUTABLE PATTERN python*.* ...) or regex'd with
FIND_PROGRAM(PYTHON_EXECUTABLE PATTERN python[0-9]+\.[0-9]+ ...). All
these functions are able to look in any corner of the system, but they
search for fixed filenames only, especially resulting in problems like
the one reported with versioned Python directories. Furthermore, this
limitation constrains the authors of several find modules to hard-code
versions, e.g. Find{Python{Interp,Libs},TCL,PNG,Boost}.cmake, bearing
the need for continuous maintenance on the developers' as well as the
users' side. Additionally, issues like 10718 occasionally prevents the
right version from being found. During the last few months, we've seen
all of the above-mentioned find modules on the mailing list because of
difficulties related to the hardcoded version numbers they incorporate.

Recently, the affiliated issue 8396 - meanwhile deferred again - has
reappeared on the scene. Here, it is my hope that Bill Hoffman et al.
can be convinced to adopt James Bigler's suggestion in [1] and equip
all four find functions with regex support. This would fit them much
better then an immediate version selection as proposed initially in
8396; the latter could probably be very well performed by a CMake-
scripted function which operates on a list of suitable candidates
returned by FIND_LIBRARY() & Co.

Another topic regarding the improvement/development of find modules
is their component-awareness. While almost every non-trivial package
gives reason for a component-aware finder, only very few of them are
actually realized in that way; some are even split with the risk of
delivering inconsistent results, e.g. FindPython{Interp,Libs}.cmake.
Although this design probably originates in CMake's own development
history, the use of components in find modules should be emphasized
more strongly. However, concerning this matter, there is hardly any
documentation with recommendations, best practices etc. A few weeks
ago, there was a quite extensive discussion about some fundamental
aspects of multi-component packages and their find modules, but to
my regret, no one else offered an opinion on the points at issue.

In summary, my vision for find modules - not only the Python one[s]:

1) Regex capabilities of FIND_{PATH,FILE,PROGRAM,LIBRARY}(), and an
   accompanying CMake-scripted function to choose the right version
   from the returned lists, e.g. SELECT_VERSION() with the VERSION_
   {EQUAL,GREATER,LESS,GREATER_OR_EQUAL,LESS_OR_EQUAL} options that
   have been initially proposed by Philip Lowman for FIND_LIBRARY().
2) If this is not feasible, perhaps due to function types, backward
   compatibility or other far-reaching consequences, a consideration
   to introduce, say, FIND_{PATHS,FILES,PROGRAMS,LIBRARIES}(), quite
   equivalent to their singular-named counterparts, but which accept
   regex patterns instead of names and return their results in lists.
3) Further discussion of component-aware find modules / config files,
   a broad consent and fixation of reasonable advices and guidelines.
4) By use of these progresses, improvement of concerned find modules,
   particularly regarding their version- and component-awareness in
   junction with the appropriate support provided by FIND_PACKAGE().
5) New find modules like the much longed for FindPython.cmake. ;-)

Best regards,

Michael

[1] http://public.kitware.com/Bug/view.php?id=8396#c23144