View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002257CMakeModulespublic2005-09-19 17:072016-06-10 14:30
ReporterMathieu Malaterre 
Assigned ToMarcus D. Hanwell 
PrioritylowSeverityfeatureReproducibilityalways
StatusclosedResolutionmoved 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0002257: FindPythonLibs.cmake deficiencies
DescriptionIt would be nice if FindPythonLibs.cmake followed the same approach as the m4 script:

http://www.python.org/pycon/dc2004/papers/42/ac/python.m4 [^]

In which case the user only has to specify which python he wants to use. And executing proper python script allow one to find out where the python installation is.

Also it will provide a more robust approach where the current script fails, see bug #2227 for more detail.


Example scripts are:
#!/usr/bin/python
import distutils.sysconfig
print distutils.sysconfig.get_config_var("MODLIBS")

 >>> print distutils.sysconfig.get_config_var("LIBPL")
/usr/lib64/python2.3/config

and
print distutils.sysconfig.get_config_vars()
TagsNo tags attached.
Attached Files? file icon FindPythonInterp.cmake [^] (2,185 bytes) 1969-12-31 19:00
? file icon FindPythonLibs.cmake.rc1 [^] (6,316 bytes) 1969-12-31 19:00
? file icon FindPythonLibs.cmake.rc2 [^] (6,488 bytes) 1969-12-31 19:00
? file icon FindPythonLibs.cmake [^] (5,751 bytes) 1969-12-31 19:00

 Relationships

  Notes
(0003665)
David Gobbi (reporter)
2006-02-01 12:15

A followup from an email I posted on the list. This might work as a quick fix:

If the VTK install prefix is /usr/local, then the python wrappers will be installed here:

 /usr/local/python2.3/site-packages/

If python is installed somewhere else, (e.g. /usr/ is a common location) then installing
the wrappers in /usr/local isn't very useful.

The default behaviour of setup.py is to query the python executable to find out where it is
installed. This behaviour is explicitly overridden by VTK/Wrapping/Python/CMakeList.txt:

 SET(VTK_PYTHON_SETUP_ARGS "--prefix=\"${DOLLAR}{CMAKE_INSTALL_PREFIX}\""
     CACHE STRING "Arguments passed to \"python setup.py install ...\" during installation.")

I think that having a customizable VTK_PYTHON_SETUP_ARGS is a good idea, but the default
value of this variable is not going to work for most people.

The best option is to query the PYTHON_EXECUTABLE to see where it was installed, and use that
as the wrapper install prefix instead of using CMAKE_INSTALL_PREFIX.

The python install prefix can be queried by running 'python -c "import sys; print sys.prefix"',
where 'python' is whatever PYTHON_EXECUTABLE has been set by cmake.
(0004152)
b_mann (reporter)
2006-05-30 15:52

Hi
ok, apparently this bug is pretty old already, but still hurts:
http://public.kitware.com/pipermail/cmake/2006-May/009383.html [^]

I have come over at least 2 or 3 (unix) systems where there is only a static library installed in python${VERSION}/config and thus cmake tries to link against the static library but fails because of the missing dependencies (e.g. -lutils).

The solution proposed here (querying the required libs from python) is probably one of the cleanest solutions to this problem - if we can assume that the python executable is always installed, if the libraries are installed (can we?)
I have implemented this proposal for my project:
http://svn.sourceforge.net/viewcvs.cgi/boson/trunk/tools/cmake/modules/boson/FindBosonPythonLibs.cmake?view=log [^]

In addition to having the dependencies fixed, this version also honors the REQUIRED and QUIET hints of find_package.
Note however that this implementation is most likely not working on windows, as I don't have a windows machine to test it on.

(and why the heck does this have a Severity of "Feature Request" ? It certainly is a bug to me, if the program linking fails)

CU
Andi
(0005649)
Tristan Carel (reporter)
2006-11-02 10:53

FindPythonLibs.cmake has to be updated on the cvs each time a new version of Python is released and the FIND_XXX commands always select the last version.

In other hand, the module `distutils' of Python provides all the necessary to know where are located the libraries, headers and particularly the list of libraries we have to link against required by the python static library.

I rewrote `FindPythonLibs.cmake' which takes idea of the python.m4 and the file http://svn.sourceforge.net/viewvc/boson/trunk/tools/cmake/modules/boson/FindBosonPythonLibs.cmake?view=markup [^]

The main cache variables defined by this module depend of the variable PythonInterp_EXECUTABLE (defined by the module FindPythonInterp.cmake). It means that each time the python executable path is changed, the cache variables of FindPythonLibs are computed. It's not done each time you launch `configure' so that it doesn't erase entries that could be specified manually by the user.

I've tested it under:
- MacOS
  - python 2.3 installed as framework
- Windows
  - python 2.3, 2.4, and 2.5 installed with .msi
- Linux Debian
  Python 2.3 and 2.4 with static/dynamic linkage

The code tries to be compliant with Modules/readme.txt and keeps compatibility with previous version.

I also updated FindPythonInterpret.cmake

The main lack is that now to use Python libraries, the user has to also have the interpreter (strange/insane case but maybe could happen)


If you're interested to put these files into the cvs, please review the 2 files and tell me what's wrong.

attachment:
FindPythonInterp.cmake.rc1
FindPythonLibs.cmake.rc1

thx by advance
(0005654)
Tristan Carel (reporter)
2006-11-03 12:17

My apologizes, I've just noticed that I mixed up the variables names by trying to keep compatibility with previous version.
If still interested, please now consider the rc2 version.
(0007590)
vjaddoe (reporter)
2007-05-08 09:41

I've attached a cmake file that uses python-config to find the include/lib paths. Especially useful if you have multiple versions of Python installed on the same system. It is assumed that the python-config that occurs first in $PATH is the preferred python version.
(0011742)
Marcus Hanwell (old account) (developer)
2008-05-07 21:58

We are getting hit by this issue even in CMake 2.6. We have a couple of projects where we need to link against Pyton but CMake incorrectly chooses the static library in /usr/lib64/python2.5/config/libpython2.5a - what I actually want to link against in in /usr/lib64/libpython2.5.so. Is there any reasonable solution to this? Why is it marked as a low priority? If we are not supposed to use FindPythonLibs then I would appreciate advice on what should be used.

Thanks.
(0011745)
Brad King (manager)
2008-05-08 08:44

The reason nothing is happening on this bug is that there is no maintainer for the FindPython* modules:

  http://www.cmake.org/Wiki/CMake:Module_Maintainers [^]

If someone wants to volunteer to maintain the modules then he/she can be given write access to make changes.
(0011765)
Brad King (manager)
2008-05-08 14:22

I just spent a little time investigating this issue. Here is information about what is available from python on each platform:

------------------------------------------------------------------------------
Windows

distutils.sysconfig.get_config_vars():

  BINLIBDEST c:\Python25\Lib
  INCLUDEPY c:\Python25\include
  LIBDEST c:\Python25\Lib
  SO .pyd
  exec_prefix c:\Python25
  prefix c:\Python25
  (there is no "config" directory)

files:

  c:/Python25/include/Python.h
  c:/Python25/libs/python25.lib

On windows there may also be a debug version named "python25_d.lib".

------------------------------------------------------------------------------
Cygwin

distutils.sysconfig.get_config_vars():

  BINLIBDEST /usr/lib/python2.5
  INCLUDEPY /usr/include/python2.5
  LIBDEST /usr/lib/python2.5
  SO .dll
  exec_prefix /usr
  prefix /usr
  LIBPL /usr/lib/python2.5/config
  (and many others with the same information)

files:

  /usr/include/python2.5/Python.h
  /usr/lib/python2.5/config/libpython2.5.dll.a
  /usr/bin/libpython2.5.dll

------------------------------------------------------------------------------
Mac OS X

distutils.sysconfig.get_config_vars():

  BINLIBDEST /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3
  INCLUDEPY /System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3
  LIBDEST /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3
  SO .so
  exec_prefix /System/Library/Frameworks/Python.framework/Versions/2.3
  prefix /System/Library/Frameworks/Python.framework/Versions/2.3
  LIBPL /System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/config
  INSTSONAME Python.framework/Versions/2.3/Python
  PYTHONFRAMEWORKINSTALLDIR /System/Library/Frameworks/Python.framework

files:

  /System/Library/Frameworks/Python.framework/Versions/2.3/include/python2.3/Python.h
  /System/Library/Frameworks/Python.framework/Versions/2.3/Python

Framework linking is typically done via "-framework Python".
The current CMake FindPythonLibs supports frameworks.

------------------------------------------------------------------------------
Linux

distutils.sysconfig.get_config_vars():

  BINLIBDEST /usr/lib/python2.5
  INCLUDEPY /usr/include/python2.5
  LIBDEST /usr/lib/python2.5
  SO .so
  exec_prefix /usr
  prefix /usr
  LIBPL /usr/lib/python2.5/config
  INSTSONAME libpython2.5.so.1.0
  (and many others with the same information)

files:

  /usr/include/python2.4/Python.h
  /usr/lib/python2.5/config/libpython2.5.a
  /usr/lib/python2.5/config/libpython2.5-pic.a
  /usr/lib/python2.5/config/libpython2.5.so -> ../../libpython2.5.so
  /usr/lib/libpython2.5.so -> libpython2.5.so.1
  /usr/lib/libpython2.5.so.1 -> libpython2.5.so.1.0
  /usr/lib/libpython2.5.so.1.0

This is from a debian installation. Other distributions may vary.
(0011766)
Brad King (manager)
2008-05-08 14:26

Unfortunately the "FindPythonLibs.cmake.rc[12]" files attached to this issue cannot be used in their current state. Finding python cannot depend only on running the interpreter because

  1.) The interpreter may not be installed
  2.) The interpreter may not be able to run on the host platform when cross compiling.

The python-config approach may work when cross compiling, but I have not checked.
(0011817)
Brad King (manager)
2008-05-12 11:18

Unfortunately it looks like the python-config script just runs the interpreter internally.
(0012397)
Brad King (manager)
2008-06-18 09:17

Here are some additional comments I received via email about trouble with finding python. Any new module should take these into account.

---------------------------------------------------------------------------------
From: Hans Meine

I just tried installing VTK-5.0.4 on a system where all software is installed
in a common NFS tree, strictly separated into shared stuff (--prefix) and
platform-specific stuff (think --exec-prefix). With Python, this means that
Python.h is in the shared include directory, but the required pyconfig.h is
in a platform-specific include directory:

[2] meine@kogspc33:~ -> ll /software/python-2.4.4/include/python2.4/Python.h
-rw-r--r-- 1 meine uucp 4178 Feb 15
2007 /software/python-2.4.4/include/python2.4/Python.h
[2] meine@kogspc33:~ -> ll /software/python-2.4.4/include/python2.4 | wc
     77 686 4167
[2] meine@kogspc33:~ -> find /software/python-2.4.4/SuSE-10.2/include
/software/python-2.4.4/SuSE-10.2/include
/software/python-2.4.4/SuSE-10.2/include/python2.4
/software/python-2.4.4/SuSE-10.2/include/python2.4/pyconfig.h

The problem is that I don't know how to pass the two required include paths to
CMake. AFAICS, VTK uses the standard FindPythonLibs.cmake (disclaimer: I am
a CMake newbie), which does not seem to support two include paths?!

---------------------------------------------------------------------------------
From: Hans Meine

In the past, I had problems with exec-prefixes not being supported (i.e.
python using two include directories, see attached mail), and now I have the
problem that a project (gizmod) using cmake does use FindPythonLibs, but not
with VTK's extension which allows to specify extra libs to be linked against.
I am referring to the following snippet from VTK/CMakeLists.txt:

> > # Some python installations on UNIX need to link to extra libraries
> > # such as zlib (-lz). It is hard to automatically detect the needed
> > # libraries so instead just give the user an easy way to specify
> > # the libraries. This should be needed only rarely. It should
> > # also be moved to the CMake FindPython.cmake module at some point.
> > IF(UNIX)
> > IF(DEFINED PYTHON_EXTRA_LIBS)
> > ELSE(DEFINED PYTHON_EXTRA_LIBS)
> > SET(PYTHON_EXTRA_LIBS "" CACHE STRING
> > "Extra libraries to link when linking to python (such as \"z\" for
> > zlib). Separate multiple libraries with semicolons.")
> > MARK_AS_ADVANCED(PYTHON_EXTRA_LIBS)
> > ENDIF(DEFINED PYTHON_EXTRA_LIBS)
> > ENDIF(UNIX)

In fact, all necessary information about header and library locations and
configuration is available from a python interpreter and I am currently using
this code to start cmake appropriately for VTK:

> > python_include_dirs() {
> > #python -c 'import distutils.sysconfig as s; print "
> > ".join(set((s.get_python_inc(), s.get_python_inc(1))))' python -c 'import
> > distutils.sysconfig as s; print s.get_python_inc(1)' }
> >
> > python_extra_libs() {
> > python -c 'import distutils.sysconfig as s; c = s.get_config_var; print
> > c("LOCALMODLIBS"), c("LIBS")' }
> >
> > python_library() {
> > python -c 'import distutils.sysconfig as s, os; c = s.get_config_var;
> > print os.path.join(c("LIBDIR"), c("LIBRARY"))' }
> >
> > cmake ...
> > -DPYTHON_INCLUDE_PATH="`python_include_dirs`" \
> > -DPYTHON_LIBRARY="`python_library`" \
> > -DPYTHON_EXTRA_LIBS="`python_extra_libs`" \

Note that python_include_dirs still contains the code for the separate include
dirs, but I had to manually symlink all header files in order to install VTK.
(0025270)
David Cole (manager)
2011-02-03 19:21

Marcus..... this FindPython* bug is really old... does it still apply after your recent work on FindPython stuff? Can it be closed out, or is there still more unifying work to be done....?

Thanks,
(0025274)
Marcus D. Hanwell (developer)
2011-02-03 21:46

I think there is still more work to be done here, things have improved but I do not optionally run the interpreter, nor have I added a method to ensure the interpreter and library found match for example.
(0028691)
Rolf Eike Beer (developer)
2012-02-21 15:52

The current next branch has better support for python version selection for both PythonInterp as well as PythonLibs. Would you try testing and reporting back if that solves your issues?
(0041299)
Kitware Robot (administrator)
2016-06-10 14:27

Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.

 Issue History
Date Modified Username Field Change
2008-05-07 21:58 Marcus Hanwell (old account) Note Added: 0011742
2008-05-08 08:44 Brad King Note Added: 0011745
2008-05-08 14:22 Brad King Note Added: 0011765
2008-05-08 14:26 Brad King Note Added: 0011766
2008-05-12 11:18 Brad King Note Added: 0011817
2008-06-18 09:17 Brad King Note Added: 0012397
2011-02-03 19:20 David Cole Assigned To Brad King => Marcus D. Hanwell
2011-02-03 19:21 David Cole Note Added: 0025270
2011-02-03 21:46 Marcus D. Hanwell Note Added: 0025274
2012-02-21 15:52 Rolf Eike Beer Note Added: 0028691
2016-06-10 14:27 Kitware Robot Note Added: 0041299
2016-06-10 14:27 Kitware Robot Status assigned => resolved
2016-06-10 14:27 Kitware Robot Resolution open => moved
2016-06-10 14:30 Kitware Robot Status resolved => closed


Copyright © 2000 - 2018 MantisBT Team