[CMake] What does `cross-platform' mean?

[Michael Jackson]

On Nov 9, 2008, at 1:30 PM, Jed Brown wrote:

> On Fri 2008-11-07 10:47, Bill Hoffman wrote:
>> So, currently CMake assumes that the environment you are working  
>> from is
>> correctly configured.  I think adding CMakeModule would go beyond the
>> scope of what CMake should be doing.   Just like you have to have a
>> working compiler in your path, you should also have a working
>> environment.
>
> Is my environment not `working' simply because I'm asking to link the
> versions of libraries that are found at a path other than the system
> defaults?  Using something else and asking the user to edit the  
> cache is
> like ignoring CC and requiring the user to edit CMAKE_C_COMPILER,
> CMAKE_LINKER, CMAKE_AR, CMAKE_RANLIB, etc to support a different
> toolchain.
>
>> However, I am sure the find_* stuff can be further refined, and extra
>> error checks put in place to make sure ABI's match.  CMake works  
>> best if
>> it can find a full path to a specific library.   The idea of a  
>> command
>> that turns -L/path -lA into /path/libA.a sounds like a useful thing  
>> to
>> add.
>
> See this:
>
>  http://github.com/jedbrown/cmake-modules/tree/master/ResolveCompilerPaths.cmake
>
> It will fail on pathological inputs (we need a way to preserve
> shell quoting and tokenizing semantics, I don't aspire to write the
> fully Posix-complient regex), but should be much better than existing
> attempts (in FindPkgConfig and FindMPI).
>
>> So, as to the title of this thread, Cross platform means Windows,  
>> OSX,
>> Linux (all distros), HPUX, IRIX, SunOS, Haiku, QNX, cygwin, mingw.
>
> Yes, but the Find* stuff requires manual intervention (like editing
> lots of advanced cache entries) when shared libraries are not  
> available
> or when the user wants to link a version in a nonstandard location  
> (when
> there is also a version in a standard location).  It's a hard problem
> that apparently hasn't been addressed.  That was the root of my
> question.
>
>> As far as I can tell, the only problem you are talking about here, is
>> when you link to a static library that depends on other static  
>> libraries
>> CMake is not figuring out the depend information for you.  That
>> information has to come form somewhere.  It comes from:
>>
>> 1. pkg-config output, which potentially needs the above command  
>> that can
>> turn -L -l into full paths for the libraries.
>>
>> 2. from makefile fragments
>>
>> 3. If it is a cmake project from that package config stuff
>>
>> 4.  Other places I am sure.  There really is no standard for this.
>>
>> So, I would appreciate constructive feedback on what specific  
>> commands
>> and enhancements can be added to CMake in order to make the writing  
>> of
>> Find* modules easier to do and get right.  I would suggest that the
>> CMake Wiki be used to come up with some ideas.   Maybe add a section
>> here:
>>
>> http://www.vtk.org/Wiki/CMake#More_Topics
>>
>> Something like
>> "Proposal for improved commands to better handle third party library
>> dependency information"
>
> I added two pages to that section
>
>  http://www.cmake.org/Wiki/CMake:Static_libraries
>  http://www.cmake.org/Wiki/CMake:Multiple_versions
>
> The second addresses the issue of making a module with subordinate  
> cache
> entries.
>
>> I would also like to comment on this quote:
>> "CMake while finding that many modules (most of which I don't
>> actually care about) are systematically broken in niche environments
>> which are disproportionately popular on clusters, an important
>> demographic for my projects."
>>
>> CMake has been used on many cluster environments by Kitware with
>> ParaView and other software, so the issues you are having are not  
>> even
>> widespread in that environment.
>
> Mandatory static libraries are certainly not rare.  Neither is having
> multiple versions of libraries and wanting a version in a non-standard
> location.  The clusters that I've used usually provide a core set of
> libraries managed by a `module' tool which sets up the environment so
> the correct version is found first, but if you need anything other  
> than
> the default build, you build it in your home directory and specify the
> path.  On every system I've worked on, lots of users have local builds
> of libraries which they want to link.
>
> Remember that ParaView bundles it's dependencies so there is little
> concern about finding the right version or tracking down dependencies.
> For instance, ParaView is bundled with a forked (at least by build
> system) version of HDF5.  It is not practical nor desirable for every
> project to do this.
>
> While it's acceptable for an application to bundle deps, it's  
> certainly
> not acceptable for a library where a selling point is that  
> applications
> can combine libraries.  That is, if libB and libC both depend on libD,
> it's not okay for them to both bundle libD (unless they provide a
> reliable way to get a non-bundled version).  AppA, on the other  
> hand, is
> guaranteed to be the end of the line so it's acceptable to bundle  
> libB,
> libC, and libD (even though AppA may never call libD directly).
>
>> However, there is always room for improvement.  So, if you are  
>> serious
>> about helping or other folks want to help with this issue, lets start
>> with a wiki article and see where this goes.
>
> Wiki started, see above.  It would be really useful for a CMake
> developer to look at these modules
>
>  http://github.com/jedbrown/cmake-modules/tree/master/ResolveCompilerPaths.cmake
>  http://github.com/jedbrown/cmake-modules/tree/master/FindPackageMultipass.cmake
>
> and decide if something similar should be added to CMake.  The most
> important thing is to decide on a stable API for doing these  
> operations
> because when/if support for static libs and multiple versions is  
> added,
> almost every module will need to be updated (unless someone is very
> clever).
>
>> One last thought...   The Find* stuff in CMake is hard!   Software is
>> installed all over the place, and often times in non-standard  
>> locations.
>> That is why there is a cache editor in CMake so if it fails to find  
>> the
>> right stuff, you can manually fix it.
>
> Find* is definitely hard.  For static libs, I see editing the advanced
> cache as a non-solution.  A nontrivial library will require the user  
> to
> fill in lots (like 40) full paths to static libraries that they may  
> not
> even know about (they're recursive dependencies).  In the best case,
> this means tracking down documentation for the interface library and
> learning how to link the dependency graph, then hand-resolving the  
> link
> line and writing it into the advanced cache entry.  Failing that, it
> means compiling with VERBOSE=1 and, reading the linker errors,  
> inferring
> the necessary library from the names of unresolved symbols, manually
> putting this library in the cache, and repeating.  Very few users have
> the patience for either.
>
>> Also, if anyone knows of a build system that does any of this, we can
>> always copy what that system does.  Although, I don't think any of
>> them do much of this stuff...
>
> Yes, the build system world is kind of a mess and resolving external
> libs is hard.  BuildSystem (http://petsc.cs.iit.edu/petsc/BuildSystem)
> is quite project-specific, but behaves sensibly with static libs and
> multiple versions, albeit often with truly horrendous link lines
> obtained by concatenating multiple recursive link lines.  A lot of
> hand-tuned automake scripts can deal as well.  CMake is definitely
> better than these once the `build' phase is reached, but the Find*  
> stuff
> is still weak.  Hopefully that can be improved.
>
> BTW, all the modules in my repository
>
>  http://github.com/jedbrown/cmake-modules
>
> are public domain and I'm willing to maintain them if they are added  
> to
> CMake.
>
> Jed
>


I have also had some experience with clusters and multiple versions of  
libraries. What I ended up doing was basically creating my own CMake  
Find* modules for the libraries that I needed. Since I only depended  
on 3 external libraries I didn't have to worry about "scaling" issues  
or writing too much CMake code. What I did since our cluster setup was  
similar (every one built their versions of their libraries in odd  
places) was that my modules look for environment variables and then  
use those environment variables to set the search path to look. Here  
is an example:

SET (HDF5_INSTALL  $ENV{HDF5_INSTALL})

SET(HDF5_INCLUDE_SEARCH_DIRS
   $ENV{HDF5_INSTALL}/include
)

SET (HDF5_LIB_SEARCH_DIRS
   $ENV{HDF5_INSTALL}/lib
)

# -- Find the Include directory for HDF5
FIND_PATH(HDF5_INCLUDE_DIR
   NAMES hdf5.h
   PATHS ${HDF5_INCLUDE_SEARCH_DIRS}
   NO_DEFAULT_PATH
)

# Look for the library.
FIND_LIBRARY(HDF5_LIBRARY_DEBUG
   NAMES ${HDF5_SEARCH_DEBUG_NAMES}
   PATHS ${HDF5_LIB_SEARCH_DIRS}
   NO_DEFAULT_PATH
   )

FIND_LIBRARY(HDF5_LIBRARY_RELEASE
   NAMES ${HDF5_SEARCH_RELEASE_NAMES}
   PATHS ${HDF5_LIB_SEARCH_DIRS}
   NO_DEFAULT_PATH
   )

So basically if you don't have the HDF5_INSTALL env variable set then  
my module probably will not find it, which is good in that instead of  
getting a wrong version of HDF5 you just don't get any at all. You may  
want to consider this approach if the number of external libraries  
that you depend on is low. You are welcome to use my modules as  
"inspiration" and modify to your liking.

---------------------------------------------------------
Mike Jackson                  mike.jackson at bluequartz.net
BlueQuartz Software                    www.bluequartz.net
Principal Software Engineer                  Dayton, Ohio