[CMake] Better BlueGene/P and cross-compile support for CMake

[Todd Gamblin]

Hi,

I'm doing research on scalable performance tools at Lawrence Livermore and my group is very enthusiastic about switching to CMake for most of our development.

I've been trying to get a proof of concept port from autotools to CMake done for a simple project, and things seem to work ok for our Linux systems but not for our BlueGene/P systems.  I'm trying to get a BlueGene/P platform file together, and while there seem to be posts on this subject to this list before, I couldn't find anything that worked.  BG/P supports shared libraries with two compiler toolchains, so this is a bit more involved than the BlueGene/L file that's already in CMake.  BG/P also has a crazier system library layout, and I'm not sure that there is sufficient support in CMake for this or for the way we typically install packages for the BG compute nodes.

I'm listing the issues I've discovered below.  Are there ways around these that I'm missing?


1. Bootstrap script doesn't seem to work quite right on the frontend.

Building CMake was a bit hairy on the Power6 front end node on our BlueGene systems.  The machine runs Linux and has both GNU and IBM XL toolchains for the frontend itself.  The CMake bootstrap script detected cc for the C compiler and xlC for the C++ compiler, and linking failed on something 50% through the build because it was passing -dynamic to xlC, which is the wrong flag.  It would be nice if CMake would prefer one or the other tool chain.

I was able to build with either if I just manually specified the compilers and linker like this:

> 	env CC=gcc C++=g++ LD=ld ./bootstrap --prefix=${CMAKE_HOME}

> 	env CC=xlc C++=xlC LD=xlC ./bootstrap --prefix=${CMAKE_HOME}

But It would've been nice if I didn't have to do all that.  Is PPC Linux not used many other places anymore?  I guess this is a sles10/ppc64 environment, so maybe that is throwing the script off.


2. The FindMPI module's library search breaks because the the cross-compiler name on BlueGene/P contains -l. Specifically, the GNU compute node compilers on BlueGene/P are named like this:

> 	powerpc-bgp-linux-g++

And the FindMPI module claims that it can't find the '-inux-g++' library, or something similar.  I was able to fix this by modifying FindMPI's search to look for -l only when it's followed by a space.  This is a little worrisome because I think that is a standard gnu naming convention, so other platforms with compilers named blah-linux are going to have the same bug.


3. Finding system libraries in directories not named 'lib'

CMake cross compile support doesn't seem robust enough for platforms where there is more than a simple system environment directory (specified with CMAKE_FIND_ROOT_PATH) and a custom user directory where platform binaries will go (also included in CMAKE_FIND_ROOT_PATH).  All the online documentation and the latest edition of the book seem to assume a system where there are a very limited number of places where backend libraries might be.

To start my port, I've tried to set up a BlueGene/P GNU tool chain file like this, based on the advice on the webpage:

> # the name of the target operating system
> set(CMAKE_SYSTEM_NAME BlueGeneP)
> 
> # set the compiler
> set(CMAKE_C_COMPILER   /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc )
> set(CMAKE_CXX_COMPILER /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-g++ )
> 
> # set the search path for the environment coming with the compiler
> # and a directory where you can install your own compiled software
> set(CMAKE_FIND_ROOT_PATH
>     /bgsys/drivers/ppcfloor/
>     /bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/
>     /bgsys/drivers/ppcfloor/runtime/SPI
>     /bgsys/drivers/ppcfloor/comm/
>     /bgsys/drivers/ppcfloor/comm/default/
>     /bgsys/drivers/ppcfloor/comm/sys/
> )
> 
> # adjust the default behaviour of the FIND_XXX() commands:
> # search headers and libraries in the target environment, search
> # programs in the host environment
> set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
> set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
> set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)


There are three problems here.  

3a. First, using this setup, FindMPI fails because the last library it needs is in /bgsys/drivers/ppcfloor/runtime/SPI, not  /bgsys/drivers/ppcfloor/runtime/SPI/lib.  CMAKE_FIND_ROOT_PATH seems to assume that its elements are just above a lib directory, but that's not how things are structured on BG/P (you can blame IBM).  So, if I just use the root path for searching, find_library fails for libs in runtime/SPI.

3b. I tried passing the root path elements as locations in a HINTS to find_library command instead.  Well, the first problem with this is that if you try this with CMAKE_FIND_ROOT_PATH_MODE_LIBRARY set to ONLY as advised, it just ignores the HINTS parameters.  I then tried BOTH, and when I do that, find_library seems to find the libraries in /bgsys/drivers/ppcfloor/runtime/SPI correctly.  However, it fails to find other libraries correctly because now it searches system directories on the front end node in addition to the backend library paths I've given above.  The result is that other libraries (like librt) resolve to their frontend versions when they should resolve to the backend versions.  If I build this way, I'll either get me a link error or a nasty runtime error.

3c. The third problem I see here is that while this particular project is self-contained, we have a lot of preinstalled libraries for the BlueGene backend that don't live in one nice, self-contained place.  The webpage for cross compiling with CMake suggests that I have *one* directory where all my binaries for the backend should go, and that I put this directory last in my CMAKE_FIND_ROOT_PATH.  

This isn't realistic for any of the large supercomputer installations I know of.  LLNL, Argonne, and ORNL all have prebuilt packages for the backends in a lot of different places, and I need a mechanism to pass those locations to the find_library command.  If I could somehow use CMAKE_FIND_ROOT_PATH and HINTS at the same time, this would be a start.  I'm assuming that PATHS is also ignored when CMAKE_FIND_ROOT_PATH_LIBRARY is set to ONLY.

Finally, is there some way to globally set NO_CMAKE_SYSTEM_PATH and NO_SYSTEM_ENVIRONMENT_PATH for every find_library call, while setting CMAKE_FIND_ROOT_PATH_MODE_LIBRARY to BOTH?  This might get me the behavior I want.


Please get back to me on these issues, as I'm really enjoying building with CMake so far, despite the BG/P headaches.  I'd like to help out by contributing a decent platform file for BlueGene/P once I'm done, and hopefully I'll get my group using CMake for most of our projects.

Thanks!

Todd Gamblin
Center for Applied Scientific Computing
Lawrence Livermore National Laboratory


























-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20100623/a962a3d8/attachment.htm>