[CMake] Better BlueGene/P and cross-compile support for CMake

[Alexander Neundorf]

Hi,

On Wednesday 23 June 2010, Todd Gamblin wrote:
> Hi,
>
> I'm doing research on scalable performance tools at Lawrence Livermore and
> my group is very enthusiastic about switching to CMake for most of our
> development.
>
> I've been trying to get a proof of concept port from autotools to CMake
> done for a simple project, and things seem to work ok for our Linux systems
> but not for our BlueGene/P systems.  I'm trying to get a BlueGene/P
> platform file together, and while there seem to be posts on this subject to
> this list before, I couldn't find anything that worked.  BG/P supports
> shared libraries with two compiler toolchains, so this is a bit more
> involved than the BlueGene/L file that's already in CMake.  BG/P also has a
> crazier system library layout, and I'm not sure that there is sufficient
> support in CMake for this or for the way we typically install packages for
> the BG compute nodes.
>
> I'm listing the issues I've discovered below.  Are there ways around these
> that I'm missing?
>
>
> 1. Bootstrap script doesn't seem to work quite right on the frontend.
>
> Building CMake was a bit hairy on the Power6 front end node on our BlueGene
> systems.  The machine runs Linux and has both GNU and IBM XL toolchains for
> the frontend itself.  The CMake bootstrap script detected cc for the C
> compiler and xlC for the C++ compiler, and linking failed on something 50%
> through the build because it was passing -dynamic to xlC, which is the
> wrong flag.  It would be nice if CMake would prefer one or the other tool
> chain.
>
> I was able to build with either if I just manually specified the compilers 
and linker like this:
> > 	env CC=gcc C++=g++ LD=ld ./bootstrap --prefix=${CMAKE_HOME}
> >
> > 	env CC=xlc C++=xlC LD=xlC ./bootstrap --prefix=${CMAKE_HOME}
>
> But It would've been nice if I didn't have to do all that.  Is PPC Linux
> not used many other places anymore?  I guess this is a sles10/ppc64
> environment, so maybe that is throwing the script off.

This doesn't seem to be related to the cross compiling part.
Bill usually chimes in on such issues...

> 2. The FindMPI module's library search breaks because the the cross-compiler 
name on BlueGene/P contains -l. Specifically, the GNU compute node compilers 
on BlueGene/P are named like this:
> > 	powerpc-bgp-linux-g++
>
> And the FindMPI module claims that it can't find the '-inux-g++' library,
> or something similar.  I was able to fix this by modifying FindMPI's search
> to look for -l only when it's followed by a space.  This is a little
> worrisome because I think that is a standard gnu naming convention, so
> other platforms with compilers named blah-linux are going to have the same
> bug.

Please add a bug report for this on http://public.kitware.com/Bug/


> 3. Finding system libraries in directories not named 'lib'
>
> CMake cross compile support doesn't seem robust enough for platforms where
> there is more than a simple system environment directory (specified with
> CMAKE_FIND_ROOT_PATH) and a custom user directory where platform binaries
> will go (also included in CMAKE_FIND_ROOT_PATH).  All the online
> documentation and the latest edition of the book seem to assume a system
> where there are a very limited number of places where backend libraries
> might be.
>
> To start my port, I've tried to set up a BlueGene/P GNU tool chain file like 
this, based on the advice on the webpage:
> > # the name of the target operating system
> > set(CMAKE_SYSTEM_NAME BlueGeneP)
> >
> > # set the compiler
> > set(CMAKE_C_COMPILER  
> > /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-gcc )
> > set(CMAKE_CXX_COMPILER
> > /bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc-bgp-linux-g++ )
> >
> > # set the search path for the environment coming with the compiler
> > # and a directory where you can install your own compiled software
> > set(CMAKE_FIND_ROOT_PATH
> >     /bgsys/drivers/ppcfloor/
> >     /bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux/
> >     /bgsys/drivers/ppcfloor/runtime/SPI
> >     /bgsys/drivers/ppcfloor/comm/
> >     /bgsys/drivers/ppcfloor/comm/default/
> >     /bgsys/drivers/ppcfloor/comm/sys/
> > )
> >
> > # adjust the default behaviour of the FIND_XXX() commands:
> > # search headers and libraries in the target environment, search
> > # programs in the host environment
> > set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
> > set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
> > set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
>
> There are three problems here.
>
> 3a. First, using this setup, FindMPI fails because the last library it
> needs is in /bgsys/drivers/ppcfloor/runtime/SPI, not 
> /bgsys/drivers/ppcfloor/runtime/SPI/lib.  CMAKE_FIND_ROOT_PATH seems to
> assume that its elements are just above a lib directory, but that's not how
> things are structured on BG/P (you can blame IBM).  So, if I just use the
> root path for searching, find_library fails for libs in runtime/SPI.

I guess you need a special Platform/BlueGeneP.cmake file, which not simply 
includes UnixPaths.cmake, but sets some special directories.
Please add a bug report for this on http://public.kitware.com/Bug/ too.

> 3b. I tried passing the root path elements as locations in a HINTS to
> find_library command instead.  Well, the first problem with this is that if
> you try this with CMAKE_FIND_ROOT_PATH_MODE_LIBRARY set to ONLY as advised,
> it just ignores the HINTS parameters.  I then tried BOTH, and when I do
> that, find_library seems to find the libraries in
> /bgsys/drivers/ppcfloor/runtime/SPI correctly.  However, it fails to find
> other libraries correctly because now it searches system directories on the
> front end node in addition to the backend library paths I've given above. 
> The result is that other libraries (like librt) resolve to their frontend
> versions when they should resolve to the backend versions.  If I build this
> way, I'll either get me a link error or a nasty runtime error.

Maybe we can solve this with 3a)

> 3c. The third problem I see here is that while this particular project is
> self-contained, we have a lot of preinstalled libraries for the BlueGene
> backend that don't live in one nice, self-contained place.  The webpage for
> cross compiling with CMake suggests that I have *one* directory where all
> my binaries for the backend should go, and that I put this directory last
> in my CMAKE_FIND_ROOT_PATH.
>
> This isn't realistic for any of the large supercomputer installations I
> know of.  LLNL, Argonne, and ORNL all have prebuilt packages for the
> backends in a lot of different places, and I need a mechanism to pass those
> locations to the find_library command.

Can you give some examples where the things are installed ?

Alex