[CMake] Better BlueGene/P and cross-compile support for CMake

[Todd Gamblin]

Some small additions to the path information I posted below:

I believe the original GNU/POSIX runtime libraries for the compute nodes are located here:

	/bgsys/drivers/ppcfloor/gnu-linux/powerpc-bgp-linux

So you'll need this directory in your prefixes to detect things like libpthread (which you ned for MPI on BG/P).  BG/P also has added support for python apps to execute on the compute nodes, and the IBM driver ships with a python installation.  For whatever reason, they install the CNK python runtime libraries it in under this prefix:

	/bgsys/drivers/ppcfloor/gnu-linux

That will need to be in your paths for python development.  I'm unsure if anyone's done much in CMake involving cross-compiling python builds, and I'm pretty unfamiliar with how CMake deals with python projects.  I'm assuming the python support in CMake knows how to search lib/python<VERSION> directories, but I'm not sure if there is more that would needs to be done to get python working seamlessly.

-Todd


On Jun 25, 2010, at 4:08 PM, Todd Gamblin wrote:

> This makes sense, but it's actually slightly more complicated for BG/P.  I just got off the phone with our lead BG sysadmin (Adam Bertsch) about this, and here's what I got about the layout of that directory:
> 
> /bgsys/drivers/ppcfloor 
> 	This is the root of the BG system software release ("driver" in IBM-speak), and it
> 	contains files for the frontends, the I/O nodes, and the compute nodes.  We probably
> 	don't want to make this the system root for the backend, because it would allow other
> 	types of binaries into the build.
> 
> /bgsys/drivers/ppcfloor/mcp-2.6.16.60
> 	This is the root of the I/O node system image.  This is the 32-bit PPC 
> 	linux (pretty sure it's SuSE too) that runs on the I/O nodes (IONs) of BG/P.  I do tools 
> 	development, and it's possible to run daemons like TotalView on the IONs, so I might
> 	at some point try to make a platform/toolchain file for this, but not in the near future, if 
> 	at all.  This shouldn't go in the plain BG/P toolchain file.
> 
> 	FYI -- the version number here is the linux kernel version, which *could* change, but
> 	typically IBM doesn't upgrade the BG ION kernels during a particular BG product's 
> 	lifetime, or at least they haven't yet, for L or P.
> 
> Now, for the Compute Nodes (CNs).  Adam essentially said that there *is* no formal documentation about where you should look to find CN system libraries.  However, there are the MPI compiler scripts in /bgsys/drivers/ppcfloor/comm/bin.  These were written by Argonne and are now included with the BG driver distribution, and they have to link against all the BG system libraries.  I ran all the compilers with -show, and unioned all the args I was getting with that command.  This narrows it down to a few directories:
> 
> 	/bgsys/drivers/ppcfloor/comm/default/lib
> 	/bgsys/drivers/ppcfloor/comm/sys/lib
> 	/bgsys/drivers/ppcfloor/runtime/SPI
> 
> That's it.  Those are really the only backend directories that contain libraries used by apps.  The CN Kernel (CNK) OS is minimal, so this makes sense, and if you look in those directories, they contain MPI libs and various lower-level comm APIs like DCMF, which BG uses.  Now, there are also directories for the particular toolchains, and I'm not sure you care about these, because the BG compilers already link against the libraries in these directories.  For GNU, the backend compiler runtime libs are in:
> 
> 	/bgsys/drivers/ppcfloor/gnu-linux
> 
> And for XL compilers, the runtime libraries seem to be in:
> 
> 	/opt/ibmcmp/<compiler>/bg/<version>/bglib