[CMake] History and ideas behind FindMPI.cmake?

[Bartlett, Roscoe A]

> -----Original Message-----
> From: Michael Wild [mailto:themiwi at gmail.com] 
> Sent: Monday, January 12, 2009 10:57 PM
> To: Bartlett, Roscoe A
> Cc: cmake at cmake.org
> Subject: Re: [CMake] History and ideas behind FindMPI.cmake?
> 
> > I am seriously considering completely writing our own version of 
> > FindMPI.cmake for Trilinos (called FindTPLMPI.cmake).  The major 
> > features of a new MPI find module would be:
> >
> > 1) It will differentiate C, C++, and Fortran MPI compilers
> >
> > 2) It will keep the MPI compilers intact and not try to extract out 
> > compile options and MPI libraries.
> >
> > 3) The MPI compilers will automatically be set as 
> > CMAKE_[C,CXX,Fortran]_COMPILER *before* the languages are 
> enabled and 
> > the compilers are checked.  This will make sure that every 
> bit of code 
> > is built and linked in a consistent way (including during 
> the various 
> > configure-time checks).  The trouble that you might have is with 
> > try_run(...) commands and some MPI implementations that require an 
> > explicit boot (like LAM MPI).  Most MPI implementations 
> allow you to 
> > just run the executable without mpirun or mpiexec and it will run 
> > correctly for a single process.
> 
> I consider this to be a "buggy" idea. It would be impossible 
> to have a project which builds both normal and MPI 
> executables. I would probably prefer a target property called 
> "MPI" which can be set to TRUE if one wants to compile a 
> specific target with MPI.

In Trilinos, an MPI configured build creates all MPI executables (in some sense).  Also, most Trilinos software is set up to automatically check to see if MPI is initialized or not and will run correctly in serial mode with MPI enabled but not initialized.

Also, most newer MPI implementations that I have used allow you to build with MPI support but never call MPI_Init(...) and never have to use mpiexec/mpirun in order to run a serial executable (as long as you don't call MPI functions).  That is how you can get away with defining CMAKE_[C,CXX,Fortran]_COMPILER as the MPI compiler wrappers.  Also, if you only use try_compile(...) tests at configure time, then you will never have any problem with this.  Even with try_run(...), as long as these tests don't call MPI functions then they should work just fine also with most newer MPI implementations.

Lastly, our project Trilinos does not really define *any* executables except for tests and examples.  To set up even single-process tests/examples to work with MPI is trivial.  There is a single object that you have to create in main(...) like:

  Teuchos::GlobalMPISession mpiSession(&argc, &argv);

and you just have to run the test/example with mpirun/mpiexec with one process in case your MPI implementation requires that you do so.

The C++ class Teuchos::GlobalMPISession works with both MPI and non-MPI enabled software.

> > 4) The standard MPI install structure of many MPI implementations 
> > (e.g. MPICH, OpenMPI, LAM MPI etc.) of prefix/[include, 
> bin] will be 
> > exploited and searching from MPI_BASE_DIR (which is set by the
> > client) will be done.  In this way, you can just set a single cache 
> > variable on most systems and get MPI totally set up.
> 
> This won't work at all. See the --includedir and --libdir 
> options of the OpenMPI configure script. Your proposal will 
> force these values to the default, making too many assumptions.

You misunderstood.  This would not require that MPI must be installed in PREFIX/[include, bin, lib] but if it is, then the user could just give the base directory.  It is not hard to set this up at all and I am surprised that the other CMake find modules don't do this.  However, if we are going to just use mpicc/mpiCC/mpiC++/mpif77 etc and mpirun/mpiexec then we just need the 'bin' directory and can thrown the rest of this out.

> > These days, I don't know if anyone still uses raw compilers 
> and then  
> > tries to manually pass in the compiler options and link 
> libraries.   
> > We can still support such a thing but I think this is the 
> 1% exception 
> > instead of the rule.  Am I wrong about this?
> 
> I know of at least one large project which does not use the 
> wrappers because the developers abstracted the communications 
> into separate libraries, one for MPI, one for GAMMA and one 
> for LVM. The user has then the choice of which library should 
> be used when he starts the program. Couldn't do that with 
> your proposal.

Is this a runtime option or a configure-time option in this "large project"?  If it is a runtime option, then you have to at least build in support for MPI so using the MPI compiler wrappers is just fine.  If it is a configure-time option, then you can configure with and without MPI.  That is exactly what Trilinos does.  You can build a serial version of Trilinos with and without MPI.  If there was some other great communication layer we could also wrap that and use that with most of Trilinos (most of Trilinos uses thin abstract adapters for communication and is not directly tied to MPI).

I can't see any downside to redefining CMAKE_[C,CXX,Fortran]_COMPILER using the MPI compiler wrappers before the language configure tests and the other configure-time tests.  If you do that, then you can gut FindMPI.cmake and in the end all it has to do is:

1) [If C is enabled] Find mpicc or the equivalent MPI C compiler wrapper using find_program(...)

2) [If C++ is enabled] Find mpiCC/mpic++ or the equivalent MPI C++ compiler wrapper using find_program(...)

3) [If Fortran is enabled] Find mpif77 or the equivalent MPI Fortran compiler wrapper using find_program(...)

4) Find mpirun/mpiexec or the equivalent using find_program(...)

5) Figure out how to run mpirun/mpiexec and what other arguments are needed (or take them from the user)

You have to take some care to try to get compatible versions of all of this but that will not be too hard and it will be set up to allow the user to select whatever they want for each major decision.

If you have to use a raw C/C++/Fortran compiler with MPI then you would be required to also set the -I paths, the libraries, and any other compile or link-time options manually.

The argument for directly using the MPI compiler wrappers is compelling (and that is why almost everyone I know uses them).

Can anyone think of any other reasons why I should not write our own simplified and generalized MPI configuration support?

Cheers,

- Ross

---------------------------------------------------------
Dr. Roscoe A. Bartlett
Senior Member of the Technical Staff
Trilinos Software Engineering Technologies and Integration Lead
Sandia National Laboratories
Phone: (505) 275-6147