[CMake] [cmake-developers] Need ideas/opinions on third party library management

[Ruslan Baratov]

On 17-Aug-16 08:36, Elizabeth A. Fischer wrote:
> > > I don't think CMake is the best place to do it,
> > Can you provide any details? I personally think that CMake is a
> > natural and the only place where it should be done.
>
> The most important reason here is because there are combinatorially
> many versions of a package you COULD install, depending on what
> versions of its dependencies you link with, and CMake provides nothing
> to address that issue.
CMake provides an abstraction. "Slots" that you need to fill:

   if(FOO_WITH_TESTS)
     # need to have GTest installed
     find_package(GTest)
   endif()
   if(FOO_WITH_OPENSSL)
     # need to have OpenSSL installed
     find_package(OpenSSL)
   endif()

And it should drive package manager. At least I find that approach 
natural and convenient, see no problems with it.
You can have as much combinations of versions/options/dependencies as 
you need:
* https://docs.hunter.sh/en/latest/overview/customization/hunter-id.html
* https://docs.hunter.sh/en/latest/overview/customization/config-id.html

>   See here for an overview of how Spack
> addresses the combinatorial versioning issue (which no other
> auto-builder does, to the best of my knowledge):
>
> http://hpcugent.github.io/easybuild/files/SC14_BoF_Spack.pdf
That's what I was talking about. I think that there is no need to 
introduce new funky syntax like "spack install mpileaks at 1.1.2 %gcc at 4.7.3 
+debug".
We already have CMAKE_CXX_COMPILER and 
CMAKE_BUILD_TYPE/CMAKE_CONFIGURATION_TYPES. Version can be set by CMake 
options too.
Effectively you can do:

   option(FOO_STABLE_BUILD "Stable build" ON)
   option(FOO_EXPERIMENTAL_BUILD "Experimental build" OFF)

   if(APPLE AND IOS AND FOO_STABLE_BUILD)
     hunter_config(BooPackage VERSION "1.0")
   endif()

   if(WIN32 AND FOO_EXPERIMENTAL_BUILD)
     hunter_config(BooPackage VERSION "2.0-beta" CMAKE_ARGS 
BOO_NEW_STUFF=YES)
   endif()

>
> Once you've built something, it's nice to be able to re-use it.  If I
> have a top-level CMake project that automatically builds three
> dependencies, will other projects be able to make use of those
> dependencies I've built?
Yes, libraries should be installed to the shared root, not to local folder:
* https://docs.hunter.sh/en/latest/overview/shareable.html

>   Or do they become private?
No. Though you can make it private by setting CMake variable. It will 
use separate directory just like virtualenv do.

>   If libraries
> cannot be shared between applications, you will get a LOT of library
> bloat, especially among the low-level libraries that get repeated
> numerous times.  Admittedly, this may not be such an issue in some
> environments where people are really only focused on building one
> thing.
>
> If you make a project, you might see it as a "top-level" project.  But 
> someone else might want to build something bigger on top of your 
> work.  You can never assume that "this package is top-level and no one 
> will ever depend on it."
No issue here, see notes above.

>
> Another obvious problem with using CMake for everything is that not
> all packages come with CMake builds; most do not, in fact. Even if we
> CAN re-write all the buils into CMake, that is a lot of extra effort.
> As Florent has discovered, upstream authors do not always see a CMake
> build in a favorable light, and these re-worked builds are not always
> as functional as the original.  Moreover... writing a Spack recipe is 
> an order of magnitude easier than writing a CMake build.  Usually, 
> it's just a matter of calling `configure` or `cmake` with the right 
> options.
Again, converting to CMake is a best option, but not the only possible 
one. E.g. OpenSSL, Boost, autotool-based package like X11 can be used as is:
* 
https://github.com/ruslo/hunter/blob/b4c370e32798cc3da74c37e4156c3bfc77add379/cmake/projects/Boost/hunter.cmake#L21
* 
https://github.com/ruslo/hunter/blob/b4c370e32798cc3da74c37e4156c3bfc77add379/cmake/projects/OpenSSL/hunter.cmake#L17
* 
https://github.com/ruslo/hunter/blob/b4c370e32798cc3da74c37e4156c3bfc77add379/cmake/projects/x11/hunter.cmake#L20

>
> Although we can maybe imagine a world in which everyone eventually
> abandons Autotools for CMake, it is still not realistic to expect that
> Python, Haskell or Java programs will ever come with CMake builds.
> This would be OK if each language exited in its own silo. But they
> don't.  Python packages (built with setuptools) routinely depend on
> C-based packages (built with Autotools or CMake).  By being agnostic
> to the build system, auto-builders (like Spack, Macports, HomeBrew,
> etc) are able to make packages work together, regardless of the build
> system chosen for each one.
That's exactly what Hunter do, but using CMake as a driver.

>
> In the sense that CMake is a Turing-complete language, there's no
> fundamental reason you CAN'T write a meta-builder in CMake.  But
> gosh... the language sure is arcane (but still better than Autotools
> by a long shot).  I like to imagine that if CMake were starting off
> today, it would be written in Python.
Language is a pain, I totally agree. But language will change nothing if 
design is originally bad. GYP use Python so what? Does the syntax sexy? 
Does it scale? No, it's just another "Works for me!" tool...

>
> > It is difficult indeed but it's the only way to reach the goal.
>
> Not sure what goal here.
Extendable, customizable and easy reusable solution.

>   My software stack requires about 70
> packages.  I'm so glad I didn't have to rewrite 70 CMake builds.  As
> long as an Autotools build works (meaning it puts stuff in prefix/bin, 
> prefix/lib, etc. at the end of the day), and I don't have to look at 
> it, why
> do I care?
That's the point - it doesn't. For example it's not possible to have 
different build types (Release/Debug) installed into one root but with 
different suffixes like CMake do. *.pc files created by autotools 
contain full path so you can't build your code on remote machine and 
download it to the local one. At least those are the problems we did 
hit, so if I missing something please let me know.

>
> > Exceptions are big libraries with a lot of components such as Qt or
> > Boost, it's unmaintainable approach, inevitable evil :(
>
> I guess I rely on a lot of evil libraries.
Me too, I'm just saying that I don't use "add new CMakeLists.txt to this 
project" approach.

Ruslo