[CMake] providing library information, what's the cmake way

[Michael Hertling]

On 11/29/2010 02:28 PM, Johannes Zarl wrote:
> Sorry for the late response, but your mail was simply to long for a
> swift response...

No problem, this topic is not exactly trivial.

> On 11/26/2010 at 05:47, Michael Hertling <mhertling at online.de> wrote: 
>> On 11/24/2010 04:51 PM, Johannes Zarl wrote:
>>> About the components question again: I played around a bit, and I think I 
>>> now more or less have comprehended this. I guess for a package XXX with 
>>> components YYY and ZZZ, one could set variables XXX_HAS_YYY and XXX_HAS_ZZZ
>>> and then use a loop like this one in the XXXConfig.cmake file:
>>>
>>> foreach( component ${XXX_FIND_COMPONENTS} )
>>>     if ( XXX_HAS_${component})
>>>         set ( XXX_${component}_FOUND TRUE )
>>>     else( XXX_HAS_${component})
>>>         if ( ${XXX_FIND_REQUIRED})
>>>             message(FATAL_ERROR "Required component ${component} not 
>> found!")
>>>         elseif ( NOT XXX_FIND_QUIETLY)
>>>             message(STATUS "Component ${component} not found!")
>>>         endif ( ${XXX_FIND_REQUIRED})
>>>     endif ( XXX_HAS_${component})
>>> endforeach( component )
>>>
>>> Correct?
>>
>> While that's a possible approach it lacks the invocation-specific
>> variables, i.e. XXX_{INCLUDE_DIRS,LIBRARIES,DEFINITIONS}, and in
>> some cases, these can't be assembled in a simple component-wise
>> manner, see below. Moreover, there are further questions w.r.t.
>> multi-component packages and their config files:
> 
> Does this mean that XXX_LIBRARIES etc. should normally incorporate the
> settings for the components as well? [...]

Yes, of course, if FIND_PACKAGE(XXX ...) returns successfully one would
expect XXX_LIBRARIES to hold all libraries necessary to link against in
order to use all components enabled by the FIND_PACKAGE() invocation,
i.e. the libraries provided by the components themselves as well as
their prerequisites.

> [...] IMO this can't work if you call
> find_package several times with different components. [...]

This is one of those questions to deal with when it comes to multi-
component packages: Does FIND_PACKAGE(XXX ...) act accumulatively?

> [...] Also, this would 
> make it impossible to link to the "base library" alone, i.e. without
> the components...

One might consider the base library as a component by itself, perhaps
enabled automatically by resolving inter-component dependencies. As an
alternative, one could a priori populate XXX_LIBRARIES et al. with the
base library stuff and add the components' contributions as required.
Furthermore, a package doesn't need to have a dedicated base library,
e.g. Boost hasn't, and if you don't specify any components the
Boost_LIBRARIES variable remains empty.

>> - Does the config file provide the component-specific variables like
>> XXX_YYY_FOUND for each available component or for the requested ones
>> only, i.e. can you rely on XXX_ZZZ_FOUND to have a definite value if
>> you just said FIND_PACKAGE(XXX COMPONENTS YYY)? With your foregoing
>> approach, you can't. That's alright, but should be mentioned in the
>> package's documentation. Imagine the following scenario: There's one
>> installation of XXX for the native system and another one for cross
>> compiling purposes. The latter has YYY and ZZZ while the former has
>> YYY only. Due to FIND_PACKAGE()'s ability to search for config files
>> in various locations, such a coexistence is easily possible. Now, a
>> FIND_PACKAGE(XXX COMPONENTS YYY ZZZ) for the cross compilation XXX
>> returns with XXX_ZZZ_FOUND=TRUE, but does a subsequent invocation of
>> FIND_PACKAGE(XXX COMPONENTS YYY) within the same scope for the native
>> XXX return with XXX_ZZZ_FOUND=FALSE, or does XXX_ZZZ_FOUND=TRUE still
>> hold from the previous invocation? Both alternatives are fine, but the
>> user should know the score. 
> 
> Thanks, I hadn't thought of this. So the code-snippet above should 
> contain a "set ( XXX_${component}_FOUND ${component}-NOTFOUND )".

Yes, one would expect that FOUND variables for requested components
receive defined values in all cases - positive as well as negative.

>> Besides, it's a good style to refer to any
>> component-related variables only if the particular component has been
>> requested explicitly.
> 
> Ok, so a better style would be to only set the XXX_HAS_* variables at 
> first (maybe unsetting them at the end of the config file?), and then
> to conditionally set the XXX_YYY_INCLUDE_DIRS etc. if XXX_YYY_FOUND
> is true.

Here, I meant it is a good *user* style to rely only on variables
related to explicitly requested components; maybe a bit ambiguous.

>> - Handling of inter-component dependencies: Imagine the user just says
>> FIND_PACKAGE(XXX COMPONENTS ZZZ), but ZZZ needs YYY. If YYY is to be
>> enabled automatically the config file must know about the dependency
>> and cannot simply add the ZZZ-specific variables to the invocation-
>> specific ones; the YYY-specific variables must be mentioned, too.
> 
> I don't really see this as a problem. Although the code at hand lacks 
> support for this kind of dependencies, in most cases you won't need it.
> And if you as the config file writer need it, you obviously know you
> have dependencies and that you have to write code supporting those
> dependencies.

Usually, handling of such dependencies means enabling some components
the user hasn't requested explicitly, so it's quite straight forward,
in fact. However, I've mentioned these dependencies to point out that
the invocation-specific variables like XXX_LIBRARIES sometimes can't
be assembled by a simple iteration over XXX_FIND_COMPONENTS; instead,
the dependencies must be resolved first. BTW, do you know an example
with a component A depending on B only if C is enabled? ;)

> Update/FYI: I just tried tried to write generic code to deal with 
> dependencies, and its not as straightforward as I thought it is. Assuming
> that there are no circular dependencies, you could recursively add the 
> dependencies to the XXX_FIND_COMPONENTS variable and remove the duplicates
> afterwards. Then you can use the above code to set XXX_*_FOUND, and
> afterwards, you have to recursively add the libraries/defines/include_dirs
> of the depended upon components to each component's includes (this
> is the tricky part IMO). I guess it is far easier to write this in a
> non-generic way...

This can be done iteratively, too: Specify all immediate inter-
component dependencies, iterate over them and add the concerned
component to XXX_FIND_COMPONENTS each time you see an unresolved
dependency; repeat this iteration until you don't see unresolved
dependencies anymore. That also works for circular dependencies.

>> - Do multiple consecutive FIND_PACKAGE(XXX ...) invocations act in an
>> accumulative manner on the invocation-specific variables? 
> 
> Generally speaking, I would say: yes. At least this is the way of the
> least surprise for the user, as in sufficiently complex projects a
> package may be included at different places with different arguments.

For the same reason, I'd tend to the opposite; look at the following:

FIND_PACKAGE(XXX COMPONENTS YYY)
...
ADD_SUBDIRECTORY(subdir)

In subdir/CMakeLists.txt:

FIND_PACKAGE(XXX COMPONENTS ZZZ)
...
TARGET_LINK_LIBRARIES(... ${XXX_LIBRARIES})

Here, the target is also linked against XXX_YYY_LIBRARY as the latter is
inherited via XXX_LIBRARIES from the parent directory, but if the target
only needs XXX_ZZZ_LIBRARY it will possibly be overlinked. Although this
can be avoided by resetting XXX_LIBRARIES before each FIND_PACKAGE()
invocation, I don't think that's the way one would like to go.

IMO, the invocation-specific results of any FIND_PACKAGE() call should
depend solely on the parameters passed in and the well-known variables
like CMAKE_PREFIX_PATH. The downside is that one must possibly process
or save the results before they could be overwritten, e.g.:

FIND_PACKAGE(XXX REQUIRED YYY)
SET(LIBRARIES ${XXX_LIBRARIES})
FIND_PACKAGE(XXX COMPONENTS ZZZ)
LIST(APPEND LIBRARIES ${XXX_LIBRARIES})
...
TARGET_LINK_LIBRARIES(... ${LIBRARIES})

Since such related calls to FIND_PACKAGE(XXX ...) usually occur nearby
each other within the same CMakeLists.txt, this little penalty should
be acceptable whereas accumulated XXX_LIBRARIES and the like may have
far reaching and surprising effects, especially in complex projects.

>> FIND_PACKAGE(XXX REQUIRED YYY)
>> FIND_PACKAGE(XXX COMPONENTS ZZZ)
> 
> Ah, this is how it's done. I was wondering why a call like
> 
> find_package(XXX COMPONENTS ZZZ REQUIRED YYY)
> 
> sets both XXX_YYY_REQUIRED and XXX_ZZZ_REQUIRED.

It's XXX_FIND_REQUIRED_{YYY,ZZZ} you talk about, I suppose, and these
variables are set if the particular component has been mentioned in the
FIND_PACKAGE() regardless whether the REQUIRED flag is specified or not.

>> When turning towards find modules, the situation becomes even more
>> complicated. From a user's perspective, find modules and config files
>> should behave the same, but the formers can't know which components are
>> available, so they must look for them; this gives rise to questions like:
>>
>> - Meaning of the REQUIRED and QUIET options: When a find module looks
>> for any component the effects of REQUIRED and QUIET depend on whether
>> the component has been requested explicitly or not. If it hasn't, the
>> find module mustn't bail out if the component isn't found even though
>> REQUIRED was specified; that's the opposite of what is expected for an
>> explicitly requested component. E.g., if FIND_PACKAGE(XXX REQUIRED YYY)
>> also looks for ZZZ on its own behalf the find module is not expected to
>> throw a fatal error if ZZZ is absent whereas the absence of YYY should
>> result in such a behaviour right away. A similar view may be taken for
>> the QUIET option: Even if it wasn't specified the user wouldn't expect
>> any messages related to components that weren't requested explicitly.
> 
> This isn't that complicated. I have written such a FindLibrary.cmake module
> (although without dependencies) and the REQUIRED and QUIET options are
> quite easy to handle. If you split the process into 2 parts, you lose much
> of the complexity: first, search for any components you can find, second
> check for the required ones, giving diagnostic/error messages as is
> appropriate.

Using two loops would indeed allow to handle the REQUIRED/QUIET flags
easily, but I would like to apply FIND_PACKAGE_HANDLE_STANDARD_ARGS()
to components in order to process them in the same standardized and
unique manner as FPHSA does with single-component packages. This is
even more interesting as FPHSA has been enhanced recently and is now
quite complicated so it's not an option anymore to provide a similar
function for components only. Since FPHSA takes care of setting the
FOUND variable, issuing messages and bailing out if necessary at one
go, the two-loop approach doesn't suit, IMO. Instead, I'd prefer to
promote the REQUIRED/QUIET flags to XXX_YYY_FIND_{REQUIRED,QUIETLY}
- if YYY has been requested explicitly - and call FPHSA like, e.g.:

FIND_PACKAGE_HANDLE_STANDARD_ARGS(
    XXX_YYY
    DEFAULT_MSG
    XXX_YYY_LIBRARY XXX_YYY_INCLUDE_DIR
)

IF YYY hasn't been requested explicitly, but the find module despite
looks for it, XXX_YYY_FIND_{REQUIRED,QUIETLY} should be set to FALSE
and TRUE, resp., so FPHSA handles the REQUIRED/QUIET flags for each
component as the user probably expects, and XXX_YYY_FOUND is set up
properly at the same time.

>> - Interpretation of XXX_FOUND: Config files can't set the value of this
>> variable, but find modules can, so one should think about what it means
>> for XXX_FOUND if a component - requested or not - hasn't been found.
> 
> As I have written above, I don't think that the components should alter
> the values of the package-wide variables (XXX_LIBRARIES etc.). The same
> applies to the XXX_FOUND variable. If you search for library XXX and 
> component YYY, XXX is still found even if it lacks the requested component.
> If you want to know if XXX was found, you use XXX_FOUND, if you want the 
> same for component YYY, you use XXX_YYY_FOUND.

While this interpretation of XXX_FOUND is absolutely right, IMO, the
intention of XXX_LIBRARIES etc. is different as I said before: These
variables should comprehend all stuff necessary to use all requested
components. E.g., suppose YYY needs zlib; where are you going to put
ZLIB_LIBRARY from the FIND_PACKAGE(ZLIB) invocation? XXX_YYY_LIBRARY
is reserved for libXXX_YYY.so or the like only. It's XXX_LIBRARIES
that takes XXX_YYY_LIBRARY along with ZLIB_LIBRARY and the other
components' libraries and prerequisites, so

ADD_DEFINITIONS(${XXX_DEFINITIONS})
INCLUDE_DIRECTORIES(${XXX_INCLUDE_DIRS})
TARGET_LINK_LIBRARIES(... ${XXX_LIBRARIES})

is basically sufficient to prepare for the use of the componentized XXX.

>> None of these questions arises for single-component packages, and while
>> there are reasonable answers to all of them, IMO, CMake's accompanying
>> documentation could perhaps be enhanced a bit w.r.t. multi-component
>> packages; particularly, a simple but yet complete example how to do
>> things would be quite helpful. 
> 
> I definitely agree. Multi-component packages (and package-finding in 
> general) are often a complex matter, and the documentation on this topic
> is not easy to read.

That's why it's very good to have another discussion about this topic. :)

Regards,

Michael