[CMake] FindCUDA ignores project dependencies when separable compilation is on

[James Bigler]

On Mon, Jan 5, 2015 at 1:57 PM, Irwin Zaid <irwin.zaid at physics.ox.ac.uk>
wrote:

> Hi James,
>
> Thanks for the quick reply! As I mentioned, we've hit two issues. The
> first is the project dependencies one, which I'll try and describe more a
> bit below. I'm not a CMake expert, so please bear with me.
>
> The second is what I've put under 2).
>
>  The only CMake build dependency changes when doing separable compilation
>> are found in CUDA_LINK_SEPARABLE_COMPILATION_OBJECTS. Basically what
>> this does is create a new rule to build an intermediate link file. For
>> everything but some versions of MSVC generators it adds a custom command
>> to generate this intermediate link file. The other case adds a custom
>> command that runs as a PRE_LINK command to generate the object file (the
>> reasons for this is a bug in VS custom command dependency scanning), but
>> this should happen during link phase and not compile phase.
>>
>> Nothing in here should change what happens before the target is built,
>> though.
>>
>
> 1) Okay, I understand that, but I do think we saw a different behaviour
> when we switched to separable compilation. Let me describe what we are
> doing.
>
> We generate part of our library from a simple program (call the simple
> program 'gen', which generates a source file 'gen.hpp') that we want to
> execute before compiling our library (call our library 'main'). We set this
> up with the following:
>
> - add_executable(...) is called for 'gen'
> - add_custom_command(...) sets up a command that executes 'gen'
> - set_source_files_properties(...) is called to set 'gen.hpp' as having
> the PROPERTY of GENERATED
> - add_dependencies(main gen) is called to establish 'main' depends on 'gen'
>
> So far, this has only failed for CUDA with separable compilation. (It has
> worked for all of our other configurations. including CUDA without
> separable compilation.)
>
> Have we done something wrong? Is there some additional information we can
> look at to figure out what's going on?
>
>
What kind of generator are you using (e.g. makefile)?

Here's what I'm thinking might be the problem, though I'm not sure why it
would have worked without CUDA_SEPARABLE_COMPILATION.

There's a dependency between gen and gen.hpp (encoded in the call to
add_custom_command(OUTPUTS gen.hpp COMMAND gen)).
There's a dependency between main and gen (can't start building main until
gen is built).
There's a dependency between gen.hpp and main (gen.hpp is an input source
to main, so it needs to be built as part of main).

What I don't see is a dependency between gen.hpp and all the cuda sources
that might use it as input.  So from a dependency graph standpoint a
makefile (if one is being used in this case) is entirely free to start
compiling the CUDA code once the gen target has been satisfied.

What you need is another target that builds the gen.hpp file which can be
forced to run before main starts to build.  There are more than one way to
do this, and I'm not sure what the best option is, but you might try this:

add_custom_command(OUTPUTS gen.hpp ....)
add_custom_target(make_gen_hpp DEPENDS gen.hpp)
add_dependency(main make_gen_hpp)


> 2) A second problem we've run into is an error when trying to link a CUDA
> shared library with separable compilation. This is specifically a Linux
> problem, on Mac it is fine. A static library is also fine, working for both
> Linux and Mac.
>
> The particular error is "relocation R_X86_64_32S against `__nv_module_id'
> can not be used when making a shared object; recompile with -fPIC".
> However, we are already compiling with -fPIC. I can confirm that -fPIC
> appears to be passed to both the host compiler for non-CUDA source and via
> -Xcompiler -fPIC for CUDA source.
>
> This error occurs when trying to link all the different object files
> together of our library.
>
> Do you have any idea of what this could be?
>
>
I'm not sure which object file wasn't compiled with -fPIC, but I would
suspect it might be the intermediate link object file.  FindCUDA is
supposed to pass this flag along (see
function(_cuda_get_important_host_flags)), but you might want to verify
this with a 'make VERBOSE=1' and look for <target_name>_intermediate_link.o
(substitute your target name in or leave it out of the search string).

James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20150105/aa34bfb1/attachment-0001.html>