[CMake] FindCUDA ignores project dependencies when separable compilation is on

[Irwin Zaid]

Alright, this is a lot of progress!

1) We are using Makefiles. I agree with you about the dependency graph, 
so I'll try and sort that out. I'll let you know what the result is.

2) I just checked and, indeed, the *_intermediate_link.o file is not 
being passed -fPIC. Is this our problem? What is the correct fix?

Irwin

James Bigler wrote:
>
>
> On Mon, Jan 5, 2015 at 1:57 PM, Irwin Zaid <irwin.zaid at physics.ox.ac.uk
> <mailto:irwin.zaid at physics.ox.ac.uk>> wrote:
>
>     Hi James,
>
>     Thanks for the quick reply! As I mentioned, we've hit two issues.
>     The first is the project dependencies one, which I'll try and
>     describe more a bit below. I'm not a CMake expert, so please bear
>     with me.
>
>     The second is what I've put under 2).
>
>         The only CMake build dependency changes when doing separable
>         compilation
>         are found in CUDA_LINK_SEPARABLE___COMPILATION_OBJECTS.
>         Basically what
>         this does is create a new rule to build an intermediate link
>         file. For
>         everything but some versions of MSVC generators it adds a custom
>         command
>         to generate this intermediate link file. The other case adds a
>         custom
>         command that runs as a PRE_LINK command to generate the object
>         file (the
>         reasons for this is a bug in VS custom command dependency
>         scanning), but
>         this should happen during link phase and not compile phase.
>
>         Nothing in here should change what happens before the target is
>         built,
>         though.
>
>
>     1) Okay, I understand that, but I do think we saw a different
>     behaviour when we switched to separable compilation. Let me describe
>     what we are doing.
>
>     We generate part of our library from a simple program (call the
>     simple program 'gen', which generates a source file 'gen.hpp') that
>     we want to execute before compiling our library (call our library
>     'main'). We set this up with the following:
>
>     - add_executable(...) is called for 'gen'
>     - add_custom_command(...) sets up a command that executes 'gen'
>     - set_source_files_properties(..__.) is called to set 'gen.hpp' as
>     having the PROPERTY of GENERATED
>     - add_dependencies(main gen) is called to establish 'main' depends
>     on 'gen'
>
>     So far, this has only failed for CUDA with separable compilation.
>     (It has worked for all of our other configurations. including CUDA
>     without separable compilation.)
>
>     Have we done something wrong? Is there some additional information
>     we can look at to figure out what's going on?
>
>
> What kind of generator are you using (e.g. makefile)?
>
> Here's what I'm thinking might be the problem, though I'm not sure why
> it would have worked without CUDA_SEPARABLE_COMPILATION.
>
> There's a dependency between gen and gen.hpp (encoded in the call to
> add_custom_command(OUTPUTS gen.hpp COMMAND gen)).
> There's a dependency between main and gen (can't start building main
> until gen is built).
> There's a dependency between gen.hpp and main (gen.hpp is an input
> source to main, so it needs to be built as part of main).
>
> What I don't see is a dependency between gen.hpp and all the cuda
> sources that might use it as input. So from a dependency graph
> standpoint a makefile (if one is being used in this case) is entirely
> free to start compiling the CUDA code once the gen target has been
> satisfied.
>
> What you need is another target that builds the gen.hpp file which can
> be forced to run before main starts to build. There are more than one
> way to do this, and I'm not sure what the best option is, but you might
> try this:
>
> add_custom_command(OUTPUTS gen.hpp ....)
> add_custom_target(make_gen_hpp DEPENDS gen.hpp)
> add_dependency(main make_gen_hpp)
>
>     2) A second problem we've run into is an error when trying to link a
>     CUDA shared library with separable compilation. This is specifically
>     a Linux problem, on Mac it is fine. A static library is also fine,
>     working for both Linux and Mac.
>
>     The particular error is "relocation R_X86_64_32S against
>     `__nv_module_id' can not be used when making a shared object;
>     recompile with -fPIC". However, we are already compiling with -fPIC.
>     I can confirm that -fPIC appears to be passed to both the host
>     compiler for non-CUDA source and via -Xcompiler -fPIC for CUDA source.
>
>     This error occurs when trying to link all the different object files
>     together of our library.
>
>     Do you have any idea of what this could be?
>
>
> I'm not sure which object file wasn't compiled with -fPIC, but I would
> suspect it might be the intermediate link object file. FindCUDA is
> supposed to pass this flag along (see
> function(_cuda_get_important_host_flags)), but you might want to verify
> this with a 'make VERBOSE=1' and look for
> <target_name>_intermediate_link.o (substitute your target name in or
> leave it out of the search string).
>
> James