[CMake] NVCC and Shared Library Full Path Linking

[MARTIN, ROBERT S CTR USAF AFMC AFRL/RQRS]

CMake Group-

I'm trying to build a CUDA project with NVCC.  The project uses 'separate
compilation'.  Unfortunately, 'separate compilation' breaks in a variety of
ways when I try to use the version in FindCUDA.cmake.  First, it doesn't
include CUDA_NVCC_FLAGS in the -dlink phase so it immediately complains that
the dlink flag can't be used without sm_20+.  That has already been noted as
a bug (http://public.kitware.com/Bug/view.php?id=14238) and there seems to
be some discussion about it being inappropriate to default to some sm_
version.  Since multiple sm_ options can be listed but only one arch flag,
it seems like this may need a more advanced way to be dealt with.  Anyway,
after modifying things to add a -arch=sm_20 to the command line, it works
for a while but then cannot dlink one of my objects because of:
'nvlink error :  Undefined reference to '_Z3minRK6float3S1_' in '(..some
path...)_generated_(...some file...).cu.o'

 I can't find this problem because I don't really understand function name
mangling, but it looks like it cannot link to a 'min' function probably.
It's not one of the "__cudaRegisterLinkedBinary..." type issues though, and
so I'm not clear why nvlink is trying to resolve all the symbols for an
intermediate object anyway.  

As a workaround for all this, I tried adding '-dc' to the normal compiles
and linking with nvcc instead of gcc so that nvcc can do device and host
linking in one pass.
(http://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#using-sepa
rate-compilation-in-cuda ... "nvcc <objects>").  This almost works except
for a few link command line incompatibilities.  First, nvcc doesn't know
what to do with '-Wl,-rpath...'.  It needs some '-Xlinker' in front of that
to know where to send it.  I can get around that by just turning off rpath
with 'set(CMAKE_SKIP_RPATH TRUE)', but it's not a great solution.    I also
have to remove the '-rdynamic' flag by setting
CMAKE_SHARED_LIBRARY_LINK_CXX_FLAGS to nothing (""). The last issue is that
for some reason on some machines CMake tries to use full paths for shared
libraries rather than -l & -L and nvcc chokes on the extension (static
libraries with .a are fine).   The error message is:

Nvlink fatal   : Unsupported file type '/usr/lib/openmpi/lib/libmpi_cxx.so'

If I replace the '/usr/lib/openmpi/lib/libmpi_cxx.so' with
'-L/usr/lib/openmpi/lib/ -lmpi_cxx', it works.  When I look at the ldd
output, it's even correctly still linking to the .so (rather than a static
library).  Interestingly enough, on a different machine with a similar setup
the FindMPI module adds the '-L -l' options instead of the full path (mpich
instead of openmpi though?).  I found this page on the cmake wiki that
suggests getting rid of '-L/-l' as an 'improvement':
http://www.cmake.org/Wiki/CMake:Improving_Find*_Modules#Converting_-L.2F-l_f
lags

Is this currently an encouraged approach?  Is there a way to control the
behavior in cmake to force CMake to choose using '-L/-l' instead of full
paths?  I realize this could also be considered more of a bug with nvlink
than with cmake, but I figured I could try to start here.  It would be a
simple enough regular expression to break the full path into the two pieces,
but I'm pretty new to CMake and I'm not sure how this could be accomplished.
If I could just get control of this behavior, I'd have something that would
work for now until FindCUDA can get sorted out a little more with respect to
all the CUDA 5+ changes and/or more tightly integrated into CMake like other
compilers tend to be.

Thanks for any help people can provide.





Robert Martin, ERC Inc.
EP Modeling and Simulation Group
In-Space Propulsion Branch
Air Force Research Laboratory
661-275-6659

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5655 bytes
Desc: not available
URL: <http://www.cmake.org/pipermail/cmake/attachments/20131216/4d6ac5b8/attachment.bin>