View Issue Details Jump to Notes ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0012284CMakeCPackpublic2011-06-17 16:572012-03-06 08:37
ReporterEric NOULARD 
Assigned ToDavid Cole 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSLinuxOS Version
Product VersionCMake 2.8.4 
Target VersionCMake 2.8.6Fixed in VersionCMake 2.8.6 
Summary0012284: User Install time created symlinks are not preserved in TGZ
DescriptionReporte by John Cary on the ML
http://www.cmake.org/pipermail/cmake/2011-June/044940.html [^]
Steps To ReproduceAttached is a minimal standalone example that illustrates the problem.

If build dir is next to the directory resulting from untarring this, do

cmake -DCMAKE_INSTALL_PREFIX=$HOME/tmp/cmtprefix ../cmt
make all install
# You will see that in the installation, the top cmtdir is a link:
numbersix.cary$ ls -l $HOME/tmp/cmtprefix/cmtdir
lrwxrwxr-x 1 cary research 10 Jun 17 11:16 /Users/cary/tmp/cmtprefix/cmtdir -> bin/cmtdir
# then do
make package
tar xzf cmt-1.1.1-Darwin.tar.gz
# and you will see that the top-level cmtdir is a directory:
numbersix.cary$ ls -l cmt-1.1.1-Darwin
total 0
drwxrwxr-x 3 cary research 102 Jun 17 11:25 bin
drwxrwxr-x 3 cary research 102 Jun 17 11:25 cmtdir

such that two copies of the binary are installed
TagsNo tags attached.
Attached Filestgz file icon cmt.tgz [^] (828 bytes) 2011-06-17 16:57
txt file icon CMakeLists.txt [^] (770 bytes) 2011-08-08 13:47 [Show Content]
patch file icon 0001-Fix-12284-Symlinks-are-not-preserved-in-CPack-tar.gz.patch [^] (814 bytes) 2011-08-27 13:11 [Show Content]
patch file icon 0002-kwsys-Glob-Fix-so-directory-symlinks-are-added-as-fi.patch [^] (1,360 bytes) 2011-08-27 13:41 [Show Content]
patch file icon 0003-cpack-Fix-12284-don-t-recurse-through-directory-syml.patch [^] (811 bytes) 2011-08-27 13:41 [Show Content]
patch file icon 0001-ArchiveWrite-Don-t-recurse-through-directory-symlink.patch [^] (966 bytes) 2011-08-27 13:41 [Show Content]

 Relationships

  Notes
(0026915)
Brad King (manager)
2011-06-20 09:30

I've never really worked on CPack myself but from a quick glance I'll guess that the call to cmSystemTools::CopyFileIfDifferent in DoPackage is the culprit, and that it was originally added here:

  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=3bce601c [^]

(and was moved to its current location by your refactoring).
(0026925)
Eric NOULARD (developer)
2011-06-20 15:34

You mean something added in 2006 :-]

I'm not sure to follow you at all.
There is currently only two places where "cmSystemTools::CopyFileIfDifferent"
appears now.

In DoPackage this is used to copy package file itself so this cannot be the culprit.

And this is used in cmCPackGenerator::InstallProjectViaInstalledDirectories
where we check if "/* If it is not a symlink then do a plain copy */" before
doing that.

I'll have a look tonight, I'm in the middle of computer change so
I may have mixed up my testing.
(0026930)
Eric NOULARD (developer)
2011-06-20 18:19

Ok tested againts current master (cmake version 2.8.4.20110620-ge85df)
The problem is the same.

I suspect some subtlities in the way symlinks are handled or detected
by libarchive.

Like I said before I am able to create TGZ with CPack which contains symlinks
(at least between files in the same directory).

in CMake/CPack libarchive is imported from upstream (http://code.google.com/p/libarchive [^]) into Utilities/cmlibarchive.
Then CPack uses: Sources/cmArchiveWrite.*

a file is added using method cmArchiveWrite::AddFile which does not seem
to handle symlink at all!!

(no reference to archive_entry_*_symlink) so I guess something is missing
here and may be it appears not to be automatically handled for symlink
NOT within the same path level.

Just a wild guess I'm no lilbarchive expert.
May be someone knowing libarchive better than me can have a look at this?
(0026931)
Brad King (manager)
2011-06-21 08:23
edited on: 2011-06-21 08:24

Using "cmake -E tar" works fine and that uses the same libarchive settings as cpack uses.

The problem is that this is a *directory* symlink but CPack recursively globs and lists all *files* (perhaps in order to do component breakdown). In the posted example CPack asks libarchive to package these paths:

  cmt-1.1.1-Linux/bin/cmtdir/cmt
  cmt-1.1.1-Linux/cmtdir/cmt

explicitly so it never actually inspects "cmtdir".

(0026932)
Brad King (manager)
2011-06-21 08:29

I think the solution is to fix the recursive file list collection to treat symlinks as files whether they point to directories or not. I'm not sure whether this will have any compatibility consequences.
(0026933)
Eric NOULARD (developer)
2011-06-21 08:53

Ok I see,

CPack needed to know the list of file to package
before component as well because each specific generator
may do something specific with this list of files.

[Group] Component packaging added an extra need.

I do not currently have much spare time
but passing symlinks in the filelist should be doable
I'll try something like that when the time comes,
I cannot ensure any deadline though.
(0026943)
Alan W. Irwin (reporter)
2011-06-21 16:41

I reported this as a bug so long ago that it predates your mantis system (!) and apparently that bug report was not propagated to that system (or at least I cannot find it now).

I haven't done any recent tests of this, but in all cases symlinks should be preserved as symlinks in the generated tarball. My FreeEOS use case is a source tree with one or more very large files in it (in this case various tables of equation-of-state results) which are symlinked multiple times in my local source tree to save a ton of disk space. The symptom of the issue was a source tarball so large that I had to work around the problem by excluding the large files and asking my users to download those separately. Obviously, this is not a very satisfactory workaround so I would very much appreciate a fix for this problem.
(0026945)
Eric NOULARD (developer)
2011-06-22 02:33

Hi Alan,

I totally agree no doubt it's a bug.
The thing is I do not currently have time to work on it.

Now if you have a test case which is somehow different then file a complement here
that way it could be work out at the same time.

I say that because you seem to be concerned by **SOURCE** tarball which is not
handled strictly the same way by CPack (CPACK_INSTALLED_DIRECTORIES is used)
so your testcase may be different from John's.
(0027183)
Johan Björk (reporter)
2011-08-08 13:46

Here is a very short example. I would love to up the severity of this bug, this makes it really hard to build proper OSX frameworks.

#cmake ../
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/phb/DEV/test/cmake-framework/build
cherimoya:build phb$ cpack
CPack: Create package using TGZ
CPack: Install projects
CPack: - Run preinstall target for: Project
CPack: - Install project: Project
CPack: Create package
CPack: - package: /Users/phb/DEV/test/cmake-framework/build/test-0.0.1-Darwin.tar.gz generated.
cherimoya:build phb$ tar -tvzf test-0.0.1-Darwin.tar.gz
lrwxr-xr-x 0 phb staff 0 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/foo -> Versions/Current/foo
-rw-r--r-- 0 phb staff 0 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/Headers/foo.h
-rw-r--r-- 0 phb staff 732 Aug 8 19:44 test-0.0.1-Darwin/foo.framework/Resources/Info.plist
-rwxr-xr-x 0 phb staff 4104 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/Versions/5/foo
-rw-r--r-- 0 phb staff 0 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/Versions/5/Headers/foo.h
-rw-r--r-- 0 phb staff 732 Aug 8 19:44 test-0.0.1-Darwin/foo.framework/Versions/5/Resources/Info.plist
-rwxr-xr-x 0 phb staff 4104 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/Versions/Current/foo
-rw-r--r-- 0 phb staff 0 Aug 8 19:45 test-0.0.1-Darwin/foo.framework/Versions/Current/Headers/foo.h
-rw-r--r-- 0 phb staff 732 Aug 8 19:44 test-0.0.1-Darwin/foo.framework/Versions/Current/Resources/Info.plist
cherimoya:build phb$ tar -tvzf test-0.0.1-Darwin.tar.gz
(0027184)
Alan W. Irwin (reporter)
2011-08-08 23:20

To Eric:

Sorry it took so long for me to reply to your request, but I now have a case that serves as a simple example that works for _source_ package creation. IOW, the old source package symlink bug that I mentioned appears to be fixed now.

Here is simple test case for those who want to review why this works while binary packaging appears to have troubles with symlinks (from what others have said here).

irwin@raven> ls -l
total 12
-rw-r--r-- 1 irwin irwin 311 Aug 8 20:05 CMakeLists.txt
lrwxrwxrwx 1 irwin irwin 14 Aug 8 20:02 CMakeLists.txt_symlink -> CMakeLists.txt
-rw-r--r-- 1 irwin irwin 310 Aug 8 20:01 CMakeLists.txt~
drwxr-xr-x 4 irwin irwin 4096 Aug 8 20:03 build_dir/

irwin@raven> cat CMakeLists.txt
cmake_minimum_required(VERSION 2.8.5 FATAL_ERROR)
project(test_source_tarball NONE)
set(CPACK_SOURCE_GENERATOR TGZ)

set(CPACK_SOURCE_PACKAGE_FILE_NAME
  "test_source_tarball"
  CACHE INTERNAL "tarball basename"
  )

set(CPACK_SOURCE_IGNORE_FILES
  "~$"
  "^${PROJECT_SOURCE_DIR}/build_dir/"
  )
include(CPack)


If you cd to build_dir, empty it with rm -rf, then run

cmake ..
make package_source

then you generate a tarball with the expected symlink to CMakeLists.txt rather than a file copy like you got in the bad old days.

Hope this helps.
(0027377)
David Cole (manager)
2011-09-06 14:47

The following series of commits was merged to 'next' today, in this order:

  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=98cb017a9deca35cd9a67e03b6bd95064b6d2fb7 [^]
  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=527a40f06fc7f0ea6aa9c1fe96fb0fe5611fa633 [^]
  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=d78bdb27832c91c775ad3782c9eb436dcd6a1e7c [^]
  http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=7b8dcdd17312a7c3ed731743468136b0ea89a6c7 [^]

These should be equivalent to the changes attaches as a patch series to this issue, but they had to be re-worked and re-ordered because the KWSys changes had to be committed via the KWSys cvs repository first.
(0027378)
David Cole (manager)
2011-09-06 14:48

See previous note for pointers to commits that fix this issue
(0028824)
David Cole (manager)
2012-03-06 08:37

Closing resolved issues that have not been updated in more than 4 months.

 Issue History
Date Modified Username Field Change
2011-06-17 16:57 Eric NOULARD New Issue
2011-06-17 16:57 Eric NOULARD File Added: cmt.tgz
2011-06-20 09:30 Brad King Note Added: 0026915
2011-06-20 15:34 Eric NOULARD Note Added: 0026925
2011-06-20 15:34 Eric NOULARD Assigned To => Eric NOULARD
2011-06-20 15:34 Eric NOULARD Status new => assigned
2011-06-20 18:19 Eric NOULARD Note Added: 0026930
2011-06-20 18:19 Eric NOULARD Assigned To Eric NOULARD =>
2011-06-21 08:23 Brad King Note Added: 0026931
2011-06-21 08:24 Brad King Note Edited: 0026931
2011-06-21 08:29 Brad King Note Added: 0026932
2011-06-21 08:53 Eric NOULARD Note Added: 0026933
2011-06-21 09:11 Brad King Status assigned => backlog
2011-06-21 16:41 Alan W. Irwin Note Added: 0026943
2011-06-22 02:33 Eric NOULARD Note Added: 0026945
2011-08-08 13:46 Johan Björk Note Added: 0027183
2011-08-08 13:47 Johan Björk File Added: CMakeLists.txt
2011-08-08 23:20 Alan W. Irwin Note Added: 0027184
2011-08-27 13:11 Johan Björk File Added: 0001-Fix-12284-Symlinks-are-not-preserved-in-CPack-tar.gz.patch
2011-08-27 13:12 Johan Björk Note Added: 0027275
2011-08-27 13:13 Johan Björk Note Deleted: 0027275
2011-08-27 13:41 Johan Björk File Added: 0002-kwsys-Glob-Fix-so-directory-symlinks-are-added-as-fi.patch
2011-08-27 13:41 Johan Björk File Added: 0003-cpack-Fix-12284-don-t-recurse-through-directory-syml.patch
2011-08-27 13:41 Johan Björk File Added: 0001-ArchiveWrite-Don-t-recurse-through-directory-symlink.patch
2011-09-06 14:15 David Cole Assigned To => David Cole
2011-09-06 14:15 David Cole Status backlog => assigned
2011-09-06 14:16 David Cole Target Version => CMake 2.8.6
2011-09-06 14:47 David Cole Note Added: 0027377
2011-09-06 14:48 David Cole Note Added: 0027378
2011-09-06 14:48 David Cole Status assigned => resolved
2011-09-06 14:48 David Cole Fixed in Version => CMake 2.8.6
2011-09-06 14:48 David Cole Resolution open => fixed
2012-03-06 08:37 David Cole Note Added: 0028824
2012-03-06 08:37 David Cole Status resolved => closed


Copyright © 2000 - 2018 MantisBT Team