View Issue Details [ Jump to Notes ] [ Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0014874CMakeCMakepublic2014-04-14 11:502015-01-05 08:38
Reporterzub 
Assigned ToBrad King 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
Platformx86_64OSLinuxOS VersionDebian sid
Product VersionCMake 2.8.12.1 
Target VersionCMake 3.1Fixed in VersionCMake 3.1 
Summary0014874: Static library can miss some object files
DescriptionWhen creating a static library using binutils (ar) and there exist a duplicate object name (e.g. a/Foo.cpp.o, b/Foo.cpp.o), the resulting static library can end up having only one of the duplicate objects.

My reading of ar man page implies ar in general is OK with duplicate object names. Also, creating a simple CMake project (just a/foo.cpp, b/foo.cpp, add_library(test STATIC a/foo.cpp b/foo.cpp) results in a correct libtest.a.

This bug only happens if there are many objects. This affects how CMake invokes ar. By default CMake invokes ar for the first time for a library like this:

ar cr libtest.a path/to/a/foo.cpp.o path/to/b/foo.cpp.o ...

and if this is the only invocation, it produces a correct libtest.a. But if there are many objects, CMake invokes ar several times, using the following for subsequent calls:

ar r libtest.a other objects...

Now if it happens that the first duplicate object is added in one invocation of ar, and then CMake invokes ar again to add another, ar just replaces the previous object.

When I set:

set(CMAKE_CXX_ARCHIVE_APPEND "<CMAKE_AR> <LINK_FLAGS> q <TARGET> <OBJECTS>")

to use "q" instead of "r", the issue is resolved for me, so the fix could be just using "q" when adding objects (CMAKE_CXX_ARCHIVE_APPEND, CMAKE_C_ARCHIVE_APPEND and perhaps also CMAKE_CXX_ARCHIVE_CREATE and CMAKE_C_ARCHIVE_CREATE, though this probably won't make a difference).

But I'm not sure how do other ar's handle this. I tested this using "GNU ar (GNU Binutils for Debian) 2.24".
Steps To ReproduceThe issue is deterministic, but it does not occur for every project with duplicate object names - it depends on the project.

To trigger this a project with many objects is required so that CMake in fact does invoke ar multiple times. Also, the duplicate objects must find their way into multiple invocations of ar, e.g.:

ar cr libtest.a objects... a/foo.cpp.o objects...
ar r libtest.a objects... b/foo.cpp.o objects...

Then ar removes the previous foo.cpp.o.

Creating a standalone testcase seems not so trivial; I run into this on a real project. If necessary, I can try to create one.
TagsNo tags attached.
Attached Files

 Relationships
related to 0006284closedBrad King Linking static libraries with long list of objects files with Makefile generators fail 
has duplicate 0014994closed Add an option to change the object file name 

  Notes
(0035695)
Brad King (manager)
2014-04-14 14:03

 Good catch, thanks. I think "q" will work because the only platform information file that uses the separate create/append steps is Windows-GNU.cmake where we know we have the GNU archiver.
(0035702)
Marcel Loose (developer)
2014-04-15 03:26

 Well, there's one problem. `ar` doesn't record path names (see, e.g. https://stackoverflow.com/questions/4907121/static-library-having-object-files-with-same-name-ar [^]). How are you going to distinguish the two identically named object files?

$ touch a.o b.o
$ ar cr foo.a a.o b.o
$ ar tv foo.a
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 b.o
$ mkdir a b
$ touch a/a.o b/b.o
$ ar q foo.a a/a.o b/b.o
$ ar tv foo.a
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:23 2014 b.o
rw-r--r-- 1000/1000 0 Apr 15 09:24 2014 a.o
rw-r--r-- 1000/1000 0 Apr 15 09:24 2014 b.o
(0035703)
zub (reporter)
2014-04-15 03:52

 Yes, the archive ends up with duplicate file names but when I look into binutils ar man page I think this is a supported usecase. binutils ar supports this modifier:

   N Uses the count parameter. This is used if there are multiple
       entries in the archive with the same name. Extract or delete
       instance count of the given name from the archive.

so it is possible to selectively extract individual objects:

$ mkdir a b
$ echo a > a/foo.o
$ echo b > b/foo.o
$ ar q foo.a a/foo.o b/foo.o
ar: creating foo.a
$ ar xvN 1 foo.a foo.o && cat foo.o
x - foo.o
a
$ ar xvN 2 foo.a foo.o && cat foo.o
x - foo.o
b

Indeed, one has to be careful when extracting the objects. But simply using the static library for linking seems to work OK. I think linker doesn't care about the object names.
(0035707)
Brad King (manager)
2014-04-15 09:02

 Patch applied:

 Windows-GNU: Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=39d0ade0 [^]
(0035721)
zub (reporter)
2014-04-15 17:05

 Unfortunately the issue is not yet resolved. I can still reproduce it with 8472ef243ffc9988ea8fb83cbc7acdf3f0daa239.

I didn't fully understand Brad King's original comment. The bug happens not only on MinGW, but also on Linux. I experienced the issue in Debian.

There seem to be several places where the ar options are set. Grepping in Modules/ brings up:

* CMakeCInformation.cmake, CMakeCXXInformation.cmake and CMakeFortranInformation.cmake - these set CMAKE_lang_ARCHIVE_... to some default value if not set elsewhere

* Platform/Windows-GNU.cmake - this has been resolved by http://cmake.org/gitweb?p=cmake.git;a=commit;h=39d0ade0 [^]

* Compiler/TI-C.cmake and Compiler/TI-CXX.cmake - for a TI compiler

If I replace the "r" with "q" in the CMakeC/CXX/FotranInformation.cmake I can no longer reproduce the issue, so I believe that fixes it with GCC on Linux. This is the patch I used and tested on my box:

http://linux.fjfi.cvut.cz/~zub/0001-GNU-Support-duplicate-object-names-in-large-archives.patch [^]

(TBH I tested CXX only, but I think it's reasonable to assume the same behavior with C and Fortran.)

As for the TI compiler - it might be good to change it there too but I don't have the TI toolchain and I have no idea about their ar, so I can't test it.
(0035722)
Brad King (manager)
2014-04-16 08:12

 Re 0014874:0035721: Ugh, I don't know how I missed those. We need only to grep for every place that sets the append rule:

$ git grep -l ARCHIVE_APPEND v3.0.0-rc3 -- Modules
v3.0.0-rc3:Modules/CMakeCInformation.cmake
v3.0.0-rc3:Modules/CMakeCXXInformation.cmake
v3.0.0-rc3:Modules/CMakeFortranInformation.cmake
v3.0.0-rc3:Modules/Platform/Windows-GNU.cmake
(0035723)
Brad King (manager)
2014-04-16 08:14

 I think "q" is a POSIX ar option. It appears in "man ar" on several old UNIX machines too. Here is the more general fix:

 Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=f54be6c0 [^]
(0035724)
zub (reporter)
2014-04-16 08:27

 That fix will solve the issue for me.

What about Compiler/TI-C.cmake and Compiler/TI-CXX.cmake?
(0035725)
Brad King (manager)
2014-04-16 08:34

 Re 0014874:0035724: The TI modules do not add any append rules, and I don't have the toolchain to test it. See 0014876 for dicussion on a rewrite of the TI modules.
(0035730)
Brad King (manager)
2014-04-16 14:07

 Re 0014874:0035723: I rebased the fix back on the commit it generalizes:

 Support duplicate object names in large archives
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=1ec6485c [^]
(0035735)
Brad King (manager)
2014-04-17 10:08

 The more general fix in 0014874:0035730 is now in 'master' and will be in 3.1.
(0037571)
Robert Maynard (manager)
2015-01-05 08:38

 Closing resolved issues that have not been updated in more than 4 months.

 Issue History
Date Modified Username Field Change
2014-04-14 11:50 zub New Issue
2014-04-14 14:02 Brad King Relationship added related to 0006284
2014-04-14 14:03 Brad King Note Added: 0035695
2014-04-15 03:26 Marcel Loose Note Added: 0035702
2014-04-15 03:52 zub Note Added: 0035703
2014-04-15 09:02 Brad King Note Added: 0035707
2014-04-15 09:03 Brad King Assigned To => Brad King
2014-04-15 09:03 Brad King Status new => resolved
2014-04-15 09:03 Brad King Resolution open => fixed
2014-04-15 09:03 Brad King Fixed in Version => CMake 3.1
2014-04-15 09:03 Brad King Target Version => CMake 3.1
2014-04-15 17:05 zub Note Added: 0035721
2014-04-15 17:05 zub Status resolved => feedback
2014-04-15 17:05 zub Resolution fixed => reopened
2014-04-16 07:40 Brad King Status feedback => assigned
2014-04-16 07:40 Brad King Resolution reopened => open
2014-04-16 07:40 Brad King Fixed in Version CMake 3.1 =>
2014-04-16 08:12 Brad King Note Added: 0035722
2014-04-16 08:14 Brad King Note Added: 0035723
2014-04-16 08:27 zub Note Added: 0035724
2014-04-16 08:34 Brad King Note Added: 0035725
2014-04-16 14:07 Brad King Note Added: 0035730
2014-04-17 10:08 Brad King Note Added: 0035735
2014-04-17 10:08 Brad King Status assigned => resolved
2014-04-17 10:08 Brad King Resolution open => fixed
2014-04-17 10:08 Brad King Fixed in Version => CMake 3.1
2014-06-26 09:16 Nils Gladitz Relationship added has duplicate 0014994
2015-01-05 08:38 Robert Maynard Note Added: 0037571
2015-01-05 08:38 Robert Maynard Status resolved => closed

Copyright © 2000 - 2018 MantisBT Team