[CMake] Poor performance of copy_if_different
David Cole
david.cole at kitware.com
Tue Nov 6 15:10:20 EST 2012
The percentage thing is a known issue (in the backlog):
http://public.kitware.com/Bug/view.php?id=11928
On Tue, Nov 6, 2012 at 3:06 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
> Well I found the problem -- it's quite silly:
>
>
> add_custom_target( ${target_name}
> DEPENDS "${output_bins}"
> )
>
> You can't have quotes around ${output_bins}... if I remove the quotes
> it works -_-
>
> Now I get the following output (the percentages don't seem to work
> though, but at least the copies happen):
>
> Scanning dependencies of target copy_dlls_Debug
> [ 0%] Processing changed debug binaries in epixmlrpc
> [ 0%] Processing changed debug binaries in ewsmanagedapi
> [ 0%] Processing changed debug binaries in icu
> [ 0%] Processing changed debug binaries in json
> [100%] Processing changed debug binaries in msredist
> [100%] Processing changed debug binaries in mssdkutils
> [100%] Processing changed debug binaries in notesapi
> [100%] Processing changed debug binaries in nspr
> [100%] Processing changed debug binaries in nss
> [100%] Processing changed debug binaries in openssl
> [100%] Processing changed debug binaries in sqlapi
> [100%] Processing changed debug binaries in stlport
> [100%] Processing changed debug binaries in we70
> [100%] Processing changed debug binaries in zlib
> [100%] Built target copy_dlls_debug
>
> On Tue, Nov 6, 2012 at 2:02 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
>> I did some more testing.
>>
>> If I only make 1 command and depend on that command's output in 1
>> target, it executes the command just fine.
>>
>> If I create 2 commands and add both commands output files as one big
>> list of depends in 1 target, the commands fail.
>>
>> Is this a bug or am I setting up my dependencies incorrectly?
>>
>> On Tue, Nov 6, 2012 at 1:21 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
>>> You're right Clinton, I want to leave them alone but for a while they
>>> were not really used a lot. I went ahead and removed my logic to set
>>> CMAKE_CONFIGURATION_TYPES and it works great now. I'm able to generate
>>> my custom targets and commands without any errors. When I execute the
>>> target from nmake now, I get a failure:
>>>
>>> Scanning dependencies of target copy_dlls_Debug
>>> NMAKE : fatal error U1073: don't know how to make 'output\bin\.\libdb47d.dll'
>>> Stop.
>>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>>> Stop.
>>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>>> Stop.
>>> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
>>> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
>>> Stop.
>>>
>>> Any idea what it is trying to say? Isn't it just supposed to run the
>>> copy command to generate the output file? Here is my updated code
>>> (below). For each of my third party libraries, I create 1 custom
>>> command to copy its DLLs (each third party library can have more than
>>> one dll to copy). I then combine all of these custom commands into 1
>>> custom target for each configuration (copy_dlls_debug,
>>> copy_dlls_release, etc).
>>>
>>>
>>> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy" )
>>>
>>> if( CMAKE_CONFIGURATION_TYPES )
>>> set( configurations ${CMAKE_CONFIGURATION_TYPES} )
>>> else()
>>> set( configurations ${CMAKE_BUILD_TYPE} )
>>> endif()
>>>
>>> status( 0 "Generating copy_dlls for configurations: ${configurations}" )
>>>
>>> foreach( config ${configurations} )
>>> string( TOUPPER ${config} upper_config )
>>>
>>> get_property( binarydirs GLOBAL PROPERTY THIRD_PARTY_DIRS_${upper_config} )
>>> foreach( dir ${binarydirs} )
>>> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>>>
>>> if( WIN32 )
>>> file( GLOB pdbs "${dir}/*.pdb" )
>>> list( APPEND binaries ${pdbs} )
>>> endif()
>>>
>>> unset( tp_source_bins )
>>> unset( tp_output_bins )
>>> unset( copy_commands )
>>>
>>> foreach( bin ${binaries} )
>>> get_filename_component( bin_file ${bin} NAME )
>>> set( output_bin
>>> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}" )
>>>
>>> list( APPEND tp_source_bins ${bin} )
>>> list( APPEND tp_output_bins ${output_bin} )
>>>
>>> list( APPEND copy_commands
>>> COMMAND ${copycmd} "${bin}" "${output_bin}"
>>> )
>>> endforeach()
>>>
>>> file( RELATIVE_PATH relative_tp_dir ${THIRD_PARTY_DIR} ${dir} )
>>> string( FIND ${relative_tp_dir} "/" slash_pos )
>>> string( SUBSTRING ${relative_tp_dir} 0 ${slash_pos} tp_name )
>>>
>>> add_custom_command(
>>> COMMENT "Processing changed binaries in ${tp_name}"
>>> OUTPUT ${tp_output_bins}
>>> DEPENDS ${tp_source_bins}
>>> ${copy_commands}
>>> )
>>>
>>> list( APPEND output_bins ${tp_output_bins} )
>>> endforeach()
>>>
>>> set( target_name copy_dlls_${config} )
>>> add_custom_target( ${target_name}
>>> COMMENT "Copying ${config} binaries..."
>>> DEPENDS "${output_bins}"
>>> )
>>>
>>> list( APPEND copy_dlls_targets ${target_name} )
>>> endforeach()
>>>
>>> On Tue, Nov 6, 2012 at 1:07 PM, Clinton Stimpson <clinton at elemtech.com> wrote:
>>>> On Tuesday, November 06, 2012 12:46:01 PM Robert Dailey wrote:
>>>>> Is there an assumption here that CMAKE_CONFIGURATION_TYPES will not be
>>>>> set for single-configuration generators?
>>>>
>>>> Yes, that's the assumption. That's how I normally check for single vs. multi
>>>> config generators.
>>>>
>>>>>
>>>>> I have some code in my root CMake script that forces only 'debug' and
>>>>> 'release' configurations to exist (I didn't want the MinRelSize and
>>>>> friends in there). I think this will force the logic that depends on
>>>>> CMAKE_CONFIGURATION_TYPES to fail. What would you do in this case?
>>>>>
>>>>> Here is the code that sets CMAKE_CONFIGURATION_TYPES:
>>>>>
>>>>> cmake_minimum_required( VERSION 2.8.8 FATAL_ERROR )
>>>>>
>>>>> set( CMAKE_BUILD_TYPE debug )
>>>>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>>>>
>>>>> project( gmms )
>>>>>
>>>>> ... etc ...
>>>>
>>>>
>>>> It doesn't make much sense to me why one could only do a release build with a
>>>> mulit-config generator... I normally leave those variables alone and let the
>>>> user set them. I find RelWithDebInfo quite useful sometimes.
>>>>
>>>> But if you had to, perhaps you could do this, after your project(gmms):
>>>>
>>>> if(CMAKE_CONFIGURATION_TYPES)
>>>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>>> endif()
>>>>
>>>> Or some code to strip out types from CMAKE_CONFIGURATION_TYPES except for
>>>> debug and release. So if it was empty to begin with, it could stay empty.
>>>>
>>>>>
>>>>> On Tue, Nov 6, 2012 at 12:41 PM, Clinton Stimpson <clinton at elemtech.com>
>>>> wrote:
>>>>> > You could base your logic on CMAKE_CONFIGURATION_TYPES and CMAKE_BUILD
>>>>> > TYPE.
>>>>> >
>>>>> > For example:
>>>>> >
>>>>> > if(CMAKE_CONFIGURATION_TYPES)
>>>>> >
>>>>> > foreach(type ${CMAKE_CONFIGURATION_TYPES})
>>>>> >
>>>>> > make_copy_rules(${type})
>>>>> >
>>>>> > endforeach()
>>>>> >
>>>>> > else()
>>>>> >
>>>>> > make_copy_rules(${CMAKE_BUILD_TYPE})
>>>>> >
>>>>> > endif()
>>>>> >
>>>>> > On Tuesday, November 06, 2012 12:34:22 PM Robert Dailey wrote:
>>>>> >> I figured out the problem.
>>>>> >>
>>>>> >> On single-configuration generators (like NMake, which I'm testing now)
>>>>> >> both release and debug configurations have the same binary output
>>>>> >> directory. Since I'm creating 1 custom target (and corresponding
>>>>> >> commands) per configuration, the 2nd configuration has the same
>>>>> >> outputs as the 1st, so the 2nd target fails because I"m adding
>>>>> >> different rules to the same output file. This isn't a problem with
>>>>> >> Visual Studio generators because my outputs go to Debug and Release.
>>>>> >>
>>>>> >> Know any good workarounds for this?
>>>>> >>
>>>>> >> On Tue, Nov 6, 2012 at 11:54 AM, Clinton Stimpson <clinton at elemtech.com>
>>>>> >
>>>>> > wrote:
>>>>> >> > Try this:
>>>>> >> > add_custom_command(
>>>>> >> >
>>>>> >> > OUTPUT ${bin_outputs}
>>>>> >> > COMMAND ${copy_commands}
>>>>> >> > DEPENDS ....
>>>>> >> >
>>>>> >> > )
>>>>> >> >
>>>>> >> > On Tuesday, November 06, 2012 11:50:54 AM Robert Dailey wrote:
>>>>> >> >> Thanks Clinton,
>>>>> >> >>
>>>>> >> >> Here is what I have so far:
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> project( copy_dlls )
>>>>> >> >>
>>>>> >> >> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy_if_different" )
>>>>> >> >>
>>>>> >> >> set( configurations debug release )
>>>>> >> >> foreach( config ${configurations} )
>>>>> >> >>
>>>>> >> >> string( TOUPPER ${config} upper_config )
>>>>> >> >>
>>>>> >> >> get_property( binarydirs GLOBAL PROPERTY
>>>>> >> >>
>>>>> >> >> THIRD_PARTY_DIRS_${upper_config} ) foreach( dir ${binarydirs} )
>>>>> >> >>
>>>>> >> >> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>>>>> >> >>
>>>>> >> >> if( WIN32 AND config STREQUAL "debug" )
>>>>> >> >>
>>>>> >> >> file( GLOB pdbs "${dir}/*.pdb" )
>>>>> >> >> list( APPEND binaries ${pdbs} )
>>>>> >> >>
>>>>> >> >> endif()
>>>>> >> >>
>>>>> >> >> foreach( bin ${binaries} )
>>>>> >> >>
>>>>> >> >> get_filename_component( bin_file ${bin} NAME )
>>>>> >> >>
>>>>> >> >> list( APPEND bin_outputs ${bin} )
>>>>> >> >> list( APPEND copy_commands
>>>>> >> >>
>>>>> >> >> COMMAND ${copycmd} "${bin}"
>>>>> >> >>
>>>>> >> >> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}"
>>>>> >> >>
>>>>> >> >> )
>>>>> >> >>
>>>>> >> >> endforeach()
>>>>> >> >>
>>>>> >> >> endforeach()
>>>>> >> >>
>>>>> >> >> add_custom_command(
>>>>> >> >>
>>>>> >> >> OUTPUT ${bin_outputs}
>>>>> >> >> ${copy_commands}
>>>>> >> >>
>>>>> >> >> )
>>>>> >> >>
>>>>> >> >> add_custom_target( copy_dlls_${config}
>>>>> >> >>
>>>>> >> >> COMMENT "Copying ${config} binaries..."
>>>>> >> >> DEPENDS ${bin_outputs}
>>>>> >> >>
>>>>> >> >> )
>>>>> >> >>
>>>>> >> >> endforeach()
>>>>> >> >>
>>>>> >> >>
>>>>> >> >> This doesn't seem to work though, CMake tells me:
>>>>> >> >>
>>>>> >> >> CMake Error: Attempt to add a custom rule to output
>>>>> >> >> "C:/Work/rdailey-hp/dpd-cmake/build-nmake-vc9/third_party/bdb/4.7.25/v
>>>>> >> >> c9s
>>>>> >> >> p1/ debug/bin/libdb47d.dll.rule" which already has a custom rule.
>>>>> >> >>
>>>>> >> >> I'm trying to stuff all the copy commands and output files into 1
>>>>> >> >> custom command, and wrap that with a custom target, so I can make
>>>>> >> >> other targets depend on the custom target.
>>>>> >> >>
>>>>> >> >> On Tue, Nov 6, 2012 at 11:44 AM, Clinton Stimpson
>>>>> >> >> <clinton at elemtech.com>
>>>>> >> >
>>>>> >> > wrote:
>>>>> >> >> > add_custom_command(
>>>>> >> >> >
>>>>> >> >> > OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>>>> >> >> > COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>>>> >> >> >
>>>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>>>> >> >> >
>>>>> >> >> > DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>>>> >> >> > )
>>>>> >> >> >
>>>>> >> >> > add_custom_target(copy_dll ALL DEPENDS
>>>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll)
>>>>> >> >> >
>>>>> >> >> > Whether you use 1 custom command per dll, or 1 for all dlls, is up
>>>>> >> >> > to
>>>>> >> >> > you.
>>>>> >> >> >
>>>>> >> >> > If they usually change together, then you could have fewer custom
>>>>> >> >> >
>>>>> >> >> > commands. In that case, the OUTPUT would have multiple files and
>>>>> >> >> > both
>>>>> >> >> > DEPENDS would have multiple files.
>>>>> >> >> >
>>>>> >> >> > Another alternative is to use add_custom_command(TARGET POST_BUILD
>>>>> >> >> > ...)
>>>>> >> >> > if
>>>>> >> >> > these are dlls that you compile yourself and need to copy after the
>>>>> >> >> > build
>>>>> >> >> > step.
>>>>> >> >> >
>>>>> >> >> > On Tuesday, November 06, 2012 11:23:03 AM Robert Dailey wrote:
>>>>> >> >> >> Also, is it safe to add 1 custom command for each DLL copy? Or
>>>>> >> >> >> would
>>>>> >> >> >> you somehow only use 1 custom command and copy them all with it?
>>>>> >> >> >> Also
>>>>> >> >> >> would I add the custom commands to the custom target?
>>>>> >> >> >>
>>>>> >> >> >> On Tue, Nov 6, 2012 at 11:11 AM, Clinton Stimpson
>>>>> >> >> >> <clinton at elemtech.com>
>>>>> >> >> >
>>>>> >> >> > wrote:
>>>>> >> >> >> > On Tuesday, November 06, 2012 11:06:45 AM Robert Dailey wrote:
>>>>> >> >> >> >> I use ${CMAKE_COMMAND} -E copy_if_different to copy DLL files to
>>>>> >> >> >> >> my
>>>>> >> >> >> >> binary output directory. The custom target runs this command
>>>>> >> >> >> >> about
>>>>> >> >> >> >> 50-100 times (for that many files).
>>>>> >> >> >> >>
>>>>> >> >> >> >> I notice that when all files are already copied, the commands
>>>>> >> >> >> >> still
>>>>> >> >> >> >> run extremely slowly. It takes just as long to copy all the
>>>>> >> >> >> >> files
>>>>> >> >> >> >> as
>>>>> >> >> >> >> it does to copy none of them.
>>>>> >> >> >> >>
>>>>> >> >> >> >> I used the Sysinternals tool called Process Monitor to see what
>>>>> >> >> >> >> CMake
>>>>> >> >> >> >> is doing as it runs copy_if_different. It's opening the full
>>>>> >> >> >> >> file
>>>>> >> >> >> >> and
>>>>> >> >> >> >> seems to be comparing actual contents instead of something
>>>>> >> >> >> >> simple,
>>>>> >> >> >> >> such as timestamp.
>>>>> >> >> >> >>
>>>>> >> >> >> >> I do not need such a thorough check, I simply want the check to
>>>>> >> >> >> >> see
>>>>> >> >> >> >> which timestamp is higher and copy if the source is newer than
>>>>> >> >> >> >> the
>>>>> >> >> >> >> target.
>>>>> >> >> >> >>
>>>>> >> >> >> >> Any reason why copy_if_different is so slow? Is my assumption
>>>>> >> >> >> >> correct?
>>>>> >> >> >> >> How can I make it faster?
>>>>> >> >> >> >
>>>>> >> >> >> > How about using plain "cmake -E copy ..." and rely on the
>>>>> >> >> >> > timestamp
>>>>> >> >> >> > check
>>>>> >> >> >> > done by a custom command (add_custom_command()).
>>>>> >> >> >> > You need to make sure the input/ouput parts of the custom command
>>>>> >> >> >> > are
>>>>> >> >> >> > set
>>>>> >> >> >> > correctly so it can do a timestamp check.
>>>>> >> >> >> >
>>>>> >> >> >> > --
>>>>> >> >> >> > Clinton Stimpson
>>>>> >> >> >> > Elemental Technologies, Inc
>>>>> >> >> >> > Computational Simulation Software, LLC
>>>>> >> >> >> > www.csimsoft.com
>>>>> >> >> >> > --
>>>>> >> >> >> >
>>>>> >> >> >> > Powered by www.kitware.com
>>>>> >> >> >> >
>>>>> >> >> >> > Visit other Kitware open-source projects at
>>>>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>>>>> >> >> >> >
>>>>> >> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>>> >> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>>> >> >> >> >
>>>>> >> >> >> > Follow this link to subscribe/unsubscribe:
>>>>> >> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>>> >> >> >
>>>>> >> >> > --
>>>>> >> >> > Clinton Stimpson
>>>>> >> >> > Elemental Technologies, Inc
>>>>> >> >> > Computational Simulation Software, LLC
>>>>> >> >> > www.csimsoft.com
>>>>> >> >> > --
>>>>> >> >> >
>>>>> >> >> > Powered by www.kitware.com
>>>>> >> >> >
>>>>> >> >> > Visit other Kitware open-source projects at
>>>>> >> >> > http://www.kitware.com/opensource/opensource.html
>>>>> >> >> >
>>>>> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>>> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>>> >> >> >
>>>>> >> >> > Follow this link to subscribe/unsubscribe:
>>>>> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>>> >> >
>>>>> >> > --
>>>>> >> > Clinton Stimpson
>>>>> >> > Elemental Technologies, Inc
>>>>> >> > Computational Simulation Software, LLC
>>>>> >> > www.csimsoft.com
>>>>> >> > --
>>>>> >> >
>>>>> >> > Powered by www.kitware.com
>>>>> >> >
>>>>> >> > Visit other Kitware open-source projects at
>>>>> >> > http://www.kitware.com/opensource/opensource.html
>>>>> >> >
>>>>> >> > Please keep messages on-topic and check the CMake FAQ at:
>>>>> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>>>> >> >
>>>>> >> > Follow this link to subscribe/unsubscribe:
>>>>> >> > http://www.cmake.org/mailman/listinfo/cmake
>>>>> >
>>>>> > --
>>>>> > Clinton Stimpson
>>>>> > Elemental Technologies, Inc
>>>>> > Computational Simulation Software, LLC
>>>>> > www.csimsoft.com
>>>>> > --
>>>>> >
>>>>> > Powered by www.kitware.com
>>>>> >
>>>>> > Visit other Kitware open-source projects at
>>>>> > http://www.kitware.com/opensource/opensource.html
>>>>> >
>>>>> > Please keep messages on-topic and check the CMake FAQ at:
>>>>> > http://www.cmake.org/Wiki/CMake_FAQ
>>>>> >
>>>>> > Follow this link to subscribe/unsubscribe:
>>>>> > http://www.cmake.org/mailman/listinfo/cmake
>>>> --
>>>> Clinton Stimpson
>>>> Elemental Technologies, Inc
>>>> Computational Simulation Software, LLC
>>>> www.csimsoft.com
>>>> --
>>>>
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.cmake.org/mailman/listinfo/cmake
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
More information about the CMake
mailing list