[CMake] Poor performance of copy_if_different
Robert Dailey
rcdailey.lists at gmail.com
Tue Nov 6 15:02:12 EST 2012
I did some more testing.
If I only make 1 command and depend on that command's output in 1
target, it executes the command just fine.
If I create 2 commands and add both commands output files as one big
list of depends in 1 target, the commands fail.
Is this a bug or am I setting up my dependencies incorrectly?
On Tue, Nov 6, 2012 at 1:21 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
> You're right Clinton, I want to leave them alone but for a while they
> were not really used a lot. I went ahead and removed my logic to set
> CMAKE_CONFIGURATION_TYPES and it works great now. I'm able to generate
> my custom targets and commands without any errors. When I execute the
> target from nmake now, I get a failure:
>
> Scanning dependencies of target copy_dlls_Debug
> NMAKE : fatal error U1073: don't know how to make 'output\bin\.\libdb47d.dll'
> Stop.
> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
> Stop.
> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
> Stop.
> NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
> Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
> Stop.
>
> Any idea what it is trying to say? Isn't it just supposed to run the
> copy command to generate the output file? Here is my updated code
> (below). For each of my third party libraries, I create 1 custom
> command to copy its DLLs (each third party library can have more than
> one dll to copy). I then combine all of these custom commands into 1
> custom target for each configuration (copy_dlls_debug,
> copy_dlls_release, etc).
>
>
> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy" )
>
> if( CMAKE_CONFIGURATION_TYPES )
> set( configurations ${CMAKE_CONFIGURATION_TYPES} )
> else()
> set( configurations ${CMAKE_BUILD_TYPE} )
> endif()
>
> status( 0 "Generating copy_dlls for configurations: ${configurations}" )
>
> foreach( config ${configurations} )
> string( TOUPPER ${config} upper_config )
>
> get_property( binarydirs GLOBAL PROPERTY THIRD_PARTY_DIRS_${upper_config} )
> foreach( dir ${binarydirs} )
> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>
> if( WIN32 )
> file( GLOB pdbs "${dir}/*.pdb" )
> list( APPEND binaries ${pdbs} )
> endif()
>
> unset( tp_source_bins )
> unset( tp_output_bins )
> unset( copy_commands )
>
> foreach( bin ${binaries} )
> get_filename_component( bin_file ${bin} NAME )
> set( output_bin
> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}" )
>
> list( APPEND tp_source_bins ${bin} )
> list( APPEND tp_output_bins ${output_bin} )
>
> list( APPEND copy_commands
> COMMAND ${copycmd} "${bin}" "${output_bin}"
> )
> endforeach()
>
> file( RELATIVE_PATH relative_tp_dir ${THIRD_PARTY_DIR} ${dir} )
> string( FIND ${relative_tp_dir} "/" slash_pos )
> string( SUBSTRING ${relative_tp_dir} 0 ${slash_pos} tp_name )
>
> add_custom_command(
> COMMENT "Processing changed binaries in ${tp_name}"
> OUTPUT ${tp_output_bins}
> DEPENDS ${tp_source_bins}
> ${copy_commands}
> )
>
> list( APPEND output_bins ${tp_output_bins} )
> endforeach()
>
> set( target_name copy_dlls_${config} )
> add_custom_target( ${target_name}
> COMMENT "Copying ${config} binaries..."
> DEPENDS "${output_bins}"
> )
>
> list( APPEND copy_dlls_targets ${target_name} )
> endforeach()
>
> On Tue, Nov 6, 2012 at 1:07 PM, Clinton Stimpson <clinton at elemtech.com> wrote:
>> On Tuesday, November 06, 2012 12:46:01 PM Robert Dailey wrote:
>>> Is there an assumption here that CMAKE_CONFIGURATION_TYPES will not be
>>> set for single-configuration generators?
>>
>> Yes, that's the assumption. That's how I normally check for single vs. multi
>> config generators.
>>
>>>
>>> I have some code in my root CMake script that forces only 'debug' and
>>> 'release' configurations to exist (I didn't want the MinRelSize and
>>> friends in there). I think this will force the logic that depends on
>>> CMAKE_CONFIGURATION_TYPES to fail. What would you do in this case?
>>>
>>> Here is the code that sets CMAKE_CONFIGURATION_TYPES:
>>>
>>> cmake_minimum_required( VERSION 2.8.8 FATAL_ERROR )
>>>
>>> set( CMAKE_BUILD_TYPE debug )
>>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>>
>>> project( gmms )
>>>
>>> ... etc ...
>>
>>
>> It doesn't make much sense to me why one could only do a release build with a
>> mulit-config generator... I normally leave those variables alone and let the
>> user set them. I find RelWithDebInfo quite useful sometimes.
>>
>> But if you had to, perhaps you could do this, after your project(gmms):
>>
>> if(CMAKE_CONFIGURATION_TYPES)
>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>> endif()
>>
>> Or some code to strip out types from CMAKE_CONFIGURATION_TYPES except for
>> debug and release. So if it was empty to begin with, it could stay empty.
>>
>>>
>>> On Tue, Nov 6, 2012 at 12:41 PM, Clinton Stimpson <clinton at elemtech.com>
>> wrote:
>>> > You could base your logic on CMAKE_CONFIGURATION_TYPES and CMAKE_BUILD
>>> > TYPE.
>>> >
>>> > For example:
>>> >
>>> > if(CMAKE_CONFIGURATION_TYPES)
>>> >
>>> > foreach(type ${CMAKE_CONFIGURATION_TYPES})
>>> >
>>> > make_copy_rules(${type})
>>> >
>>> > endforeach()
>>> >
>>> > else()
>>> >
>>> > make_copy_rules(${CMAKE_BUILD_TYPE})
>>> >
>>> > endif()
>>> >
>>> > On Tuesday, November 06, 2012 12:34:22 PM Robert Dailey wrote:
>>> >> I figured out the problem.
>>> >>
>>> >> On single-configuration generators (like NMake, which I'm testing now)
>>> >> both release and debug configurations have the same binary output
>>> >> directory. Since I'm creating 1 custom target (and corresponding
>>> >> commands) per configuration, the 2nd configuration has the same
>>> >> outputs as the 1st, so the 2nd target fails because I"m adding
>>> >> different rules to the same output file. This isn't a problem with
>>> >> Visual Studio generators because my outputs go to Debug and Release.
>>> >>
>>> >> Know any good workarounds for this?
>>> >>
>>> >> On Tue, Nov 6, 2012 at 11:54 AM, Clinton Stimpson <clinton at elemtech.com>
>>> >
>>> > wrote:
>>> >> > Try this:
>>> >> > add_custom_command(
>>> >> >
>>> >> > OUTPUT ${bin_outputs}
>>> >> > COMMAND ${copy_commands}
>>> >> > DEPENDS ....
>>> >> >
>>> >> > )
>>> >> >
>>> >> > On Tuesday, November 06, 2012 11:50:54 AM Robert Dailey wrote:
>>> >> >> Thanks Clinton,
>>> >> >>
>>> >> >> Here is what I have so far:
>>> >> >>
>>> >> >>
>>> >> >> project( copy_dlls )
>>> >> >>
>>> >> >> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy_if_different" )
>>> >> >>
>>> >> >> set( configurations debug release )
>>> >> >> foreach( config ${configurations} )
>>> >> >>
>>> >> >> string( TOUPPER ${config} upper_config )
>>> >> >>
>>> >> >> get_property( binarydirs GLOBAL PROPERTY
>>> >> >>
>>> >> >> THIRD_PARTY_DIRS_${upper_config} ) foreach( dir ${binarydirs} )
>>> >> >>
>>> >> >> file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>>> >> >>
>>> >> >> if( WIN32 AND config STREQUAL "debug" )
>>> >> >>
>>> >> >> file( GLOB pdbs "${dir}/*.pdb" )
>>> >> >> list( APPEND binaries ${pdbs} )
>>> >> >>
>>> >> >> endif()
>>> >> >>
>>> >> >> foreach( bin ${binaries} )
>>> >> >>
>>> >> >> get_filename_component( bin_file ${bin} NAME )
>>> >> >>
>>> >> >> list( APPEND bin_outputs ${bin} )
>>> >> >> list( APPEND copy_commands
>>> >> >>
>>> >> >> COMMAND ${copycmd} "${bin}"
>>> >> >>
>>> >> >> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}"
>>> >> >>
>>> >> >> )
>>> >> >>
>>> >> >> endforeach()
>>> >> >>
>>> >> >> endforeach()
>>> >> >>
>>> >> >> add_custom_command(
>>> >> >>
>>> >> >> OUTPUT ${bin_outputs}
>>> >> >> ${copy_commands}
>>> >> >>
>>> >> >> )
>>> >> >>
>>> >> >> add_custom_target( copy_dlls_${config}
>>> >> >>
>>> >> >> COMMENT "Copying ${config} binaries..."
>>> >> >> DEPENDS ${bin_outputs}
>>> >> >>
>>> >> >> )
>>> >> >>
>>> >> >> endforeach()
>>> >> >>
>>> >> >>
>>> >> >> This doesn't seem to work though, CMake tells me:
>>> >> >>
>>> >> >> CMake Error: Attempt to add a custom rule to output
>>> >> >> "C:/Work/rdailey-hp/dpd-cmake/build-nmake-vc9/third_party/bdb/4.7.25/v
>>> >> >> c9s
>>> >> >> p1/ debug/bin/libdb47d.dll.rule" which already has a custom rule.
>>> >> >>
>>> >> >> I'm trying to stuff all the copy commands and output files into 1
>>> >> >> custom command, and wrap that with a custom target, so I can make
>>> >> >> other targets depend on the custom target.
>>> >> >>
>>> >> >> On Tue, Nov 6, 2012 at 11:44 AM, Clinton Stimpson
>>> >> >> <clinton at elemtech.com>
>>> >> >
>>> >> > wrote:
>>> >> >> > add_custom_command(
>>> >> >> >
>>> >> >> > OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>> >> >> > COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>> >> >> >
>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>>> >> >> >
>>> >> >> > DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>>> >> >> > )
>>> >> >> >
>>> >> >> > add_custom_target(copy_dll ALL DEPENDS
>>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll)
>>> >> >> >
>>> >> >> > Whether you use 1 custom command per dll, or 1 for all dlls, is up
>>> >> >> > to
>>> >> >> > you.
>>> >> >> >
>>> >> >> > If they usually change together, then you could have fewer custom
>>> >> >> >
>>> >> >> > commands. In that case, the OUTPUT would have multiple files and
>>> >> >> > both
>>> >> >> > DEPENDS would have multiple files.
>>> >> >> >
>>> >> >> > Another alternative is to use add_custom_command(TARGET POST_BUILD
>>> >> >> > ...)
>>> >> >> > if
>>> >> >> > these are dlls that you compile yourself and need to copy after the
>>> >> >> > build
>>> >> >> > step.
>>> >> >> >
>>> >> >> > On Tuesday, November 06, 2012 11:23:03 AM Robert Dailey wrote:
>>> >> >> >> Also, is it safe to add 1 custom command for each DLL copy? Or
>>> >> >> >> would
>>> >> >> >> you somehow only use 1 custom command and copy them all with it?
>>> >> >> >> Also
>>> >> >> >> would I add the custom commands to the custom target?
>>> >> >> >>
>>> >> >> >> On Tue, Nov 6, 2012 at 11:11 AM, Clinton Stimpson
>>> >> >> >> <clinton at elemtech.com>
>>> >> >> >
>>> >> >> > wrote:
>>> >> >> >> > On Tuesday, November 06, 2012 11:06:45 AM Robert Dailey wrote:
>>> >> >> >> >> I use ${CMAKE_COMMAND} -E copy_if_different to copy DLL files to
>>> >> >> >> >> my
>>> >> >> >> >> binary output directory. The custom target runs this command
>>> >> >> >> >> about
>>> >> >> >> >> 50-100 times (for that many files).
>>> >> >> >> >>
>>> >> >> >> >> I notice that when all files are already copied, the commands
>>> >> >> >> >> still
>>> >> >> >> >> run extremely slowly. It takes just as long to copy all the
>>> >> >> >> >> files
>>> >> >> >> >> as
>>> >> >> >> >> it does to copy none of them.
>>> >> >> >> >>
>>> >> >> >> >> I used the Sysinternals tool called Process Monitor to see what
>>> >> >> >> >> CMake
>>> >> >> >> >> is doing as it runs copy_if_different. It's opening the full
>>> >> >> >> >> file
>>> >> >> >> >> and
>>> >> >> >> >> seems to be comparing actual contents instead of something
>>> >> >> >> >> simple,
>>> >> >> >> >> such as timestamp.
>>> >> >> >> >>
>>> >> >> >> >> I do not need such a thorough check, I simply want the check to
>>> >> >> >> >> see
>>> >> >> >> >> which timestamp is higher and copy if the source is newer than
>>> >> >> >> >> the
>>> >> >> >> >> target.
>>> >> >> >> >>
>>> >> >> >> >> Any reason why copy_if_different is so slow? Is my assumption
>>> >> >> >> >> correct?
>>> >> >> >> >> How can I make it faster?
>>> >> >> >> >
>>> >> >> >> > How about using plain "cmake -E copy ..." and rely on the
>>> >> >> >> > timestamp
>>> >> >> >> > check
>>> >> >> >> > done by a custom command (add_custom_command()).
>>> >> >> >> > You need to make sure the input/ouput parts of the custom command
>>> >> >> >> > are
>>> >> >> >> > set
>>> >> >> >> > correctly so it can do a timestamp check.
>>> >> >> >> >
>>> >> >> >> > --
>>> >> >> >> > Clinton Stimpson
>>> >> >> >> > Elemental Technologies, Inc
>>> >> >> >> > Computational Simulation Software, LLC
>>> >> >> >> > www.csimsoft.com
>>> >> >> >> > --
>>> >> >> >> >
>>> >> >> >> > Powered by www.kitware.com
>>> >> >> >> >
>>> >> >> >> > Visit other Kitware open-source projects at
>>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>>> >> >> >> >
>>> >> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>> >> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>> >> >> >> >
>>> >> >> >> > Follow this link to subscribe/unsubscribe:
>>> >> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>> >> >> >
>>> >> >> > --
>>> >> >> > Clinton Stimpson
>>> >> >> > Elemental Technologies, Inc
>>> >> >> > Computational Simulation Software, LLC
>>> >> >> > www.csimsoft.com
>>> >> >> > --
>>> >> >> >
>>> >> >> > Powered by www.kitware.com
>>> >> >> >
>>> >> >> > Visit other Kitware open-source projects at
>>> >> >> > http://www.kitware.com/opensource/opensource.html
>>> >> >> >
>>> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>>> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>> >> >> >
>>> >> >> > Follow this link to subscribe/unsubscribe:
>>> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>>> >> >
>>> >> > --
>>> >> > Clinton Stimpson
>>> >> > Elemental Technologies, Inc
>>> >> > Computational Simulation Software, LLC
>>> >> > www.csimsoft.com
>>> >> > --
>>> >> >
>>> >> > Powered by www.kitware.com
>>> >> >
>>> >> > Visit other Kitware open-source projects at
>>> >> > http://www.kitware.com/opensource/opensource.html
>>> >> >
>>> >> > Please keep messages on-topic and check the CMake FAQ at:
>>> >> > http://www.cmake.org/Wiki/CMake_FAQ
>>> >> >
>>> >> > Follow this link to subscribe/unsubscribe:
>>> >> > http://www.cmake.org/mailman/listinfo/cmake
>>> >
>>> > --
>>> > Clinton Stimpson
>>> > Elemental Technologies, Inc
>>> > Computational Simulation Software, LLC
>>> > www.csimsoft.com
>>> > --
>>> >
>>> > Powered by www.kitware.com
>>> >
>>> > Visit other Kitware open-source projects at
>>> > http://www.kitware.com/opensource/opensource.html
>>> >
>>> > Please keep messages on-topic and check the CMake FAQ at:
>>> > http://www.cmake.org/Wiki/CMake_FAQ
>>> >
>>> > Follow this link to subscribe/unsubscribe:
>>> > http://www.cmake.org/mailman/listinfo/cmake
>> --
>> Clinton Stimpson
>> Elemental Technologies, Inc
>> Computational Simulation Software, LLC
>> www.csimsoft.com
>> --
>>
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.cmake.org/mailman/listinfo/cmake
More information about the CMake
mailing list