[CMake] Poor performance of copy_if_different

Robert Dailey rcdailey.lists at gmail.com
Tue Nov 6 14:21:39 EST 2012 

You're right Clinton, I want to leave them alone but for a while they
were not really used a lot. I went ahead and removed my logic to set
CMAKE_CONFIGURATION_TYPES and it works great now. I'm able to generate
my custom targets and commands without any errors. When I execute the
target from nmake now, I get a failure:

Scanning dependencies of target copy_dlls_Debug
NMAKE : fatal error U1073: don't know how to make 'output\bin\.\libdb47d.dll'
Stop.
NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"c:\Program Files (x86)\Microsoft Visual
Studio 9.0\VC\BIN\nmake.exe"' : return code '0x2'
Stop.

Any idea what it is trying to say? Isn't it just supposed to run the
copy command to generate the output file? Here is my updated code
(below). For each of my third party libraries, I create 1 custom
command to copy its DLLs (each third party library can have more than
one dll to copy). I then combine all of these custom commands into 1
custom target for each configuration (copy_dlls_debug,
copy_dlls_release, etc).


set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy" )

if( CMAKE_CONFIGURATION_TYPES )
    set( configurations ${CMAKE_CONFIGURATION_TYPES} )
else()
    set( configurations ${CMAKE_BUILD_TYPE} )
endif()

status( 0 "Generating copy_dlls for configurations: ${configurations}" )

foreach( config ${configurations} )
    string( TOUPPER ${config} upper_config )

    get_property( binarydirs GLOBAL PROPERTY THIRD_PARTY_DIRS_${upper_config} )
    foreach( dir ${binarydirs} )
        file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )

        if( WIN32 )
            file( GLOB pdbs "${dir}/*.pdb" )
            list( APPEND binaries ${pdbs} )
        endif()

        unset( tp_source_bins )
        unset( tp_output_bins )
        unset( copy_commands )

        foreach( bin ${binaries} )
            get_filename_component( bin_file ${bin} NAME )
            set( output_bin
"${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}" )

            list( APPEND tp_source_bins ${bin} )
            list( APPEND tp_output_bins ${output_bin} )

            list( APPEND copy_commands
                COMMAND ${copycmd} "${bin}" "${output_bin}"
            )
        endforeach()

        file( RELATIVE_PATH relative_tp_dir ${THIRD_PARTY_DIR} ${dir} )
        string( FIND ${relative_tp_dir} "/" slash_pos )
        string( SUBSTRING ${relative_tp_dir} 0 ${slash_pos} tp_name )

        add_custom_command(
            COMMENT "Processing changed binaries in ${tp_name}"
            OUTPUT ${tp_output_bins}
            DEPENDS ${tp_source_bins}
            ${copy_commands}
        )

        list( APPEND output_bins ${tp_output_bins} )
    endforeach()

    set( target_name copy_dlls_${config} )
    add_custom_target( ${target_name}
        COMMENT "Copying ${config} binaries..."
        DEPENDS "${output_bins}"
    )

    list( APPEND copy_dlls_targets ${target_name} )
endforeach()

On Tue, Nov 6, 2012 at 1:07 PM, Clinton Stimpson <clinton at elemtech.com> wrote:
> On Tuesday, November 06, 2012 12:46:01 PM Robert Dailey wrote:
>> Is there an assumption here that CMAKE_CONFIGURATION_TYPES will not be
>> set for single-configuration generators?
>
> Yes, that's the assumption.  That's how I normally check for single vs. multi
> config generators.
>
>>
>> I have some code in my root CMake script that forces only 'debug' and
>> 'release' configurations to exist (I didn't want the MinRelSize and
>> friends in there). I think this will force the logic that depends on
>> CMAKE_CONFIGURATION_TYPES to fail. What would you do in this case?
>>
>> Here is the code that sets CMAKE_CONFIGURATION_TYPES:
>>
>> cmake_minimum_required( VERSION 2.8.8 FATAL_ERROR )
>>
>> set( CMAKE_BUILD_TYPE debug )
>> set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
>>
>> project( gmms )
>>
>> ... etc ...
>
>
> It doesn't make much sense to me why one could only do a release build with a
> mulit-config generator...  I normally leave those variables alone and let the
> user set them.  I find RelWithDebInfo quite useful sometimes.
>
> But if you had to, perhaps you could do this, after your project(gmms):
>
> if(CMAKE_CONFIGURATION_TYPES)
>   set( CMAKE_CONFIGURATION_TYPES debug release CACHE INTERNAL "" FORCE )
> endif()
>
> Or some code to strip out types from CMAKE_CONFIGURATION_TYPES except for
> debug and release.  So if it was empty to begin with, it could stay empty.
>
>>
>> On Tue, Nov 6, 2012 at 12:41 PM, Clinton Stimpson <clinton at elemtech.com>
> wrote:
>> > You could base your logic on CMAKE_CONFIGURATION_TYPES and CMAKE_BUILD
>> > TYPE.
>> >
>> > For example:
>> >
>> > if(CMAKE_CONFIGURATION_TYPES)
>> >
>> >   foreach(type ${CMAKE_CONFIGURATION_TYPES})
>> >
>> >     make_copy_rules(${type})
>> >
>> >   endforeach()
>> >
>> > else()
>> >
>> >   make_copy_rules(${CMAKE_BUILD_TYPE})
>> >
>> > endif()
>> >
>> > On Tuesday, November 06, 2012 12:34:22 PM Robert Dailey wrote:
>> >> I figured out the problem.
>> >>
>> >> On single-configuration generators (like NMake, which I'm testing now)
>> >> both release and debug configurations have the same binary output
>> >> directory. Since I'm creating 1 custom target (and corresponding
>> >> commands) per configuration, the 2nd configuration has the same
>> >> outputs as the 1st, so the 2nd target fails because I"m adding
>> >> different rules to the same output file. This isn't a problem with
>> >> Visual Studio generators because my outputs go to Debug and Release.
>> >>
>> >> Know any good workarounds for this?
>> >>
>> >> On Tue, Nov 6, 2012 at 11:54 AM, Clinton Stimpson <clinton at elemtech.com>
>> >
>> > wrote:
>> >> > Try this:
>> >> >     add_custom_command(
>> >> >
>> >> >         OUTPUT ${bin_outputs}
>> >> >         COMMAND ${copy_commands}
>> >> >         DEPENDS ....
>> >> >
>> >> >     )
>> >> >
>> >> > On Tuesday, November 06, 2012 11:50:54 AM Robert Dailey wrote:
>> >> >> Thanks Clinton,
>> >> >>
>> >> >> Here is what I have so far:
>> >> >>
>> >> >>
>> >> >> project( copy_dlls )
>> >> >>
>> >> >> set( copycmd "${CMAKE_COMMAND}" ARGS "-E" "copy_if_different" )
>> >> >>
>> >> >> set( configurations debug release )
>> >> >> foreach( config ${configurations} )
>> >> >>
>> >> >>     string( TOUPPER ${config} upper_config )
>> >> >>
>> >> >>     get_property( binarydirs GLOBAL PROPERTY
>> >> >>
>> >> >> THIRD_PARTY_DIRS_${upper_config} ) foreach( dir ${binarydirs} )
>> >> >>
>> >> >>         file( GLOB binaries "${dir}/*${CMAKE_SHARED_LIBRARY_SUFFIX}" )
>> >> >>
>> >> >>         if( WIN32 AND config STREQUAL "debug" )
>> >> >>
>> >> >>             file( GLOB pdbs "${dir}/*.pdb" )
>> >> >>             list( APPEND binaries ${pdbs} )
>> >> >>
>> >> >>         endif()
>> >> >>
>> >> >>         foreach( bin ${binaries} )
>> >> >>
>> >> >>             get_filename_component( bin_file ${bin} NAME )
>> >> >>
>> >> >>             list( APPEND bin_outputs ${bin} )
>> >> >>             list( APPEND copy_commands
>> >> >>
>> >> >>                 COMMAND ${copycmd} "${bin}"
>> >> >>
>> >> >> "${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${CMAKE_CFG_INTDIR}/${bin_file}"
>> >> >>
>> >> >>             )
>> >> >>
>> >> >>         endforeach()
>> >> >>
>> >> >>     endforeach()
>> >> >>
>> >> >>     add_custom_command(
>> >> >>
>> >> >>         OUTPUT ${bin_outputs}
>> >> >>         ${copy_commands}
>> >> >>
>> >> >>     )
>> >> >>
>> >> >>     add_custom_target( copy_dlls_${config}
>> >> >>
>> >> >>         COMMENT "Copying ${config} binaries..."
>> >> >>         DEPENDS ${bin_outputs}
>> >> >>
>> >> >>     )
>> >> >>
>> >> >> endforeach()
>> >> >>
>> >> >>
>> >> >> This doesn't seem to work though, CMake tells me:
>> >> >>
>> >> >> CMake Error: Attempt to add a custom rule to output
>> >> >> "C:/Work/rdailey-hp/dpd-cmake/build-nmake-vc9/third_party/bdb/4.7.25/v
>> >> >> c9s
>> >> >> p1/ debug/bin/libdb47d.dll.rule" which already has a custom rule.
>> >> >>
>> >> >> I'm trying to stuff all the copy commands and output files into 1
>> >> >> custom command, and wrap that with a custom target, so I can make
>> >> >> other targets depend on the custom target.
>> >> >>
>> >> >> On Tue, Nov 6, 2012 at 11:44 AM, Clinton Stimpson
>> >> >> <clinton at elemtech.com>
>> >> >
>> >> > wrote:
>> >> >> > add_custom_command(
>> >> >> >
>> >> >> >   OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>> >> >> >   COMMAND cmake -E copy ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>> >> >> >
>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll
>> >> >> >
>> >> >> >   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/my.dll
>> >> >> >   )
>> >> >> >
>> >> >> > add_custom_target(copy_dll ALL DEPENDS
>> >> >> > ${CMAKE_CURRENT_BINARY_DIR}/my.dll)
>> >> >> >
>> >> >> > Whether you use 1 custom command per dll, or 1 for all dlls, is up
>> >> >> > to
>> >> >> > you.
>> >> >> >
>> >> >> >  If they usually change together, then you could have fewer custom
>> >> >> >
>> >> >> > commands.  In that case, the OUTPUT would have multiple files and
>> >> >> > both
>> >> >> > DEPENDS would have multiple files.
>> >> >> >
>> >> >> > Another alternative is to use add_custom_command(TARGET POST_BUILD
>> >> >> > ...)
>> >> >> > if
>> >> >> > these are dlls that you compile yourself and need to copy after the
>> >> >> > build
>> >> >> > step.
>> >> >> >
>> >> >> > On Tuesday, November 06, 2012 11:23:03 AM Robert Dailey wrote:
>> >> >> >> Also, is it safe to add 1 custom command for each DLL copy? Or
>> >> >> >> would
>> >> >> >> you somehow only use 1 custom command and copy them all with it?
>> >> >> >> Also
>> >> >> >> would I add the custom commands to the custom target?
>> >> >> >>
>> >> >> >> On Tue, Nov 6, 2012 at 11:11 AM, Clinton Stimpson
>> >> >> >> <clinton at elemtech.com>
>> >> >> >
>> >> >> > wrote:
>> >> >> >> > On Tuesday, November 06, 2012 11:06:45 AM Robert Dailey wrote:
>> >> >> >> >> I use ${CMAKE_COMMAND} -E copy_if_different to copy DLL files to
>> >> >> >> >> my
>> >> >> >> >> binary output directory. The custom target runs this command
>> >> >> >> >> about
>> >> >> >> >> 50-100 times (for that many files).
>> >> >> >> >>
>> >> >> >> >> I notice that when all files are already copied, the commands
>> >> >> >> >> still
>> >> >> >> >> run extremely slowly. It takes just as long to copy all the
>> >> >> >> >> files
>> >> >> >> >> as
>> >> >> >> >> it does to copy none of them.
>> >> >> >> >>
>> >> >> >> >> I used the Sysinternals tool called Process Monitor to see what
>> >> >> >> >> CMake
>> >> >> >> >> is doing as it runs copy_if_different. It's opening the full
>> >> >> >> >> file
>> >> >> >> >> and
>> >> >> >> >> seems to be comparing actual contents instead of something
>> >> >> >> >> simple,
>> >> >> >> >> such as timestamp.
>> >> >> >> >>
>> >> >> >> >> I do not need such a thorough check, I simply want the check to
>> >> >> >> >> see
>> >> >> >> >> which timestamp is higher and copy if the source is newer than
>> >> >> >> >> the
>> >> >> >> >> target.
>> >> >> >> >>
>> >> >> >> >> Any reason why copy_if_different is so slow? Is my assumption
>> >> >> >> >> correct?
>> >> >> >> >> How can I make it faster?
>> >> >> >> >
>> >> >> >> > How about using plain "cmake -E copy ..." and rely on the
>> >> >> >> > timestamp
>> >> >> >> > check
>> >> >> >> > done by a custom command (add_custom_command()).
>> >> >> >> > You need to make sure the input/ouput parts of the custom command
>> >> >> >> > are
>> >> >> >> > set
>> >> >> >> > correctly so it can do a timestamp check.
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > Clinton Stimpson
>> >> >> >> > Elemental Technologies, Inc
>> >> >> >> > Computational Simulation Software, LLC
>> >> >> >> > www.csimsoft.com
>> >> >> >> > --
>> >> >> >> >
>> >> >> >> > Powered by www.kitware.com
>> >> >> >> >
>> >> >> >> > Visit other Kitware open-source projects at
>> >> >> >> > http://www.kitware.com/opensource/opensource.html
>> >> >> >> >
>> >> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>> >> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>> >> >> >> >
>> >> >> >> > Follow this link to subscribe/unsubscribe:
>> >> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>> >> >> >
>> >> >> > --
>> >> >> > Clinton Stimpson
>> >> >> > Elemental Technologies, Inc
>> >> >> > Computational Simulation Software, LLC
>> >> >> > www.csimsoft.com
>> >> >> > --
>> >> >> >
>> >> >> > Powered by www.kitware.com
>> >> >> >
>> >> >> > Visit other Kitware open-source projects at
>> >> >> > http://www.kitware.com/opensource/opensource.html
>> >> >> >
>> >> >> > Please keep messages on-topic and check the CMake FAQ at:
>> >> >> > http://www.cmake.org/Wiki/CMake_FAQ
>> >> >> >
>> >> >> > Follow this link to subscribe/unsubscribe:
>> >> >> > http://www.cmake.org/mailman/listinfo/cmake
>> >> >
>> >> > --
>> >> > Clinton Stimpson
>> >> > Elemental Technologies, Inc
>> >> > Computational Simulation Software, LLC
>> >> > www.csimsoft.com
>> >> > --
>> >> >
>> >> > Powered by www.kitware.com
>> >> >
>> >> > Visit other Kitware open-source projects at
>> >> > http://www.kitware.com/opensource/opensource.html
>> >> >
>> >> > Please keep messages on-topic and check the CMake FAQ at:
>> >> > http://www.cmake.org/Wiki/CMake_FAQ
>> >> >
>> >> > Follow this link to subscribe/unsubscribe:
>> >> > http://www.cmake.org/mailman/listinfo/cmake
>> >
>> > --
>> > Clinton Stimpson
>> > Elemental Technologies, Inc
>> > Computational Simulation Software, LLC
>> > www.csimsoft.com
>> > --
>> >
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Please keep messages on-topic and check the CMake FAQ at:
>> > http://www.cmake.org/Wiki/CMake_FAQ
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.cmake.org/mailman/listinfo/cmake
> --
> Clinton Stimpson
> Elemental Technologies, Inc
> Computational Simulation Software, LLC
> www.csimsoft.com
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

More information about the CMake mailing list