[CMake] Question on ExternalProject_Add and different dependency directives

[David Cole]

On Fri, Dec 7, 2012 at 2:20 PM, Bill Katz <billkatz at gmail.com> wrote:

> Hi,
>
> I love the ExternalProject_Add module and use it extensively in an
> open-source build system I'm developing for our apps:
>
> http://github.com/janelia-flyem/buildem<https://github.com/janelia-flyem/buildem>
>
> The idea is to make a build environment from source, using cached
> tarballs, and allow different version dependency graphs as commits in a git
> repository of all build modules.  I don't use FindPackage because we want
> to know exactly how dependencies are built and modify parameters for some
> of them.  Also, just about everything above the language compiler level
> gets built into one build prefix directory.  The goal is to run at most two
> "cmake..; make..." commands and have the system completely download and
> build all necessary components from archived source.
>
>
Why not just one command? ;-)
  ( See item 2 of the 12-item Joel Test:
http://www.joelonsoftware.com/articles/fog0000000043.html )


In looking through the CMake message boards, I think I understand that
> external projects aren't real targets but can still be used as
> dependencies.  As long as I use the DEPENDS in ExternalProject_Add(), the
> dependencies seem to work as expected.  However, I've run into an issue
> where I'm using add_dependencies(EP1) where EP1 is an external project and
> it's being rebuilt regardless of whether it was built already.
>

It's not that external projects aren't "real" targets. But they are custom
targets that have a chain of associated custom commands based on your
parameters to ExternalProject_Add.



>
> So I was hoping to get clarification on the different ways of specifying
> dependencies and what happens after those statements.
>
> DEPENDS in ExternalProject_Add lets you specify other external project
> targets.  The dependencies only get downloaded and built once.  Question:
> how does ExternalProject_Add determine when an external project needs to go
> through its stages again?  I read that git repo external projects are
> always pulled unless you explicitly use an empty UPDATE_COMMAND.
>
>
The dependencies are strictly time-stamp based. Each step has a stamp file
that it writes out after the step executes successfully. The step should
not rerun, unless a step that it depends on is re-run, or it is marked as
an ALWAYS re-run step... If it does, there's a bug somewhere.



> If two different CMakeLists.txt include the same cmake file with a fixed
> ExternalProject_Add's PREFIX directory, could there be some thrashing going
> on?
>
>
I suppose so, but I don't really understand the question. Could you give an
example? (Perhaps a link to a specific line of code in your github project?)



> add_dependencies(myexe EP1 EP2...) seems to work efficiently for most of
> my modules (i.e, doesn't try to download, configure, etc, if it's done so
> already), but for some more complicated ones, the external projects and its
> dependencies always get rebuilt.  I'm assuming there's something in my
> modules that forces rebuilding of an external project even if its built.
>  In the suspect CMakeLists.txt for external project EP1 that I own, I have
> a number of commands of the form:
>
> project(EP1)
>
> add_library(EP1 ${SOURCES})
> add_dependencies(EP1 ep3 ep4)
>
> add_custom_command(TARGET EP1
>                                          DEPENDS EP2 EP3
>                                          COMMAND dostuff)
>
> add_custom_command(OUTPUT some_constants.h ${more_autogen_include_files}
>       DEPENDS thrift_EP
>       COMMAND generate_includes)
>
> add_executable(foo foo.cpp)
> target_link_libraries(foo EP1 ${foolibs})
>
> set_source_files_properties(somestuff.cpp
>    PROPERTIES OBJECT_DEPENDS ${generated_file_dir}/some_constants.h)
>
>
> I think there's either a deficiency in my understanding and/or one of the
> above is forcing rebuilds.  Any help would be appreciated in figuring out
> where I'm going wrong.
>
> Thanks,
> Bill
>
>
Be careful with your target names and add_dependencies. Be aware of this
bug: http://public.kitware.com/Bug/view.php?id=9188 -- also, I notice you
have both "ep3" and "EP3" -- those are different because the case is
different, but on Windows, if any files are based on target names, the
case-insensitive file system will have you trouncing files if both names
are actually used. Do you actually have such differences, or is that just a
typo in the email?

I always use the OUTPUT form of add_custom_command, and then hook up the
output to a DEPENDS clause of an add_custom_target. Then I specify
inter-target dependencies with add_dependencies. I have not yet had a
problem with this technique, which is used extensively in the
ExternalProject chain of custom commands.

I have not used the add_custom_command(TARGET signature very much. Tracing
what changes are causing build re-runs is not an easy task.


Hope this helps figure something out for you....

David





> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20121214/74803176/attachment-0001.htm>