<div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div dir="ltr">Hi all,<div><br></div><div>I faced exactly the same issue with Box Backup, and "solved" it by caching completely built packages in tarballs, replacing the ExternalProject with a different one that uses the cached package if it hasn't been invalidated (which we determine based on the CMakeLists.txt having changed).</div><div><br></div><div>You can see the superbuild file here:</div><div><a href="https://github.com/boxbackup/boxbackup/blob/s3_support/infrastructure/cmake/windows/CMakeLists.txt" target="_blank">https://github.com/boxbackup/<wbr>boxbackup/blob/s3_support/<wbr>infrastructure/cmake/windows/<wbr>CMakeLists.txt<br></a></div><div><br></div><div>We define a function that extracts a cached package:</div><div><pre style="color:rgb(0,0,0);text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">function(ExternalProject_Use_<wbr>Cache project_name package_file install_path)
message(STATUS "Will use cached package file: ${package_file}")
ExternalProject_Add(${project_<wbr>name}
DOWNLOAD_COMMAND ${CMAKE_COMMAND} -E echo
"No download step needed (using cached package)"
CONFIGURE_COMMAND ${CMAKE_COMMAND} -E echo
"No configure step needed (using cached package)"
BUILD_COMMAND ${CMAKE_COMMAND} -E echo
"No build step needed (using cached package)"
INSTALL_COMMAND ${CMAKE_COMMAND} -E echo
"No install step needed (using cached package)"
)
# We want our tar files to contain the Install/<package> prefix (not for any
# very special reason, only for consistency and so that we can identify them
# in the extraction logs) which means that we must extract them in the
# binary (top-level build) directory to have them installed in the right
# place for subsequent ExternalProjects to pick them up. It seems that the
# only way to control the working directory is with Add_Step!
ExternalProject_Add_Step(${<wbr>project_name} extract
ALWAYS 1
COMMAND
${CMAKE_COMMAND} -E echo
"Extracting ${package_file} to ${install_path}"
COMMAND
${CMAKE_COMMAND} -E tar xzvf ${package_file}
${install_path}
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
)
ExternalProject_Add_<wbr>StepTargets(${project_name} extract)
endfunction()<br></pre>And a function to create a new cached package:<pre style="color:rgb(0,0,0);text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">function(ExternalProject_<wbr>Create_Cache project_name package_file install_path)
if(NOT EXISTS ${package_file})
# Note: because this is evaluated when CMake is run, not when make is run, you will
# need to rerun CMake to stop it repackaging your already-packaged sources. This
# works well enough for the main use case, which is speeding up AppVeyor builds,
# since the cache from a previous run is always restored before CMake is run, so if
# we already have a package then this code will never be run.
message(STATUS "Will create cached package file: ${package_file}")
ExternalProject_Add_Step(${<wbr>project_name} package
DEPENDEES install
BYPRODUCTS ${package_file}
COMMAND ${CMAKE_COMMAND} -E echo "Updating cached package file: ${package_file}"
COMMAND ${CMAKE_COMMAND} -E tar czvf ${package_file}
${install_path}
)
ExternalProject_Add_<wbr>StepTargets(${project_name} package)
endif()
endfunction()</pre>And call them like this:</div><div><pre style="color:rgb(0,0,0);text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"><pre style="text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word"><pre style="text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">file(MD5 ${CMAKE_CURRENT_LIST_FILE} cmake_lists_hash)</pre>set(zlib_install_dir "${install_dir}/zlib")
set(zlib_package_file "${cache_dir}/zlib_${cmake_<wbr>lists_hash}.tgz")</pre>if(EXISTS ${zlib_package_file})
ExternalProject_Use_Cache(zlib ${zlib_package_file} ${zlib_install_dir})
else()
ExternalProject_Add(...)
ExternalProject_Create_Cache(<wbr>zlib ${zlib_package_file} ${zlib_install_dir})
endif()</pre><div>These package files are cached and restored by our <a href="https://github.com/boxbackup/boxbackup/blob/s3_support/appveyor.yml#L30" target="_blank">AppVeyor configuration</a>, and also the source tarballs used to rebuild them, but not the CMake build directory.</div><div><br></div><div>This cuts about 10 minutes off our normal build time for each configuration on AppVeyor (80 minutes total per commit tested, with 8 different configurations). But it does make local superbuilds a bit more tricky, as described in the comments: we often have to rerun CMake to use previously built cached packages instead of rebuilding them.</div><div><br></div>I hope this is interesting/helpful to others too.</div><div><br></div><div>Thanks, Chris.</div></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 6 September 2018 at 05:30, Isaiah Norton <span dir="ltr"><<a href="mailto:isaiah.norton@gmail.com" target="_blank">isaiah.norton@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">CMake notices the downloaded tarball is up-to-date and<br>doesn't download it again, but it still extracts it again </blockquote><div><br></div></span><div>From what I can tell, the 'check|download tarball' and 'extract tarball' commands are independent parts of the "download step": as long as the download step runs at all, it will re-extract the tarball, even if it skipped re-downloading. So the issue (design questions aside) is what triggers the download step. The minimal dependency for that step looks like "LIBNAME-gitinfo.txt" file somewhere in the stamp directory, so you could check the mtime preservation there. I don't know if there are other dependencies added for the download target added separately.</div></div></div><div class="m_-5066595681939314633HOEnZb"><div class="m_-5066595681939314633h5"><br><div class="gmail_quote"><div dir="ltr">On Wed, Sep 5, 2018 at 8:06 AM Antoine Pitrou <<a href="mailto:antoine@python.org" target="_blank">antoine@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hello,<br>
<br>
On our project (Apache Arrow - <a href="https://arrow.apache.org/" rel="noreferrer" target="_blank">https://arrow.apache.org/</a>) we're using<br>
CMake for the C++ source tree and have many external dependencies<br>
fetched using ExternalProject. In turn building those dependencies can<br>
make up a significant portion of build times on CI services, especially<br>
AppVeyor. So I've been looking for a solution to cache those<br>
third-party builds from one CI run to the other.<br>
<br>
Right now, what I'm trying to do is to set EP_BASE to a well-known base<br>
directory and ask AppVeyor to cache and restore that directory in each<br>
new build. The AppVeyor caching seems to work fine (the EP_BASE<br>
directory is saved and restored). However, it seems that nevertheless<br>
CMake will rebuild all those projects again, despite the cached build<br>
results.<br>
<br>
This is with CMake 3.12.1 on Windows.<br>
<br>
Here is the log for an example build step, here the zstd library:<br>
<a href="https://ci.appveyor.com/project/pitrou/arrow/build/1.0.700/job/i4tj6tifp4xq1mjn?fullLog=true#L803" rel="noreferrer" target="_blank">https://ci.appveyor.com/projec<wbr>t/pitrou/arrow/build/1.0.700/<wbr>job/i4tj6tifp4xq1mjn?fullLog=<wbr>true#L803</a><br>
<br>
As you can see, CMake notices the downloaded tarball is up-to-date and<br>
doesn't download it again, but it still extracts it again (why?) and<br>
builds the source code anew. Yet the entire EP_BASE directory (here<br>
"C:/Users/appveyor/arrow-exter<wbr>nals") is cached and restored by AppVeyor.<br>
<br>
Did someone manage to make this work, and/or is there another solution?<br>
<br>
Thank you<br>
<br>
Regards<br>
<br>
Antoine.<br>
-- <br>
<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" rel="noreferrer" target="_blank">http://www.cmake.org/Wiki/CMak<wbr>e_FAQ</a><br>
<br>
Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br>
<br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/su<wbr>pport.html</a><br>
CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/co<wbr>nsulting.html</a><br>
CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/tr<wbr>aining.html</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensou<wbr>rce/opensource.html</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://cmake.org/mailman/listinfo/cmake" rel="noreferrer" target="_blank">https://cmake.org/mailman/list<wbr>info/cmake</a><br>
</blockquote></div>
</div></div><br>-- <br>
<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" rel="noreferrer" target="_blank">http://www.cmake.org/Wiki/CMak<wbr>e_FAQ</a><br>
<br>
Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br>
<br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/su<wbr>pport.html</a><br>
CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/co<wbr>nsulting.html</a><br>
CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/tr<wbr>aining.html</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/opensou<wbr>rce/opensource.html</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://cmake.org/mailman/listinfo/cmake" rel="noreferrer" target="_blank">https://cmake.org/mailman/list<wbr>info/cmake</a><br>
<br></blockquote></div><br></div>
</div></div></div><br></div>