[cmake-developers] Compile targets affected by changeset

Florent Castelli florent.castelli at gmail.com
Mon May 22 18:55:47 EDT 2017 

Depending on your setup and the amount of build machines, you may have 
even better results with sccache from Mozilla. It's a new implementation 
similar to ccache with an optional distributed cache in S3 or Redis.

The downside is if you have long running tests in the list, it won't 
help you much (unless you have a way to checksum the test programs and 
skip them maybe?). But still, it's always a good thing to rerun them as 
you may have some unstable tests and more runs means more confidence.

/Florent

On 22/05/2017 23:06, Craig Scott wrote:
> I highly recommend Florent's suggestion. We use ccache on our CI 
> system and for local development. We've stopped worrying about how 
> long builds take now, since only files that are changed or that rely 
> on things that changed contribute any meaningful amount to the build 
> time. It also works for Make, Ninja and Xcode, so it's more flexible 
> than relying on some feature of Make and it is also pretty easy to set 
> up. You can set it up system wide, or you can take the approach 
> discussed in this article 
> <https://crascit.com/2016/04/09/using-ccache-with-cmake/> for systems 
> where you don't have access to set up ccache globally.
>
>
> On Tue, May 23, 2017 at 4:54 AM, Florent Castelli 
> <florent.castelli at gmail.com <mailto:florent.castelli at gmail.com>> wrote:
>
>
>>     On 22 May 2017, at 20:07, Robert Patterson via cmake-developers
>>     <cmake-developers at cmake.org <mailto:cmake-developers at cmake.org>>
>>     wrote:
>>
>>     We understand that CMake and make already can rebuild targets
>>     which depend on changed files, and this behavior works exactly as
>>     expected for us. Our issue is not that make is rebuilding targets
>>     that it shouldn't. We would like this 'compile targets which
>>     changed' behavior to work from a clean state, not just for
>>     subsequent rebuilds. That is, from the first time cmake / make is
>>     invoked, only the targets that depend on the given set of files
>>     will be built, and no more.
>>
>>     When a developer submits a changeset to our continuous
>>     integration tool, a fresh copy of the repository is checked out
>>     by the worker at the commit of the changeset. We can access the
>>     set of changed files from the commit data. Currently, cmake and
>>     make correctly identify that the system is in a clean state and
>>     that everything should be built if the current top-level target
>>     is specified. This is the point where we would like our behavior
>>     to differ. We want the rebuild changed targets behavior as if
>>     everything were already built, but from this clean state.
>>
>>     Once we have determined the targets to build, we could simply
>>     specify these targets as goals to make in the command line, but
>>     we are hampered by the following limitation of make.
>>
>>     We are using GNU make 3.82. Invoking 'make -j target1 target2
>>     target3' on the command line specifies the targets as goals to
>>     make. If several goals are specified, 'make' processes each of
>>     them in turn, in the order you name them.
>>     https://www.gnu.org/software/make/manual/html_node/Goals.html
>>     <https://www.gnu.org/software/make/manual/html_node/Goals.html>
>>
>>     Ninja does not currently work for our project.
>
>     It would be interesting to fix Ninja for your project then. Do you
>     know what isn’t currently working?
>     Also, Ninja has some APIs to expose all the dependency graph,
>     which you could probably query somehow to find out which target
>     you need to rebuild.
>
>     Another possibility, albeit a bit different, could be to use a
>     compilation cache to just rebuild everything but much faster, to
>     the point that it might not be relevant anymore.
>
>     /Florent
>
>>
>>>     On May 19, 2017, at 2:32 AM, Simon Richter
>>>     <Simon.Richter at hogyros.de <mailto:Simon.Richter at hogyros.de>> wrote:
>>>
>>>     Hi,
>>>
>>>     On 18.05.2017 23:48, Robert Patterson via cmake-developers wrote:
>>>
>>>>     My company has a large, predominately C++ codebase, with
>>>>     hundreds of
>>>>     targets, both for product and unit tests.  In an effort to
>>>>     improve the
>>>>     compile and test time for developers, which utilizes a continuous
>>>>     integration infrastructure, it is desirable to compile only the
>>>>     targets
>>>>     that are affected by developer's change sets.
>>>
>>>     Erm, it should already work this way. If Make rebuilds a target
>>>     that it
>>>     shouldn't, the first step would be investigating why it thinks the
>>>     target needs to be rebuilt.
>>>
>>>     Dependency tracking is one of the oldest problems, and cmake
>>>     should use
>>>     an appropriate solution for the actual build system you use. The
>>>     approach used for Make is a bit more conservative than you would
>>>     require
>>>     for GNU Make, but should nevertheless still work.
>>>
>>>     http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
>>>     <http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/>
>>>
>>>     is a good primer on dependency list generation with Make -- CMake
>>>     generates rules that are similar to these.
>>>
>>>>     'make' has a limitation where if 'make target1 target2 target3' is
>>>>     invoked, target1, target2, and target3 are built serially, not in
>>>>     parallel.
>>>
>>>     Which version of make are you using?
>>>
>>>       Simon
>>>
>>>     -- 
>>>
>>>     Powered by www.kitware.com <http://www.kitware.com>
>>>
>>>     Please keep messages on-topic and check the CMake FAQ at:
>>>     http://www.cmake.org/Wiki/CMake_FAQ
>>>     <http://www.cmake.org/Wiki/CMake_FAQ>
>>>
>>>     Kitware offers various services to support the CMake community.
>>>     For more information on each offering, please visit:
>>>
>>>     CMake Support: http://cmake.org/cmake/help/support.html
>>>     <http://cmake.org/cmake/help/support.html>
>>>     CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>     <http://cmake.org/cmake/help/consulting.html>
>>>     CMake Training Courses:
>>>     http://cmake.org/cmake/help/training.html
>>>     <http://cmake.org/cmake/help/training.html>
>>>
>>>     Visit other Kitware open-source projects at
>>>     http://www.kitware.com/opensource/opensource.html
>>>     <http://www.kitware.com/opensource/opensource.html>
>>>
>>>     Follow this link to subscribe/unsubscribe:
>>>     http://public.kitware.com/mailman/listinfo/cmake-developers
>>>     <http://public.kitware.com/mailman/listinfo/cmake-developers>
>>
>>     -- 
>>
>>     Powered by www.kitware.com <http://www.kitware.com>
>>
>>     Please keep messages on-topic and check the CMake FAQ at:
>>     http://www.cmake.org/Wiki/CMake_FAQ
>>     <http://www.cmake.org/Wiki/CMake_FAQ>
>>
>>     Kitware offers various services to support the CMake community.
>>     For more information on each offering, please visit:
>>
>>     CMake Support: http://cmake.org/cmake/help/support.html
>>     <http://cmake.org/cmake/help/support.html>
>>     CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>     <http://cmake.org/cmake/help/consulting.html>
>>     CMake Training Courses: http://cmake.org/cmake/help/training.html
>>     <http://cmake.org/cmake/help/training.html>
>>
>>     Visit other Kitware open-source projects at
>>     http://www.kitware.com/opensource/opensource.html
>>     <http://www.kitware.com/opensource/opensource.html>
>>
>>     Follow this link to subscribe/unsubscribe:
>>     http://public.kitware.com/mailman/listinfo/cmake-developers
>>     <http://public.kitware.com/mailman/listinfo/cmake-developers>
>
>
>     --
>
>     Powered by www.kitware.com <http://www.kitware.com>
>
>     Please keep messages on-topic and check the CMake FAQ at:
>     http://www.cmake.org/Wiki/CMake_FAQ
>     <http://www.cmake.org/Wiki/CMake_FAQ>
>
>     Kitware offers various services to support the CMake community.
>     For more information on each offering, please visit:
>
>     CMake Support: http://cmake.org/cmake/help/support.html
>     <http://cmake.org/cmake/help/support.html>
>     CMake Consulting: http://cmake.org/cmake/help/consulting.html
>     <http://cmake.org/cmake/help/consulting.html>
>     CMake Training Courses: http://cmake.org/cmake/help/training.html
>     <http://cmake.org/cmake/help/training.html>
>
>     Visit other Kitware open-source projects at
>     http://www.kitware.com/opensource/opensource.html
>     <http://www.kitware.com/opensource/opensource.html>
>
>     Follow this link to subscribe/unsubscribe:
>     http://public.kitware.com/mailman/listinfo/cmake-developers
>     <http://public.kitware.com/mailman/listinfo/cmake-developers>
>
>
>
>
> -- 
> Craig Scott
> Melbourne, Australia
> https://crascit.com


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20170523/9b2e2b8c/attachment-0001.html>

More information about the cmake-developers mailing list