[CMake] Incremental builds with ExternalProject_Add

[Attila Krasznahorkay]

Dear All,

I ran into an issue with incremental builds that use ExternalProject_Add(...), which I'm very surprised isn't a more widely advertised issue. So I thought I'd ask here if people are aware of this behaviour...

For our software projects we build a good number of externals using separate "externals project".

https://gitlab.cern.ch/atlas/atlasexternals

Most of these externals we build by downloading tar files of their sources from various locations, and building those. Like for instance:

https://gitlab.cern.ch/atlas/atlasexternals/blob/master/External/ROOT/CMakeLists.txt

(Just to demonstrate how not trivial some of these builds already are...)

With one of the latest updates of this repository I started seeing some very unnerving compilation errors in our CI system, which lead me to realise that we've been using ExternalProject_Add(...) in a way so far that's fundamentally not CI compatible.

Let's take an example where I have two versions of an external. Let's download these versions in files external-1.0.0.tar.gz, and external-1.0.1.tar.gz. And let's assume that version 1.0.0 was created in January, while 1.0.1 in February of some year.

Now, if we just create a very simple ExternalProject_Add(...) configuration, that will just not be incremental build compatible in this setup. Since let's just walk through the build steps.
  - Let's say that it's March by now.
  - We first build version 1.0.0 of the external. When ExternalProject_Add(...) unpacks external-1.0.0.tar.gz, the source directory of this project will have modification dates from January. The built files on the other hand will have modification dates from March. (The current date.)
  - Now we update our build to use version 1.0.1 of the external. When ExternalProject_Add(...) unpacks external-1.0.1.tar.gz, the source files get updated to have modification dates from February. But remember, the build results have a modification date from March. So any reasonable build system that external-1.0.1.tar.gz may use, will not do anything. It will assume that the files in the build directory are already up to date.

Of course the issue is even worse when we try to downgrade the version of the external in an update of our own repository.

So my question is: Are people not running into this issue all the time? For now I'm thinking of solving this problem like:

https://gitlab.cern.ch/akraszna/IncrementalBuildTest/blob/master/CMakeLists.txt

Which is a quite drastic approach, but I don't know of a better one. And I wonder if ExternalProject_Add(...) should just behave like this out of the box by default. That whenever it downloads/unpacks a different version of an external than it built before, it should always purge the previous build completely.

What does everybody think about this? Am I missing something? Have I been using ExternalProject_Add(...) wrong all along?

Cheers,
           Attila