[CMake] Parallel build & test problem

[Michael Hertling]

On 05/23/2011 01:39 PM, Marcel Loose wrote:
> On Mon, 2011-05-23 at 07:21 -0400, David Cole wrote:
>> On Mon, May 23, 2011 at 4:13 AM, Marcel Loose <loose at astron.nl> wrote:
>>         Hi all,
>>         
>>         A colleague of mine reported a bug in our CMake-base build
>>         system when
>>         doing a parallel build of multiple targets where one of the
>>         targets is
>>         'test'.
>>         
>>         <quote>
>>                Running 'make -j16 tMutex test' (or any test other than
>>         tMutex)
>>                for example will result in building tMutex in parallel
>>         to
>>                testing it.
>>         
>>                Expected behaviour is first building tMutex, followed
>>         by running
>>                the tests.
>>         </quote>
>>         
>>         Is this indeed a bug? Either in our build system or in CMake?
>>         AFAIK it is not possible to define dependencies between the
>>         'test'
>>         target and the target to build the test program. Correct?
>>         
>>         Best regards,
>>         Marcel Loose.
>>         
>>         
>>         --
>>         Marcel Loose
>>         Senior Software Engineer, Computing Group R&D, Astron, the
>>         Netherlands
>>         
>>         _______________________________________________
>>         Powered by www.kitware.com
>>         
>>         Visit other Kitware open-source projects at
>>         http://www.kitware.com/opensource/opensource.html
>>         
>>         Please keep messages on-topic and check the CMake FAQ at:
>>         http://www.cmake.org/Wiki/CMake_FAQ
>>         
>>         Follow this link to subscribe/unsubscribe:
>>         http://www.cmake.org/mailman/listinfo/cmake
>>
>> To the best of my knowledge you can only do a parallel make of one
>> target at a time.
>>
>>
>> The best way to do what you want is to do two parallel make runs in
>> sequence, like this:
>>
>>
>>   make -j16 tMutex
>>   make -j16 test
>>
>>
>> The test target is defined by CMake, though, and runs all tests.
>>
>>
>> HTH,
>> David
>>
>>
> 
> Hi David,
> 
> I vaguely remembered that limitation of 'make' as well, but I couldn't
> find any relevant pointers on this, and colleagues questioned this
> alleged limitation. Therefore I thought it might be a limitation of
> CMake, or maybe I should say: the combination of CMake and make.
> 
> Anyway, my response was also to do two separate 'make' calls, one for
> each target.
> 
> Regards,
> Marcel Loose.

Look at the following project for an example:

# CMakeLists.txt:
CMAKE_MINIMUM_REQUIRED(VERSION 2.8 FATAL_ERROR)
PROJECT(PARALLEL C)
FILE(WRITE ${CMAKE_BINARY_DIR}/generate.txt "0\n")
ADD_CUSTOM_COMMAND(
    OUTPUT ${CMAKE_BINARY_DIR}/generate.txt
    COMMAND ${CMAKE_COMMAND}
    -DGENERATE=${CMAKE_BINARY_DIR}/generate.txt
    -P ${CMAKE_SOURCE_DIR}/generate.cmake)
FILE(WRITE ${CMAKE_BINARY_DIR}/f.c "void f(void){}\n")
FILE(WRITE ${CMAKE_BINARY_DIR}/g.c "void g(void){}\n")
ADD_LIBRARY(f SHARED f.c ${CMAKE_BINARY_DIR}/generate.txt)
ADD_LIBRARY(g SHARED g.c ${CMAKE_BINARY_DIR}/generate.txt)

# generate.cmake:
IF(EXISTS ${GENERATE})
  FILE(STRINGS ${GENERATE} VAR)
  MATH(EXPR VAR ${VAR}+1)
  FILE(WRITE ${GENERATE} "${VAR}\n")
ELSE()
  FILE(WRITE ${GENERATE} "1\n")
ENDIF()

After configuring, enter the following command:

while true; do
(make clean; make -j1) | grep "generate\.txt";
echo "generate.txt: $(cat generate.txt)";
done

You'll endlessly see "... Generating generate.txt" followed by
"generate.txt: 1" which is expected. Now, switch to parallel:

while true; do
(make clean; make -j2) | grep "generate\.txt";
echo "generate.txt: $(cat generate.txt)";
done

On my system, findings are:

1. Two messages "... Generating generate.txt". This means that both
Make processes run the custom command, i.e. none of these processes
finds the generated.txt file already generated by the other process.
2. The generated.txt file's content varies between 1 and 2: If the
IF(EXISTS) command of one process is executed after the FILE(WRITE)
command of the other process, the result in generate.txt will be 2,
but if the one's IF(EXISTS) command is executed between the other's
IF(EXISTS) and FILE(WRITE) commands, both processes will find the
generate.txt file absent and write it with a content of 1. That's
a typical race condition among the Make processes with j2 or more.
3. If I replace '| grep "generate\.txt"' with '> /dev/null' in the
above-noted command, the following error occurs from time to time:

> CMake Error at .../generate.cmake:3 (MATH):
>   math cannot parse the expression: "+1": syntax error, unexpected exp_PLUS,
>   expecting exp_OPENPARENT or exp_NUMBER (1)

Supposedly, the reason is that the FILE(STRINGS) command of one process
is executed during the other's FILE(WRITE) command, i.e. between open()
and close() when the generate.txt file is open for writing but not yet
closed and, thus, empty, so the VAR variable will be empty, too, and
the MATH() command will fail. That's also a typical race condition.

This example shows that CMake-generated Makefiles might rate targets as
sufficiently independent to be built in parallel although these targets
are coupled by a custom command which - due to its construction - is
sensitive to parallel execution and gives rise to a race condition.
So, the warning w.r.t. building multiple targets in parallel must
be taken absolutely seriously.

However, David, things can even get worse: Enhance the above-noted
PARALLEL project's CMakeLists.txt with the following three lines:

FILE(WRITE ${CMAKE_BINARY_DIR}/main.c "int main(void){return 0;}\n")
ADD_EXECUTABLE(main main.c)
TARGET_LINK_LIBRARIES(main f g)

Most certainly, this is a quite common configuration, and IMO, it's
perfectly legal to build with "make -j2 main", i.e. explicitly only
one target in parallel. When I enter the following lines

while true; do
(make clean; make -j2 main) > /dev/null;
echo "generate.txt: $(cat generate.txt)";
done

I see plenty of "generate.txt: 2" messages, i.e. the custom command is
still executed twice and usually in sequence, but sometimes there's a
"generate.txt: 1" message. Obviously, the race condition hasn't gone.

What does this mean in regard to parallel building a single target?
Are independent targets the only ones that can be reliably built in
parallel, i.e. do I have to build f,g and main individually in the
correct order by hand? That would be strange, IMO. Maybe, someone
can shed light upon this issue.

Regards,

Michael