[CMake] FW: Parallel GNU make issue

[Chuck Atkins]

Hi Gary,

Do you see either of these two warning messages show up:

"warning: -jN forced in submake: disabling jobserver mode."
or
"warning: jobserver unavailable: using -j1. Add `+' to parent make rule."

These warnings often accompany the forced serialization of a parallel make
build, although usually they show up regardless of the launch method, i.e.
ctests vs manual make.  These warnings are often indicative of a particular
problem in super-build scenarios..

- Chuck

On Thu, Sep 11, 2014 at 2:29 PM, Hennigan, Gary L <glhenni at sandia.gov>
wrote:

>  I have a strange, and very frustrating, problem. I have a pretty large
> piece of software that I build nightly as part of regression testing of my
> own software. All of the software uses CMake and I use a ctest script, via
> “ctest –S [script file]”, for my nightly regression testing . As I stated,
> this is a pretty large collection of software but during development it’s
> not a huge issue because the build is quite parallelizable via GNU make’s
> “-j N” option. On my nightly test platform, a 64-core machine, I can build
> the whole thing in about an hour.  A nice manageable amount of time for a
> nightly regression test. Unfortunately when I run the build process via
> ctest something is causing the parallel make to fail and I’m lucky if the
> build takes under 15 hours. Barely practical for a nightly test.
>
> I’m not sure how to find out what’s going on. After the ctest build I can
> go into the build directory, do a “make clean” and then a “make –j 12”, for
> example, and the build flies. Of course I can build the software entirely
> outside of ctest and it too flies. Only when the build happens as part of
> ctest does it seem to revert to, essentially, a “make –j 1” and slow to a
> crawl.
>
> I can look at the process tree, via “ps –ef”, during the ctest build and I
> see the root invocation of gmake and it’s fine. For example, it typically
> looks something like:
>
>   PID  PPID  C STIME TTY          TIME CMD
>  6141  6283 96 11:41 pts/0    00:24:40 ctest -VV -S ctest_nightly.cmake
> -DPROCESSORCOUNT=12
>  8032  6141  0 11:42 pts/0    00:00:00 /usr/bin/gmake -i *-j 12*
>  8035  8032  0 11:42 pts/0    00:00:00 /usr/bin/gmake -f
> CMakeFiles/Makefile2 all
>   851  8035  0 11:52 pts/0    00:00:00 /usr/bin/gmake -f
> packages/ml/src/CMakeFiles/ml.dir/build.make
> 27797  8035  0 11:46 pts/0    00:00:00 /usr/bin/gmake -f
> packages/moocho/src/CMakeFiles/moocho.dir/build.make
>
> You can see that the parent make, PID 8032 which is started via ctest (PID
> 6141), has the appropriate flag, “-j 12”, but at this point in the build
> it’s compiling one file at a time. Another odd thing is that I think the
> build starts out fine, invoking multiple file compilations simultaneously,
> but after a couple of minutes it reverts to essentially the “make –j 1”
> behavior. It’s like the GNU make jobserver is failing, but I’m not getting
> any error messages from GNU make to that affect.
>
> If anyone has any suggestions on how I can figure this out I’d appreciate
> it.
>
> Apologies for the lengthy explanation. I’ve been pulling my hair out
> trying to figure out how to solve this issue.
>
> Thanks in advance,
> Gary
>
>
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20140911/7934fa6f/attachment.html>