[cmake-developers] Faster automoc with parallel processing?

[Thomas Sondergaard]

On 2012-06-19 22:13, Alexander Neundorf wrote:
> Considering a 8 or 16 core machine, this could give already 256 mocs
> running on 16 cores. I think this would then really be a bit much.
>
> I'd actually have to check first whether most of the time is spent in
> executing moc, or in parsing the files for stuff to be moc'ed.
>
> Also the parsing can still be optimized by caching results from earlier
> runs (which it doesn't do right now).
>
> For executing moc, this could also be limited to some number, like 2 or
> 4, which should already bring a lot.
>
> Anyway, currently executing moc is very simple, it's a system-style
> call, i.e. it starts the executable and blocks until the process has exited.
>
> Making this non-blocking is a bit of work.

Right, as it turns out after I have automoc'ed all our projects I am 
getting better parallelity. As you can see on this image I am getting 
good value for my four cores and an incredibuild of our complete package 
is down from about 12 minutes to 8 minutes and 30 seconds.

http://spear.medical-insight.com/~ts/automoc_build_time.png

So a big thanks to you and the CMake crew for giving us automoc support.


>
>  > automoc could start a background process and use that to handle all
>  > moc'ing. The process could stop after a few seconds of inactivity. That
>  > way moc would never use more than the number of cores in the machine,
>  > but it might overlap with other build activity. It doesn't seem sensible
>  > that automoc should only be concerned about overutilization from other
>  > mocs. Perhaps the best way is simply to look at the load-average, like
>  > make --load-average?
>
> This must be cross-platform, i.e. work on Windows, OSX, Linux, *BSD,
> AIX, etc.
>
> This would be quite an effort I think.

I agree, before spending time on more parallelization it would be 
interesting to take a look at moc itself. It appears to be significantly 
slower on Windows than on Linux.

Thanks again,

Thomas