[cmake-developers] Faster automoc with parallel processing?

[Alexander Neundorf]

On Tuesday 19 June 2012, Thomas Sondergaard wrote:
> Hi Alexander,
> 
> Thanks for replying.
> 
> On 2012-06-18 21:58, Alexander Neundorf wrote:
> > On Sunday 17 June 2012, Thomas Sondergaard wrote:
> >  > On Windows with MSVC 2010 Generator, moc seems to be very slow and it
> >  > is a paint to see it run sequentially. Could automoc be taught to do
> >  > this in parallel using all the cores on the machine?
> > 
> > is "certainly" good enough as an answer ? ;-)
> 
> It's a good start :-)
> 
> > The thing is, for each target there is an accompanying automoc target.
> > So if you have 4 cores, and 4 targets are built in parallel, then also 4
> > (auto)mocs should be running in parallel.
> > 
> > If you have only 1 target, consisting of many files, only 1 automoc
> > target will be executed, processing one moc after the other.
> 
> Interesting. I haven't actually fully switched to automoc, I have only
> tested it on one target in a project with +50 targets. Perhaps the
> situation will look better if I automoc everything.
> 
> > Parallelizing this is not obvious. Simply running as many mocs on a
> > machine as it has cores might lead to 16 mocs running on 4 cores (for 4
> > different targets).
> 
> I think overparallization is better than underparallization. I don't
> think running 16 parallel mocs at a time on a 4 core machine should be
> dramatically slower than running 4. An experiment could show.

Considering a 8 or 16 core machine, this could give already 256 mocs running 
on 16 cores. I think this would then really be a bit much.
I'd actually have to check first whether most of the time is spent in 
executing moc, or in parsing the files for stuff to be moc'ed.
Also the parsing can still be optimized by caching results from earlier runs 
(which it doesn't do right now).
For executing moc, this could also be limited to some number, like 2 or 4, 
which should already bring a lot.

Anyway, currently executing moc is very simple, it's a system-style call, i.e. 
it starts the executable and blocks until the process has exited.
Making this non-blocking is a bit of work.

> > So, it is possible, it is some implementation work, and I'm not clear on
> > how to do the logic.
> > 
> > Do you have some ideas how to do this in a good way ?
> 
> automoc could start a background process and use that to handle all
> moc'ing. The process could stop after a few seconds of inactivity. That
> way moc would never use more than the number of cores in the machine,
> but it might overlap with other build activity. It doesn't seem sensible
> that automoc should only be concerned about overutilization from other
> mocs. Perhaps the best way is simply to look at the load-average, like
> make --load-average?

This must be cross-platform, i.e. work on Windows, OSX, Linux, *BSD, AIX, etc.
This would be quite an effort I think.

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20120619/fea4f87c/attachment.html>