[cmake-developers] dependency scanning speed

[Alexander Neundorf]

Hi,

On Sunday 10 September 2006 17:41, Alexander Neundorf wrote:
> On Sunday 10 September 2006 17:07, Bill Hoffman wrote:
> > At 05:58 AM 9/10/2006, Alexander Neundorf wrote:
> > >I tried callgrind, but it doesn't work.
> > >I have kubuntu, valgrind 3.0.1 and callgrind 0.10.1.
> > >It starts and finishes after approx 3 seconds (instead of 30). 10
> > > percent of the time are spent in cmake, 50 in ld and 30 percent in
> > > libc. So it seems cmake quits immediately after starting.
> > >Any idea what's wrong ?
> >
> > Often times when the problem is due to disk access calgrind is not much
> > help. The overhead of calgrind makes it so the disk can keep up with the
> > process. The only way to debug, is it add timers in the code directly. 
> > So, it sounds like the problem is accessing all of those files.
>
> I got callgrind working, it was a version problem between callgrind and
> valgrind.
> Since using a map<string, bool> cache to get rid of 2,5 million access()
> calls didn't help but made things worse, it seems it's not the file IO
> which is slow.
> Now kcachegrind shows that 99 % are spent in WriteDependencies(), from
> which 53 % are spent in std::string operator+= and the std::string dtor. I
> was able to get the time from 28 seconds down to 24 seconds, i.e. approx.
> 15% by replacing the std::string temp which is used when iterating through
> the include dirs with a simple char buffer.
> While 15 % doesn't sound bad, it's no perceivable difference.
> I'll try to cache the location of the actual file so that not everytime all
> include dirs have to be iterated.

with the attached patch the time for dependency scanning in unity goes down 
from 28 seconds to 15 seconds.
It does three things:

1) renames fileCache to FileCache, so that it fits better with the naming 
style of the rest

2) it introduces a cache map<std::string, std::string> HeaderLocationCache, 
which assigns a found header location to a key created from the complete 
include path and the header. So if the same header is searched with another 
include path, it gets a different key. This saves 2 millions iterations for 
unity, makes 10 seconds.

3) it uses std::string::resize(4*1024) to preallocate space for the strings,  
and then assigns always c_str() instead of the std::string itself, since 
assigning the std::string apparently throws away the preallocated memory so 
that the memory has to be allocated again if the string grows (which it does 
often when creating the key for the header location cache). This saves 
another 4 seconds.

So, how about applying ?

Bye
Alex
-- 
Work: alexander.neundorf AT jenoptik.com - http://www.jenoptik-los.de
Home: neundorf AT kde.org                - http://www.kde.org
      alex AT neundorf.net               - http://www.neundorf.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cmDepends.patch
Type: text/x-diff
Size: 4660 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20060910/8b1567a1/attachment.patch>