MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0014898CMakeCMakepublic2014-04-27 08:282016-06-10 14:31
ReporterAndrey Morozov 
Assigned ToKitware Robot 
PrioritynormalSeverityfeatureReproducibilityN/A
StatusclosedResolutionmoved 
PlatformALLOSALLOS Version
Product Version 
Target VersionFixed in Version 
Summary0014898: Improve the performance of ninja build
DescriptionIf all target(in ninja file) will be sorted by size of source(first build big files) then build will be faster.
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2014-04-27 08:28Andrey MorozovNew Issue
2014-04-27 12:30Peter KuemmelNote Added: 0035792
2014-04-27 12:45Andrey MorozovNote Added: 0035793
2014-04-27 14:11Peter KuemmelNote Added: 0035794
2014-04-27 16:39Andrey MorozovNote Added: 0035795
2014-04-28 00:04Ben BoeckelNote Added: 0035797
2014-04-28 02:26Peter KuemmelNote Added: 0035798
2014-04-28 03:25Andrey MorozovNote Added: 0035799
2014-04-28 12:17Ben BoeckelNote Added: 0035800
2014-05-05 16:06Ben BoeckelNote Added: 0035818
2016-06-10 14:29Kitware RobotNote Added: 0042542
2016-06-10 14:29Kitware RobotStatusnew => resolved
2016-06-10 14:29Kitware RobotResolutionopen => moved
2016-06-10 14:29Kitware RobotAssigned To => Kitware Robot
2016-06-10 14:31Kitware RobotStatusresolved => closed

Notes
(0035792)
Peter Kuemmel   
2014-04-27 12:30   
Any explanation for this? Do you see this also when the fs-cache is hot?
(0035793)
Andrey Morozov   
2014-04-27 12:45   
Ok. When we have a lot of little files and sever a big files this problem is actual.

For example: we have 4 file with build time 4sec, 3sec, 10sec, 2sec and two core(for build).So.

not sorted target:
1thread: 4s + 10s = 14s
2thread: 3s + 2s = 5s
total time: 14s
(one thread is sleep around 9 sec - this is a BIG problem)

But if all target will be sorted (10sec, 4sec, 3sec, 2sec)

sorted target:
1thread: 10s = 10s
2thread: 4s + 3s+ 2s = 9s
total time is 10s.


I understand that the build time and file size are not related, but it's better than nothing.
(0035794)
Peter Kuemmel   
2014-04-27 14:11   
Ninja supports "pools":
http://martine.github.io/ninja/manual.html#ref_pool [^]

Which are available in next cmake version:

http://www.cmake.org/cmake/help/git-next/prop_gbl/JOB_POOLS.html#prop_gbl:JOB_POOLS [^]

You could add all expensive files to one target (e.g. a static lib) and limit the parallel compiles for this target, this guaranties at least that not two expensive files are compiled in parallel. Not sure if it solves completely your problem.
(0035795)
Andrey Morozov   
2014-04-27 16:39   
I think it's not solve my problem.

The solving is just sorted target by file size(first is biggest) before write to build.ninja
(0035797)
Ben Boeckel   
2014-04-28 00:04   
File size is not a good indicator of compile speed. C compiles faster than C++ and non-templated C++ code compiles faster than templated C++ code. Not to mention that headers may hide thousands of lines of code.

Ninja keeps stats on compilation (at least the command line and time). Adding the time it took might to it then have ninja do some naive bin-packing to try and pack the compile jobs into the available parallelism.
(0035798)
Peter Kuemmel   
2014-04-28 02:26   
Yes, for instance have a look at this 16kB file

http://www.cs.rpi.edu/~musser/design/blitz/fft.html [^]

By setting an arbitrary large size of the FFT you could get any compile time (at least until the compiler dies or you run out of memory).
(0035799)
Andrey Morozov   
2014-04-28 03:25   
then we need a tool to determine the order (priority) of the assembly.
(0035800)
Ben Boeckel   
2014-04-28 12:17   
I don't think this is CMake's job. Also, if you want to rescan the files, you'll need to rerun CMake to get the new build estimates; effectively making the configure/generate step depend on every source file. Also, how would you order files between different libraries? Link targets should also be ordered.

Sorry, I don't see why this isn't better done in Ninja itself since it has *all* the information and can get actual timings rather than just guessing. It also helps non-CMake projects using Ninja.
(0035818)
Ben Boeckel   
2014-05-05 16:06   
Bug filed for ninja: https://github.com/martine/ninja/issues/759 [^]
(0042542)
Kitware Robot   
2016-06-10 14:29   
Resolving issue as `moved`.

This issue tracker is no longer used. Further discussion of this issue may take place in the current CMake Issues page linked in the banner at the top of this page.