<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I manage a very large project using CMake that I have done significant restructuring to lower the CMake configure times. I will say the one thing that surprised
me the most is that for one piece of it where we have repeated work done say “for each folder in this subdir build 11 executables with a ton of code generation.” If I structure the build as such:<br>
<br>
One top level CMakeLists.txt and one single line CMakeLists.txt in each subdir containing do_work(${SUBDIR}):<br>
foreach( SUBDIR ${LIST_OF_SUBDIRS} )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> add_subdirectory(${SUBDIR})<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">endforeach()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Versus:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><br>
One top level CMakeLists.txt:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">foreach( SUBDIR ${LIST_OF_SUBDIRS} )<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> do_work(${SUBDIR})<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">endforeach()<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In short, the decision to do all the work in the same CMakeLists.txt file or do I enter and leave a CMakeLists.txt for each directory causes a dramatic performance
and memory difference.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The add_subdirectory method saves many Gigs of RAM and reduces our configure time from 30 minutes to 5 minutes. Please note I am writing this from memory so
my numbers may be a bit off but scale is accurate.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I don’t know if this is applicable to your case but it was the biggest saving in my project and took a long time to figure out.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Kris<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> CMake [mailto:cmake-bounces@cmake.org]
<b>On Behalf Of </b>Isaiah Norton<br>
<b>Sent:</b> Friday, May 04, 2018 7:51 PM<br>
<b>To:</b> Patrick E Gartung <gartung@fnal.gov><br>
<b>Cc:</b> cmake@cmake.org<br>
<b>Subject:</b> Re: [CMake] Memory usage and configuration time for large project<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#222222;background:white">As one ballpark datapoint: a "superbuild" of 3D Slicer (<a href="http://slicer.org">slicer.org</a>) has a similar object and library count:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><span style="background:white">macbook-pro:s5nj inorton$ find ./ -name *.o | wc -l<br>
14127<o:p></o:p></span></p>
</blockquote>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><span style="background:white">macbook-pro:s5nj inorton$ find ./ -name *.dylib -or -name *.so | wc -l<br>
2158<o:p></o:p></span></p>
</blockquote>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#222222;background:white">Based on a few quick tests, the aggregate cmake time is probably about 6-8 minutes for the ninja generator, over the multi-hour build (each dependency is configured
and built as a separate sub-project via CMake's ExternalProject mechanism) -- with the caveat that each of those cmake runs is doing lengthy checks that should only be done once if your project strictly uses `add_subdirectory`.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As a more concrete point of comparison, building VTK generates 5747 object files, and a clean configure on my 2-core macbook takes about 90s.<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">Isaiah<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, May 4, 2018 at 2:16 PM, Patrick E Gartung <<a href="mailto:gartung@fnal.gov" target="_blank">gartung@fnal.gov</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Just to be clear, the memory and time used are just to configure and generate the makefiles or Ninja file. The build itself can take several hours.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On 4/30/18, 4:55 PM, "R0b0t1" <<a href="mailto:r030t1@gmail.com">r030t1@gmail.com</a>> wrote:<br>
<br>
On Mon, Apr 30, 2018 at 4:44 PM, Patrick E Gartung <<a href="mailto:gartung@fnal.gov">gartung@fnal.gov</a>> wrote:<br>
> Hi,<br>
><br>
> We have a large c/c++/fortran project, CMSSW, that is built with a custom tool, scram.
<br>
><br>
> <a href="https://github.com/cms-sw/cmssw" target="_blank">https://github.com/cms-sw/cmssw</a><br>
><br>
> The project might move to a cmake based build in the future. The scripts to convert to CmakeLists.tx has been written<br>
><br>
> <a href="https://github.com/cms-sw/cmssw2cmake" target="_blank">https://github.com/cms-sw/cmssw2cmake</a><br>
><br>
> Tests show that with the cmake files generated with this script, configuring the project uses up to 1.5GB of ram and takes 11 minutes when using -GMakefiles. Using -GNinja and using the latest cmake, this time can be reduced to 8 minutes.<br>
><br>
> The project builds 14k object files, 2.2k libraries, ~600 binaries, 500 generated source files with links to ~100 external libraries.<br>
><br>
> Is this amount of memory usage and time typical for a project of this size?<br>
><br>
<br>
I'm inclined to say "yes" as many builds such as Firefox, its<br>
supporting libraries, and Chrome all take lots of time and memory.<br>
Chrome uses Ninja, I might add. But the issue is not CMake itself.<br>
CMake tends to produce better builds.<br>
<br>
As I am not intimately familiar with your project, I can't make good<br>
concrete suggestions. You may enjoy reading<br>
<a href="https://news.ycombinator.com/item?id=14733829" target="_blank">https://news.ycombinator.com/item?id=14733829</a> and searching for build<br>
optimization strategies.<br>
<br>
Keep in mind a lot of the blame falls on C++ and Windows, should you<br>
be using Windows. If you aren't using Windows, then the advice in the<br>
comments above should still be relevant, and give you something to go<br>
on.<br>
<br>
Cheers,<br>
R0b0t1<br>
<br>
> Patrick Gartung<br>
> Fermilab<br>
><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
-- <br>
<br>
Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" target="_blank">
http://www.cmake.org/Wiki/CMake_FAQ</a><br>
<br>
Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br>
<br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" target="_blank">
http://cmake.org/cmake/help/support.html</a><br>
CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" target="_blank">
http://cmake.org/cmake/help/consulting.html</a><br>
CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" target="_blank">
http://cmake.org/cmake/help/training.html</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">
http://www.kitware.com/opensource/opensource.html</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="https://cmake.org/mailman/listinfo/cmake" target="_blank">https://cmake.org/mailman/listinfo/cmake</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction,
disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security
or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments
is free of viruses.<br>
</font>
</body>
</html>