<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Yes you are correct on all your observations. We already use ninja and ccache wherever we can. In fact we have an issue about the whole
end-to-end build performance where we track all effort on this throughout the project (if you're interested:
<a href="https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649">https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1649</a>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">There are indeed quick and easy wins to be had with those tools (and we are working on it). But my original question is not about that.
It's about speeding up the configure step with makefile generator. I still don't think it's normal that it does hundreds of thousands of I/O on files that are a few bytes or even empty. However it's possible that it's because we do something incorrect in our
CMakeLists.txt and not CMake's fault.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">It's true that a few minutes of configure is not much when doing a full build, but consider incremental builds where all you do is change
one cxx file (and cmake is triggered because you changed git branch or something). Then the generate step is a significant part of the time you wait.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Victor Poughon<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">De :</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Eric Noulard <eric.noulard@gmail.com>
<br>
<b>Envoyé :</b> me</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">rcredi 14 novembre 2018 12:34<br>
<b>À :</b> Poughon Victor <Victor.Poughon@cnes.fr><br>
<b>Cc :</b> CMake Mailinglist <cmake@cmake.org><br>
<b>Objet :</b> Re: [CMake] Tips on reducing configure time<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Le mer. 14 nov. 2018 à 12:06, Poughon Victor <<a href="mailto:Victor.Poughon@cnes.fr">Victor.Poughon@cnes.fr</a>> a écrit :<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for the replies</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">You can see configure times on our dashboard (switch to advanced view):
<a href="https://dash.orfeo-toolbox.org/index.php?project=OTB" target="_blank">https://dash.orfeo-toolbox.org/index.php?project=OTB</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">It ranges from 1-3min. There's even one windows machine that's at 4m30s.</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Somes windows system are known to be slow on many files accesses.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">However on this particular machine (<a href="https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008">https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342008</a> I guess) you already seem<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to be using NInja + MSVC. so many small files access to ?makefiles? should not be the culprit. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">That said whatever the system your build and test time seems to dominate the overall time (in the CI).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have had great speed-up experience using CMake+ninja+ccache for our build in CI. We did go from 1 hour down to less than 10min.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">We only have various linux build (no Windows) it is really a big time savior combo.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We even share the compil' cache between the CI and the developers. So everybody is feeding compile cache and takes benefits from others including CI.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You seem to be using that as well here:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><a href="https://dash.orfeo-toolbox.org/buildSummary.php?buildid=341993">https://dash.orfeo-toolbox.org/buildSummary.php?buildid=341993</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and you get less than 5min build time which corroborate my suggestion.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The same machine build with clang: <a href="https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033">https://dash.orfeo-toolbox.org/buildSummary.php?buildid=342033</a> is lasting 30min+<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">and is not using ccache.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span><o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for the tip about Ninja generator, it's indeed much faster. But switching the entire dev team
+ CI servers from make to ninja is not really an option for us right now because of a bug in ninja that affects our SuperBuild configuration (see here if you're interested in the details:
<a href="https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625" target="_blank">
https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/issues/1625</a>)</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I see.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Not using ninja in the Superbuild case does not mean you cannot use it in other cases, right?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">May be the one-size fits all is a little too much to ask at this point? My opinion though.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Did you report the issue upstream (ninja or cmake)?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I was impressed in the past by the amount of tests orfeo is doing, I'm still are !!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Are you running those tests in parallel (with ctest --parallel) ? It can be a time savior as well is the running machine has a fair amount of cores and memory.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Eric<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>