[CMake] CMake Project Generation Speedup

[Kuba Ober]

The best advice I had for myself: profile the code and tweak it. The last I looked, and that was a while ago, the generators could use a threaded write queue so that the file output would overlap with file generation. Those probably could be memory mapped files, too, to further minimize the overhead: writing to mapped pages that are dumped to disk via DMA (that’s how page flushes work) is cheaper than copying all the data in little chunks into a small buffer in the file output code, and then once again into the page that ends up flushed to disk. Never mind that the write latencies are on the hot path, where they don’t belong.

If generators could be const-correct (not writing anywhere into the Makefile data structures), they could all be parallelized as well – but I’m not sure whether that’s a minor hack or a major rework. 

Cheers, Kuba

> 20 mars 2019 kl. 16:53 skrev J. Caleb Wherry <calebwherry at gmail.com>:
> 
> Did anything ever come of this?
> 
> I am in a similar boat: we have >800 targets on our full build (native C++, Managed C++, C#, Java, CUDA, etc) and the majority of the time for the configure/generate steps takes place in the generate step (>70%).
> 
> I understand there is a lot of IO since the generate step has to write the project files and filters for each C++ project (the majority of our projects) for VS generators (what we use). I'm just looking to see if there is anything to look at or potentially speedup up the generate step besides "get a faster drive".
> 
> Thanks!
> -Caleb
> 
>> On Thu, Nov 17, 2016 at 11:15 AM Damian <damian.campeanu at gmail.com> wrote:
>> Hi all,
>> 
>> We are still in the process of switching our large Make-based build to CMake. One of the issues we're running into is the time it takes to reparse and regenerate the CMake  project (whether ninja, VS, or make) after touching any CMake file. To give you an idea, we have about 1000 targets and that takes a good 2 min for CMake to rerun.
>> 
>> Are there any plans to speed this up? Maybe parallelize it in some way or do a better job regenerating only what needs regenerating? Is there anything we can do on our side to reduce our regeneration times?
>> 
>> For example, if using a VS generator, each directory in the source that has a CMakeLists.txt gets a .vcproj and .sln generated. Ideally, if I touch one of those CMakeLists.txt, only that .sln/.vcproj would get regenerated.
>> 
>> Thanks for any help.
>> -- 
>> 
>> Powered by www.kitware.com
>> 
>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>> 
>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>> 
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>> 
>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake
> 
> 
> -- 
> J. Caleb Wherry
> Scientific Software Engineer
> 
> http://www.calebwherry.com
> +1 (615) 708-5651
> calebwherry at gmail.com
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> https://cmake.org/mailman/listinfo/cmake
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://cmake.org/pipermail/cmake/attachments/20190321/78f363c7/attachment-0001.html>