MantisBT - CMake
View Issue Details
0008826CMakeCMakepublic2009-03-31 14:422009-09-22 13:05
Wojciech Migda 
Brad King 
normalminoralways
closedfixed 
CMake-2-6 
 
0008826: CMake performance improvement on Cygwin
1. Cygwin is known for its poor performance with regard to some i/o operations, due to necessity to maintain *nix-like access rights management on ntfs filesystem.
2. there are know approaches to tackle this problem, such as recent improvements in git (http://kerneltrap.org/index.php?q=mailarchive/git/2008/9/27/3427614 [^])
3. CMake profiling during cmake_depends option execution shows that indeed calls to POSIX access function is the main bottleneck.
4. Simple benchmarking (approx. 700 files in a single directory, 80000 random executions of access) shows that it takes around 36 secs on cygwin/ntfs system regardless of the full/relative path being passed as access' argument.
5. the main source of calls to POSIX access in CMake is a function checking for file existence: SystemTools::FileExists
6. since on Cygwin file can be read even if a user has no reading rights replacing access with GetFileAttributesExA can be made to check for file existence. This requires conversion of cygwin paths to native Win32 form using a cygwin_conv_to_win32_path utility provided with the cygwin dll
7. this change alone brings drastic improvements in CMake performance during cmake_depends execution, however, the next bottleneck becomes the cygwin_conv_to_win32_path function. Fortunately we can cache path translations what moves this problem aside.
8. the attached patches for files cmake-2.6.3/Source/kwsys/SystemTools.cxx and cmake-2.6.3/Source/kwsys/SystemTools.hxx.in attempt to introduce basic changes to the way CMake checks for file existence. For one of the targets we have in our build system (ca. 130 .c files, 78,000 executions of access to check existnce of some of almost 700 headers) this cahnge has reduced cmake_depends timing from 36 secs to ca. 6 secs !!
solution
zip patches.zip (2,075) 2009-03-31 14:42
https://public.kitware.com/Bug/file/2148/patches.zip
Issue History
2009-03-31 14:42Wojciech MigdaNew Issue
2009-03-31 14:42Wojciech MigdaFile Added: patches.zip
2009-03-31 16:29Wojciech MigdaTag Attached: solution
2009-09-11 17:40Bill HoffmanStatusnew => assigned
2009-09-11 17:40Bill HoffmanAssigned To => Bill Hoffman
2009-09-11 17:41Bill HoffmanAssigned ToBill Hoffman => Brad King
2009-09-22 13:05Brad KingNote Added: 0017704
2009-09-22 13:05Brad KingStatusassigned => closed
2009-09-22 13:05Brad KingResolutionopen => fixed

Notes
(0017704)
Brad King   
2009-09-22 13:05   
I've committed changes based on your patch. I kept the cygwin->windows path mapping and cache code. I re-wrote SystemTools::FileExists almost completely. It is now optimized on native Windows by using GetFileAttributesEx directly (instead of MSVC's access emulation API), and on cygwin by using your approach.

Optimize KWSys SystemTools::FileExists on Windows
/cvsroot/CMake/CMake/Source/kwsys/SystemTools.cxx,v <-- Source/kwsys/SystemTools.cxx
new revision: 1.249; previous revision: 1.248
/cvsroot/CMake/CMake/Source/kwsys/SystemTools.hxx.in,v <-- Source/kwsys/SystemTools.hxx.in
new revision: 1.79; previous revision: 1.78