From ben.boeckel at kitware.com Mon Jan 1 09:42:53 2018 From: ben.boeckel at kitware.com (Ben Boeckel) Date: Mon, 1 Jan 2018 09:42:53 -0500 Subject: [cmake-developers] Build failure due to {cd bindir} failed on Windows/Ninja In-Reply-To: References: <20171229150223.GA26168@megas.kitware.com> Message-ID: <20180101144253.GA23312@megas.kitware.com> On Fri, Dec 29, 2017 at 13:32:54 -0800, Jom O'Fisher wrote: > >> Ninja should be using relative paths for the build tree and absolute > for anything outside of it. > That appears to be the case, so I need a better theory. I noticed another > difference between compilation phase and link phase. The link phase is > bracketed by cmd.exe /C "cd . && && cd .". > I'm wondering problem those "cd ." calls are solving? Their presences is > what's causing everything to be wrapped in "cmd /C" They're the "no-op" of the PRE_LINK and POST_LINK commands for targets. > Here's the error that encapsulates the failure > > CMake Error at D:/Android/sdk/cmake/3.6.3155560/share/cmake-3.6/ > Modules/CMakeTestCCompiler.cmake:61 (message): > The C compiler > "D:/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/ > windows-x86_64/bin/clang.exe" > is not able to compile a simple test program. > It fails with the following output: > Change Dir: D:/src/MyApplication/app/.externalNativeBuild/cmake/ > debug/armeabi/CMakeFiles/CMakeTmp > > Run Build Command:"D:\Android\sdk\cmake\3.6.3155560\bin\ninja.exe" > "cmTC_0dcd8" > [1/2] Building C object CMakeFiles/cmTC_0dcd8.dir/testCCompiler.c.o > [2/2] Linking C executable cmTC_0dcd8 > FAILED: cmd.exe /C "cd . && > D:\Android\sdk\ndk-bundle\toolchains\llvm\prebuilt\ > windows-x86_64\bin\clang.exe > --target=armv5te-none-linux-androideabi > --gcc-toolchain=D:/Android/sdk/ndk-bundle/toolchains/arm- > linux-androideabi-4.9/prebuilt/windows-x86_64 > --sysroot=D:/Android/sdk/ndk-bundle/sysroot -isystem > D:/Android/sdk/ndk-bundle/sysroot/usr/include/arm-linux-androideabi > -D__ANDROID_API__=14 -g -DANDROID -ffunction-sections -funwind-tables > -fstack-protector-strong -no-canonical-prefixes -march=armv5te > -mtune=xscale -msoft-float -fno-integrated-as -mthumb -Wa,--noexecstack > -Wformat -Werror=format-security -Wl,--exclude-libs,libgcc.a --sysroot > D:/Android/sdk/ndk-bundle/platforms/android-14/arch-arm -Wl,--build-id > -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--no-undefined > -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now > -Wl,--gc-sections -Wl,-z,nocopyreloc > CMakeFiles/cmTC_0dcd8.dir/testCCompiler.c.o -o cmTC_0dcd8 -lm && cd ." > clang.exe: error: no such file or directory: > 'CMakeFiles/cmTC_0dcd8.dir/testCCompiler.c.o' > ninja: build stopped: subcommand failed. You can pass `--debug-trycompile` to keep the output of the try compile step. Running the command manually may help to narrow down the problem. The tools mentioned here may also help: https://stackoverflow.com/questions/3847745/systrace-for-windows --Ben From domen.vrankar at gmail.com Tue Jan 2 17:39:31 2018 From: domen.vrankar at gmail.com (Domen Vrankar) Date: Tue, 2 Jan 2018 23:39:31 +0100 Subject: [cmake-developers] Problem with created debian package In-Reply-To: <1692378.GO9LxbZiVc@amd64> References: <9411017.E9NYm8Fdsl@amd64> <1692378.GO9LxbZiVc@amd64> Message-ID: 2017-12-30 6:29 GMT+01:00 Kornel Benko : > > 2) use gnu tar instead by setting CPACK_DEBIAN_ARCHIVE_TYPE to gnutar > > Thanks, this is really nice. Meanwhile I found out what was going wrong. > The package manager seems to not understand the way some filenames > containing > non-ascii chars are coded by cpack. > I prepared a minimal example showing the effect here. > > 1.) Create a dir and copy the attached to it > 2.) mkdir build; cd bild > 3.) cmake .. -DCPACK_BINARY_DEB:BOOL=ON > 4.) sudo dpkg -i wrongfilename-0.1.1-Linux.deb > (Reading database ... 422665 files and directories currently > installed.) > Preparing to unpack wrongfilename-0.1.1-Linux.deb ... > Unpacking wrongfilename (0.1.1) ... > dpkg: error processing archive wrongfilename-0.1.1-Linux.deb > (--install): > corrupted filesystem tarfile - corrupted package archive > Errors were encountered while processing: > wrongfilename-0.1.1-Linux.deb > Thanks for the example. The "use gnu tar option with CPack" produces a valid package for your example: set(CPACK_DEBIAN_ARCHIVE_TYPE "gnutar") It would seem that non gnu version doesn't handle non ascii filenames correctly. I'm not certain if it's an libarchive bug or expectedly unsupported feature (it would seem that it's a bug...) - I'll have to investigate further. Thanks, Domen -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at johnwordsworth.com Wed Jan 3 16:27:37 2018 From: john at johnwordsworth.com (John Wordsworth) Date: Wed, 3 Jan 2018 22:27:37 +0100 Subject: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS Message-ID: I have recently been reviewing ways to improve build times for our project, which is comprised of a number of static libraries. I stumbled across this post on the CMake tracker from 2012/13 (https://cmake.org/Bug/view.php?id=13799 ). It suggests adding a COMPILE_DEPENDS target property to to explicitly set dependencies for STATIC libraries instead of always using all linked libraries as build-order dependencies. Having done a draft implementation in a local CMake repository it has shaved off 20% of our 120s build time. I expect the savings to be much more dramatic when I test with Incredibuild (approximately 50% based on tests done previously from just deleting dependencies manually in Visual Studio). I don?t really want to refactor our code to use ?OBJECT? libraries as the inability to link with other targets means that propagating compile options / include directories etc down the chain of linked libs becomes painful. This method allows me to switch between static and shared libs using a config option and none of my CMake scripts need to change. Anyway, I was wondering whether there was any interest in me pushing my solution back to Git / submitting a Pull request so that it might be merged in at some point. If there is - any advice on any gotchas I might watch for instead of just adding some fairly simple code to cmComputeTargetDepends.cxx would be gratefully received - especially as this is my first time poking around in CMake code. // John Wordsworth -------------- next part -------------- An HTML attachment was scrubbed... URL: From craig.scott at crascit.com Thu Jan 4 04:48:53 2018 From: craig.scott at crascit.com (Craig Scott) Date: Thu, 4 Jan 2018 20:48:53 +1100 Subject: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS In-Reply-To: References: Message-ID: On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth wrote: > I have recently been reviewing ways to improve build times for our > project, which is comprised of a number of static libraries. I stumbled > across this post on the CMake tracker from 2012/13 ( > https://cmake.org/Bug/view.php?id=13799). It suggests adding a > COMPILE_DEPENDS target property to to explicitly set dependencies for > STATIC libraries instead of always using all linked libraries as > build-order dependencies. > > Having done a draft implementation in a local CMake repository it has > shaved off 20% of our 120s build time. I expect the savings to be much > more dramatic when I test with Incredibuild (approximately 50% based on > tests done previously from just deleting dependencies manually in Visual > Studio). > > I don?t really want to refactor our code to use ?OBJECT? libraries as the > inability to link with other targets means that propagating compile options > / include directories etc down the chain of linked libs becomes painful. > This method allows me to switch between static and shared libs using a > config option and none of my CMake scripts need to change. > There's a couple more choices here. If your project consists of lots of small (static) libraries, consider whether you can combine some of them to result in a smaller number of larger libraries. This isn't always a gain, but in terms of ability to compile sources in parallel, it will often lead to more efficient builds. You just need to be careful you don't end up with so many objects being combined into one library that you start to hit max open file limits during linking/archiving (something I've hit on multiple platforms lately, so it's not just a hypothetical example). Use of target_sources() can be quite helpful if you want to try out this path (you may find this article helpful). Another choice is to go in the opposite direction and take advantage of the optimisation made for the Ninja generator (if that's a choice open to you) that was introduced in CMake 3.9.0 where if no custom commands exist on a target A, then compilation steps of another target B linking to A are allowed to proceed without waiting for A's link step to complete. Only B's link step will depend on A's link step. In your project, if you have custom commands, see if you can split up that target into just those sources that need the results of the custom command and another target that doesn't. The latter will then be able to compile earlier, so fewer sources have to wait for earlier linking steps. This might be hard to do, it really depends on how your project is structured. Both of the above choices allow you to retain the automatic propagation of compile options, include directories, etc. and to switch between shared/static easily, but the latter is specific to the Ninja generator and may not be an acceptable change for you. > > Anyway, I was wondering whether there was any interest in me pushing my > solution back to Git / submitting a Pull request so that it might be merged > in at some point. If there is - any advice on any gotchas I might watch for > instead of just adding some fairly simple code to > cmComputeTargetDepends.cxx would be gratefully received - especially as > this is my first time poking around in CMake code. > The existing behaviour is conservative and any change would have to also be conservative, meaning that it must not introduce any possibility of breaking existing projects. If I'm understanding your proposed feature correctly, it sounds like you want to relax the build-order dependencies by default when a COMPILE_DEPENDS target property is defined. Basically, if COMPILE_DEPENDS is defined, you are taking over responsibility for the build-order dependencies. This would be something I'd usually discourage projects from doing because such manual dependencies would be a prime candidate for not being kept up to date as a project evolves, leading to subtle, hard-to-trace build errors. Some judicious project restructuring can normally give a pretty efficient parallel build without having to resort to such measures, so I'm wary of adding a feature like this (though I can understand the desire for it). In my experience, you can get some considerable speedups using tools like ccache (and its equivalents for other platforms/compilers). These obviously only help for subsequent builds of things that have been built previously, but for everyday development where you switch between branches or for CI servers doing lots of similar builds, the savings can be impressively big. -- Craig Scott Melbourne, Australia https://crascit.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.noulard at gmail.com Thu Jan 4 06:16:43 2018 From: eric.noulard at gmail.com (Eric Noulard) Date: Thu, 4 Jan 2018 12:16:43 +0100 Subject: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS In-Reply-To: References:

Message-ID: 2018-01-04 10:48 GMT+01:00 Craig Scott : > > > On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth > wrote: > >> I have recently been reviewing ways to improve build times for our >> project, which is comprised of a number of static libraries. I stumbled >> across this post on the CMake tracker from 2012/13 ( >> https://cmake.org/Bug/view.php?id=13799). It suggests adding a >> COMPILE_DEPENDS target property to to explicitly set dependencies for >> STATIC libraries instead of always using all linked libraries as >> build-order dependencies. >> >> Having done a draft implementation in a local CMake repository it has >> shaved off 20% of our 120s build time. I expect the savings to be much >> more dramatic when I test with Incredibuild (approximately 50% based on >> tests done previously from just deleting dependencies manually in Visual >> Studio). >> > You said you tested with Incredibuild but with what kind of configuration? AFAIK Incredibuild takes its power from distributing the build? So does your CMake test uses something that distribute the build too ? > >> I don?t really want to refactor our code to use ?OBJECT? libraries as the >> inability to link with other targets means that propagating compile options >> / include directories etc down the chain of linked libs becomes painful. >> This method allows me to switch between static and shared libs using a >> config option and none of my CMake scripts need to change. >> > > There's a couple more choices here. If your project consists of lots of > small (static) libraries, consider whether you can combine some of them to > result in a smaller number of larger libraries. This isn't always a gain, > but in terms of ability to compile sources in parallel, it will often lead > to more efficient builds. You just need to be careful you don't end up with > so many objects being combined into one library that you start to hit max > open file limits during linking/archiving (something I've hit on multiple > platforms lately, so it's not just a hypothetical example). Use of > target_sources() can be quite helpful if you want to try out this path (you > may find this article > > helpful). > > Another choice is to go in the opposite direction and take advantage of > the optimisation made for the Ninja generator (if that's a choice open to > you) that was introduced in CMake 3.9.0 > where if no > custom commands exist on a target A, then compilation steps of another > target B linking to A are allowed to proceed without waiting for A's link > step to complete. Only B's link step will depend on A's link step. In your > project, if you have custom commands, see if you can split up that target > into just those sources that need the results of the custom command and > another target that doesn't. The latter will then be able to compile > earlier, so fewer sources have to wait for earlier linking steps. This > might be hard to do, it really depends on how your project is structured. > I fully agree with Craig here. We use CMake + Ninja and we get really efficient parallel build (on 10+ cores machines). This is on Linux though but it seems MS is putting effort to get CMake + ninja build work with Visual Studio: https://blogs.msdn.microsoft.com/vcblog/2017/05/10/cmake-support-in-visual-studio-whats-new-in-2017-15-3-update/ Another way to improve your build may be to review your dependency graph carefully. You can use cmake --grahviz options to dump the graph and see whether you can cut some dependencies due to transitivity of dependency. I have seen many projects with "overspecified deps". I think this impair build time when using a lot of static libs because in that case you may get poor performance link time. Namelly if you have many unit tests linking to too many static libs because of overspecified deps. That said you did not specify whether if the 120s build time is only for the library or if this is a "global" build time which includes linking somes executables and possible unit tests programs. Note that even if this is not the case having a "big-picture" look at the dependency build graph generated by CMake may be insightful. Note that with CMake 3.10 you can now see difference between PUBLIC, PRIVATE and INTERFACE deps: https://cmake.org/cmake/help/v3.10/release/3.10.html > > Both of the above choices allow you to retain the automatic propagation of > compile options, include directories, etc. and to switch between > shared/static easily, but the latter is specific to the Ninja generator and > may not be an acceptable change for you. > > >> >> Anyway, I was wondering whether there was any interest in me pushing my >> solution back to Git / submitting a Pull request so that it might be merged >> in at some point. If there is - any advice on any gotchas I might watch for >> instead of just adding some fairly simple code to >> cmComputeTargetDepends.cxx would be gratefully received - especially as >> this is my first time poking around in CMake code. >> > > The existing behaviour is conservative and any change would have to also > be conservative, meaning that it must not introduce any possibility of > breaking existing projects. If I'm understanding your proposed feature > correctly, it sounds like you want to relax the build-order dependencies by > default when a COMPILE_DEPENDS target property is defined. Basically, if > COMPILE_DEPENDS is defined, you are taking over responsibility for the > build-order dependencies. This would be something I'd usually discourage > projects from doing because such manual dependencies would be a prime > candidate for not being kept up to date as a project evolves, leading to > subtle, hard-to-trace build errors. Some judicious project restructuring > can normally give a pretty efficient parallel build without having to > resort to such measures, so I'm wary of adding a feature like this (though > I can understand the desire for it). > > In my experience, you can get some considerable speedups using tools like > ccache (and its equivalents for other platforms/compilers). These obviously > only help for subsequent builds of things that have been built previously, > but for everyday development where you switch between branches or for CI > servers doing lots of similar builds, the savings can be impressively big. > +1 again. ccache boost our build. It can be more than 10x faster (depends on cache hit). You can even share (on a local network) the ccache directory between developers and the CI and get very high hit rate (> 80%). Now I'm not working on windows nor with Visual Studio and AFAIK ccache does not work with MSVC. There seem to be alternative, https://github.com/frerich/clcache, but as you guessed I did never used that. -- Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From john at johnwordsworth.com Fri Jan 5 04:09:43 2018 From: john at johnwordsworth.com (John Wordsworth) Date: Fri, 5 Jan 2018 10:09:43 +0100 Subject: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS In-Reply-To: References:

Message-ID: I realise I wasn't very clear in my previous explanation. Hopefully this paints a better picture; - We have 40 or so developers spread across Visual Studio 2015 (MSBuild - Windows), Xcode (macOS) and Make/Clang (Linux). 80% use Windows. - We have a core set of libs in use by a number of projects (~30 static libs and ~10 different projects), but the projects mix and match which libs they use - merging the static libs (beyond a few of them) isn't something we want to do. - I have recently been revising our CMake structure and have boiled it down to a fairly simple "add_library(x), target_link_libraries(x, default_compile_settings, dependencies)" which means compile settings, include directories and macros propagate through the libs nicely. - We sometimes want to build and use certain libs as shared libs instead of static libs. With regards to the COMPILE_DEPENDS feature that I have mocked up and started testing with our team; - If and only if COMPILE_DEPENDS is set for a STATIC library, then when building the target dependencies for that library, it uses the list of provided targets instead of those that would have been inferred from previous calls to "target_link_libraries" or "add_dependencies". The reality is, the libs which are being built statically can nearly all build in parallel with only one or two "real" dependencies. I understand that our case is rather specific, but having implemented "COMPILE_DEPENDS" and written ~10 lines of CMake in our project (hard coding the one or two actual build order dependencies and disabling the rest), our compile graph looks much nicer and saves a significant amount of time building on a single machine (see https://www.johnwordsworth.com/temp/cmake_compile_ depends.jpg). We see similar savings on macOS using Xcode but have not tested on Linux yet. When we allow Incredibuild to distribute building of compilation units across 15-20 agents, the proportional build time drops more dramatically between the two (the same project goes from ~2.6mins -> ~1.6mins). I understand that just "hard overriding" the target dependencies is a bit messy, but I'd also be happy to explore other ways I could add a similar feature to CMake if there is a potentially better way to do this. What I am ideally looking for in a solution is that I can continue to propagate include directories / compile settings / preprocessor macros through the chain of libs and that it improves building static libs in parallel across all our build systems (MSVC / XCode / make). Ideally, I wouldn't have to restructure my CMake project too much either - as the draft I have for our new structure feels super clean now and refactoring it just to remove build order dependencies would be shame. If there are any alternative ideas for how to implement this, I'd love to discuss. I'm no CMake expert, but maybe if we could use "target_link_libraries" with OBJECT libraries to grab compile settings, or perhaps I could look into a mechanism for not adding build order dependencies on VS2015/Xcode if there is no need too? Thanks again for the feedback and discussion so far. On Thu, Jan 4, 2018 at 12:16 PM, Eric Noulard wrote: > > 2018-01-04 10:48 GMT+01:00 Craig Scott : > >> >> >> On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth >> wrote: >> >>> I have recently been reviewing ways to improve build times for our >>> project, which is comprised of a number of static libraries. I stumbled >>> across this post on the CMake tracker from 2012/13 ( >>> https://cmake.org/Bug/view.php?id=13799). It suggests adding a >>> COMPILE_DEPENDS target property to to explicitly set dependencies for >>> STATIC libraries instead of always using all linked libraries as >>> build-order dependencies. >>> >>> Having done a draft implementation in a local CMake repository it has >>> shaved off 20% of our 120s build time. I expect the savings to be much >>> more dramatic when I test with Incredibuild (approximately 50% based on >>> tests done previously from just deleting dependencies manually in Visual >>> Studio). >>> >> > You said you tested with Incredibuild but with what kind of configuration? > AFAIK Incredibuild takes its power from distributing the build? So does > your CMake test uses something that distribute the build too ? > > >> >>> I don?t really want to refactor our code to use ?OBJECT? libraries as >>> the inability to link with other targets means that propagating compile >>> options / include directories etc down the chain of linked libs becomes >>> painful. This method allows me to switch between static and shared libs >>> using a config option and none of my CMake scripts need to change. >>> >> >> There's a couple more choices here. If your project consists of lots of >> small (static) libraries, consider whether you can combine some of them to >> result in a smaller number of larger libraries. This isn't always a gain, >> but in terms of ability to compile sources in parallel, it will often lead >> to more efficient builds. You just need to be careful you don't end up with >> so many objects being combined into one library that you start to hit max >> open file limits during linking/archiving (something I've hit on multiple >> platforms lately, so it's not just a hypothetical example). Use of >> target_sources() can be quite helpful if you want to try out this path (you >> may find this article >> >> helpful). >> >> Another choice is to go in the opposite direction and take advantage of >> the optimisation made for the Ninja generator (if that's a choice open to >> you) that was introduced in CMake 3.9.0 >> where if no >> custom commands exist on a target A, then compilation steps of another >> target B linking to A are allowed to proceed without waiting for A's link >> step to complete. Only B's link step will depend on A's link step. In your >> project, if you have custom commands, see if you can split up that target >> into just those sources that need the results of the custom command and >> another target that doesn't. The latter will then be able to compile >> earlier, so fewer sources have to wait for earlier linking steps. This >> might be hard to do, it really depends on how your project is structured. >> > > I fully agree with Craig here. We use CMake + Ninja and we get really > efficient parallel build (on 10+ cores machines). > This is on Linux though but it seems MS is putting effort to get CMake + > ninja build work with Visual Studio: > https://blogs.msdn.microsoft.com/vcblog/2017/05/10/cmake- > support-in-visual-studio-whats-new-in-2017-15-3-update/ > > Another way to improve your build may be to review your dependency graph > carefully. You can use cmake --grahviz options to dump the graph and see > whether you can cut > some dependencies due to transitivity of dependency. I have seen many > projects with "overspecified deps". I think this impair build time when > using a lot of static libs because in that case you may get > poor performance link time. Namelly if you have many unit tests linking to > too many static libs because of overspecified deps. > > That said you did not specify whether if the 120s build time is only for > the library or if this is a "global" build time which includes linking > somes executables and possible unit tests programs. > > Note that even if this is not the case having a "big-picture" look at the > dependency build graph generated by CMake may be insightful. > Note that with CMake 3.10 you can now see difference between PUBLIC, > PRIVATE and INTERFACE deps: > https://cmake.org/cmake/help/v3.10/release/3.10.html > > > >> >> Both of the above choices allow you to retain the automatic propagation >> of compile options, include directories, etc. and to switch between >> shared/static easily, but the latter is specific to the Ninja generator and >> may not be an acceptable change for you. >> >> >>> >>> Anyway, I was wondering whether there was any interest in me pushing my >>> solution back to Git / submitting a Pull request so that it might be merged >>> in at some point. If there is - any advice on any gotchas I might watch for >>> instead of just adding some fairly simple code to >>> cmComputeTargetDepends.cxx would be gratefully received - especially as >>> this is my first time poking around in CMake code. >>> >> >> The existing behaviour is conservative and any change would have to also >> be conservative, meaning that it must not introduce any possibility of >> breaking existing projects. If I'm understanding your proposed feature >> correctly, it sounds like you want to relax the build-order dependencies by >> default when a COMPILE_DEPENDS target property is defined. Basically, if >> COMPILE_DEPENDS is defined, you are taking over responsibility for the >> build-order dependencies. This would be something I'd usually discourage >> projects from doing because such manual dependencies would be a prime >> candidate for not being kept up to date as a project evolves, leading to >> subtle, hard-to-trace build errors. Some judicious project restructuring >> can normally give a pretty efficient parallel build without having to >> resort to such measures, so I'm wary of adding a feature like this (though >> I can understand the desire for it). >> >> In my experience, you can get some considerable speedups using tools like >> ccache (and its equivalents for other platforms/compilers). These obviously >> only help for subsequent builds of things that have been built previously, >> but for everyday development where you switch between branches or for CI >> servers doing lots of similar builds, the savings can be impressively big. >> > > +1 again. > > ccache boost our build. It can be more than 10x faster (depends on cache > hit). > You can even share (on a local network) the ccache directory between > developers and the CI and get very high hit rate (> 80%). > > Now I'm not working on windows nor with Visual Studio and AFAIK ccache > does not work with MSVC. > There seem to be alternative, https://github.com/frerich/clcache, but as > you guessed I did never used that. > > -- > Eric > -------------- next part -------------- An HTML attachment was scrubbed... URL: From seblist at xwmw.org Fri Jan 5 14:34:53 2018 From: seblist at xwmw.org (Sebastian Holtermann) Date: Fri, 05 Jan 2018 20:34:53 +0100 Subject: [cmake-developers] kwsysProcess threadsafety Message-ID: <4776173.KzqDUZkdIB@blauwal> Hello! As you might have noticed I tried to parallelize AUTOMOC/UIC. https://gitlab.kitware.com/cmake/cmake/merge_requests/1632 The issue that's blocking it now is that the kwsysProcess framework isn't thread safe. As a consequence it is not possible for threads to start processes concurrently using the kwsysProcess framework. There are three options I see 1) Make kwsysProcess thread safe I don't think it's impossible but it's not a small task. Looking through the code I found a few issues that would block the whole idea for me. - OpenVMS: I have little to no knowledge about OpenVMS. Wikipedia says it is basically abandoned. Can the OpenVMS code be removed? I couldn't test it anyway. - C vs. C++ There's a lot of C in processUNIX.c (list allocation etc.). I would prefer to use C++11 (or higher), especially std::array, std::vector, std::thread, std::mutex, etc.. Is C++11 acceptable in kwsysProcess? This would make processUNIX.c processUNIX.cpp? 2) Use libuv instead Using the libuv event loop is overkill and would probably imply that more threads are started than necessary (the libuv thread pool will be started anyway). But it would allow to start multiple concurrent (moc/uic) processes. Is there a reason to not use libuv? 3) Abandon the idea to parallelize AUTOMOC/UIC Well, I liked the idea. Any thoughts? -Sebastian From seblist at xwmw.org Fri Jan 5 15:25:17 2018 From: seblist at xwmw.org (Sebastian Holtermann) Date: Fri, 05 Jan 2018 21:25:17 +0100 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <817653610.13054045.1515182430156.JavaMail.zimbra@elemtech.com> References: <4776173.KzqDUZkdIB@blauwal> <817653610.13054045.1515182430156.JavaMail.zimbra@elemtech.com> Message-ID: <1811653.G2k9T0IVT5@blauwal> On Freitag, 5. Januar 2018 13:00:30 CET clinton at elemtech.com wrote: > Cool. I'd like to have parallel automoc and uic. > > How about this 4th option? > Don't use threads. A single thread is able to spawn multiple processes, and > wait on multiple processes, and react when 1 or more processes change > state. I'm probably not familiar enough with kwsysProcess to know if this > approach will work, but it might be easier and simpler(?) than making it > thread safe. > > Clint > AFAIK there are two interfaces inside CMake that allow to start a process, kwsysProcess and libuv. kwsysProcess allows to start a single process and wait for it to finish. I thinks it's also possible to start multiple parallel processes. But there's no interface to select/poll a number of parallel processes. So the 4th option would boil down to use libuv for process management. Sebastian From clinton at elemtech.com Fri Jan 5 15:00:30 2018 From: clinton at elemtech.com (clinton at elemtech.com) Date: Fri, 5 Jan 2018 13:00:30 -0700 (MST) Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <4776173.KzqDUZkdIB@blauwal> References: <4776173.KzqDUZkdIB@blauwal> Message-ID: <817653610.13054045.1515182430156.JavaMail.zimbra@elemtech.com> Cool. I'd like to have parallel automoc and uic. How about this 4th option? Don't use threads. A single thread is able to spawn multiple processes, and wait on multiple processes, and react when 1 or more processes change state. I'm probably not familiar enough with kwsysProcess to know if this approach will work, but it might be easier and simpler(?) than making it thread safe. Clint ----- On Jan 5, 2018, at 12:34 PM, Sebastian Holtermann seblist at xwmw.org wrote: > Hello! > > As you might have noticed I tried to parallelize AUTOMOC/UIC. > https://gitlab.kitware.com/cmake/cmake/merge_requests/1632 > The issue that's blocking it now is that the kwsysProcess framework > isn't thread safe. As a consequence it is not possible for threads to > start processes concurrently using the kwsysProcess framework. > > There are three options I see > > 1) Make kwsysProcess thread safe > > I don't think it's impossible but it's not a small task. > Looking through the code I found a few issues that > would block the whole idea for me. > - OpenVMS: I have little to no knowledge about OpenVMS. > Wikipedia says it is basically abandoned. > Can the OpenVMS code be removed? I couldn't test it anyway. > > - C vs. C++ > There's a lot of C in processUNIX.c (list allocation etc.). > I would prefer to use C++11 (or higher), especially > std::array, std::vector, std::thread, std::mutex, etc.. > Is C++11 acceptable in kwsysProcess? > This would make processUNIX.c processUNIX.cpp? > > 2) Use libuv instead > > Using the libuv event loop is overkill and would probably imply that > more threads are started than necessary > (the libuv thread pool will be started anyway). > But it would allow to start multiple concurrent (moc/uic) processes. > Is there a reason to not use libuv? > > 3) Abandon the idea to parallelize AUTOMOC/UIC > > Well, I liked the idea. > > > Any thoughts? > > -Sebastian > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > https://cmake.org/mailman/listinfo/cmake-developers From ben.boeckel at kitware.com Fri Jan 5 16:47:25 2018 From: ben.boeckel at kitware.com (Ben Boeckel) Date: Fri, 5 Jan 2018 16:47:25 -0500 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <4776173.KzqDUZkdIB@blauwal> References: <4776173.KzqDUZkdIB@blauwal> Message-ID: <20180105214725.GA17149@megas.kitware.com> On Fri, Jan 05, 2018 at 20:34:53 +0100, Sebastian Holtermann wrote: > 2) Use libuv instead libuv for process management is on the list. I think it is waiting for porting to all of CMake's platforms to actually happen. I'm seeing this PR hung up on process: https://github.com/libuv/libuv/pull/1527 I don't know how much followup work there is. I thought there was a CMake issue tracking it, but I don't see one. --Ben From seblist at xwmw.org Sat Jan 6 02:31:26 2018 From: seblist at xwmw.org (Sebastian Holtermann) Date: Sat, 06 Jan 2018 08:31:26 +0100 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <20180105214725.GA17149@megas.kitware.com> References: <4776173.KzqDUZkdIB@blauwal> <20180105214725.GA17149@megas.kitware.com> Message-ID: <8020824.cVAZQE1hnl@blauwal> > > 2) Use libuv instead I just saw the libuv library in the CMake sources. > libuv for process management is on the list. I think it is waiting for > porting to all of CMake's platforms to actually happen. Does that mean it's only there for some specific scenarios and shouldn't be used in general? > I'm seeing this PR hung up on process: > > https://github.com/libuv/libuv/pull/1527 > > I don't know how much followup work there is. I thought there was a > CMake issue tracking it, but I don't see one. IIUC this is about the CPU affinity for child processes? Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child processes are running and to get a notification when one has finished. I think that this level of control is available in libuv already. Sebastian From jeanmichael.celerier at gmail.com Mon Jan 8 04:32:22 2018 From: jeanmichael.celerier at gmail.com (=?UTF-8?Q?Jean=2DMicha=C3=ABl_Celerier?=) Date: Mon, 8 Jan 2018 10:32:22 +0100 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <4776173.KzqDUZkdIB@blauwal> References: <4776173.KzqDUZkdIB@blauwal> Message-ID: > 3) Abandon the idea to parallelize AUTOMOC/UIC please no ! moc is a huge bottleneck in my builds (to the point that using verdigris instead ends up having faster build time overall). Best, Jean-Micha?l ------- Jean-Micha?l Celerier http://www.jcelerier.name On Fri, Jan 5, 2018 at 8:34 PM, Sebastian Holtermann wrote: > Hello! > > As you might have noticed I tried to parallelize AUTOMOC/UIC. > https://gitlab.kitware.com/cmake/cmake/merge_requests/1632 > The issue that's blocking it now is that the kwsysProcess framework > isn't thread safe. As a consequence it is not possible for threads to > start processes concurrently using the kwsysProcess framework. > > There are three options I see > > 1) Make kwsysProcess thread safe > > I don't think it's impossible but it's not a small task. > Looking through the code I found a few issues that > would block the whole idea for me. > - OpenVMS: I have little to no knowledge about OpenVMS. > Wikipedia says it is basically abandoned. > Can the OpenVMS code be removed? I couldn't test it anyway. > > - C vs. C++ > There's a lot of C in processUNIX.c (list allocation etc.). > I would prefer to use C++11 (or higher), especially > std::array, std::vector, std::thread, std::mutex, etc.. > Is C++11 acceptable in kwsysProcess? > This would make processUNIX.c processUNIX.cpp? > > 2) Use libuv instead > > Using the libuv event loop is overkill and would probably imply that > more threads are started than necessary > (the libuv thread pool will be started anyway). > But it would allow to start multiple concurrent (moc/uic) processes. > Is there a reason to not use libuv? > > 3) Abandon the idea to parallelize AUTOMOC/UIC > > Well, I liked the idea. > > > Any thoughts? > > -Sebastian > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at http://www.kitware.com/ > opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > https://cmake.org/mailman/listinfo/cmake-developers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brad.king at kitware.com Mon Jan 8 06:56:57 2018 From: brad.king at kitware.com (Brad King) Date: Mon, 8 Jan 2018 06:56:57 -0500 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <8020824.cVAZQE1hnl@blauwal> References: <4776173.KzqDUZkdIB@blauwal> <20180105214725.GA17149@megas.kitware.com> <8020824.cVAZQE1hnl@blauwal> Message-ID: On 01/06/2018 02:31 AM, Sebastian Holtermann wrote: >>> 2) Use libuv instead > > I just saw the libuv library in the CMake sources. > >> libuv for process management is on the list. I think it is waiting for >> porting to all of CMake's platforms to actually happen. > > Does that mean it's only there for some specific scenarios and shouldn't > be used in general? A goal is to eventually fully replace use of KWSys Process with libuv. I'd prefer not to make any major changes to KWSys Process in the meantime. Currently libuv is used only for the server mode, but soon it will be used for CTest too: https://gitlab.kitware.com/cmake/cmake/merge_requests/1455 > Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child > processes are running and to get a notification when one has finished. Use of libuv's process management for that would be good. It supports fully asynchronous operation and so can manage many child processes from one thread. Thanks, -Brad From seblist at xwmw.org Mon Jan 8 07:26:53 2018 From: seblist at xwmw.org (Sebastian Holtermann) Date: Mon, 08 Jan 2018 13:26:53 +0100 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: References: <4776173.KzqDUZkdIB@blauwal> <8020824.cVAZQE1hnl@blauwal> Message-ID: <4461098.1BX4KFueEi@blauwal> On Montag, 8. Januar 2018 06:56:57 CET Brad King wrote: > On 01/06/2018 02:31 AM, Sebastian Holtermann wrote: > >>> 2) Use libuv instead > > > > I just saw the libuv library in the CMake sources. > > > >> libuv for process management is on the list. I think it is waiting for > >> porting to all of CMake's platforms to actually happen. > > > > Does that mean it's only there for some specific scenarios and shouldn't > > be used in general? > > A goal is to eventually fully replace use of KWSys Process with libuv. > I'd prefer not to make any major changes to KWSys Process in the meantime. > > Currently libuv is used only for the server mode, but soon it will be > used for CTest too: > > https://gitlab.kitware.com/cmake/cmake/merge_requests/1455 > > > Looks nice, but in AUTOMOC/UIC it's sufficient to just known which child > > processes are running and to get a notification when one has finished. > > Use of libuv's process management for that would be good. It supports > fully asynchronous operation and so can manage many child processes from > one thread. Okay. Then I'm going to refactor cmQtAutoGeneratorMocUic to use a libuv event loop internally. -Sebastian From brad.king at kitware.com Mon Jan 8 07:33:12 2018 From: brad.king at kitware.com (Brad King) Date: Mon, 8 Jan 2018 07:33:12 -0500 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: <4461098.1BX4KFueEi@blauwal> References: <4776173.KzqDUZkdIB@blauwal> <8020824.cVAZQE1hnl@blauwal> <4461098.1BX4KFueEi@blauwal> Message-ID: <235154e2-93a7-969a-2b67-8004a9d8e0f8@kitware.com> On 01/08/2018 07:26 AM, Sebastian Holtermann wrote: > Then I'm going to refactor cmQtAutoGeneratorMocUic > to use a libuv event loop internally. Great. Take a look at Source/cmUVHandlePtr.h for some C++ helpers. -Brad From neundorf at kde.org Mon Jan 8 14:50:06 2018 From: neundorf at kde.org (Alexander Neundorf) Date: Mon, 08 Jan 2018 20:50:06 +0100 Subject: [cmake-developers] kwsysProcess threadsafety In-Reply-To: References: <4776173.KzqDUZkdIB@blauwal> Message-ID: <7358593.hyGOZflnW6@linux-l7nd> On 2018 M01 8, Mon 10:32:22 CET Jean-Micha?l Celerier wrote: > > 3) Abandon the idea to parallelize AUTOMOC/UIC > > please no ! moc is a huge bottleneck in my builds (to the point that using > verdigris instead ends up having > faster build time overall). you could also just not use automoc and use the "manual" moc-wrapper macros. This should make the build-time overhead go away. Instead the developer has to add the header file to be moc'ed to the CMakeLists.txt (personally I consider this acceptable). Alex From craig.scott at crascit.com Tue Jan 9 05:30:05 2018 From: craig.scott at crascit.com (Craig Scott) Date: Tue, 9 Jan 2018 21:30:05 +1100 Subject: [cmake-developers] Optionally disabling static lib dependencies with COMPILE_DEPENDS In-Reply-To: References:

Message-ID: Perhaps keep an eye on the work going on in merge request 1524 , which may allow you to use object libraries in target_link_libraries(). Not sure if it would cut the dependencies in the way you are seeking, but maybe take a look and see. On Fri, Jan 5, 2018 at 8:08 PM, John Wordsworth wrote: > First of all - thanks for the suggestions. I intend to look into ccache > for our Linux builds in the future and there are some useful tips here for > me to look into. I wasn't aware of clcache and it looks very interesting, > especially combined with network sharing. > > I realise I wasn't very clear in my previous explanation. Hopefully this > paints a better picture; > > - We have 40 or so developers spread across Visual Studio 2015 (MSBuild - > Windows), Xcode (macOS) and Make/Clang (Linux). 80% use Windows. > - We have a core set of libs in use by a number of projects (~30 static > libs and ~10 different projects), but the projects mix and match which libs > they use - merging the static libs (beyond a few of them) isn't something > we want to do. > - I have recently been revising our CMake structure and have boiled it > down to a fairly simple "add_library(x), target_link_libraries(x, > default_compile_settings, dependencies)" which means compile settings, > include directories and macros propagate through the libs nicely. > - We sometimes want to build and use certain libs as shared libs instead > of static libs. > > With regards to the COMPILE_DEPENDS feature that I have mocked up and > started testing with our team; > > - If and only if COMPILE_DEPENDS is set for a STATIC library, then when > building the target dependencies for that library, it uses the list of > provided targets instead of those that would have been inferred from > previous calls to "target_link_libraries" or "add_dependencies". > > The reality is, the libs which are being built statically can nearly all > build in parallel with only one or two "real" dependencies. I understand > that our case is rather specific, but having implemented "COMPILE_DEPENDS" > and written ~10 lines of CMake in our project (hard coding the one or two > actual build order dependencies and disabling the rest), our compile graph > looks much nicer and saves a significant amount of time building on a > single machine (see https://www.johnwordsworth.com/temp/cmake_compile_ > depends.jpg). We see similar savings on macOS using Xcode but have not > tested on Linux yet. When we allow Incredibuild to distribute building of > compilation units across 15-20 agents, the proportional build time drops > more dramatically between the two (the same project goes from ~2.6mins -> > ~1.6mins). > > I understand that just "hard overriding" the target dependencies is a bit > messy, but I'd also be happy to explore other ways I could add a similar > feature to CMake if there is a potentially better way to do this. What I am > ideally looking for in a solution is that I can continue to propagate > include directories / compile settings / preprocessor macros through the > chain of libs and that it improves building static libs in parallel across > all our build systems (MSVC / XCode / make). Ideally, I wouldn't have to > restructure my CMake project too much either - as the draft I have for our > new structure feels super clean now and refactoring it just to remove build > order dependencies would be shame. If there are any alternative ideas for > how to implement this, I'd love to discuss. I'm no CMake expert, but maybe > if we could use "target_link_libraries" with OBJECT libraries to grab > compile settings, or perhaps I could look into a mechanism for not adding > build order dependencies on VS2015/Xcode if there is no need too? > > Thanks again for the feedback and discussion so far. > > // John W > > On Thu, Jan 4, 2018 at 12:16 PM, Eric Noulard > wrote: > >> >> 2018-01-04 10:48 GMT+01:00 Craig Scott : >> >>> >>> >>> On Thu, Jan 4, 2018 at 8:27 AM, John Wordsworth >> > wrote: >>> >>>> I have recently been reviewing ways to improve build times for our >>>> project, which is comprised of a number of static libraries. I stumbled >>>> across this post on the CMake tracker from 2012/13 ( >>>> https://cmake.org/Bug/view.php?id=13799). It suggests adding a >>>> COMPILE_DEPENDS target property to to explicitly set dependencies for >>>> STATIC libraries instead of always using all linked libraries as >>>> build-order dependencies. >>>> >>>> Having done a draft implementation in a local CMake repository it has >>>> shaved off 20% of our 120s build time. I expect the savings to be much >>>> more dramatic when I test with Incredibuild (approximately 50% based on >>>> tests done previously from just deleting dependencies manually in Visual >>>> Studio). >>>> >>> >> You said you tested with Incredibuild but with what kind of configuration? >> AFAIK Incredibuild takes its power from distributing the build? So does >> your CMake test uses something that distribute the build too ? >> >> >>> >>>> I don?t really want to refactor our code to use ?OBJECT? libraries as >>>> the inability to link with other targets means that propagating compile >>>> options / include directories etc down the chain of linked libs becomes >>>> painful. This method allows me to switch between static and shared libs >>>> using a config option and none of my CMake scripts need to change. >>>> >>> >>> There's a couple more choices here. If your project consists of lots of >>> small (static) libraries, consider whether you can combine some of them to >>> result in a smaller number of larger libraries. This isn't always a gain, >>> but in terms of ability to compile sources in parallel, it will often lead >>> to more efficient builds. You just need to be careful you don't end up with >>> so many objects being combined into one library that you start to hit max >>> open file limits during linking/archiving (something I've hit on multiple >>> platforms lately, so it's not just a hypothetical example). Use of >>> target_sources() can be quite helpful if you want to try out this path (you >>> may find this article >>> >>> helpful). >>> >>> Another choice is to go in the opposite direction and take advantage of >>> the optimisation made for the Ninja generator (if that's a choice open to >>> you) that was introduced in CMake 3.9.0 >>> where if no >>> custom commands exist on a target A, then compilation steps of another >>> target B linking to A are allowed to proceed without waiting for A's link >>> step to complete. Only B's link step will depend on A's link step. In your >>> project, if you have custom commands, see if you can split up that target >>> into just those sources that need the results of the custom command and >>> another target that doesn't. The latter will then be able to compile >>> earlier, so fewer sources have to wait for earlier linking steps. This >>> might be hard to do, it really depends on how your project is structured. >>> >> >> I fully agree with Craig here. We use CMake + Ninja and we get really >> efficient parallel build (on 10+ cores machines). >> This is on Linux though but it seems MS is putting effort to get CMake + >> ninja build work with Visual Studio: >> https://blogs.msdn.microsoft.com/vcblog/2017/05/10/cmake-sup >> port-in-visual-studio-whats-new-in-2017-15-3-update/ >> >> Another way to improve your build may be to review your dependency graph >> carefully. You can use cmake --grahviz options to dump the graph and see >> whether you can cut >> some dependencies due to transitivity of dependency. I have seen many >> projects with "overspecified deps". I think this impair build time when >> using a lot of static libs because in that case you may get >> poor performance link time. Namelly if you have many unit tests linking >> to too many static libs because of overspecified deps. >> >> That said you did not specify whether if the 120s build time is only for >> the library or if this is a "global" build time which includes linking >> somes executables and possible unit tests programs. >> >> Note that even if this is not the case having a "big-picture" look at the >> dependency build graph generated by CMake may be insightful. >> Note that with CMake 3.10 you can now see difference between PUBLIC, >> PRIVATE and INTERFACE deps: >> https://cmake.org/cmake/help/v3.10/release/3.10.html >> >> >> >>> >>> Both of the above choices allow you to retain the automatic propagation >>> of compile options, include directories, etc. and to switch between >>> shared/static easily, but the latter is specific to the Ninja generator and >>> may not be an acceptable change for you. >>> >>> >>>> >>>> Anyway, I was wondering whether there was any interest in me pushing my >>>> solution back to Git / submitting a Pull request so that it might be merged >>>> in at some point. If there is - any advice on any gotchas I might watch for >>>> instead of just adding some fairly simple code to >>>> cmComputeTargetDepends.cxx would be gratefully received - especially as >>>> this is my first time poking around in CMake code. >>>> >>> >>> The existing behaviour is conservative and any change would have to also >>> be conservative, meaning that it must not introduce any possibility of >>> breaking existing projects. If I'm understanding your proposed feature >>> correctly, it sounds like you want to relax the build-order dependencies by >>> default when a COMPILE_DEPENDS target property is defined. Basically, if >>> COMPILE_DEPENDS is defined, you are taking over responsibility for the >>> build-order dependencies. This would be something I'd usually discourage >>> projects from doing because such manual dependencies would be a prime >>> candidate for not being kept up to date as a project evolves, leading to >>> subtle, hard-to-trace build errors. Some judicious project restructuring >>> can normally give a pretty efficient parallel build without having to >>> resort to such measures, so I'm wary of adding a feature like this (though >>> I can understand the desire for it). >>> >>> In my experience, you can get some considerable speedups using tools >>> like ccache (and its equivalents for other platforms/compilers). These >>> obviously only help for subsequent builds of things that have been built >>> previously, but for everyday development where you switch between branches >>> or for CI servers doing lots of similar builds, the savings can be >>> impressively big. >>> >> >> +1 again. >> >> ccache boost our build. It can be more than 10x faster (depends on cache >> hit). >> You can even share (on a local network) the ccache directory between >> developers and the CI and get very high hit rate (> 80%). >> >> Now I'm not working on windows nor with Visual Studio and AFAIK ccache >> does not work with MSVC. >> There seem to be alternative, https://github.com/frerich/clcache, but as >> you guessed I did never used that. >> >> -- >> Eric >> > > -- Craig Scott Melbourne, Australia https://crascit.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From gjasny at googlemail.com Tue Jan 16 05:19:01 2018 From: gjasny at googlemail.com (Gregor Jasny) Date: Tue, 16 Jan 2018 11:19:01 +0100 Subject: [cmake-developers] iOS friendly CMake Message-ID: <9e9d5a26-03bf-2bf7-11a1-4a960b083e38@googlemail.com> Hi, I created a CMake branch where I attempt to provide an easy way to configure CMake projects for iOS (and all other Xcode SDKs). All you need to build for iOS is: cmake -DCMAKE_SYSTEM_NAME=Darwin -DCMAKE_OSX_SYSROOT=iphoneos -GXcode . In case you build code that must be code signed also add your development team: -DCMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM=ABCDEFG The merge request is here: https://gitlab.kitware.com/cmake/cmake/merge_requests/1664 Please give it a try and add any comments you have to the merge requests. I only tested it with Xcode 9 on High Sierra so far. Thanks, Gregor From eric.noulard at gmail.com Tue Jan 16 07:10:07 2018 From: eric.noulard at gmail.com (Eric Noulard) Date: Tue, 16 Jan 2018 13:10:07 +0100 Subject: [cmake-developers] Undocumented command line flags (-H, -B) why? Message-ID: Hi there, I came accross -H and -B undocumented command line flags for cmake: https://stackoverflow.com/questions/24460486/cmake-build-type-not-being-used-in-cmakelists-txt/24470998#24470998 Is there any reason to keep those undocumented? -- Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From cristian.adam at gmail.com Tue Jan 16 08:42:32 2018 From: cristian.adam at gmail.com (Cristian Adam) Date: Tue, 16 Jan 2018 14:42:32 +0100 Subject: [cmake-developers] Undocumented command line flags (-H, -B) why? In-Reply-To: References: Message-ID: There was an attempt to document them: https://www.mail-archive.com/cmake-developers at cmake.org/msg16693.html I don't know why it wasn't a pull request on gitlab though. Cheers, Cristian. On Tue, Jan 16, 2018 at 1:10 PM, Eric Noulard wrote: > Hi there, > > I came accross -H and -B undocumented command line flags for cmake: > https://stackoverflow.com/questions/24460486/cmake- > build-type-not-being-used-in-cmakelists-txt/24470998#24470998 > > Is there any reason to keep those undocumented? > > -- > Eric > > -- > > Powered by www.kitware.com > > Please keep messages on-topic and check the CMake FAQ at: > http://www.cmake.org/Wiki/CMake_FAQ > > Kitware offers various services to support the CMake community. For more > information on each offering, please visit: > > CMake Support: http://cmake.org/cmake/help/support.html > CMake Consulting: http://cmake.org/cmake/help/consulting.html > CMake Training Courses: http://cmake.org/cmake/help/training.html > > Visit other Kitware open-source projects at http://www.kitware.com/ > opensource/opensource.html > > Follow this link to subscribe/unsubscribe: > https://cmake.org/mailman/listinfo/cmake-developers > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.boeckel at kitware.com Tue Jan 16 08:46:21 2018 From: ben.boeckel at kitware.com (Ben Boeckel) Date: Tue, 16 Jan 2018 08:46:21 -0500 Subject: [cmake-developers] Undocumented command line flags (-H, -B) why? In-Reply-To: References: Message-ID: <20180116134621.GB10587@megas.kitware.com> On Tue, Jan 16, 2018 at 13:10:07 +0100, Eric Noulard wrote: > I came accross -H and -B undocumented command line flags for cmake: > https://stackoverflow.com/questions/24460486/cmake-build-type-not-being-used-in-cmakelists-txt/24470998#24470998 > > Is there any reason to keep those undocumented? They're internal and not meant for use outside of CMake itself. There have been threads on this before. Here's one with a patch to document it that was rejected as-is: https://www.mail-archive.com/cmake-developers at cmake.org/msg16693.html For those who want to submit a patch, there's a TODO list here: https://www.mail-archive.com/cmake-developers at cmake.org/msg16720.html --Ben From eric.noulard at gmail.com Tue Jan 16 09:14:37 2018 From: eric.noulard at gmail.com (Eric Noulard) Date: Tue, 16 Jan 2018 15:14:37 +0100 Subject: [cmake-developers] Undocumented command line flags (-H, -B) why? In-Reply-To: <20180116134621.GB10587@megas.kitware.com> References: <20180116134621.GB10587@megas.kitware.com> Message-ID: Thank you very much for the explanation. 2018-01-16 14:46 GMT+01:00 Ben Boeckel : > On Tue, Jan 16, 2018 at 13:10:07 +0100, Eric Noulard wrote: > > I came accross -H and -B undocumented command line flags for cmake: > > https://stackoverflow.com/questions/24460486/cmake- > build-type-not-being-used-in-cmakelists-txt/24470998#24470998 > > > > Is there any reason to keep those undocumented? > > They're internal and not meant for use outside of CMake itself. There > have been threads on this before. Here's one with a patch to document it > that was rejected as-is: > > https://www.mail-archive.com/cmake-developers at cmake.org/msg16693.html > > For those who want to submit a patch, there's a TODO list here: > > https://www.mail-archive.com/cmake-developers at cmake.org/msg16720.html > > --Ben > -- Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben.boeckel at kitware.com Tue Jan 16 09:36:04 2018 From: ben.boeckel at kitware.com (Ben Boeckel) Date: Tue, 16 Jan 2018 09:36:04 -0500 Subject: [cmake-developers] Undocumented command line flags (-H, -B) why? In-Reply-To: References:

Message-ID: <20180116143604.GA937@megas.kitware.com> On Tue, Jan 16, 2018 at 14:42:32 +0100, Cristian Adam wrote: > I don't know why it wasn't a pull request on gitlab though. > Thu, 30 Jun 2016 12:24:02 -0700 This predates Gitlab usage for patch submission (which started around July 2016). --Ben From Lenindarbi.Jayakumar at magna.com Wed Jan 17 03:52:54 2018 From: Lenindarbi.Jayakumar at magna.com (Jayakumar, Lenindarbi) Date: Wed, 17 Jan 2018 08:52:54 +0000 Subject: [cmake-developers] CMAKE :: mingw make files with response file for includes Message-ID: <869a0ca961ff4ba58ec22a3cf4c93493@FRAMSEX06.magna.global> Hello, I have encountered a problem with response file usage for includes. My project is using tasking compiler. Issue: Response file link flag "CMAKE_C_RESPONSE_FILE_LINK_FLAG" is not customizable for include paths. Cmake generated the *.rsp file always using @ but this is not accepted by tasking compiler. Tasking compiler expects -f instead of @. ... C_INCLUDES = @CMakeFiles/dummyTarget/includes_C.rsp This issue is only with includes response file. For objects, CMAKE_C_RESPONSE_FILE_LINK_FLAG works. Project setup: set(CMAKE_C_RESPONSE_FILE_LINK_FLAG "-f ") set(CMAKE_C_USE_RESPONSE_FILE_FOR_OBJECTS 1) set(CMAKE_C_USE_RESPONSE_FILE_FOR_INCLUDES 1) Possible cause: The @ symbol is hardcoded in "cmMakefileTargetGenerator::AddIncludeFlags" function in file https://gitlab.kitware.com/cmake/cmake/blob/master/Source/cmMakefileTargetGenerator.cxx. if (useResponseFile) { std::string name = "includes_"; name += lang; name += ".rsp"; std::string arg = "@" + this->CreateResponseFile(name.c_str(), includeFlags, this->FlagFileDepends[lang]); this->LocalGenerator->AppendFlags(flags, arg); } However, the implementation should be like in "cmMakefileTargetGenerator::CreateObjectLists" function. // Lookup the response file reference flag. std::string responseFlagVar = "CMAKE_"; responseFlagVar += this->GeneratorTarget->GetLinkerLanguage(this->ConfigName); responseFlagVar += "_RESPONSE_FILE_LINK_FLAG"; const char* responseFlag = this->Makefile->GetDefinition(responseFlagVar); if (!responseFlag) { responseFlag = "@"; } Regards, Lenindarbi Jayakumar Software Engineer Magna Electronics Europe GmbH & Co. OHG Kurf?rst-Eppstein-Ring 9 63877 Sailauf OFFICE +49 6093 9942 6462 FAX +49 6093 9942 6001 MOBILE Lenindarbi.Jayakumar at magna.com http://www.magna.com Magna Electronics Europe GmbH & Co. OHG Gesch?ftsf?hrende Gesellschaft: Magna Electronics Europe Verwaltungs GmbH Gesch?ftsf?hrer: Rainer Bruns, Cornelius Wolber Sitz der gesch?ftsf?hrenden Gesellschaft: Sailauf, Deutschland Registergericht: Aschaffenburg / HRB 8778 Sitz der Gesellschaft (OHG): 63877 Sailauf, Deutschland Registergericht: Aschaffenburg / HRA 4616 Diese Nachricht ist ausschlie?lich f?r den oben bezeichneten Adressaten bestimmt und enth?lt m?glicherweise vertrauliche Informationen. Sollten Sie nicht der oben bezeichnete Adressat sein oder diese Nachricht irrt?mlich erhalten haben, ersuchen wir Sie, diese Nachricht nicht weiterzugeben, zu kopieren oder im Vertrauen darauf zu handeln, sondern uns unter Lenindarbi.Jayakumar at magna.com zu verst?ndigen und diese Nachricht samt allf?lliger Anlagen sofort zu l?schen. Vielen Dank. This message is intended only for use by the named addressee and may contain privileged and/or confidential information. If you are not the named addressee, you should not disseminate, copy, or take any action in reliance on it. If you have received this message in error, please immediately notify Lenindarbi.Jayakumar at magna.com and delete this message and any attachment. Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From claus.klein at arcormail.de Wed Jan 17 16:14:32 2018 From: claus.klein at arcormail.de (Claus Klein) Date: Wed, 17 Jan 2018 22:14:32 +0100 Subject: [cmake-developers] Cannot control the name of dependfile set in Ninja generator Message-ID: <2D71D7E4-F520-4287-9926-AF736B5E7290@arcormail.de> Hi all, there was mail with this issues a view years ago: https://cmake.org/pipermail/cmake/2014-October/058963.html quote: ????????????????????????????????????????????- Ninja generator sets the name to objectpath.d cmGlobalNinjaGenerator::EncodeDepfileSpace(objectFileName + ".d"); WindRiver sets the dep file name to objectDir/SourceName.d, so the ".obj" in "DEP_FILE = path/BaseName.c.obj.d" must be removed Armcc just uses base name in the same directory as the command runs, "BaseName.d". -depend_dir= can be used to control the directory for C but not asm. The workaround is to patch the generated build.ninja A proper solution is using new variables similar to above. (Just examples, not figured out etc) WindRiver (not added to CMake): SET (CMAKE___DEPFILE_PATTERN /.d ) Armcc (RVDS patch exists for CMake): SET (CMAKE_DEPFILE_FLAGS_C -depend_dir= --depend_format=unix) SET (CMAKE_C_ARMCC_DEPFILE_PATTERN /.d ) SET (CMAKE_DEPFILE_FLAGS_ASM --depend_format=unix) SET (CMAKE_ASM_ARMCC_DEPFILE_PATTERN .d ) Note: