[cmake-developers] Introduce 'Unix Ninja' generator

[Daniel Levin]

Brad, below is the list of variables set manually by me before testing QNX
SDK environment variables, the qcc will fail if any of them are missing:

set QNX_CONFIGURATION=d:\opt\qnx-software-systems
set QNXLM_LICENSE_FILE=@10.144.209.10
set QNX_HOST=d:\opt\qnx650sp1\host\win32\x86
set QNX_TARGET=d:\projects\pasa\ford\r6\QNX_SDK\target\qnx6
set PATH=d:\opt\qnx650sp1\host\win32\x86\usr\bin;%PATH%

> Do you mean some kind of dll injection to intercept CreateProcess calls?

I doubt, but that does not mean it does not. I remember errors from Ninja
when it was trying to execute "%1" in CreateProcess call, but suspect that
was a problem in escape characters in .ninja file.

I did short research again and would want to explain from scratch.

1. My story takes start from the existing QNX project based on Makefiles
and my effort was to inject CMake into that build tree. QNX SDK always uses
Unix environment which means:
- Makefiles always use Unix syntax
- environment guarantees that you can use shell scripts in your Makefiles
as commands

2. When you want to reuse existing .sh scripts in your CMake target rules
it apparently fails because Ninja does not know how to execute them. QNX
SDK make tool does.

3. The most problematic issue for me was that qcc and gcc tools from QNX
SDK cannot run without QNX_HOST and QNX_TARGET variables set. Also qcc will
fail if QNX_CONFIGURATION and QNXLM_LICENSE_FILE are missing or license
server is not available (but that is another story). CMake build works in
two stages: generation and native build. Setting listed env variables at
generation stage is absolutely valid. But setting them again later at build
stage leads to issues:
- you need to explicitly load environment variables each time before build
- if you have multiple QNX SDKs you might accidentally load wrong variable
set
- if you use IDE launched from clean shell with default variables you
cannot source env variables saved in some script anymore
- if you use IDE you probably need to explicitly replicate env variables in
some IDE specific way

CMake has solution to overcome problems like this: cache. But apparently
cache does not work when you need to save environment variables which need
to be loaded _before_ build starts.

So the only way to fix that was to wrap compiler into script, that does: a)
environment variable setup, b) launch real compiler, forwarding passed
arguments to it.

This wrapper script cannot be common, because for each kind of build env
variable set will be different. So it need to be created at CMake
generation time saving current QNX* env variables into this wrapper script.
So next time you can open clean shell and run 'make' or 'ninja' and it will
properly use environment variables defined previously at generation stage.

I implemented that in my toolchain file, redirecting CMAKE_C_COMPILER to my
gcc_wrapper.sh and that was working perfectly for 'Unix Makefiles'
generator. 'Ninja' generator was failing because it cannot execute .sh
files. Fine, lets replace CMAKE_C_COMPILER="gcc_wrapper.sh" with
CMAKE_C_COMPILER="/path/to/sh.exe gcc_wrapper.sh". But that does not work,
because CMAKE_C_COMPILER expects full path to executable, not command line.
I reworked wrapper as Windows batch file: gcc_wrapper.bat. Now it works
under 'Ninja' but fails under 'Unix Makefiles', because QNX SDK make
overrides .bat processing somehow.

Maybe there were another reasons, but at some point I decided that the most
appropriate way will be to stop playing with Windows environment. If QNX
SDK applications always expect Unix environment we should give it to them.
That is why I added 'Unix Ninja' generator that will always use sh as shell
tool.

Thanks,
Daniel


On Mon, Jan 19, 2015 at 2:50 PM, Brad King <brad.king at kitware.com> wrote:

> On 01/16/2015 01:13 PM, Daniel Levin wrote:
> > The CMake and Ninja were part of the bigger build script,
> > which was running under the QNX SDK sh.exe.
> > When running under this shell it overrides some environment
> > variables (compare attached files).
>
> The main differences I see are:
>
> * The Windows shell is a 64-bit cmd and the MSYS one is 32-bit
> * PATH has been converted to MSYS-style and prepended with /usr/bin
> * Added: QNX_HOST=c:\opt\qnx650-gcc-4.8.1\host\win32\x86
> * Added: QNX_TARGET=c:\opt\qnx650-gcc-4.8.1\target\qnx6
> * Added: TERM=cygwin, consistent with MSYS shells
> * Added: SHLVL=1
> * The "TEMP" and "TMP" variables are /tmp instead of the Windows dirs
>
> So this is a MSYS shell.  AFAIK ninja under MSYS sh.exe normally
> works even for pure Windows builds.  Perhaps their shell does more.
>
> > But as far as I remember it also does some nasty launcher overrides,
> > intercepting calls to cmd, bat and sh and tries to process them
> > somehow else.
>
> Do you mean some kind of dll injection to intercept CreateProcess calls?
>
> I'd really like to have a deep understanding of the situation before
> adding a lot of escaping code for a specialized environment.  It
> may be that ninja simply needs to learn how to isolate itself in
> this case.
>
> Thanks,
> -Brad
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20150120/c284735f/attachment-0001.html>