MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0012957CMakeCMakepublic2012-02-12 09:582012-09-03 16:02
ReporterAndras Lasso 
Assigned ToBrad King 
PrioritynormalSeveritymajorReproducibilityrandom
StatusclosedResolutionfixed 
PlatformOSOS VersionWin7 (64bit)
Product VersionCMake 2.8.7 
Target VersionCMake 2.8.8Fixed in VersionCMake 2.8.8 
Summary0012957: Compiler tests randomly fail on Win7
DescriptionThe compiler tests during configure step randomly fail on Windows7. The problems are reprodusible with older CMake versions (CMake 2.8.6, 2.8.5, and 2.8.4 - I haven't tried with any older).

The failure may happen in any phase of the process:

=> change dir:

CMake Error at c:/Program Files (x86)/CMake 2.8/share/cmake-2.8/Modules/CMakeTestCCompiler.cmake:52 (MESSAGE):
  The C compiler "cl" is not able to compile a simple test program.
  It fails with the following output:
   Change Dir: C:/Users/lasso/devel/Slicer4-cmake2873/cmcurl-build/CMakeFiles/CMakeTmp

=> linking (cannot open the exe or pdb):

  Microsoft (R) Visual Studio Version 9.0.21022.8.
  Copyright (C) Microsoft Corp. All rights reserved.
  1>------ Build started: Project: cmTryCompileExec, Configuration: Debug
  Win32 ------
  1>Compiling...
  1>Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 15.00.21022.08 for
  80x86
  1>Copyright (C) Microsoft Corporation. All rights reserved.
  1>cl /Od /D "WIN32" /D "_WINDOWS" /D "_DEBUG" /D "CMAKE_INTDIR=\"Debug\""
  /D "_MBCS" /FD /RTC1 /MDd /Fo"cmTryCompileExec.dir\Debug\\"
  /Fd"C:\Users\lasso\devel\Slicer4-cmake2873\cmcurl-build\CMakeFiles\CMakeTmp\Debug/cmTryCompileExec.pdb"
  /W3 /c /Zi /TC /Zm1000
  1> ".\testCCompiler.c"
  1>testCCompiler.c
  1>Compiling manifest to resources...
  1>Microsoft (R) Windows (R) Resource Compiler Version 6.0.5724.0
  1>Copyright (C) Microsoft Corporation. All rights reserved.
  1>Linking...
  1>LINK : fatal error LNK1104: cannot open file
  'C:\Users\lasso\devel\Slicer4-cmake2873\cmcurl-build\CMakeFiles\CMakeTmp\Debug\cmTryCompileExec.exe'
  Run Build Command:c:\PROGRA~2\MICROS~1.0\Common7\IDE\devenv.com
  CMAKE_TRY_COMPILE.sln /build Debug /project cmTryCompileExec

Although it's not an acceptable workaround, I've turned of the antivirus and file indexing features and it seemed to reduce the occurrence of the error, but it kept happening. It seems that if anything runs in parallel on the computer then it increases the chance of error. It happens on several computers (desktop, laptop), in different institutions (it's not likely to be caused by some specific local software or malware infection; it could still be caused by utilities that are commonly found on developer's computers). I think it's the same issue as the one kstrike155 reported in http://cmake.org/Bug/view.php?id=12410. [^]

The problem is particularly severe when building complex projects (with several external projects), such as Slicer, because:
-several external projects build run in parallel (therefore the chance of failure is high),
-tens of projects are built (therefore it is quite likely that one of them fails), and
-if there is an error in configuration phase then even a rebuild of the project doesn't solve the problem (because CMakeCache contains the result of some incorrect compiler test).

In a superbuild environment the workaround is quite complex: identiy the failed project, delete the CMakeCache.txt of that project, reconfigure the project with superbuild (not by manually running CMake, because then some CMake variables may not be set correctly), rebuild.
Steps To ReproduceRepeat the following steps many times:
1. Delete the CMakeCache.txt (del cmakecache.txt)
2. Run a cmake to configure a project ("c:\Program Files (x86)\CMake 2.8\bin\cmake.exe" ../cmcurl)

Do some other work in parallel to make the failure happen sooner. If there is intensive work in the background then the failure rate is about 10-20%.
Additional Information
TagsNo tags attached.
Relationships
related to 0012410closed Bill Hoffman CMake randomly fails to detect Visual Studio 2010 C/CXX compiler since CMake 2.8.5. rc3 
related to 0013160closed Brad King try_compile leaves temporary directories 
Attached Filespatch 16b1a6e4+0001-try_compile-Use-random-executable-file-name-12957.patch (2,316) 2012-02-13 10:07
https://public.kitware.com/Bug/file/4217/16b1a6e4%2B0001-try_compile-Use-random-executable-file-name-12957.patch
patch cmCoreTryCompile-2.8.7-rev02.patch (1,399) 2012-02-16 09:30
https://public.kitware.com/Bug/file/4223/cmCoreTryCompile-2.8.7-rev02.patch
patch 0002-try_compile-Use-random-executable-file-name-12957.patch (2,282) 2012-02-16 10:15
https://public.kitware.com/Bug/file/4224/0002-try_compile-Use-random-executable-file-name-12957.patch
Issue History
Date ModifiedUsernameFieldChange
2012-02-12 09:58Andras LassoNew Issue
2012-02-12 10:18Andras LassoNote Added: 0028563
2012-02-13 08:39Bill HoffmanNote Added: 0028567
2012-02-13 09:06Brad KingNote Added: 0028570
2012-02-13 10:07Brad KingFile Added: 16b1a6e4+0001-try_compile-Use-random-executable-file-name-12957.patch
2012-02-13 10:09Brad KingNote Added: 0028572
2012-02-13 12:43Bill HoffmanRelationship addedrelated to 0012410
2012-02-14 00:42Andras LassoNote Added: 0028582
2012-02-15 16:28Andras LassoNote Added: 0028597
2012-02-15 16:38Andras LassoNote Added: 0028598
2012-02-15 16:50Brad KingNote Added: 0028599
2012-02-15 17:08Andras LassoNote Edited: 0028597bug_revision_view_page.php?bugnote_id=28597#r540
2012-02-15 17:38Andras LassoNote Added: 0028600
2012-02-15 19:21Andras LassoNote Added: 0028601
2012-02-16 09:30Andras LassoFile Added: cmCoreTryCompile-2.8.7-rev02.patch
2012-02-16 09:42Andras LassoNote Added: 0028604
2012-02-16 10:15Brad KingFile Added: 0002-try_compile-Use-random-executable-file-name-12957.patch
2012-02-16 10:17Brad KingNote Added: 0028606
2012-02-16 10:47Andras LassoNote Added: 0028607
2012-02-17 10:10Andras LassoNote Added: 0028614
2012-02-17 11:57Brad KingAssigned To => Brad King
2012-02-17 11:57Brad KingStatusnew => assigned
2012-02-17 11:58Brad KingNote Added: 0028618
2012-02-17 11:58Brad KingStatusassigned => resolved
2012-02-17 11:58Brad KingFixed in Version => CMake 2.8.8
2012-02-17 11:58Brad KingResolutionopen => fixed
2012-04-19 15:50David ColeTarget Version => CMake 2.8.8
2012-04-23 10:18Brad KingRelationship addedrelated to 0013160
2012-09-03 16:02David ColeNote Added: 0030894
2012-09-03 16:02David ColeStatusresolved => closed

Notes
(0028563)
Andras Lasso   
2012-02-12 10:18   
Example during Slicer4 build:

8>1>c:\users\lasso\devel\slicer4-bin\cmcurl\urldata.h(585) : error C2061: syntax error : identifier 'ssize_t'

Caused by these failed tests:
8>-- Check size of long
8>-- Check size of long - failed
8>-- Check size of __int64
8>-- Check size of __int64 - failed

Normally (90% of the cases) you get:
8>-- Check size of long
8>-- Check size of long - done
8>-- Check size of __int64
8>-- Check size of __int64 - done
(0028567)
Bill Hoffman   
2012-02-13 08:39   
This is a really hard problem. I just ran the curl test 10 times on my windows 7 box with no failure. I am not sure what does it but SOME win 7 machines seem to have this issue, and some do not. This sort of thing started showing up with vista and never totally went away. For some reason some machines are much worst than others.
(0028570)
Brad King   
2012-02-13 09:06   
I do lots of development on Win7 64-bit (with an AV tool) and have never seen this problem. I have VS 6, 7.1, 8, 9, 10 installed and they all work. Sometimes I run the entire CMake test suite in parallel on 3 of the IDEs at once.

I run as a "Standard" user account that is not in the power users or administrators group. Under what kind of account are you running Andras?
(0028572)
Brad King   
2012-02-13 10:09   
Please build CMake from source after applying attached patch

  16b1a6e4+0001-try_compile-Use-random-executable-file-name-12957.patch

It teaches try_compile to use a different file name every time to avoid conflict when the filesystem hangs on to an old file for any reason.
(0028582)
Andras Lasso   
2012-02-14 00:42   
Thanks for all the replies.

I am a local admin on my computers.

To reproduce the problem you can try deleting cmakecache & running cmake continuously while doing CPU and I/O intensive work.

I'll apply the patch and let you know if it helped.
(0028597)
Andras Lasso   
2012-02-15 16:28   
(edited on: 2012-02-15 17:08)
Unfortunately, after applying the 16b1a6e4+0001-try_compile-Use-random-executable-file-name-12957.patch to CMake-2.8.7 the problem is still reproducible:

14>47>c:\users\lasso\devel\slicer4-cm287p-bin\vtk\common\vtkDataArrayTemplate.txx(837) : error C2491: 'vtkDataArrayTemplate<T>::SetComponent' : definition of dllimport function not allowed

Caused by these failed tests:
-- Checking support for full template specialization syntax
-- Checking support for full template specialization syntax -- yes
-- Checking support for C++ explicit template instantiation
-- Checking support for C++ explicit template instantiation -- no
 
Normally you get:
-- Checking support for full template specialization syntax
-- Checking support for full template specialization syntax -- yes
-- Checking support for C++ explicit template instantiation
-- Checking support for C++ explicit template instantiation -- yes


CMakeError.log:

-----

Determining if the C++ compiler supports explict template instantiation failed with the following output:
Change Dir: C:/Users/lasso/devel/Slicer4-cm287p2-bin/VTK-build/CMakeTmp/TestExplicitInstantiation/Build

Run Build Command:c:\PROGRA~2\MICROS~1.0\Common7\IDE\devenv.com EXPLICIT.sln /build Debug /project cmTryCompileExec


Microsoft (R) Visual Studio Version 9.0.21022.8.

Copyright (C) Microsoft Corp. All rights reserved.



Invalid project

-----

=> so, it seems that the patch does not change the project name for this special case (EXPLICIT.sln), as normally you get something like this (not the project name: "cmTryCompileExec3309943481"):

Determining if the SO_REUSEADDR exist failed with the following output:
Change Dir: C:/Users/lasso/devel/Slicer4-cm287p2-bin/VTK-build/CMakeFiles/CMakeTmp

Run Build Command:c:\PROGRA~2\MICROS~1.0\Common7\IDE\devenv.com CMAKE_TRY_COMPILE.sln /build Debug /project cmTryCompileExec3309943481
(0028598)
Andras Lasso   
2012-02-15 16:38   
Today I've got a somewhat different pattern as well (during Slicer4 build):

19>------ Build started: Project: CTK, Configuration: Release Win32 ------
...
19>-- Looking for Q_WS_MAC - not found.
19>-- Found Qt4: C:/Qt/4.7.3/bin/qmake.exe (found version "4.7.3")
19>CMake Error at CMakeLists.txt:631 (message):
19> Failed to compile DGraph application.
19> Change Dir:
19> C:/Users/lasso/devel/Slicer4-cm287p-bin/CTK-build/Utilities/DGraph
19>
19> Run Build Command:c:\PROGRA~2\MICROS~1.0\Common7\IDE\devenv.com DGraph.sln
19> /build Debug /project cmTryCompileExec
19>
19> Microsoft (R) Visual Studio Version 9.0.21022.8.
19> Copyright (C) Microsoft Corp. All rights reserved.
19>
19> Invalid project
19>
19> Use:
19> devenv [solutionfile | projectfile | anyfile.ext] [switches]
19>
19> The first argument for devenv is usually a solution file or project file.
19> You can also use any other file as the first argument if you want to have
...
19> instead of IDE paths for VC++ builds.
19>
19> To attach the debugger from the command line, use:
19> VsJITDebugger.exe -p <pid>
19>-- Configuring incomplete, errors occurred!
19>Project : error PRJ0019: A tool returned an error code from "Performing configure step for 'CTK'"
19>Build log was saved at "file://c:\Users\lasso\devel\Slicer4-cm287p-bin\CTK.dir\Release\BuildLog.htm" [^]
19>CTK - 1 error(s), 0 warning(s)


The directory (C:/Users/lasso/devel/Slicer4-cm287p-bin/CTK-build/Utilities/DGraph) and the solution file (c:\Users\lasso\devel\Slicer4-cm287p-bin\CTK-build\Utilities\DGraph\DGraph.sln) does exist on the computer and the sln file can be built without errors. However, due to some reason somehow the build from CMake failed.
(0028599)
Brad King   
2012-02-15 16:50   
Re 0012957:0028597: If the patch did not solve the problem then I do not think this is due to rapid creation and deletion of the same executable.

Please try creating a "Standard" user account your Win7 machine for testing purposes. Can you reproduce it under said account?
(0028600)
Andras Lasso   
2012-02-15 17:38   
In both error cases the failed project was called "cmTryCompileExec" without the postfixes that the patch supposed to add. So maybe the patch could help, but there are some special cases when still not modifies the cmTryCompileExec name or having a different name than cmTryCompileExec might cause some problems.

I'll do some more testing of the patch and then try to reproduce the problem with "Standard" user account.
(0028601)
Andras Lasso   
2012-02-15 19:21   
Maybe the patch fixes the random cmTryCompileExec problem. Now the problem is that the build always fails during the vtkTestExplicitInstantiation.cmake test. It seems that this test is not compatible with the patch, as "Run Build Command:c:\PROGRA~2\MICROS~1.0\Common7\IDE\devenv.com EXPLICIT.sln /build Debug /project cmTryCompileExec" always fails with "Invalid project" (as described in comment 0028597).

Brad, could advise how to modify vtkTestExplicitInstantiation.cmake script or make the CMake patch compatible with this script?
(0028604)
Andras Lasso   
2012-02-16 09:42   
I have good news!

With some small changes I've fixed the patch (see cmCoreTryCompile-2.8.7-rev02.patch). The problem was that the default targetName was set to "cmTryCompileExec" instead of NULL; this way when the user did not specify targetName then the "cmTryCompileExec" target attempted to be built instead of "ALL_BUILD". I've kept the randomization, but set default targetName to NULL, as it was before.

Most importantly: it seems that the randomization solved the random configuration failures. I've created an intensive test scenario (two parallel Slicer4 builds running in parallel with a configuration step continuously repeated for the cmcurl project). Without the patch the configuration problem occurred 4 out of 4 cases. With the patched CMake the configuration problem occurred 0 out of 4 cases. I'll repeat the test 6 more times with both the patched and the unpatched version to confirm the findings.
(0028606)
Brad King   
2012-02-16 10:17   
Re 0012957:0028604: Great! I forgot when writing the patch that keeping targetName=0 in some code paths is necessary. OTOH in your patch the buffer goes out of scope while targetName still points at it. I fixed my original patch. Please try "0002-try_compile-Use-random-executable-file-name-12957.patch".
(0028607)
Andras Lasso   
2012-02-16 10:47   
OK, thanks! I'll do all the further testing with 0002-try_compile-Use-random-executable-file-name-12957.patch and let you know the statistics.
(0028614)
Andras Lasso   
2012-02-17 10:10   
I've completed 20 full Slicer4 builds (under heavy parallel load, on Win7 64-bit). The results:
* Without the patch: 1 successful, 9 failed
* With the patch: 10 successful, 0 failed

Based on this I'm confident that the patch fixes these random build problems and there doesn't seem to be any regressions.

Who would integrate the patch into the main branch? When we can expect to have a new release that includes this patch?
(0028618)
Brad King   
2012-02-17 11:58   
Re 0012957:0028614: Wonderful, thanks for testing!

I was just waiting for confirmation that it resolves the problem. I've merged the fix and expect it to be in 2.8.8 (to be released in the next couple months).

 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=4fbdce2b [^]
(0030894)
David Cole   
2012-09-03 16:02   
Closing resolved issues that have not been updated in more than 4 months.