MantisBT - CMake
View Issue Details
IDProjectCategoryView StatusDate SubmittedLast Update
0014807CMakeCMakepublic2014-03-14 05:052014-10-06 10:32
ReporterJohn Szakmeister 
Assigned ToBrad King 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionCMake 3.1Fixed in VersionCMake 3.1 
Summary0014807: file(DOWNLOAD) progress goes wild
DescriptionWe're using ExternalProject_Add to help build dependencies, but it can lead to nearly 3,000 lines of progress output as seen here: https://travis-ci.org/neovim/neovim/jobs/20601268#L826 [^]

It'd be nice if CMake provided a way to disable that output. Plus, the output is bogus. The first output was 100%, then 0%, then 739%, and ending at 3069066%.

Finally, this issue would've been easy to work around had the download, verify, and extract steps been exposed separately. Since they're not, it a substantial amount of work, just to avoid seeing 3,000 lines of progress. :-(
Steps To Reproduce
Additional Information
TagsNo tags attached.
Relationships
Attached Files
Issue History
Date ModifiedUsernameFieldChange
2014-03-14 05:05John SzakmeisterNew Issue
2014-03-14 08:59Brad KingNote Added: 0035402
2014-03-14 08:59Brad KingSummaryExternalProject_Add needs a way to silence progress output (and progress output is bogus) => file(DOWNLOAD) progress goes wild
2014-03-14 09:36John SzakmeisterNote Added: 0035405
2014-03-14 09:39Brad KingNote Added: 0035406
2014-03-14 15:20John SzakmeisterNote Added: 0035412
2014-03-17 11:10Brad KingNote Added: 0035419
2014-03-17 11:31Brad KingNote Added: 0035420
2014-03-17 11:31Brad KingAssigned To => Brad King
2014-03-17 11:31Brad KingStatusnew => assigned
2014-03-17 11:31Brad KingTarget Version => CMake 3.1
2014-03-17 11:34John SzakmeisterNote Added: 0035421
2014-03-17 11:38Brad KingNote Added: 0035422
2014-03-20 09:27Brad KingStatusassigned => resolved
2014-03-20 09:27Brad KingResolutionopen => fixed
2014-03-20 09:27Brad KingFixed in Version => CMake 3.1
2014-10-06 10:32Robert MaynardNote Added: 0036947
2014-10-06 10:32Robert MaynardStatusresolved => closed

Notes
(0035402)
Brad King   
2014-03-14 08:59   
Those messages come from the file(DOWNLOAD) command. See the UpdatePercentage method here:

 http://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/cmFileCommand.cxx;hb=v3.0.0-rc1#l2535 [^]

and the call to it here:

 http://cmake.org/gitweb?p=cmake.git;a=blob;f=Source/cmFileCommand.cxx;hb=v3.0.0-rc1#l2569 [^]

The callbacks and numbers come from curl. Normally it only calls back a few times and gets to 100% so it is not supposed to be so verbose. I would be interested in knowing why it is saying more data have been downloaded than the total. That is likely the underlying problem.
(0035405)
John Szakmeister   
2014-03-14 09:36   
Perhaps the bundled curl needs to be updated? The release notes for 7.22.0 (http://curl.haxx.se/changes.html#7_22_0 [^]) have an interesting entry:

    * progress: reset flags at transfer start to avoid wrong CURLINFO_CONTENT_LENGTH_DOWNLOAD

It looks like 7.20.0 fixed some issues too:

    * progress meter percentage and transfer time estimates fixes
    * progress callback called repeatedly during slow connects
(0035406)
Brad King   
2014-03-14 09:39   
Re 0014807:0035405: We support building against a system-installed curl. Can you try doing that with a newer curl to see if it resolves the issue? If so then I'll look at updating the builtin curl too.
(0035412)
John Szakmeister   
2014-03-14 15:20   
Brad, I just checked and the Ubuntu 12.04 CMake is being built against the system curl, which is at 7.22.0 (plus a couple other patches). So, it's not likely either one of the fixes I mention above--though there are other fixes in later releases of curl for progress-related bits.

It's worth noting that the frequency of updates changed in the latest curl, and probably not for the better when using Travis. So having an option to disable the progress updates with ExternalProject_Add is still desirable.
(0035419)
Brad King   
2014-03-17 11:10   
Re 0014807:0035412: It looks like we already round progress to the nearest integer percentage and report only when that changes. That should limit progress to at most 101 lines. However, when progress goes beyond 100% then there is no limit. Here is a fix for that:

 file: Avoid runaway DOWNLOAD/UPLOAD progress reports
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=413282c0 [^]
(0035420)
Brad King   
2014-03-17 11:31   
I've added a DOWNLOAD_NO_PROGRESS option to ExternalProject_Add:

 ExternalProject: Add option to disable download progress
 http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=7d35b550 [^]
(0035421)
John Szakmeister   
2014-03-17 11:34   
Thank you Brad!
(0035422)
Brad King   
2014-03-17 11:38   
Re 0014807:0035421: The remaining problem is to figure out why Curl provides the wrong numbers in the first place. I tried file(DOWNLOAD) on https://github.com/joyent/libuv/archive/v0.11.21.tar.gz [^] locally and it gets expected progress reports. This is with a system-installed curl 7.34.0. Perhaps it is a bug that has been fixed in Curl since 7.22.0?
(0036947)
Robert Maynard   
2014-10-06 10:32   
Closing resolved issues that have not been updated in more than 4 months.