On Fri, Sep 28, 2012 at 4:42 PM, Robert Dailey <span dir="ltr"><<a href="mailto:rcdailey.lists@gmail.com" target="_blank">rcdailey.lists@gmail.com</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On Fri, Sep 28, 2012 at 2:58 PM, David Cole <<a href="mailto:david.cole@kitware.com">david.cole@kitware.com</a>> wrote:<br>
> On Fri, Sep 28, 2012 at 3:30 PM, Robert Dailey <<a href="mailto:rcdailey.lists@gmail.com">rcdailey.lists@gmail.com</a>><br>
> wrote:<br>
>><br>
>> CMake downloads our third party libraries from a central repository<br>
>> and we have a "manifest.cmake" module where we define the following:<br>
>><br>
>> - Library alias (the library's base name, such as "boost", "bdb",<br>
>> "openssl")<br>
>> - Library version (e.g. 2.1.5)<br>
>> - Library iteration (A counter that is incremented if a library<br>
>> changes remotely without version # increasing (such as if we rebuild<br>
>> the same version of the library and it must be re-served))<br>
>><br>
>> My third party download logic knows to download the following files:<br>
>><br>
>> <repo>/<alias>/<version>/include.7z<br>
>> <repo>/<alias>/<version>/<platform>.7z<br>
>><br>
>> In this case, platform will represent the toolchain -- such as<br>
>> vc9sp1.7z for the lib & bin files for visual studio 2008 SP1.<br>
>><br>
>> I have 2 files here, so I'd need 2 MD5 values recorded in my manifest<br>
>> somewhere, but since I have 1 line per "library" (not per file that<br>
>> will be downloaded) it wouldn't work out very well.<br>
>><br>
>> I want to keep my manifest simple and easy to look at and modify,<br>
>> adding a bunch of MD5 values will make it messy and harder to upgrade<br>
>> libraries (right now I just drop files on a server and add or modify a<br>
>> line in the manifest. Having MD5s would mean I would have to run<br>
>> another tool to calculate the MD5 and then stuff it somewhere in the<br>
>> manifest module)<br>
>><br>
>> If you have some ideas on how to make this fit well into my system I'm<br>
>> all for that, but I guess if not then I'll have to rely on assumptions<br>
>> :(<br>
>><br>
>> However I strongly believe that CMake's file DOWNLOAD should do more<br>
>> checks to make sure that the data received is valid. I will look at<br>
>> the code later to see if there is more that can be done.<br>
>><br>
>> On Wed, Sep 26, 2012 at 11:20 PM, David Cole <<a href="mailto:david.cole@kitware.com">david.cole@kitware.com</a>><br>
>> wrote:<br>
>> > On Wed, Sep 26, 2012 at 7:32 PM, Robert Dailey<br>
>> > <<a href="mailto:rcdailey.lists@gmail.com">rcdailey.lists@gmail.com</a>> wrote:<br>
>> >> To do MD5 checks, I need to somehow record the expected MD5 somewhere,<br>
>> >> which isn't very maintainable.<br>
>> >><br>
>> >> I provide a list of third party libraries that CMake should download<br>
>> >> from a central third party repository here at work. It is a trusted<br>
>> >> source, because we know it is, so we don't need to verify the MD5.<br>
>> >> However, if I could request the MD5 first, and then download, then<br>
>> >> compare the MD5 the server gave me with what I actually downloaded,<br>
>> >> that would certainly work just to verify the complete file was<br>
>> >> downloaded.<br>
>> >><br>
>> >> Other than that, I'll have to rely on the status of the operation...<br>
>> >> but I don't like that the destination file is created prior to any<br>
>> >> writes being possible by CMake (it can't write anything if no data was<br>
>> >> received, so why doesn't it create the file once it has a write<br>
>> >> buffer?)<br>
>> >><br>
>> ><br>
>> > Recording the MD5 somewhere is the only way to have a reasonable<br>
>> > re-assurance that what you've asked for is what you're getting from a<br>
>> > network operation. It seems to me that it could be made "maintainable"<br>
>> > if you centralize the knowledge of the checksums in a file that is<br>
>> > changed whenever any of the downloadable files is changed.<br>
>> ><br>
>> > I guess we figure it's no use downloading bits over the network if you<br>
>> > can't even open a (presumably local) output file for writing... so we<br>
>> > try to open the output file for writing first, and if it succeeds,<br>
>> > then we start grabbing bits from the network and writing them into the<br>
>> > file as we receive them.<br>
>> ><br>
>> > There is room for improvement in the file(DOWNLOAD implementation, but<br>
>> > it is the way it is right now (and will be for 2.8.10 as well...)<br>
>> ><br>
>> > Start proposing improvements for it now, and submitting patches to<br>
>> > make stuff better for 2.8.11 and/or beyond. :-)<br>
>> ><br>
>> ><br>
>> > HTH,<br>
>> > David<br>
><br>
><br>
><br>
> You can rely on the STATUS to see if there were any errors during the<br>
> download. If the error code is 0, then you got whatever was on the server.<br>
> You can rely on that.<br>
><br>
> So, if you don't want to use a hash, you can rely on STATUS. I do not know<br>
> of any case that reports a "0" status code, but gives an incorrect file<br>
> result.<br>
><br>
><br>
> What you *can't* rely on is that the correct thing was on the server. And to<br>
> validate that, you should use checksums of some sort. (If you can't or don't<br>
> want to, that's fine. To each his own.) Starting with CMake 2.8.10, there<br>
> will be EXPECTED_HASH and you can use the hashing algorithm of your choice<br>
> rather than just the MD5 that we had in 2.8.9 and earlier...<br>
><br>
> Also new in 2.8.10, the Kitware provided pre-built binaries will link to<br>
> OpenSSL such that we can handle downloading files from "https://" URLs.<br>
<br>
</div></div>In my tests, I've found that redirects can affect the return code of<br>
STATUS. For example, if I try to initiate a download of a file that<br>
doesn't really exist, the HTTP server may return a "dummy" file, in<br>
that case it would be downloaded just fine no matter what the URL or<br>
filename is, and status wouldn't know the difference.<br>
<br>
However for FTP URLs, it is generally more honest (since HTTP can do<br>
funny things, like lie to you).<br>
</blockquote></div><div><br></div><br><div>What version of CMake are you using? Server side redirects should be followed properly since this commit:</div><div><br></div><div> <a href="http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=ef491f78218e255339278656bf6dc26073fef264">http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=ef491f78218e255339278656bf6dc26073fef264</a></div>
<div><br></div><div>Which has been in CMake for more than 2 years since 2.8.2...</div><div><br></div>