[CMake] file(DOWNLOAD) and HTTP 302 (redirect)

Michael Wild themiwi at gmail.com
Sat Jun 5 03:01:18 EDT 2010


Thanks a lot

Michael

PS: Any chance of getting built-in SSL support? Looks like the US is fairly liberal now w.r.t. to open source software:

http://old.cdt.org/crypto/admin/000112commercefactsheet.shtml
https://developer.mozilla.org/en/Mozilla_Crypto_FAQ

And the actual thing (I think 740.13(e) is the relevant part here): http://www.access.gpo.gov/bis/ear/pdf/740.pdf
As far as I understand it, you are not allowed to export crypto-software _knowingly_ to banned countries (group E:1 in the Supplement No. 1 to part 740 of the EAR). Making the software available on an internet-site or via FTP does not fall under the "knowing" export restriction.

Further you must notify BIS and the ENC Encryption Request Coordinator via e-mail (crypt at bis.doc.gov and enc at nsa.gov) of where the source-code is available online at the time (or before) you make it publicly available.

On 4. Jun, 2010, at 20:49 , David Cole wrote:

> Thanks, Michael and Aaron!
> 
> I pushed your patch (+ a SetError line) here in CMake 'next' :
> http://cmake.org/gitweb?p=cmake.git;a=commitdiff;h=ef491f78218e255339278656bf6dc26073fef264
> 
> 
> On Fri, Jun 4, 2010 at 11:38 AM, Michael Wild <themiwi at gmail.com> wrote:
> 
>> The following patch does work for me:
>> 
>> diff --git a/Source/cmFileCommand.cxx b/Source/cmFileCommand.cxx
>> index 5611527..3b18e0b 100644
>> --- a/Source/cmFileCommand.cxx
>> +++ b/Source/cmFileCommand.cxx
>> @@ -2629,6 +2629,14 @@
>> cmFileCommand::HandleDownloadCommand(std::vector<std::string>
>>    return false;
>>    }
>> 
>> +  res = ::curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
>> +  if (res != CURLE_OK)
>> +    {
>> +    std::string errstring = "FILE(DOWNLOAD ) error; cannot set
>> follow-redirect option: ";
>> +    errstring += ::curl_easy_strerror(res);
>> +    return false;
>> +    }
>> +
>>  if(verboseLog.size())
>>    {
>>    res = ::curl_easy_setopt(curl, CURLOPT_VERBOSE, 1);
>> --
>> 
>> However, I didn't test it thoroughly, and am not sure whether it would be a
>> good idea to restrict the protocols and number of redirects.
>> 
>> Michael
>> 
>> On 4. Jun, 2010, at 17:08 , <aaron.meadows at thomsonreuters.com> <
>> aaron.meadows at thomsonreuters.com> wrote:
>> 
>>> I don't know what options are compiled in for CMake, but it looks like
>>> this is the libcurl option required to follow redirects:
>>> 
>>> 
>>> 
>>> CURLOPT_FOLLOWLOCATION
>>> 
>>> 
>>> 
>>> A parameter set to 1 tells the library to follow any Location: header
>>> that the server sends as part of an HTTP header.
>>> 
>>> 
>>> 
>>> This means that the library will re-send the same request on the new
>>> location and follow new Location: headers all the way until no more such
>>> headers are returned. CURLOPT_MAXREDIRS can be used to limit the number
>>> of redirects libcurl will follow.
>>> 
>>> 
>>> 
>>> Since 7.19.4, libcurl can limit what protocols it will automatically
>>> follow. The accepted protocols are set with CURLOPT_REDIR_PROTOCOLS and
>>> it excludes the FILE protocol by default.
>>> 
>>> 
>>> 
>>> Aaron C. Meadows
>>> 
>>> ________________________________
>>> 
>>> From: cmake-bounces at cmake.org [mailto:cmake-bounces at cmake.org] On Behalf
>>> Of David Cole
>>> Sent: Friday, June 04, 2010 9:47 AM
>>> To: Michael Wild
>>> Cc: CMake List
>>> Subject: Re: [CMake] file(DOWNLOAD) and HTTP 302 (redirect)
>>> 
>>> 
>>> 
>>> On Fri, Jun 4, 2010 at 8:03 AM, Michael Wild <themiwi at gmail.com> wrote:
>>> 
>>> The log looks like this (I've put a message("log = ${log}") into
>>> ExternalProject.cmake to get this, since the numeric error code is 0):
>>> 
>>> <snip>
>>> -- downloading...
>>> 
>>> src='http://downloads.sourceforge.net/project/freefoam/ThirdParty/zlib/z
>>> lib-1.2.5.tar.gz'
>>> 
>>> dst='/Users/mwild/Projects/FreeFOAM/build-pu/ThirdParty/zlib/src/zlib-1.
>>> 2.5.tar.gz'
>>>   timeout='none'
>>> log = About to connect() to downloads.sourceforge.net port 80 (#0)
>>> Trying 216.34.181.59... connected
>>> Connected to downloads.sourceforge.net (216.34.181.59) port 80 (#0)
>>> GET /project/freefoam/ThirdParty/zlib/zlib-1.2.5.tar.gz HTTP/1.1
>>> Host: downloads.sourceforge.net
>>> Accept: */*
>>> 
>>> HTTP/1.1 302 Found
>>> X-Powered-By: PHP/5.2.9
>>> Content-Disposition: attachment; filename="zlib-1.2.5.tar.gz"
>>> Location:
>>> http://surfnet.dl.sourceforge.net/project/freefoam/ThirdParty/zlib/zlib-
>>> 1.2.5.tar.gz
>>> Content-type
>>> <http://surfnet.dl.sourceforge.net/project/freefoam/ThirdParty/zlib/zlib
>>> -1.2.5.tar.gz%0d%0aContent-type> : text/html
>>> Content-Length: 0
>>> Date: Fri, 04 Jun 2010 07:52:13 GMT
>>> Server: lighttpd/1.4.26
>>> 
>>> Connection #0 to host downloads.sourceforge.net left intact
>>> Closing connection #0
>>> 
>>> -- downloading... done
>>> </snip>
>>> 
>>> So you see, CMake really thinks this succeeded.
>>> 
>>> 
>>> 
>>> Hmm. I wonder if there's some sort of curl option we should be passing
>>> to get redirects to work automatically...?
>>> 
>>> 
>>> 
>>> 
>>>      I really am looking forward to the EXPECTED_MD5 feature, please
>>> also make it available through ExternalProject_Add! Currently I'm doing
>>> this manually via an additional step, but it's awfully fragile...
>>> 
>>> 
>>> 
>>> It is. That was actually the main motivation for the additional
>>> file(DOWNLOAD arg... In CMake 'next' there's a URL_MD5 arg to
>>> ExternalProject_Add now.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>      Michael
>>> 
>>> 
>>>      On 4. Jun, 2010, at 13:57 , David Cole wrote:
>>> 
>>>> I'm pretty sure the original intent was for file(DOWNLOAD ...)
>>> to not raise
>>>> errors so that scripts can analyze status and log and decide
>>> what to do next
>>>> rather than forcing a CMake error...
>>>> 
>>>> I recently added EXPECTED_MD5 to file(DOWNLOAD in the 'next'
>>> branch of
>>>> CMake. If a download fails when there's an expected MD5 sum
>>> given, then it
>>>> *does* raise an error. Perhaps you'll find that useful in the
>>> next
>>>> release...
>>>> 
>>>> Cheers,
>>>> David C.
>>>> 
>>>> 
>>>> On Fri, Jun 4, 2010 at 7:55 AM, David Cole
>>> <david.cole at kitware.com> wrote:
>>>> 
>>>>> What if you do something this...? What do status and log tell
>>> you?
>>>>> 
>>>>> set(remote "http://......")
>>>>> set(local "some file name")
>>>>> 
>>>>> file(DOWNLOAD
>>>>> 
>>>>> "${remote}"
>>>>> 
>>>>> "${local}"
>>>>> 
>>>>> STATUS status
>>>>> 
>>>>> LOG log)
>>>>> 
>>>>> 
>>>>> list(GET status 0 status_code)
>>>>> 
>>>>> list(GET status 1 status_string)
>>>>> 
>>>>> 
>>>>> if(NOT status_code EQUAL 0)
>>>>> 
>>>>> message(FATAL_ERROR "error: downloading '${remote}' failed
>>>>> 
>>>>> status_code: ${status_code}
>>>>> 
>>>>> status_string: ${status_string}
>>>>> 
>>>>> log: ${log}
>>>>> 
>>>>> ")
>>>>> 
>>>>> endif()
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jun 4, 2010 at 4:00 AM, Michael Wild
>>> <themiwi at gmail.com> wrote:
>>>>> 
>>>>>> When downloading something from
>>> http://downloads.sourceforge.net/. <http://downloads.sourceforge.net/>
>>> .. you
>>>>>> always get a 302 redirect to one of the many mirror servers.
>>> Unfortunately,
>>>>>> it seems that file(DOWNLOAD ...) doesn't follow the
>>> redirect, but just stops
>>>>>> and leaves an empty file behind. No error, no nothing. Is
>>> this intentional?
>>>>>> 
>>>>>> I'd rather not use a specific mirror...
>>>>>> 
>>>>>> Michael
>>>>>> _______________________________________________
>>>>>> Powered by www.kitware.com
>>>>>> 
>>>>>> Visit other Kitware open-source projects at
>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>> 
>>>>>> Please keep messages on-topic and check the CMake FAQ at:
>>>>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>>>> 
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://www.cmake.org/mailman/listinfo/cmake
>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> This email was sent to you by Thomson Reuters, the global news and
>> information company.
>>> Any views expressed in this message are those of the individual sender,
>> except where the sender specifically states them to be the views of Thomson
>> Reuters.
>>> 
>>> _______________________________________________
>>> Powered by www.kitware.com
>>> 
>>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>> 
>>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>> 
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.cmake.org/mailman/listinfo/cmake
>> 
>> 



More information about the CMake mailing list