[cmake-developers] cmake -E tar not working correctly with xz-compressed tarballs on MSYS

[Alan W. Irwin]

On 2013-07-25 16:16-0700 Alan W. Irwin wrote:

> On 2013-07-25 15:31-0400 Brad King wrote:
>
>> On 07/24/2013 03:27 PM, Alan W. Irwin wrote:
>>> The first priority is to verify on Microsoft Windows that this is
>>> actually an issue using the the simple test I outlined in my previous
>>> post.
>> 
>> At a MSYS prompt under MS Windows this works:
>> 
>> $ cat a.tar.xz | unxz | tar x
>> 
>> but this hangs:
>> 
>> $ cmake -E tar xvf a.tar.xz
>> 
>> One can see in the process tree that there is a child process of
>> "cmake" called "unxz" that was started with no command line arguments.
>> One can attach a debugger to see CMake blocked inside libarchive code.
>> 
>> This confirms that the hang occurs on MS Windows just as in Wine.
>> I suggest getting upstream libarchive and building its sample
>> tools but without lzma support built-in.  Then try using that
>> binary to extract the .tar.xz file.
>
> Thanks, Brad, for that Microsoft Windows confirmation, and I will
> follow your advice for the rest to try and track down whether the
> problem is for the libarchive upstream version or the libarchive
> version within CMake.  With luck I should be able to come up with a
> fix as well (since drop-through to command-line handling of *.tar.xz
> should be completely straightforward) assuming the MSYS version of gdb
> works similarly to the Linux version (which I am familiar with).

Hi Brad:

Here is progress so far.  I used git to obtain the latest git version
of libarchive (v3.1.2-40-g4b5f651).  I built and tested the
result on Linux using

cmake ... -DENABLE_TEST:BOOL=ON ...
make VERBOSE=1 -j4 >& all.out
make VERBOSE=1 -j4 test >& test.out
make VERBOSE=1 -j4 install >& install.out

In some cases (e.g., lzma) I did not have the required development
version of the linux libraries installed so it (presumably) used the fallback
command-line tools instead.  All was well including 100 per cent
passage of the tests so this seems to be a pretty good version of
libarchive.

For MinGW-4.7.2/MSYS/Wine-1.6-rc4 I went through exactly the same
build and test procedure for "MSYS Makefiles".  In virtually all cases
(including lzma) it could not find the libraries so it (presumably)
used the fallback command-line tools instead.  The "all" and "install"
targets ran without issues, but 36 tests failed out of 400.  I don't
know how many of those are due to general issues for libarchive on the
MSYS platform and how many are due to Wine-1.6-rc4, but the track
record of Wine so far for my build and test experients is most/all of
those will be due to MSYS. I have attached a compressed tarball
containing the *.out files in case you would like to compare with your
own MSYS build.

>From the perspective of the current issue, the important result from the
above tests is that the lzma tests did pass and also all extraction
tests.  I also tried two simple tests of the installed bsdtar
executable:

bash.exe-3.1$ ../install/bin/bsdtar.exe xJf \
/z/home/wine/newstart/test_tar_h/test_xz/test.tar.xz
bash.exe-3.1$ ../install/bin/bsdtar.exe tJvf \
/z/home/wine/newstart/test_tar_h/test_xz/test.tar.xz

Those tests had no issues.

So it appears to me that for extraction at least and also in particular
for *.tar.xz extraction, the upstream git version (v3.1.2-40-g4b5f651) of
libarchive is fine on MSYS (and also Linux).

So would you be willing to import v3.1.2-40-g4b5f651 into CMake to
see if that solves the cmake -E tar issue that I found with
xz-compressed tarballs?

Alan
__________________________
Alan W. Irwin

Astronomical research affiliation with Department of Physics and Astronomy,
University of Victoria (astrowww.phys.uvic.ca).

Programming affiliations with the FreeEOS equation-of-state
implementation for stellar interiors (freeeos.sf.net); the Time
Ephemerides project (timeephem.sf.net); PLplot scientific plotting
software package (plplot.sf.net); the libLASi project
(unifont.org/lasi); the Loads of Linux Links project (loll.sf.net);
and the Linux Brochure Project (lbproject.sf.net).
__________________________

Linux-powered Science
__________________________
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libarchive_build_test_results.tar.gz
Type: application/x-gtar-compressed
Size: 26605 bytes
Desc: compressed tarball containing *.out files from MSYS build and test of libarchive
URL: <http://public.kitware.com/pipermail/cmake-developers/attachments/20130726/08caf283/attachment.tgz>