[rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin

[Ken Brown via RT]

Fri Sep 23 15:51:35 2016: Request 118053 was acted upon.
Transaction: Ticket created by ***@cornell.edu
Queue: PAR-Packer
Subject: Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >


Hi Roderich,

Phil Kime suggested that I raise a ticket concerning a problem I'm
having with a packed archive for biber
(https://sourceforge.net/projects/biblatex-biber/) on 64-bit Cygwin.
The problem doesn't occur on 32-bit Cygwin.

Here's the short version: I use PAR::Packer to create a packed archive
biber.exe. It unpacks itself and runs fine in the Cygwin distribution
on which I created it. But if I run it on a fresh, bare-bones Cygwin
installation (which in particular does not have perl installed), it
fails as follows:

$ ./biber.exe
C:/cygwin64a/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/biber:
error while loading shared libraries: cygperl5_24.dll: cannot open
shared object file: No such file or directory

When I look at the cache, I see the following:

$ ls -l
/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/
total 2136
-rwxr-x--- 1 kbrown None 53248 Sep 23 15:01 biber
-rwxr-x--- 1 kbrown None 2133011 Sep 23 15:01 cygperl5_24.dll

So nothing has been unpacked except biber and one DLL. And then the
system tries to load that DLL and fails, because its dependent DLLs have
not yet been unpacked. In this case the DLL cygcrypt-0.dll is the
problematic one, as we can see by running cygcheck:

$ cygcheck
/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/cygperl5_24.dll
C:\cygwin64a\tmp\par-6b62726f776e\cache-128a18f7440676b8f4632879a6f42b099bad8a05\cygperl5_24.dll
C:\cygwin64a\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
C:\cygwin64a\bin\cygssp-0.dll
cygcheck: track_down: could not find cygcrypt-0.dll

[Note that cygperl5_24.dll also depends on cygssp-0.dll, which also
should have been unpacked; but in this case it doesn't matter, because
/usr/bin/cygssp-0.dll exists in the bare-bones Cygwin install.]

Do you have any idea why biber.exe doesn't fully unpack itself before
trying to load cygperl5_24.dll?

The rest of this message provides more details about how I built
biber.exe. Please let me know if there's anything else you need.

My system (both build and test):

$ uname -a
CYGWIN_NT-10.0 moufang 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin

My perl on the build system:

$ perl -v

This is perl 5, version 24, subversion 0 (v5.24.0) built for
x86_64-cygwin-threads-multi

I built this perl myself, because biber requires perl 5.24, and Cygwin's
perl is currently at 5.22.

I created the build installation by doing the following:

1. Start with a fresh Cygwin installation, install my build of
perl-5.24, and install a few other packages that are needed for the next
steps.

2. Use cpan to install the build prerequisites for biber and to install
PAR::Packer. (I repeated this process today to make sure my build
environment was clean, so all modules are at their latest versions as of
today.)

3. Clone the biber git repo, checkout v2.6, and build and install biber.

4. Create the packed biber archive by running the attached script
build.sh, which uses the attached auxiliary file biber.files.

Thanks in advance for any help you can offer.

Ken

[Ken Brown via RT]

Sat Sep 24 21:06:22 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] AutoReply: Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Greetings,
This message has been automatically generated in response to the
"Problem with packed perl archive for biber on 64-bit Cygwin",
a summary of which appears below.
There is no need to reply to this message right now. Your ticket has been
assigned an ID of [rt.cpan.org #118053]. Your ticket is accessible
https://rt.cpan.org/Ticket/Display.html?id=118053
[rt.cpan.org #118053]
in the subject line of all future correspondence about this issue. To do so,
you may reply to this message.
Thank you,
-------------------------------------------------------------------------
Hi Roderich,
Phil Kime suggested that I raise a ticket concerning a problem I'm
having with a packed archive for biber
(https://sourceforge.net/projects/biblatex-biber/) on 64-bit Cygwin.
The problem doesn't occur on 32-bit Cygwin.
Here's the short version: I use PAR::Packer to create a packed archive
biber.exe. It unpacks itself and runs fine in the Cygwin distribution
on which I created it. But if I run it on a fresh, bare-bones Cygwin
installation (which in particular does not have perl installed), it
$ ./biber.exe
error while loading shared libraries: cygperl5_24.dll: cannot open
shared object file: No such file or directory
$ ls -l
/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/
total 2136
-rwxr-x--- 1 kbrown None 53248 Sep 23 15:01 biber
-rwxr-x--- 1 kbrown None 2133011 Sep 23 15:01 cygperl5_24.dll
So nothing has been unpacked except biber and one DLL. And then the
system tries to load that DLL and fails, because its dependent DLLs have
not yet been unpacked. In this case the DLL cygcrypt-0.dll is the
$ cygcheck
/tmp/par-6b62726f776e/cache-128a18f7440676b8f4632879a6f42b099bad8a05/cygperl5_24.dll
C:\cygwin64a\tmp\par-6b62726f776e\cache-128a18f7440676b8f4632879a6f42b099bad8a05\cygperl5_24.dll
C:\cygwin64a\bin\cygwin1.dll
C:\WINDOWS\system32\KERNEL32.dll
C:\WINDOWS\system32\ntdll.dll
C:\WINDOWS\system32\KERNELBASE.dll
C:\cygwin64a\bin\cygssp-0.dll
cygcheck: track_down: could not find cygcrypt-0.dll
[Note that cygperl5_24.dll also depends on cygssp-0.dll, which also
should have been unpacked; but in this case it doesn't matter, because
/usr/bin/cygssp-0.dll exists in the bare-bones Cygwin install.]
Do you have any idea why biber.exe doesn't fully unpack itself before
trying to load cygperl5_24.dll?
The rest of this message provides more details about how I built
biber.exe. Please let me know if there's anything else you need.
$ uname -a
CYGWIN_NT-10.0 moufang 2.6.0(0.304/5/3) 2016-08-31 14:32 x86_64 Cygwin
$ perl -v
This is perl 5, version 24, subversion 0 (v5.24.0) built for
x86_64-cygwin-threads-multi
I built this perl myself, because biber requires perl 5.24, and Cygwin's
perl is currently at 5.22.
1. Start with a fresh Cygwin installation, install my build of
perl-5.24, and install a few other packages that are needed for the next
steps.
2. Use cpan to install the build prerequisites for biber and to install
PAR::Packer. (I repeated this process today to make sure my build
environment was clean, so all modules are at their latest versions as of
today.)
3. Clone the biber git repo, checkout v2.6, and build and install biber.
4. Create the packed biber archive by running the attached script
build.sh, which uses the attached auxiliary file biber.files.
Thanks in advance for any help you can offer.
Ken

I understand a little better how the packed executable is supposed to
work. First the embedded files are extracted, which are a perl
interpreter (named biber) and cygperl5_24.dll in; it's then supposed to
be possible to run the extracted biber, which will extract the rest of
the files. Roderich, is this right?

So it seems that the solution should be to add cygcrypt-0.dll as an
embedded file. I haven't tried to do this in the PAR::Packer source yet
but I did test the theory as follows: I installed the libcrypt0 package
(which provides cygcrypt-0.dll) in my bare-bones test installation, and
biber.exe then unpacked itself and ran.

The reason I didn't see this problem in 32-bit Cygwin is that for some
reason my 32-bit build of cygperl5_24.dll doesn't depend on cygcrypt-0.dll.

Roderich, should the build of PAR::Packer have detected the need to
embed cygcrypt-0.dll? If not, what's the right fix for this?

David, I don't think this explains your symptoms. I'll write separately
to you about that, since I don't think it's related to this PAR::Packer
issue.

Ken

[Roderich Schupp via RT]

Sun Sep 25 03:06:46 2016: Request 118053 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: PAR-Packer
Subject: Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
I understand a little better how the packed executable is supposed to
work. First the embedded files are extracted, which are a perl
interpreter (named biber) and cygperl5_24.dll in; it's then supposed
to
be possible to run the extracted biber, which will extract the rest of
the files. Roderich, is this right?

Your analysis is correct.

Post by Ken Brown via RT
So it seems that the solution should be to add cygcrypt-0.dll as an
embedded file.

Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer source to

*is_system_lib = sub { shift =~ m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i };

i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library" anymore.
Then re-build and install PAR:Packer and re-pack biber.

Cheers, Roderich

[Ken Brown via RT]

Sun Sep 25 11:24:43 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
<URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
I understand a little better how the packed executable is supposed to
work. First the embedded files are extracted, which are a perl
interpreter (named biber) and cygperl5_24.dll in; it's then supposed
to
be possible to run the extracted biber, which will extract the rest of
the files. Roderich, is this right?

Your analysis is correct.

Post by Ken Brown via RT
So it seems that the solution should be to add cygcrypt-0.dll as an
embedded file.

Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer source to
*is_system_lib = sub { shift =~ m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i };
i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library" anymore.
Then re-build and install PAR:Packer and re-pack biber.
Cheers, Roderich

Thanks, that fixes it.

Ken

[Ken Brown via RT]

Sun Sep 25 13:37:42 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT

Post by Ken Brown via RT
<URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
I understand a little better how the packed executable is supposed to
work. First the embedded files are extracted, which are a perl
interpreter (named biber) and cygperl5_24.dll in; it's then supposed
to
be possible to run the extracted biber, which will extract the rest of
the files. Roderich, is this right?

Your analysis is correct.

Post by Ken Brown via RT
So it seems that the solution should be to add cygcrypt-0.dll as an
embedded file.

Yes. Try changing line 43 of myldr/embed_files.pl in the PAR::Packer
source to
*is_system_lib = sub { shift =~
m{^/usr/bin/(?!cygcrypt\b)|^\Q$system_root\E/}i };
i.e. /usr/bin/cygcrypt-0.dll is _not_ regarded as a "system library"
anymore.
Then re-build and install PAR:Packer and re-pack biber.
Cheers, Roderich

Thanks, that fixes it.

Actually, there's still one glitch. If I run biber with no arguments,
it unpacks itself and gives me a usage message. It also works fine if I
run 'biber --help'. But if I give it a file as an argument (e.g.,
'biber test.bcf') before the cache is created, it unpacks itself and
then hangs. After I kill it, I can rerun it with no problem.

David Carlisle, who is also testing, finds that it still hangs even
after the cache is created, but that may be a separate issue.

Do you have any idea what might cause this? For me it's just a glitch.
For David, it means that biber is unusable.

Thanks.

Ken

[Roderich Schupp via RT]

Mon Sep 26 06:16:57 2016: Request 118053 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: PAR-Packer
Subject: Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
Actually, there's still one glitch. If I run biber with no arguments,
it unpacks itself and gives me a usage message. It also works fine if I
run 'biber --help'. But if I give it a file as an argument (e.g.,
'biber test.bcf') before the cache is created, it unpacks itself and
then hangs. After I kill it, I can rerun it with no problem.

Sorry, no clue. If "biber --help" works, then basically PAR::Packer's job
is done (though there might still be missing files etc, but typically that
doesn't result in a hang). Does this hang occur also on the machine
where you packed biber?

Cheers, Roderich

[Ken Brown via RT]

Mon Sep 26 09:51:06 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
<URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
Actually, there's still one glitch. If I run biber with no arguments,
it unpacks itself and gives me a usage message. It also works fine if I
run 'biber --help'. But if I give it a file as an argument (e.g.,
'biber test.bcf') before the cache is created, it unpacks itself and
then hangs. After I kill it, I can rerun it with no problem.

Sorry, no clue. If "biber --help" works, then basically PAR::Packer's job
is done (though there might still be missing files etc, but typically that
doesn't result in a hang). Does this hang occur also on the machine
where you packed biber?

It did originally, but I just repacked biber (which a slight change),
and now I can't reproduce the problem anymore. The change was to remove
these two arguments from the invocation of pp:

--link=/usr/bin/cygssp-0.dll
--link=/usr/bin/cygcrypt-0.dll

The first is not needed because cygssp-0.dll is contained in a minimal
Cygwin install, and the second is not needed now that cygcrypt-0.dll is
embedded.

I'm not sure whether (or why) these changes made a difference, but in
any case the problem seems to be gone.

Thanks for your help.

Ken

[Ken Brown via RT]

Mon Sep 26 13:23:05 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >

Post by Ken Brown via RT
<URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >
On Mon, Sep 26, 2016 at 3:51 PM, Ken Brown via RT <

Post by Ken Brown via RT
The change was to remove
--link=/usr/bin/cygssp-0.dll
--link=/usr/bin/cygcrypt-0.dll

I was going to suggest that as well: these result in a second copy of the
DLL to be unpacked - while the first copy is probably already in use.
Which Windows doesn't allow i(the Linux equivalent error would be EBUSY).
But then I convinced myself that the two copies would go to different
directories (the first to the root of the cache area, the second to
cache/shlibs or similar)...

No, they were actually both going to the root of the cache directory.
So you're explanation makes perfect sense.

Thanks.

Ken

[Ken Brown via RT]

Tue Sep 27 11:23:32 2016: Request 118053 was acted upon.
Transaction: Correspondence added by ***@cornell.edu
Queue: PAR-Packer
Subject: Re: [rt.cpan.org #118053] Problem with packed perl archive for biber on 64-bit Cygwin
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cornell.edu
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=118053 >


It looks like David Carlisle's issue is completely unrelated, so I think
you can apply your change and close this ticket.

Thanks again for your help.

Ken