[rt.cpan.org #91530] Missing MIME\types.db from pp-generated code

[ric via RT]

Thu Dec 19 19:46:55 2013: Request 91530 was acted upon.
Transaction: Ticket created by ***@mclink.it
Queue: PAR-Packer
Subject: Missing MIME\types.db from pp-generated code
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@mclink.it
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=91530 >



I've an issue, for a missing of MIME:Types.db from code generated via pp
of PAR::Packer.

Development environment is:

XP SP3 machine within VirtualBox
Strawberry Perl 5.18.1.1

PAR::Packer 1.017
MIME:Lite 3.030


Interpreted code (i.e. code invoked via "perl -w scriptname.pl") runs
fine within this environment, so I would exclude issues on the script.
pp compilation is invoked in a very plain way as

pp -v -o scriptname.exe scriptname.pl


and does not output any warning.

When the executable gets transferred on a "real" environment (an
independent, XP SP3 "true" box), program fails with a

"cannot open type database in MIME\types.db: No such file or directory"


As a matter of fact, if I enter the cache folder within TEMP (where
pp-compiled scripts expand their contents), and navigate downwards to
\inc\lib\MIME, I find 4 modules there (base64.pm, lite.pm, type.pm,
types.pm), but no types.db whatsoever.

I've also tried to include explicitly MIME:Lite via -M

pp -M MIME:Lite -v -o scriptname.exe scriptname.pl


and also, specifying source and destination of the file to be included,

pp -a
"c:\strawberry\perl\site\lib\MIME\types.db;/inc/lib/MIME/types.db"
-v -o scriptname.exe scriptname.pl

still, to no extent. In the last form, I've also opened the zipped
executable, and requested file is not where it should be.

Can't say whether this is an issue or a pitfall on my side; whatever the
case, any suggestion/workaround is greatly appreciated.

regards
ric

[Roderich Schupp via RT]

Sat Dec 21 06:52:52 2013: Request 91530 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: Module-ScanDeps
Subject: Missing MIME\types.db from pp-generated code
Broken in: 1.12
Severity: (no value)
Owner: RSCHUPP
Requestors: ***@mclink.it
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=91530 >

Post by ric via RT
I've an issue, for a missing of MIME:Types.db from code generated via pp
of PAR::Packer.
...
Can't say whether this is an issue or a pitfall on my side; whatever the
case, any suggestion/workaround is greatly appreciated.

Definitely an issue, but it's complicated.
If you don't want to read through my ramblings below, skip to the bottom for a workaround :)

When a pp-packed executable is missing a module or data file, this can usually
be worked around by using pp options -M (for a missing module) or -a (for a plain file), though to get the latter "right" may need some experimentation
(or insight how some module searches for its data file).

BTW, even if your workaround works, you might submit a bug report for
Module::ScanDeps (which is used by pp to scan your script for all modules
it uses). I can usually make the workaround unnecessery by adding another
explicit rule to Module::ScanDeps.

In your case, MIME::Types (it's not in MIME::Lite, but used by MIME::Lite
under the hood) uses the data file MIME/types.db.
Your idea to use "pp -a ..." to pack this file explicitly is correct,
but can't work because MIME::Types does the following to open this file:

open FH, "<:encoding(utf8)", File::Spec->catfile(dirname(__FILE__), "types.db")

This assumes that (when perl parses MIME/Types.pm) __FILE__ is set to the
absolute pathname where MIME/Types.pm was found. This is probably true for
99% of all use cases, but it always breaks in a packed executable.
The reason is that any module is transformed ("filtered") when it is packed.
The default filter (PodStrip) strips POD from the module. To compensate for that
(so that a die() or warn() from inside the module shows the "correct" line number)
it adds "# line N ..." statements. Especially it adds

# line 1 "MIME/Types.pm"

at the start of the packed MIME/Types.pm. As a side effect this sets
__FILE__ to "MIME/Types.pm". So we must use the following options for pp:

-a '/path/to/your/MIME/types.db;lib/MIME/types.db'

to pack MIME/types.db as a "sibling" of MIME/Types.pm AND suppress
the application of the PodStrip filter for MIME/Types.pm:

-F PatchContent=MIME/Types.pm

(or set PAR_VERBATIM=1 in the environment when packing to generally suppress filtering).

But wait, there's more :( With the above you will still get

cannot open type database in /tmp/par-726f646572696368/cache-5be06ead4dff6a6e16c2df7d03b59c0dbf5a8c02/inc/lib/MIME/types.db: No such file or directory

even though this file exists! The reason is the argument "<:encoding(utf8)"
to open(). It implicitly does "require PerlIO::encoding". Since this module hasn't been packed, open() fails even before actually trying to open the file.
Actually, this is a bug in Module::ScanDeps, it's supposed to look out for
for patterns like this and infer a dependency on PerlIO and PerlIO::encoding.
I've fixed this in Module::ScanDeps, but until its next release the workaround
is to use the pp option

-M PerlIO::encoding

So the whole workaround is

pp -a "/path/to/your/MIME/types.db;lib/MIME/types.db" \
-F PatchContent=MIME/Types.pm \
-M PerlIO::encoding ...

(use forward slashes even under Windows)

Cheers, Roderich

[ric via RT]

Mon Dec 23 09:48:56 2013: Request 91530 was acted upon.
Transaction: Correspondence added by ***@mclink.it
Queue: Module-ScanDeps
Subject: Re: [rt.cpan.org #91530] Missing MIME\types.db from pp-generated code
Broken in: 1.12
Severity: (no value)
Owner: RSCHUPP
Requestors: ***@mclink.it
Status: resolved
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=91530 >


Thanks Roderich, for the detailed explanation and the workaround.

Unfortunately, the workaround does not work as expected: despite your
guidelines, there's still no trace of the missing types.db, neither
within the compiled .exe, nor in the cache.

Is there any other chance to force it where it should be?

TIA
ric

So the whole workaround is pp -a
"/path/to/your/MIME/types.db;lib/MIME/types.db" -F
PatchContent=MIME/Types.pm -M PerlIO::encoding ...
Cheers, Roderich

[Roderich Schupp via RT]

Wed Dec 25 15:04:31 2013: Request 91530 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: Module-ScanDeps
Subject: Missing MIME\types.db from pp-generated code
Broken in: 1.12
Severity: (no value)
Owner: RSCHUPP
Requestors: ***@mclink.it
Status: resolved
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=91530 >

Post by ric via RT
Unfortunately, the workaround does not work as expected: despite your
guidelines, there's still no trace of the missing types.db, neither
within the compiled .exe, nor in the cache.

Check your -a option for pp. For my Strawberry Perl (installed in C:\strawberry) the following works:

pp -a "C:\strawberry\perl\vendor\lib\MIMEtypes.db;lib/MIME/types.db" ...

Cheers, Roderich