[rt.cpan.org #111455] unable to use -x and -u with pp

[Shawn Laffan via RT]

Wed Jan 27 01:39:00 2016: Request 111455 was acted upon.
Transaction: Ticket created by SLAFFAN
Queue: PAR-Packer
Subject: unable to use -x and -u with pp
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cpan.org
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=111455 >


Running pp with both the -u and -x options fails. It appears that some of the text files under the unicore directory are being executed by Module::ScanDeps. See code and results below.

The "used once" warnings appear to be from Module::ScanDeps when it processes utf8_heavy.pl, and are harmless, but I have not looked into where the text files are executed.

Tested on Strawberry perl 5.16.3 and 5.22.1.

Regards,
Shawn.



C:\some_folder\x>pp -u -e "print 'blort'"

C:\some_folder\x>a.exe
blort

C:\some_folder\x>pp -x -u -e "print 'blort'"
Name "utf8::loose_property_to_file_of" used only once: possible typo at C:/strawberry_51613_x64/perl/lib/utf8_heavy.pl line 438.
Name "utf8::file_to_swash_name" used only once: possible typo at C:/strawberry_51613_x64/perl/lib/utf8_heavy.pl line 440.
Name "utf8::loose_property_name_of" used only once: possible typo at C:/strawberry_51613_x64/perl/lib/utf8_heavy.pl line 210.
Name "utf8::stricter_to_file_of" used only once: possible typo at C:/strawberry_51613_x64/perl/lib/utf8_heavy.pl line 393.
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 30, near "007F"
(Missing operator before F?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 31, near "00FF"
(Missing operator before FF?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 31, near "1 Supplement"
(Missing operator before Supplement?)
Number found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 32, near "Supplement
0100"
(Missing semicolon on previous line?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 32, near "017F"
(Missing operator before F?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 33, near "024F"
(Missing operator before F?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 34, near "02AF"
(Missing operator before AF?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 35, near "02B0"
(Missing operator before B0?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 35, near "02FF"
(Missing operator before FF?)
Number found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 36, near "Letters
0300"
(Missing semicolon on previous line?)
Bareword found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 36, near "036F"
(Missing operator before F?)
Number found where operator expected at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 37, near "Marks
0370"
(Missing semicolon on previous line?)
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 30, near "007F"
Illegal octal digit '8' at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 31, at end of line
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 31, near "00FF"
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 32, near "1 Supplement
"
Illegal octal digit '8' at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 33, at end of line
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 33, near "024F"
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 34, near "02AF"
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 35, near "02B0"
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 36, near "Letters
0300"
syntax error at C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt line 37, near "Marks
0370"
C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt has too many errors.
SYSTEM ERROR in executing C:/strawberry_51613_x64/perl/lib/unicore/Blocks.txt : 65280 at C:/strawberry_51613_x64/perl/site/lib/Module/ScanDeps.pm line 1422.

[Roderich Schupp via RT]

Wed Jan 27 08:38:40 2016: Request 111455 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: PAR-Packer
Subject: unable to use -x and -u with pp
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cpan.org
Status: new
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=111455 >

Post by Shawn Laffan via RT
Running pp with both the -u and -x options fails. It appears that some
of the text files under the unicore directory are being executed by
Module::ScanDeps. See code and results below.

Yeah, "pp -u ..." is just short for "pp -M utf8_heavy.pl ..." so combining
this with "-x" gets the same result :( In fact, "pp -x -M Foo::Bar ..." with
a Foo::Bar that has a %Preload rule in Module/ScanDeps.pm that adds anything
NOT perlish will provoke the same problem. Other examples are charnames or
LWP::MediaTypes.

Turns out that "pp -x" - counter to what I would expect - does not only
execute the actual input file(s), but also every module added with -M and
everything it "depends" on according to the %Preload rules. The problem
has been lurking for ages, but - since running "perl /path/to/module.pm"
usually doesn't generate an error - nobody noticed.

BTW, adding dependencies from "-M" arguments to the list of input files
affects not only "-x", but the default use case as well.
This behaviour also explains what's going on in #110480:
try "pp -M utf8_heavy.pl ..." (no "-x") and look at the contents of the
generated exe (as a zip file). It contains every non-pm file, e.g. 1000.pl,
below the "unicore" directory TWICE, once as "lib/unicore/lib/Nv/1000.pl"
(correct) and also as "lib/1000.pl".

Can you try the patch below?

Cheers, Roderich


Index: lib/PAR/Packer.pm
===================================================================
--- lib/PAR/Packer.pm (revision 1642)
+++ lib/PAR/Packer.pm (working copy)
@@ -737,7 +737,6 @@
skip => \%skip,
# warn_missing => $args->{warn_missing},
);
- push @files, map {&$inc_find($_)} @preload;
}
push @files, @$input;

[Shawn Laffan via RT]

Wed Jan 27 14:59:36 2016: Request 111455 was acted upon.
Transaction: Correspondence added by SLAFFAN
Queue: PAR-Packer
Subject: unable to use -x and -u with pp
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=111455 >


Thanks Roderich,

The patch makes the failure go away.

The used once warnings still remain, but they appear harmless.

Someone at a later date might use -M and want all the associated dependencies, but that could lead to all sorts of issues in itself. Maybe something can be added to the documentation for -M to clarify its interaction with -x?


Regards,
Shawn.

Post by Roderich Schupp via RT

Post by Shawn Laffan via RT
Running pp with both the -u and -x options fails. It appears that some
of the text files under the unicore directory are being executed by
Module::ScanDeps. See code and results below.

Yeah, "pp -u ..." is just short for "pp -M utf8_heavy.pl ..." so combining
this with "-x" gets the same result :( In fact, "pp -x -M Foo::Bar ..." with
a Foo::Bar that has a %Preload rule in Module/ScanDeps.pm that adds anything
NOT perlish will provoke the same problem. Other examples are charnames or
LWP::MediaTypes.
Turns out that "pp -x" - counter to what I would expect - does not only
execute the actual input file(s), but also every module added with -M and
everything it "depends" on according to the %Preload rules. The problem
has been lurking for ages, but - since running "perl /path/to/module.pm"
usually doesn't generate an error - nobody noticed.
BTW, adding dependencies from "-M" arguments to the list of input files
affects not only "-x", but the default use case as well.
try "pp -M utf8_heavy.pl ..." (no "-x") and look at the contents of the
generated exe (as a zip file). It contains every non-pm file, e.g. 1000.pl,
below the "unicore" directory TWICE, once as "lib/unicore/lib/Nv/1000.pl"
(correct) and also as "lib/1000.pl".
Can you try the patch below?
Cheers, Roderich
Index: lib/PAR/Packer.pm
===================================================================
--- lib/PAR/Packer.pm (revision 1642)
+++ lib/PAR/Packer.pm (working copy)
@@ -737,7 +737,6 @@
skip => \%skip,
# warn_missing => $args->{warn_missing},
);
}

[Roderich Schupp via RT]

Thu Jan 28 04:48:39 2016: Request 111455 was acted upon.
Transaction: Correspondence added by RSCHUPP
Queue: PAR-Packer
Subject: unable to use -x and -u with pp
Broken in: (no value)
Severity: (no value)
Owner: Nobody
Requestors: ***@cpan.org
Status: open
Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=111455 >

Post by Shawn Laffan via RT
The patch makes the failure go away.

Thanks for testing, will be in the next release.

Post by Shawn Laffan via RT
The used once warnings still remain, but they appear harmless.

Yes, essentially we're doing

$ perl .../utf8_heavy.pl

which produces the same warnings.

Post by Shawn Laffan via RT
Someone at a later date might use -M and want all the associated
dependencies, but that could lead to all sorts of issues in itself.

I don't think we're loosing anything here:
for "pp -M Foo::Bar ..." we still do

- scan Foo/Bar.pm
- add any %Preload dependencies of Foo/Bar.pm to the list of dependencies

If "-x" was also specified, we still run Foo/Bar.pm (pointless,
but harmless except for possible warnings).

But we don't:

- scan all the %Preload dependencies of Foo/Bar.pm explicitly, but AFAICT
Module::ScanDeps will scan them anyway
- with "-x" run all the %Preload dependencies of Foo/Bar.pm (even more
pointless, and harmful as they might include files that aren't Perl source)

Cheers, Roderich