packaging a script that uses DBIx::Class

[RAPPAZ Francois]

Hi

I wrote an application using DBIx::Class to access data on mysql. The script work well (atlast) but the packaging with pp hangs: the "pp: Writing PAR on C:\Users\ ... " when verbose is on stay for 20 minutes, whereas the same application using sql and DBI is packed in a few minutes.

I suspect it's because DBIC use a lot of "use base '...'" and "__PACKAGE__->load_namespaces()" commands

My script calls "use Dbc::Schema;"

which is the only call to the Dbix::Class stuff.

The Dbc/Schema.pm file is 7 lines long:

use utf8;
package Dbc::Schema;
use strict;
use warnings;
use base 'DBIx::Class::Schema';
__PACKAGE__->load_namespaces;
__PACKAGE__->load_namespaces( default_resultset_class => 'ResultSet', );


My call to pp is basically

set file=cmd_dokpe_dbc
set dir=dokpe_i02_dbc
set inputfile=cmdui.pl
pp -v 3 -B -a glade -I C:\strawberry\c\bin\ @ppopts.txt -o %file%.par %inputfile%

Thanks for any advices on correcting this (I'm on w7)

François Rappaz

Centre de documentation de la Faculté des Sciences Université de Fribourg DokPe - Dokumentationszentrum der Naturwissenschaftlichen Fakultät Universität Freiburg Pérolles CH-1700 Fribourg Switzerland http://www.unifr.ch/dokpe/
Tel.: [phone]
Fax.: [phone]

[RAPPAZ Francois]

Retry: With the script test.pl given below
#!/usr/bin/perl -w
use strict;
use warnings;
use Config::YAML::Tiny;
use Dbc::Schema;

my $usage=<<END;
usage $0 config_file
END
die $usage unless ($ARGV[0]);


my ($s, $conf ) = get_schema($ARGV[0]);

my $href2 = {
'+select' => ['ComUser.nom' , 'ComCred.nom_credit'],
'+as' => [qw/nom nom_credit/],
order_by => 't_create',
join => ['ComCred', 'ComUser']};


my $href = {id_commande => {'<' => 60}};

my $rs1 = $s->resultset('Commande')->search_rs($href, $href2);
my $i;
while ( my $doc = $rs1->next) {
print "user: ", $doc->get_column('nom'), " credit : ", $doc->get_column('nom_credit'), " titre: ", $doc->get_column('titre'), "\n";
#last if ($i++ == 3);

}

print "count1: ", $rs1->count , "\n";

sub get_schema {
my $config = shift;
my $cfg = Config::YAML::Tiny->new(config=>$config);
my $dsn = $cfg->get_dsn . ";" . $cfg->get_database . ";" . ($cfg->get_socket?$cfg->get_socket:"");
#$dsn = "DBI:mysql:host=localhost;database=dokpe_i02;mysql_socket=/opt/lampp/var/mysql/mysql.sock";
my $gladefiles;

if ($ENV{PAR_TEMP}){
$gladefiles = $ENV{PAR_TEMP}. "/inc/" . $cfg->get_gladefiles;
$cfg->set_gladefiles($gladefiles);
}

#$globals->{ConnectionName}= $conn->{Name};
my $s = Dbc::Schema->connect(
$dsn,
$cfg->get_usr,
$cfg->get_psw,
{
PrintError => 0,
RaiseError => 1,
AutoCommit => 1,
mysql_enable_utf8=>1
}
);
return ($s, $cfg);
}
and

U:\docs\perl\dokpe_i02_dbc>pp -v 3 -o test test.pl


I got
C:\strawberry\perl\site\bin/pp: Packing test.pl
C:\strawberry\perl\site\bin/pp: Writing PAR on C:\Users\rappazf\LOCALS~1\Temp\ppYSfpk.par

... and things stop here.

Thanks for any help on packaging this.

François

-----Original Message-----
Sent: vendredi, 8. août 2014 16:50
Subject: packaging a script that uses DBIx::Class
Hi
I wrote an application using DBIx::Class to access data on mysql. The
Writing PAR on C:\Users\ ... " when verbose is on stay for 20 minutes,
whereas the same application using sql and DBI is packed in a few
minutes.
I suspect it's because DBIC use a lot of "use base '...'" and
"__PACKAGE__->load_namespaces()" commands
My script calls "use Dbc::Schema;"
which is the only call to the Dbix::Class stuff.
use utf8;
package Dbc::Schema;
use strict;
use warnings;
use base 'DBIx::Class::Schema';
__PACKAGE__->load_namespaces;
__PACKAGE__->load_namespaces( default_resultset_class => 'ResultSet',
);
My call to pp is basically
set file=cmd_dokpe_dbc
set dir=dokpe_i02_dbc
set inputfile=cmdui.pl
%inputfile%
Thanks for any advices on correcting this (I'm on w7)
François Rappaz
Centre de documentation de la Faculté des Sciences Université de
Fribourg DokPe - Dokumentationszentrum der Naturwissenschaftlichen
Fakultät Universität Freiburg Pérolles CH-1700 Fribourg Switzerland
http://www.unifr.ch/dokpe/
Tel.: [phone]
Fax.: [phone]

[Roderich Schupp]

Post by RAPPAZ Francois
Retry: With the script test.pl given below

Hi Francois,

no clue what's happening here - I'll try to reproduce this, but that won't
happen before the weekend.
I assume that you have already checked that you can pack a "hello world"
one-line script. Did you build PAR::Packer yourself and did you run the
testsuite?

Cheers, Roderich

[RAPPAZ Francois]

Hi Roderich

Yes I can pp –o hello –e « print ‘Hello’ ; »

From the directory where my script is, if I run
U:\docs\perl\dokpe_i02_dbc>scandeps.pl -V -e "use Dbc::Schema;"

I have no output and no return,, where as

U:\docs\perl\dokpe_i02_dbc>scandeps.pl -V -e "use Data::Dumper;"
gives a lot of dependencies in a few seconds

François

From: Roderich Schupp [mailto:***@gmail.com]
Sent: mardi, 12. août 2014 19:13
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class



On Tuesday, August 12, 2014, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
Retry: With the script test.pl<http://test.pl> given below

Hi Francois,

no clue what's happening here - I'll try to reproduce this, but that won't happen before the weekend.
I assume that you have already checked that you can pack a "hello world" one-line script. Did you build PAR::Packer yourself and did you run the testsuite?

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
From the directory where my script is, if I run
U:\docs\perl\dokpe_i02_dbc>scandeps.pl -V -e "use Dbc::Schema;"

Looks like infinite recursion in Module::ScanDeps ...

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
From the directory where my script is, if I run
U:\docs\perl\dokpe_i02_dbc>scandeps.pl -V -e "use Dbc::Schema;"
I have no output and no return

Patience! There's nothing wrong with this (as of Module::ScanDeps 1.14), it
just takes a while.
On my machine (with the system - including all Perl stuff - on an SSD) it
took 60 seconds
and scanned over 2500 files.

And your previously posted test.pl packed successfully in about the same
tiime.

Cheers, Roderich

[RAPPAZ Francois]

Thanks for testing !

My « scandeps.pl –V test.pl » is running for half an hour without any output

I must miss something somewhere 


I have Module::Scandeps version 1.07

How can I solve this ?

Thanks

François

From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: samedi, 16. août 2014 13:46
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class

On Wed, Aug 13, 2014 at 7:50 AM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
From the directory where my script is, if I run
U:\docs\perl\dokpe_i02_dbc>scandeps.pl<http://scandeps.pl> -V -e "use Dbc::Schema;"

I have no output and no return

Patience! There's nothing wrong with this (as of Module::ScanDeps 1.14), it just takes a while.
On my machine (with the system - including all Perl stuff - on an SSD) it took 60 seconds
and scanned over 2500 files.
And your previously posted test.pl<http://test.pl> packed successfully in about the same tiime.
Cheers, Roderich

[RAPPAZ Francois]

Well, at the end you were right
 scandeps.pl return after some 40 minutes

And my pp script terminate also after a long time.

I tried the –cd File option with pp and received an error, something like “cannot write to FILE, permission denied”, whether I gave a relative or an absolute path to an existing directory.

That’s a file name I have to give ?

Thanks

François



From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mardi, 19. août 2014 09:32
To: RAPPAZ Francois
Subject: Re: packaging a script that uses DBIx::Class

On Tue, Aug 19, 2014 at 9:05 AM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
My « scandeps.pl<http://scandeps.pl> –V test.pl<http://test.pl> » is running for half an hour without any output

I must miss something somewhere 


I have Module::Scandeps version 1.07

How can I solve this ?

First, upgrade to the latest version of Module::ScanDeps:)
If that doesn't solve the problem stick the following line into Module/ScanDeps.pm (around line 770)

sub scan_file{
my $file = shift;
my %found;
my $FH;
open $FH, $file or die "Cannot open $file: $!";
print STDERR "scan_file $file ...\n"; # <---

and repeat the above test while capturing stderr.
Check if there are duplicate lines in the output.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
Well, at the end you were right
 scandeps.pl return after some 40 minutes


Is your Perl installation (or anything in @INC) on a network share?
Your test script pulls in several of the worst offenders, e.g. DateTime
causes lots of
DateTime/Locale/* and DateTime/Timezone/* files to be scanned, there's also
Moose, Perl unicore...

Post by RAPPAZ Francois
I tried the –cd File option with pp and received an error, something like
“cannot write to FILE, permission denied”, whether I gave a relative or an
absolute path to an existing directory.
That’s a file name I have to give ?

Yes, if it doesn't exist, Module::ScanDeps will create it.
Sorry, I currently can't try this myself, can you check with a simple
example.pl whether

scandeps.pl -C cache_file example.pl

works and then try

pp --cd cache_file example.pl


Cheers, Roderich

[RAPPAZ Francois]

Giving a file name is ok.
I put every files on my c drive and It’s somewhat faster

I got the following error when I run the exe

Could not find a suitable B::Hooks::EndOfScope implementation:
Can't locate Variable/Magic.pm in @INC (
) at Module/Runtime.pm line 317.
Can't locate B/Hooks/EndOfScope/PP/FieldHash.pm in @INC ( 
 ) at Module/Runtime.pm line 317.


Shall I add –M Variable/Magic.pm – M B/Hooks/EndOfScope/PP/FieldHash.pm ?

Best
François

From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mardi, 19. août 2014 14:05
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class

On Tue, Aug 19, 2014 at 1:11 PM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
Well, at the end you were right
 scandeps.pl<http://scandeps.pl> return after some 40 minutes


Is your Perl installation (or anything in @INC) on a network share?
Your test script pulls in several of the worst offenders, e.g. DateTime causes lots of
DateTime/Locale/* and DateTime/Timezone/* files to be scanned, there's also Moose, Perl unicore...

I tried the –cd File option with pp and received an error, something like “cannot write to FILE, permission denied”, whether I gave a relative or an absolute path to an existing directory.

That’s a file name I have to give ?

Yes, if it doesn't exist, Module::ScanDeps will create it.
Sorry, I currently can't try this myself, can you check with a simple example.pl<http://example.pl> whether

scandeps.pl<http://scandeps.pl> -C cache_file example.pl<http://example.pl>

works and then try

pp --cd cache_file example.pl<http://example.pl>


Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
Shall I add –M Variable/Magic.pm – M B/Hooks/EndOfScope/PP/FieldHash.pm ?

It should read -M Variable::Magic -M B::Hooks::EndOfScope::PP
(the latter should not be necessary if you were using the latest
Module::ScanDeps).

Alternatively, you could pack with "pp -x ..." as M::SD may not detect all
stuff
loaded by Module::Runtime by using static analysis (it's called a *runtime*
loader for a reason).
Note that this will run your script once.

Cheers, Roderich

[RAPPAZ Francois]

I progress : I did manage to have a zip archive that could be transformed in a working exe.
But I correct the zip file with adding by hand the missing modules.

I‘m trying to have the right zip from the start, but I’m struggling with the modules that described my tables: they are located in a Dbc folder, below the working directory instead of in my site/lib perl tree.

I tried to include these with

pp -v 3 -B -cd .\ppcache -I .\Dbc\Schema -I .\Dbc\Schema\Result -M Dbc::Schema -M Dbc::Schema::Result 


without success

What am I missing ?

I will try the –x option when my static compilation is ok. My first tests were failing with a runtime error in the script execution.

François

From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mardi, 19. août 2014 15:59
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class

On Tue, Aug 19, 2014 at 3:43 PM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
Shall I add –M Variable/Magic.pm – M B/Hooks/EndOfScope/PP/FieldHash.pm ?

It should read -M Variable::Magic -M B::Hooks::EndOfScope::PP
(the latter should not be necessary if you were using the latest Module::ScanDeps).

Alternatively, you could pack with "pp -x ..." as M::SD may not detect all stuff
loaded by Module::Runtime by using static analysis (it's called a *runtime* loader for a reason).
Note that this will run your script once.

Cheers, Roderich

[RAPPAZ Francois]

OK : I did gave a directory DBC/Schema/Result and it search for a file DBC/Schema/Result.pm

Is there a way to include as modules all the pm files in the . DBC/Schema/Result folder ?

F.

From: RAPPAZ Francois [mailto:***@unifr.ch]
Sent: mercredi, 20. août 2014 08:39
To: ***@perl.org
Subject: RE: packaging a script that uses DBIx::Class

I progress : I did manage to have a zip archive that could be transformed in a working exe.
But I correct the zip file with adding by hand the missing modules.

I‘m trying to have the right zip from the start, but I’m struggling with the modules that described my tables: they are located in a Dbc folder, below the working directory instead of in my site/lib perl tree.

I tried to include these with

pp -v 3 -B -cd .\ppcache -I .\Dbc\Schema -I .\Dbc\Schema\Result -M Dbc::Schema -M Dbc::Schema::Result 


without success

What am I missing ?

I will try the –x option when my static compilation is ok. My first tests were failing with a runtime error in the script execution.

François

From: ***@gmail.com<mailto:***@gmail.com> [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mardi, 19. août 2014 15:59
To: RAPPAZ Francois
Cc: ***@perl.org<mailto:***@perl.org>
Subject: Re: packaging a script that uses DBIx::Class

On Tue, Aug 19, 2014 at 3:43 PM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
Shall I add –M Variable/Magic.pm – M B/Hooks/EndOfScope/PP/FieldHash.pm ?

It should read -M Variable::Magic -M B::Hooks::EndOfScope::PP
(the latter should not be necessary if you were using the latest Module::ScanDeps).

Alternatively, you could pack with "pp -x ..." as M::SD may not detect all stuff
loaded by Module::Runtime by using static analysis (it's called a *runtime* loader for a reason).
Note that this will run your script once.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
OK : I did gave a directory DBC/Schema/Result and it search for a file DBC/Schema/Result.pm

Nonsense.

I noticed that sometimes you say "DBC" and sometimes "Dbc".

Post by RAPPAZ Francois
Is there a way to include as modules all the pm files in the .
DBC/Schema/Result folder ?

No. Again: how are these pm files loaded by your script . Maybe you don't
explicitly,
but some DBIx::Class machinery does that for you? Perhaps it's using
something
like Module::Runtime's require_module()? That's one example of a dependency
that Module::ScanDeps can't find out by static analysis.

Cheers, Roderich

[RAPPAZ Francois]

Now I have a working exe with including each missing modules with the –M option.

Running
c:\docs\dokpe_i02_dbc>scandeps.pl -xVB cmdui.pl .commandes_localhost > scan_out_dyn.txt

I got SYSTEM ERROR in executing cmdui.pl: 65280 at C:/strawberry/perl/site/lib/Module/ScanDeps.pm line 1351.

François


From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mercredi, 20. août 2014 09:20
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class

On Wed, Aug 20, 2014 at 8:47 AM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
OK : I did gave a directory DBC/Schema/Result and it search for a file DBC/Schema/Result.pm

Nonsense.

I noticed that sometimes you say "DBC" and sometimes "Dbc".
Is there a way to include as modules all the pm files in the . DBC/Schema/Result folder ?

No. Again: how are these pm files loaded by your script . Maybe you don't explicitly,
but some DBIx::Class machinery does that for you? Perhaps it's using something
like Module::Runtime's require_module()? That's one example of a dependency
that Module::ScanDeps can't find out by static analysis.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
I got SYSTEM ERROR in executing cmdui.pl: 65280 at
C:/strawberry/perl/site/lib/Module/ScanDeps.pm line 1351.

"scandeps -x ... cmdui.pl" tries to run your script once (under some
instrumentation),
ie. it does "perl ... cmdui.pl". If your script insists on arguments and
dies otherwise, this would be the result.
And sorry, there's no way to pass arguments down.

Cheers, Roderich

[RAPPAZ Francois]

Without the argument the scandeps -C File –xVB 
 runs ok !

Now that I have a cache file from a dynamic scan, may I use it with pp without the –x check ?
For example, will this include the modules detected by scandeps.pl –x ?
pp -B -cd .\ppcache_from_scandeps -o myarchive.par myscript.pl

Thanks

François


From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mercredi, 20. août 2014 13:33
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class


On Wed, Aug 20, 2014 at 12:48 PM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
I got SYSTEM ERROR in executing cmdui.pl<http://cmdui.pl>: 65280 at C:/strawberry/perl/site/lib/Module/ScanDeps.pm line 1351.

"scandeps -x ... cmdui.pl<http://cmdui.pl>" tries to run your script once (under some instrumentation),
ie. it does "perl ... cmdui.pl<http://cmdui.pl>". If your script insists on arguments and dies otherwise, this would be the result.
And sorry, there's no way to pass arguments down.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
Now that I have a cache file from a dynamic scan, may I use it with pp
without the –x check ?
For example, will this include the modules detected by scandeps.pl –x ?
pp -B -cd .\ppcache_from_scandeps -o myarchive.par myscript.pl

Presumably :)

BTW, you're aware that the .par file created by the above command can only
be run on a machine
with a Perl installation that's ABI compatible with the Perl on your
machine. In this case, -B is useless.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
I‘m trying to have the right zip from the start, but I’m struggling with
the modules that described my tables: they are located in a Dbc folder,
below the working directory instead of in my site/lib perl tree.
I tried to include these with
pp -v 3 -B -cd .\ppcache -I .\Dbc\Schema -I .\Dbc\Schema\Result -M
Dbc::Schema -M Dbc::Schema::Result 


If your script does

use Dbc::Schema;

and that works (when running it from your working directory), then
Module::ScanDeps will pick it up
and it will be packed (when packing from your working directory). This
assumes that your @INC includes
"." (check with "perl -V"). In any case, check the output of "scandeps.pl
-V -B your_script.pl".
In any case, the options

-I .\Dbc\Schema -I .\Dbc\Schema\Result

don't make sense: the pp option -I works like the same option for perl
itself: it adds to @INC,
Hence to locate module Dbc::Schema below some_dir (i.e. the file
some_dir/Dbc/Schema.pm),
you should add "-I some-dir". But if you do all stuff in your working
directory, some_dir is "." and
that is already implicity in @INC.

If your script does NOT

use Dbc::Schema;

to load module Dbc::Schema, or if the loading depends on compile or runtime
manipulation of @INC
(either explicitly or thru "use lib ..." or similar), tell me what you did.

Cheers, Roderich

[RAPPAZ Francois]

What I wanted is to have the script running on a machine with the same os and no perl installation.
By the past, it works

What am I missing?

François

From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mercredi, 20. août 2014 16:35
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: {Spam?} Re: packaging a script that uses DBIx::Class



BTW, you're aware that the .par file created by the above command can only be run on a machine
with a Perl installation that's ABI compatible with the Perl on your machine. In this case, -B is useless.

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
What I wanted is to have the script running on a machine with the same os
and no perl installation.

Then why create a .par and not an .exe? Or do you create the .exe from the
.par in a second step?

Cheers, Roderich

[RAPPAZ Francois]

Yes I do,
And between the par making and the exe making, I unzip the par file and do some renaming to have Gtk2 gui working (and old story, remember ?)

See
http://www.mail-archive.com/par%40perl.org/msg05033.html


rd /s /q c:\docs\%dir%\tmp
7z x %file%.par -aoa -oC:\docs\%dir%\tmp
c:
cd c:\docs\%dir%\tmp\lib\auto
ren Gtk2 _Gtk2
ren Glib _Glib
ren Pango _Pango
cd c:\docs\%dir%\tmp
7z a %file%.zip .\*
move /Y %file%.zip c:\docs\%dir%
cd c:\docs\%dir%
call pp -o %file%.exe %file%.zip

François

From: ***@gmail.com [mailto:***@gmail.com] On Behalf Of Roderich Schupp
Sent: mercredi, 20. août 2014 17:13
To: RAPPAZ Francois
Cc: ***@perl.org
Subject: Re: packaging a script that uses DBIx::Class

On Wed, Aug 20, 2014 at 4:58 PM, RAPPAZ Francois <***@unifr.ch<mailto:***@unifr.ch>> wrote:
What I wanted is to have the script running on a machine with the same os and no perl installation.

Then why create a .par and not an .exe? Or do you create the .exe from the .par in a second step?

Cheers, Roderich

[Roderich Schupp]

Post by RAPPAZ Francois
And between the par making and the exe making, I unzip the par file and do
some renaming to have Gtk2 gui working (and old story, remember ?)

Yeah, old story :) Unfortunately, the root cause, i.e. PAR::Packer
extracting glue DLLs with mangled names,
still persists.

Cheers, Roderich