Error while reloading a par file

[Amit Dev]

Hi,

We have a perl program packaged into a par. The par is executed and it
works fine. Sometime we shutdown the process and run it again or run an
updated copy of the par file. And intermittently it fails with "Compilation
error..." in logs.
Looks like the issue is somewhere within temp folder (par-root/cache-*).
Once the issue happens I see a "inc.lock" folder created within. Not sure
how the whole thing works. Looks like setting PAR_GLOBAL_CLEAN might fix
the issue, but interested to know why this is happening. Does someone know
how the unpacking really works, and is it required to clean the temp
directory before loading another par program?

[Roderich Schupp]

Post by Amit Dev
We have a perl program packaged into a par. The par is executed and it

By "program packaged into a par" do you mean a standalone executable
created
with "pp -o some.exe script.pl" or a .par file created with "pp -p -o
some.par script.pl"
(that you run with "perl -MPAR=some.par ...")?

Post by Amit Dev
Sometime we shutdown the process and run it again or run an
updated copy of the par file. And intermittently it fails with "Compilation
error..." in logs.
Looks like the issue is somewhere within temp folder (par-root/cache-*).
Once the issue happens I see a "inc.lock" folder created within. Not sure
how the whole thing works.

For the case of the standalone executable:

If you run it for the first time (i.e. /tmp/par-USER/cache-DIGEST doesn't
exit),
this cache directory is created and some stuff (a custom perl interpreter,
modules, DLLs, data files) is extracted from the executable into it.
Then the custom perl interpreter is executed, finally running the original
script.

The next time you run the same executable (as the same user, as the
cache directory is per-user) we see that the cache directory already exists,
so the extraction phase is skipped and the perl interpreter is executed
right away.

Actually that's the simplified version - what happens if the first and
second
invocations happen in rapid succession, so that the first is still
extracting stuff
while the second is started? So the actual criterion for "stuff has
completely been
extracted (hence we may skip extraction)" is "does
/tmp/par-USER/cache-DIGEST/inc
exist". If not, acquire some kind of lock, create .../inc, extract stuff
and release the lock.
Acquiring the lock here is implemented as "mkdir .../inc.lock"
(successfully)
and releasing it is "rmdir .../inc.lock" (probably because mkdir is the
only portable
atomic file operation).

Cheers, Roderich

If you update the executable it should get a different hash DIGEST and hence
the cache directory shoudl be different, so no problem here.

[Amit Dev]

On Mon, Dec 17, 2012 at 10:54 PM, Roderich Schupp <

Post by Roderich Schupp

Post by Amit Dev
We have a perl program packaged into a par. The par is executed and it

By "program packaged into a par" do you mean a standalone executable
created
with "pp -o some.exe script.pl
" or a .par file created with "pp -p -o some.par script.pl
"
(that you run with "perl -MPAR=some.par ...")?

It is of the second type, "pp -c -M AnyEvent (and bunch of other -M..) -o
some.par script.pl".

Post by Roderich Schupp
If you run it for the first time (i.e. /tmp/par-USER/cache-DIGEST doesn't
exit),
this cache directory is created and some stuff (a custom perl interpreter,
modules, DLLs, data files) is extracted from the executable into it.
Then the custom perl interpreter is executed, finally running the original
script.
The next time you run the same executable (as the same user, as the
cache directory is per-user) we see that the cache directory already exists,
so the extraction phase is skipped and the perl interpreter is executed
right away.
Actually that's the simplified version - what happens if the first and
second
invocations happen in rapid succession, so that the first is still
extracting stuff
while the second is started? So the actual criterion for "stuff has
completely been
extracted (hence we may skip extraction)" is "does
/tmp/par-USER/cache-DIGEST/inc
exist". If not, acquire some kind of lock, create .../inc, extract stuff
and release the lock.
Acquiring the lock here is implemented as "mkdir .../inc.lock"
(successfully)
and releasing it is "rmdir .../inc.lock" (probably because mkdir is the
only portable
atomic file operation).

Thanks Roderich. What's happening seems to be that it is failing somewhere
in between (since only mkdir happened and rmdir didn't).
We are thinking of using PAR_GLOBAL_CLEAN to avoid this whole thing, but
curious how it could fail. Some corruption is happening since
sometimes files inside the cache (say x.pm) is of 0 bytes.

[Roderich Schupp]

Post by Amit Dev
It is of the second type, "pp -c -M AnyEvent (and bunch of other -M..) -o
some.par script.pl".

OK, this works basically the same as for a standalone executable: replace
"executable"
in my description with ".par file" and omit the part about the custom perl
interpreter.

What's happening seems to be that it is failing somewhere in between (since

Post by Amit Dev
only mkdir happened and rmdir didn't).

Looks like it, though without the exact error message it's hard to guess.
Note that mkdir/rmdir .../inc.lock only happens on the very first
"invocation " of the .par file.

Cheers, Roderich