dl_unload_all_files() revisited

[p5pRT]

Migrated from rt.perl.org#2968 (status was 'open')

Searchable as RT2968$

[p5pRT]

From @AlanBurlison

My current understanding is that this patch broke DBD​::Oracle on Linux -
I'm not quite sure why. I can see a potential problem if a module uses
call_atexit(), and that then uses a bit of XS that has already been
dlclosed, but for DBD​::Oracle this doesn't seem to be the case.

I (respectfully!) disagree with Sarathay's assertion that dlclosing()
stuff when the interpreter exits is an 'unnecessary overhead' - it is an
*essential* part of the cleanup, much in the same way as freeing up
everything that has been malloced is. It is alright to not free and not
dlclose if you know that the process is going to exit anyway, but in the
case of an embedded perl interpreter this is not necessarily the case.

I'd like to get the dlclose stuff into a state where it on by default,
as not having it there generates a particulary abstruse type of bug - it
took the best part of a year to figure out why mod_perl dumped core when
built with APXS.

I'd like some informed suggestions of the best way of doing this
whithout causing collateral damage - anyone have any ideas?

Alan Burlison

[p5pRT]

From @timbunce

On Sun, Apr 02, 2000 at 11​:05​:41AM +0100, Alan Burlison wrote​:

My current understanding is that this patch broke DBD​::Oracle on Linux -
I'm not quite sure why.

I'll happily help as far as I can if you send me details.

(Oracle does very weird things with it's libraries, and then does
different weird things in the following release).

I can see a potential problem if a module uses
call_atexit(), and that then uses a bit of XS that has already been
dlclosed, but for DBD​::Oracle this doesn't seem to be the case.

I (respectfully!) disagree with Sarathay's assertion that dlclosing()
stuff when the interpreter exits is an 'unnecessary overhead' - it is an
*essential* part of the cleanup, much in the same way as freeing up
everything that has been malloced is. It is alright to not free and not
dlclose if you know that the process is going to exit anyway, but in the
case of an embedded perl interpreter this is not necessarily the case.

I think Sarathy meant it's an 'unnecessary overhead' when _not_ embedding.

I'd like to get the dlclose stuff into a state where it on by default,

When embedding, yes.

as not having it there generates a particulary abstruse type of bug - it
took the best part of a year to figure out why mod_perl dumped core when
built with APXS.

I'd like some informed suggestions of the best way of doing this
whithout causing collateral damage - anyone have any ideas?

Do dl_unload_all_files() if Perl_destruct_level > 0.

Tim.

[p5pRT]

From @Hugmeir

On Sun Apr 02 22​:51​:44 2000, RT_System wrote​:

On Sun, Apr 02, 2000 at 11​:05​:41AM +0100, Alan Burlison wrote​:

My current understanding is that this patch broke DBD​::Oracle on Linux -
I'm not quite sure why.

I'll happily help as far as I can if you send me details.

(Oracle does very weird things with it's libraries, and then does
different weird things in the following release).

I can see a potential problem if a module uses
call_atexit(), and that then uses a bit of XS that has already been
dlclosed, but for DBD​::Oracle this doesn't seem to be the case.

I (respectfully!) disagree with Sarathay's assertion that dlclosing()
stuff when the interpreter exits is an 'unnecessary overhead' - it is an
*essential* part of the cleanup, much in the same way as freeing up
everything that has been malloced is. It is alright to not free and not
dlclose if you know that the process is going to exit anyway, but in the
case of an embedded perl interpreter this is not necessarily the case.

I think Sarathy meant it's an 'unnecessary overhead' when _not_ embedding.

I'd like to get the dlclose stuff into a state where it on by default,

When embedding, yes.

as not having it there generates a particulary abstruse type of bug - it
took the best part of a year to figure out why mod_perl dumped core when
built with APXS.

I'd like some informed suggestions of the best way of doing this
whithout causing collateral damage - anyone have any ideas?

Do dl_unload_all_files() if Perl_destruct_level > 0.

Tim.

I can't find the context behind this report. There.. seems to be a
solution for the poster's issue at the end, but does anyone know if this
is still an issue?

Does anyone know what the issue _was_? : /

[p5pRT]

From @tonycoz

On Fri May 04 08​:52​:57 2012, Hugmeir wrote​:

On Sun Apr 02 22​:51​:44 2000, RT_System wrote​:

On Sun, Apr 02, 2000 at 11​:05​:41AM +0100, Alan Burlison wrote​:

My current understanding is that this patch broke DBD​::Oracle on
Linux -
I'm not quite sure why.

I'll happily help as far as I can if you send me details.

(Oracle does very weird things with it's libraries, and then does
different weird things in the following release).

I can see a potential problem if a module uses
call_atexit(), and that then uses a bit of XS that has already been
dlclosed, but for DBD​::Oracle this doesn't seem to be the case.

I (respectfully!) disagree with Sarathay's assertion that dlclosing()
stuff when the interpreter exits is an 'unnecessary overhead' - it
is an
*essential* part of the cleanup, much in the same way as freeing up
everything that has been malloced is. It is alright to not free
and not
dlclose if you know that the process is going to exit anyway, but
in the
case of an embedded perl interpreter this is not necessarily the case.

I think Sarathy meant it's an 'unnecessary overhead' when _not_
embedding.

I'd like to get the dlclose stuff into a state where it on by default,

When embedding, yes.

as not having it there generates a particulary abstruse type of
bug - it
took the best part of a year to figure out why mod_perl dumped
core when
built with APXS.

I'd like some informed suggestions of the best way of doing this
whithout causing collateral damage - anyone have any ideas?

Do dl_unload_all_files() if Perl_destruct_level > 0.

Tim.

I can't find the context behind this report. There.. seems to be a
solution for the poster's issue at the end, but does anyone know if this
is still an issue?

Does anyone know what the issue _was_? : /

From reading the thread, apparently we (were|are) not releasing loaded
shared libraries when we clean up the interpreter.

This isn't a problem when perl is running standalone, but is a problem
when perl is embedded, since we're continuing to use resources after the
interpreter has been deleted.

As to the proposed solution, I wonder if freeing the libraries on
interpreter destruction is safe - it should be sort-of-safe where the
underlying library management tools reference count shared libraries
(true for dlopen and MSWin32), but I don't think it's safe otherwise.

Also, there doesn't seem to be an XS entry point (like boot_Foo for
module Foo) to do clean up for a loaded XS module, which means memory
may leak anyway (and other worse things, like dangling signal handlers.)

Tony

[p5pRT]

From @nwc10

TL;DR​: It's a mess.

On Fri, Aug 09, 2013 at 12​:35​:41AM -0700, Tony Cook via RT wrote​:

On Fri May 04 08​:52​:57 2012, Hugmeir wrote​:

On Sun Apr 02 22​:51​:44 2000, RT_System wrote​:

On Sun, Apr 02, 2000 at 11​:05​:41AM +0100, Alan Burlison wrote​:

[whether at exit the interpreter should attempt to dl_unload() all files it
dl_load()ed]

I'd like some informed suggestions of the best way of doing this
whithout causing collateral damage - anyone have any ideas?

Do dl_unload_all_files() if Perl_destruct_level > 0.

Tim.

I can't find the context behind this report. There.. seems to be a
solution for the poster's issue at the end, but does anyone know if this
is still an issue?

As an aside, I can't figure out how this has actually come to be recorded
as a ticket in RT, given that it seems to be a 2 message thread sent just to
the mailing list, back in 2000​:

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-04/msg00091.html

(Back before the ticketing system was on RT. So what's the old bug number?)

Does anyone know what the issue _was_? : /

From reading the thread, apparently we (were|are) not releasing loaded
shared libraries when we clean up the interpreter.

This isn't a problem when perl is running standalone, but is a problem
when perl is embedded, since we're continuing to use resources after the
interpreter has been deleted.

As to the proposed solution, I wonder if freeing the libraries on
interpreter destruction is safe - it should be sort-of-safe where the
underlying library management tools reference count shared libraries
(true for dlopen and MSWin32), but I don't think it's safe otherwise.

Also, there doesn't seem to be an XS entry point (like boot_Foo for
module Foo) to do clean up for a loaded XS module, which means memory
may leak anyway (and other worse things, like dangling signal handlers.)

I've looked at this and the related thread "perl_atexit() considered useless"

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-09/msg00846.html

Relevant commits from the period are​:

commit abb9e9d
Author​: Gurusamy Sarathy <gsar@​cpan.org>
Date​: Wed Mar 1 00​:46​:44 2000 +0000

  unload extension shared objects when exiting, implemented
  only for dl_dlopen.xs (from Alan Burlison)
 
  p4raw-id​: //depot/perl@​5381

commit 23d2500
Author​: Gurusamy Sarathy <gsar@​cpan.org>
Date​: Wed Mar 22 18​:41​:50 2000 +0000

  make unloading of extension shared objects (change#5381) a build
  option (use "Configure -Accflags=-DDL_UNLOAD_ALL_AT_EXIT" to enable)
 
  p4raw-link​: @​5381 on //depot/perl​: abb9e9d
 
  p4raw-id​: //depot/perl@​5885

Both are just before v5.6.0 was released. Nothing has been changed in the code
since. (The documentation failed to note the second commit until yesterday)

It's a mess.

I can't actually work out what the problem was that Alan was reporting in the
second thread ("perl_atexit() considered useless").

  I've been looking at memory errors in perl, and found that if
  DL_UNLOAD_ALL_AT_EXIT is defined, it causes heap problems on exit. The
  cause is that the AV holding the references to the dlopen() handles has
  been reclaimed before the perl atexit processing takes place to
  dlclose() the handles. I could fix this by keeping a list of the dlopen
  handles locally in DynaLoader.xs, but I think there is a more general
  problem here - what use is perl_atexit() if it is called after globals
  have been destroyed?

perl_atexit() is called after object destruction, but *before* all the
unblessed SVs gets destroyed, and the AV in question is
@​DynaLoader​::dl_librefs, which is not blessed. So what he describes shouldn't
happen.

I've also built blead from that era with gcc 4.8.1 and ASAN.
Yes, this works. I was a bit surprised how easy it was.

The build and most of the tests pass. There are a couple of bugs tickled by
regression tests which ASAN spots, but mostly 5.6.0 was pretty clean.
(No doubt in large part because Alan did a lot of work with purify around
that time to identify and fix problems.)
But ASAN can't find any sort of heap error. So I'm confused.

I think I can see how the DBD​::Oracle problem might be happening. It's
likely the same sort of thing as this bit of code in DynaLoader.t​:

  SKIP​: {
  skip "unloading unsupported on $^O", 2
  if ($old_darwin || $^O eq 'VMS');
  my $module = pop @​loaded_modules;
  skip "File​::Glob sets PL_opfreehook", 2 if $module eq 'File​::Glob';
  my $r = eval { DynaLoader​::dl_unload_file($libref) };

in that, if DBD​::Oracle is using something (probably tie magic) that is a
pointer into the shared object held by an SV (or similar) that survives
until late in global destruction, then the point at which perl_atexit() runs
is before then, and so it creates dangling pointers.

DL_UNLOAD_ALL_AT_EXIT is not the default. I'm not sure if anyone is building
with it. Certainly, no-one is building with it and ithreads, because there is
an explosion of fail if you do. The problem is that DynaLoader is using
perl_atexit() to unmap shared libraries, and that's per *interpreter* exit.
So as soon as an ithread is spawned, its exit triggers the unmapping of all
shared objects, and the next time the parent thread tries to use one, kaboom!

I suspect that no-one is building with it, because since this (innocent)
commit,

commit 667763b
Author​: Father Chrysostomos <sprout@​cpan.org>
Date​: Thu Oct 4 21​:56​:00 2012 -0700

  Make PerlIO​::encoding more resilient to buffer changes

ext/PerlIO-encoding/t/encoding.t will start SEGVing, because something still
in use during global destruction is unmapped.

The issues are sort-of threefold

1) As ithreads are effectively forks as far as the interpreter goes, but are
  just threads as far as the OS knows, just calling perl_atexit() is the
  wrong point to call any sort of unloading, because it will happen in each
  ithreads
  We (at least) need to only do it in the top level interpreter
  However, we'd still leak if child interpreters dl_open() more things, so I
  guess that this means that ideally it ought to be a shared structure
  tracking this stuff
2) perl_atexit() fires too early during global destruction to be the right
  place to run this. As it's C-only code that needs running, it ought to be
  really late in global destruction, after all SVs have been freed.
  But the point at which perl_atexit() runs is (differently) useful so it
  can't be changed. It's sort of feeling like we need two (or more?) hook
  points in the global destruction sequence
3) as to dl_unload_file() at any other time, and as demonstrated by
  File​::Glob, we have a mess. Right now, it's never safe to unload arbitrary
  modules
  a) because we don't have a way to tell them to free up things
  b) possibly because they know that they can't (eg OP free hooks)
  c) definitely because the way we currently do hooks, eg OP free hooks, we
  encourage modules to store the old hook value, and restore it later.
  Which means that any other module might have a pointer to File​::Glob's
  innards, which File​::Glob has no idea about

Point (3) doesn't matter for last-stage global destruction unloading, which
I think was the specific problem that Alan was describing, and I suspect is
the least impossible to fix.

I think to properly solve the "at interpreter exit" cleanup we'd really need
to change dl_load_file() to internally track what it's been called for
(tracking shared across threads), and (I think) how often, with ownership
being upped on thread clone. At which point, interpreter exit is safe to call
dl_unload_all_files(). But only really late.

The current code is *in XS*, and kicks back to more XS code. I think that it
could be replaced by pure Perl code (as-is) with no loss of functionality.
But no actual bug fixes either.

#ifdef DL_UNLOAD_ALL_AT_EXIT
/* Close all dlopen'd files */
static void
dl_unload_all_files(pTHX_ void *unused)
{
  CV *sub;
  AV *dl_librefs;
  SV *dl_libref;

  if ((sub = get_cvs("DynaLoader​::dl_unload_file", 0)) != NULL) {
  dl_librefs = get_av("DynaLoader​::dl_librefs", 0);
  while ((dl_libref = av_pop(dl_librefs)) != &PL_sv_undef) {
  dSP;
  ENTER;
  SAVETMPS;
  PUSHMARK(SP);
  XPUSHs(sv_2mortal(dl_libref));
  PUTBACK;
  call_sv((SV*)sub, G_DISCARD | G_NODEBUG);
  FREETMPS;
  LEAVE;
  }
  }
}
#endif

Part of the fun of changing things is that it's provable that other code out
there already manipulates @​Dynaloader​::dl_librefs directly​:

  http​://grep.cpan.me/?q=dl_librefs

but I'm not sure if all code doing so

1) only adds *all* handles it got by calling dl_load_file()
2) takes no action based on @​dl_librefs - it thinks that it's just doing
  booking for DynaLoader

ie - I've not looked at the code on CPAN closely enough to work out whether
it would work to change dl_load_file() to internally track what it was called
with, and then have dl_unload_all_files() uses that and ignore
@​DynaLoader​::dl_librefs.

We possibly also need *two* hooks into shared objects. One for "this
interpreter is closing down", which would probably be XS code, and called
just before object cleanup. And a second, C code, for "the last interpreter
is now about to unload you".
Although, botherit, we might also want an XS level hook for "we're about to
start global destruction and you will be freed at the end".

Problem is that, like PerlIO, we want to be able to do things both when there
still is an interpreter (to be able to use it), and after it's gone (because
it was relying on things we provide)

We'd still be trusting shared objects not to leave signal handers live.
(and not to use atexit() routines, and I guess pthread_atfork() handlers,
as it looks like neither can be de-registered)

To solve the more general problem of dl_unload_file() - ie to be able to
unload shared objects *before* global destruction, I think that we'd also
need to revisit how we're doing hooks (*all* of them), and provide some
mechanism to register handlers for op_free, op_check, opcode overrides,
so that *only* the interpreter is remembering pointers to extension code.
We'd then need to "fix" everything on CPAN to be conformant.

And I think we'd also need those hooks described above, so that XS modules
can be told "you need to clean up" and undo whatever is necessary to unlink
themselves from the interpreter.

It's hard work to fix the first problem.
The "fix everything on CPAN" means that it's impossible to ever completely
fix the second. However, having a better hook system would be useful in
its own right, even if we never get to the point of doing enough to fix
the general dl_unload_file() issue.

Nicholas Clark

[p5pRT]

From @bulk88

On Fri Aug 09 00​:35​:40 2013, tonyc wrote​:

From reading the thread, apparently we (were|are) not releasing loaded
shared libraries when we clean up the interpreter.

This isn't a problem when perl is running standalone, but is a problem
when perl is embedded, since we're continuing to use resources after the
interpreter has been deleted.

As to the proposed solution, I wonder if freeing the libraries on
interpreter destruction is safe - it should be sort-of-safe where the
underlying library management tools reference count shared libraries
(true for dlopen and MSWin32), but I don't think it's safe otherwise.

Also, there doesn't seem to be an XS entry point (like boot_Foo for
module Foo) to do clean up for a loaded XS module, which means memory
may leak anyway (and other worse things, like dangling signal handlers.)

END block? package var with a DESTROY? I think I once did it with an XS
module by having an OS mutex protect a C reference count, to run the
unloader, whatever Perl level thing that triggered it has to obtain the
lock, and the count must be 0, objects cant initialize from other perl
threads because the mutex is being held. When the first thread clears
all the static resources and releases the lock, the 2nd thread will
reinitialize the DLL again. Also -1 could be placed in the reference
count to indicate to never reinitialize the DLL again in the process and
return an error/undef on Perl lang level if an object new was tried. I
am not referring to the Windows DLL loader, just Perl XS concepts.

Windows has DllMain which is always called at DLL unload time, but I
dont think that exists on any other OS.

Tony

related link
http​://www.nntp.perl.org/group/perl.perl5.porters/2000/02/msg8051.html

--
bulk88 ~ bulk88 at hotmail.com

[p5pRT]

From @nwc10

On Thu, Aug 22, 2013 at 07​:09​:36AM -0700, bulk88 via RT wrote​:

On Fri Aug 09 00​:35​:40 2013, tonyc wrote​:

From reading the thread, apparently we (were|are) not releasing loaded
shared libraries when we clean up the interpreter.

This isn't a problem when perl is running standalone, but is a problem
when perl is embedded, since we're continuing to use resources after the
interpreter has been deleted.

As to the proposed solution, I wonder if freeing the libraries on
interpreter destruction is safe - it should be sort-of-safe where the
underlying library management tools reference count shared libraries
(true for dlopen and MSWin32), but I don't think it's safe otherwise.

Also, there doesn't seem to be an XS entry point (like boot_Foo for
module Foo) to do clean up for a loaded XS module, which means memory
may leak anyway (and other worse things, like dangling signal handlers.)

END block? package var with a DESTROY? I think I once did it with an XS

As Ilya Z noted in one of the threads, Alan's initial suggestion of END is
way to early. END is run before objects are cleaned up. A package var with
a DESTROY would trigger *while* objects were being cleared up, which is still
too early, as any other objects relying on C or XS code in that library
might not yet be cleared up. (This is the same bug as Reini figured out in
Storable)

Ilya Z noted that it really needs to be done after everything Perl-like is
cleared up​:

  Alan Burlison writes​:
  > > Couldn't it be done in an END block inside DynaLoader?

  Too early. The only safe place (if any) should be
  after-the-end-of-global-destruction (assuming the list is kept in a C
  structure, thus is indestructible).

  Ilya

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-01/msg00843.html

which sort of gets asked again by Sarathy, to which Alan replies "pass"​:

  >> There was some doubt in my mind whether call_atexit() would do the job
  >> properly. What happens if the dll has allocated SVs? The call_atexit()
  >> callbacks are called after objects are destroyed but *before* the SV
  >> arenas are deallocated. Do you foresee any problems from that?
  >
  >Pass. Does that mean that @​dl_librefs will have been reclaimed?
 
  Not unless @​dl_librefs is an object.
 
  > If so,
  >the patch seems to work remarkably well considering it is popping stuff
  >of a free'd AV. I don't think the dll allocating SVs has anything to do
  >with it - dlclose() doesn't do any sort of cleanup [...]
 
  OK, I think that answers my question.
 
  >I wrote the patch based on what I thought was the consensus on the
  >correct way to do this - was I mislead?
 
  Doesn't look much like it. I just want to be sure we aren't overlooking
  any sharp corners.

http​://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/2000-02/msg01594.html

I think that we are overlooking potential sharp corners, and that Ilya Z
remains correct - if we are going to unload dynamic libraries, we have to do
it very late, after we've cleaned up all perl-related data structures, just
before we free the interpreter structure.

http​://www.nntp.perl.org/group/perl.perl5.porters/2000/02/msg8051.html

Thanks for the link

Nicholas Clark

[bulk88]

perl5/ext/DynaLoader/t/DynaLoader.t

Line 27 in 0eda2a5

%modules = (

perl5/ext/DynaLoader/t/DynaLoader.t

Line 157 in 0eda2a5

skip( "File::Glob sets PL_opfreehook", 2 ) if $module eq 'File::Glob';

Still a problem, adding any more XS deps, to those 6 XS modules, if #7 module, has an XSUB END(), ext/DynaLoader/t/DynaLoader.t SEGVs.