Undef loop while condition, loop code motion, and bad warning line number? 5.10-RC1 + 5.8.8

[p5pRT]

Migrated from rt.perl.org#47632 (status was 'open')

Searchable as RT47632$

[p5pRT]

From William.Ricker@FMR.COM

Created by a273121@fmr.com

Summary -

A uninitialized undef Value in
  while($mightbeundef ... ){...
results in warning that have strange line number, in some cases.

User ID --

a273121@​fmr.com on Linux = William.Ricker@​FMR.com more generally
  = is also known as use.perl.org/~n1vux , Boston.pm type person

Version + BugLevel -

BOTH 5.10-RC1 (Linux) + 5.8.8/5.9.5 (Windows), possibly others.

"Medium" may be over-high for a misleading warning that 5.10 *has*
improved.
However, this Bug applies to 5.8.8 as well, where the better messages in
5.10 doesn't help figure out the bad line number, so more serious on
5.8.8. I'd be happy with LOW on 5.10 and MEDIUM on 5.8.8.

Figured we should send it in during 5.10-RC1 so you have a chance to get
the fix into 5.10 if it's easy, since we can reproduce it in RC1.

Speculation -

This may be related to code-motion optimization in loop, as it uses the
line# of last statement of body as line# for control expression many
lines before in source. (Simplest versions of code get correct line #.)

Background / detail -

A user adopting a complex non-CPAN published module (COBOLIO.pm)
reported
a strange uninitialized value warning for a line unrelated to any
variable.

Inserting !defined in loop control expression of the loop of which
the line numer given was last executable statement made the warning go
away.

(This *will* [has been?] be reported to module author separately as a
module bug.
This report is about the *bad* line# on the warning.)

Using locally built 5.9.5 (Windows) or 5.10.0_RC1 (Linux) gives a
better
warning, with name of variable, great progress, but still not right line
number.

REPRODUCTION TESTCASE --

Our first attempt to remove all other bits of the module to make
a simple test case failed; the simplest test case warning warned the
"right" line.
But we found something just simple enough.

Here's the simple reproduction that tricks it into appearing.
The following should warn on use at line 12, warns at 14.

[perl-5.10.0-RC1]$ cat bad2.pl
#! bleedperl/bin/perl

use strict;
use warnings;

our @​recLevel= ();
our $currLevel=9;
our $level=3;

sub foo {
  my (@​elsewhere, $c); # temp
  while($currLevel > $level) { # 12
  $c = $currLevel = pop @​recLevel; # 13
  push @​elsewhere, $c; # 14
  } # 15

}

  foo();===EOF===

[perl-5.10.0-RC1]$ perl510 bad2.pl
Use of uninitialized value $currLevel in numeric gt (>) at bad2.pl line
14.

Note there is no $currLevel in line #14. (If not using $c temp,
there would be and it would be MORE confusing!)

IMPACT UNDER 5.8.8 --

If any other uses of '>' occur in loop, the 5.8.8 version of this
message is

  Use of uninitialized value in numeric gt (>) at bad2.pl line 14.

Which is even more misleading. This confusion was such that our user
downloaded the tarball for 5.9.5 to get the 5.10 better message,
which solved the conundrum. Hence I think this is long term
more important for a 5.8.x maintenance release; if it is triaged
to miss 5.10.0 and is fixed in 5.8.9 and ported to 5.10.1,
that's completely understandable, we just want the p5p's to have
the option.

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.10.0:

Configured by a273121 at Tue Nov 20 15:16:33 EST 2007.

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.6.9-55.elsmp, archname=x86_64-linux
    uname='linux lmro273 2.6.9-55.elsmp #1 smp fri apr 20 16:36:54 edt
2007 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des
-Dprefix=/export/home/a273121/perl-5.10.0-RC1/bleedperl
-Dusesitecustomize -Duserelocatableinc'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -pipe -I/usr/local/include
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='3.4.6 20060404 (Red Hat 3.4.6-8)',
gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64
/usr/local/lib64
    libs=-lnsl -ldl -lm -lcrypt -lutil -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc
    libc=/lib/libc-2.3.4.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.3.4'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib'

Locally applied patches:
    RC1


@INC for perl 5.10.0:
 
/export/home/a273121/perl-5.10.0-RC1/bleedperl/lib/5.10.0/x86_64-linux
    /export/home/a273121/perl-5.10.0-RC1/bleedperl/lib/5.10.0
 
/export/home/a273121/perl-5.10.0-RC1/bleedperl/lib/site_perl/5.10.0/x86_
64-linux
    /export/home/a273121/perl-5.10.0-RC1/bleedperl/lib/site_perl/5.10.0
    .


Environment for perl 5.10.0:
    HOME=/export/home/a273121
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
 
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/fisc/uts/bin:/usr/X
11R6/bin
    PERL_BADLANG (unset)
    SHELL=/bin/ksh

[p5pRT]

From @schwern

Ricker, William (via RT) wrote​:

A uninitialized undef Value in
while($mightbeundef ... ){...
results in warning that have strange line number, in some cases.

Thanks for your very through report.

This is like a consequence of a very old issue that nobody's quite figured out
how to resolve. Perl tends to get a little confused with regard to getting
the line number right around a block condition.

However, I notice that 5.6.2 got this right. Andreas, could you apply your
binary patch searcher to this problem?

--
ROCKS FALL! EVERYONE DIES!
  http​://www.somethingpositive.net/sp05032002.shtml

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @andk

On Wed, 21 Nov 2007 13​:32​:58 -0800, Michael G Schwern <schwern@​pobox.com> said​:

  > Ricker, William (via RT) wrote​:

A uninitialized undef Value in
while($mightbeundef ... ){...
results in warning that have strange line number, in some cases.

  > Thanks for your very through report.

  > This is like a consequence of a very old issue that nobody's quite figured out
  > how to resolve. Perl tends to get a little confused with regard to getting
  > the line number right around a block condition.

  > However, I notice that 5.6.2 got this right. Andreas, could you apply your
  > binary patch searcher to this problem?

19610 looks very suspicious​:

Change 19610 by jhi@​kosh on 2003/05/24 06​:42​:52

  Subject​: [PATCH #2] Re​: [perl #22181] goto undefines my() variables
  From​: Dave Mitchell <davem@​fdgroup.com>
  Date​: Thu, 22 May 2003 10​:13​:19 +0100
  Message-ID​: <20030522091319.GA4568@​fdgroup.com>
 
  Subject​: Re​: [PATCH #2] Re​: [perl #22181] goto undefines my() variables
  From​: Dave Mitchell <davem@​fdgroup.com>
  Date​: Fri, 23 May 2003 17​:09​:44 +0100
  Message-ID​: <20030523160944.GC9194@​fdgroup.com>

  ----Program----

  use strict;
  use warnings;

  our @​recLevel= ();
  our $currLevel=9;
  our $level=3;

  sub foo {
  my (@​elsewhere, $c); # temp
  while($currLevel > $level) { # 12
  $c = $currLevel = pop @​recLevel; # 13
  push @​elsewhere, $c; # 14
  } # 15

  }
  $SIG{__WARN__} = sub { my $warn = shift; print $warn =~ /line (\d+)/ ? "->$1<-\n" : $warn; };

  foo();

  ----Output of .../poKIX8P/perl-5.8.0@​19608/bin/perl----
  ->12<-

  ----EOF ($?='0')----
  ----Output of .../ppLsGGj/perl-5.8.0@​19610/bin/perl----
  ->14<-

  ----EOF ($?='0')----

--
andreas

[p5pRT]

From @iabyn

On Fri, Nov 23, 2007 at 10​:45​:32PM +0100, Andreas J. Koenig wrote​:

On Wed, 21 Nov 2007 13​:32​:58 -0800, Michael G Schwern <schwern@​pobox.com> said​:

Ricker, William (via RT) wrote​:

A uninitialized undef Value in
while($mightbeundef ... ){...
results in warning that have strange line number, in some cases.

Thanks for your very through report.

This is like a consequence of a very old issue that nobody's quite figured out
how to resolve. Perl tends to get a little confused with regard to getting
the line number right around a block condition.

However, I notice that 5.6.2 got this right. Andreas, could you apply your
binary patch searcher to this problem?

19610 looks very suspicious​:

This is mainly a red herring. The basic problem is that line numbers are
only recorded once per statement rather than once per op. This gives rise
to quite a few line-numbering bugs (especially with if/then and while
loops). Change 19610 just happened to alter when and where nextstate gets
executed as as side-effect of fixing a bug.

The real fix is to record the line number in every op, at the cost of
making ops bigger. Something to do post 5.10.0 ...

--
Never work with children, animals, or actors.

[p5pRT]

From @nwc10

On Sat, Nov 24, 2007 at 01​:28​:31AM +0000, Dave Mitchell wrote​:

The real fix is to record the line number in every op, at the cost of
making ops bigger. Something to do post 5.10.0 ...

Or at least in every op where the line number changed.

Could one get away with a null op in the same family as nextstate that would
be able to record that line number change without getting in the way of
execution?

Nicholas Clark

[p5pRT]

From William.Ricker@FMR.COM

1) Obviously I intend a newline (or #) after foo(); and before
===EOF=== annotation,
  perhaps I should use __END__ ...
  (VI famously omits NL at EOF, and I pasted ... oopsie)

2) Same result with perl-5.10-RC2 [although RC2 was harder to build
relocatable]
  Use of uninitialized value $currLevel in numeric gt (>) at bad2.pl
line 14.
  (should be @​ 12)

  while($currLevel > $level) { # 12
  $c = $currLevel = pop @​recLevel; # 13
  push @​elsewhere, $c; # 14
  } # 15

3) Before someone asks, Yes, 'make test' succeeds on my RC1 & RC2 Linux
builds.

4) I finally found an AIX5.3 development system with the newer VAC 6
and space on $HOME.
  I get same result there too, with RC2. ('make test' ok)

William Ricker
Principal (swe), Architecture Services
Fidelity Investments / FPCMS / Systems
617-563-0648
M/S Z1E
William.Ricker@​FMR.com
1141547 Skytel
http​://www.linkedin.com/in/n1vux

[p5pRT]

From William.Ricker@FMR.COM

Michael, Andreas, Dave, Nick --

Thanks for the great explanation.

We are not surprised that it's not an easy fix for 5.10.0; that's pretty
much what we expected from dark-ages compiler theory dimly remembered.
But the ideas for alternative future solutions are fascinating, and is
more hopeful than we'd dared hope.

I have not come up with an example that is not erroneous; but then,
warnings are to help the erroneous. At least 5.10 tells us WHAT variable
became undef, even if not the right line sometimes. The module that
evoked this warning should probably have a
  last unless $currLevel; # hit bottom early, so done
inserted at the bottom or
  defined($currLevel) and
in the condition at the top, to avoid runtime blather to STDERR and
scaring module users.
We _have_ reported this, but haven't submitted a patch since we're not
Absolutely certain that that is the correct fix to the module.

[for BCC​:'s - see discussion at
http​://rt.perl.org/rt3/Public/Bug/Display.html?id=47632 ]

Cheers,

BILL in Boston

William Ricker
Principal (swe), Architecture Services
Fidelity Investments / FPCMS / Systems
+1 617-563-0648
William.Ricker@​FMR.com
http​://www.linkedin.com/in/n1vux
Aka http​://use.perl.org/~n1vux
  + http​://boston.pm.org/kwiki/index.cgi?BillRicker
  + bill.n1vux@​gmail.com and maybe even uunet!world!wdr