Problem with bleadperl of 7/18/00

[p5pRT]

Migrated from rt.perl.org#3552 (status was 'resolved')

Searchable as RT3552$

[p5pRT]

From schinder@pobox.com

Created by schinder@c22234-c.scllg1.pa.home.com

I rsynced a copy of bleadperl last night EDT, 7/18/00. Looks like there's
a problem with one of the tests, comp/require.t​:

linux% ./perl -I ../lib comp/require.t
1..23
ok 1
ok 2
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
ok 21
ok 22
Out of memory!

Here's the perl -V for bleadperl (since I was too lazy to run perlbug
under it)​:

linux% ./perl -I../lib -V
Summary of my perl5 (revision 5.0 version 6 subversion 0) configuration​:
  Platform​:
  osname=linux, osvers=2.4.0-test1-ac21, archname=ppc-linux
  uname='linux c22234-c.scllg1.pa.home.com 2.4.0-test1-ac21 #1 wed jun 21 23​:15​:08 edt 2000 ppc unknown '
  config_args=''
  hint=recommended, useposix=true, d_sigaction=define
  usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
  useperlio=undef d_sfio=undef uselargefiles=undef
  use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=undef
  Compiler​:
  cc='gcc', optimize='-O2', gccversion=2.95.3 19991030 (prerelease/franzo/20000625)
  cppflags='-fno-strict-aliasing -I/usr/local/include'
  ccflags ='-fno-strict-aliasing -I/usr/local/include'
  stdchar='char', d_stdstdio=define, usevfork=false
  intsize=4, longsize=4, ptrsize=4, doublesize=8
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=4
  alignbytes=8, usemymalloc=n, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -L/usr/local/lib'
  libpth=/usr/local/lib /lib /usr/lib
  libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lc -lposix -lcrypt
  libc=/lib/libc-2.1.3.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
  cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​:
  Built under linux
  Compiled at Jul 18 2000 22​:05​:33
  @​INC​:
  ../lib
  /usr/local/lib/perl5/5.6.0/ppc-linux
  /usr/local/lib/perl5/5.6.0
  /usr/local/lib/perl5/site_perl/5.6.0/ppc-linux
  /usr/local/lib/perl5/site_perl/5.6.0
  /usr/local/lib/perl5/site_perl/5.5.670/ppc-linux
  /usr/local/lib/perl5/site_perl/5.5.670
  /usr/local/lib/perl5/site_perl/5.5.660/ppc-linux
  /usr/local/lib/perl5/site_perl/5.5.660
  /usr/local/lib/perl5/site_perl/5.5.650/ppc-linux
  /usr/local/lib/perl5/site_perl/5.5.650
  /usr/local/lib/perl5/site_perl/5.5.640/ppc-linux
  /usr/local/lib/perl5/site_perl/5.5.640
  /usr/local/lib/perl5/site_perl/5.005/ppc-linux
  /usr/local/lib/perl5/site_perl/5.005
  /usr/local/lib/perl5/site_perl
  .

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl v5.6.0:

Configured by schinder at Thu Mar 23 08:21:29 EST 2000.

Summary of my perl5 (revision 5.0 version 6 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.2.14, archname=ppc-linux
    uname='linux c22234-c.scllg1.pa.home.com 2.2.14 #2 sun jan 9 20:34:00 est 2000 ppc unknown '
    config_args='-Dcc=gcc -Dprefix=/usr/local -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=undef d_sfio=undef uselargefiles=define 
    use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=undef
  Compiler:
    cc='gcc', optimize='-O2', gccversion=2.95.2 19991024 (release/franzo)
    cppflags='-fno-strict-aliasing -I/usr/local/include'
    ccflags ='-fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    stdchar='char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, usemymalloc=n, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lc -lposix -lcrypt
    libc=/lib/libc-2.1.3.so, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl v5.6.0:
    /usr/local/lib/perl5/5.6.0/ppc-linux
    /usr/local/lib/perl5/5.6.0
    /usr/local/lib/perl5/site_perl/5.6.0/ppc-linux
    /usr/local/lib/perl5/site_perl/5.6.0
    /usr/local/lib/perl5/site_perl/5.5.670/ppc-linux
    /usr/local/lib/perl5/site_perl/5.5.670
    /usr/local/lib/perl5/site_perl/5.5.660/ppc-linux
    /usr/local/lib/perl5/site_perl/5.5.660
    /usr/local/lib/perl5/site_perl/5.5.650/ppc-linux
    /usr/local/lib/perl5/site_perl/5.5.650
    /usr/local/lib/perl5/site_perl/5.5.640/ppc-linux
    /usr/local/lib/perl5/site_perl/5.5.640
    /usr/local/lib/perl5/site_perl/5.005/ppc-linux
    /usr/local/lib/perl5/site_perl/5.005
    /usr/local/lib/perl5/site_perl
    .


Environment for perl v5.6.0:
    HOME=/usr/local/home/schinder
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=.:/usr/local/bin:/bin:/sbin:/usr/bin:/usr/etc:/usr/local/sbin:/usr/sbin:/usr/X11R6/bin:/usr/local/home/schinder/bin:/var/qmail/bin
    PERL_BADLANG (unset)
    SHELL=/usr/local/bin/tcsh

[p5pRT]

From @jhi

On Wed, Jul 19, 2000 at 03​:41​:43PM -0000, schinder@​pobox.com wrote​:

This is a bug report for perl from schinder@​c22234-c.scllg1.pa.home.com,
generated with the help of perlbug 1.28 running under perl v5.6.0.

-----------------------------------------------------------------
[Please enter your report here]
I rsynced a copy of bleadperl last night EDT, 7/18/00. Looks like there's
a problem with one of the tests, comp/require.t​:

linux% ./perl -I ../lib comp/require.t
1..23

The subtest #23 of the comp/require test is broken. I haven't yet had
the time to sit down and figure out what's wrong. If anybody feels
inteersted, take a look at swallow_bom() in toke.c.

[p5pRT]

From [Unknown Contact. See original ticket]

[ Note to bugmongers​: this fixes the following other bug reports​:
  20000716.016 20000716.021 20000718.002 20000719.001 20000721.002
]

Jarkko Hietaniemi <jhi@​iki.fi> wrote

The subtest #23 of the comp/require test is broken. I haven't yet had
the time to sit down and figure out what's wrong. If anybody feels
inteersted, take a look at swallow_bom() in toke.c.

The attached patch fixes this. It's a long way from ideal for the
reasons listed below, but it does prevent the SEGVs and does allow
"make test" to run cleanly.

Looking at the BOM code and at the UTF16 stuff, there appears to be a
lot wrong with it. I gave up studying it in detail, so this list
certainly isn't exhaustive.

i) Shouldn't the swallow_bom() call in toke.c be earlier, e.g.
  immediately after where bof is set? Currently you can't have
  e.g. a UTF16 sharpbang line.

ii) utf16_to_utf8() in utf8.c uses U16 as if were *exactly* 2 bytes.
  But of course it isn't

iii) utf16_to_utf8() doesn't seem to check for odd length

iv) Shouldn't the warning "Malformed UTF-16 surrogate" be a hard error?
  And it should the offending chars be included in the message?
  Also it doesn't have a perldiag entry

v) This comment in utf8.c is out of date :-)
  /* XXX NOTHING CALLS THE FOLLOWING TWO ROUTINES YET!!! */

vi) The tests are currently a joke. E.g. the UTF16 BOM really ought
  to be followed by valid UTF16 characters.

vii) I didn't bother to explore exactly *why* PL_bufend needed refreshing.
  As a general principle, since PL_bufend is so fragile, shouldn't
  it's use be avoided except in "hot" code? swallow_bom()
  certainly isn't hot.

Mike Guy

Inline Patch

--- ./t/comp/require.t.orig	Tue Jul 11 22:19:23 2000
+++ ./t/comp/require.t	Tue Jul 25 11:01:17 2000
@@ -132,7 +132,8 @@
 do_require("$utf8\nprint \"ok $i\n\"; 1;\n");
 $i++;
 do_require("$utf16\n1;");
-print "ok $i\n" if $@ =~ /Unsupported script encoding/;
+print "not " unless $@ =~ /^Unrecognized character /;
+print "ok $i\n";
 
 END { 1 while unlink 'bleah.pm'; 1 while unlink 'bleah.do'; }
 
--- ./toke.c.orig	Tue Jul 25 09:29:37 2000
+++ ./toke.c	Tue Jul 25 09:41:46 2000
@@ -2519,6 +2519,7 @@
 		}
 	    } 
 	    if (bof)
+		PL_bufend = SvPVX(PL_linestr) + SvCUR(PL_linestr);
 		s = swallow_bom((U8*)s);
 	    incline(s);
 	} while (PL_doextract);

End of patch

[p5pRT]

From [Unknown Contact. See original ticket]

Mike

@​@​ -2519,6 +2519,7 @​@​
}
}
if (bof)
+ PL_bufend = SvPVX(PL_linestr) + SvCUR(PL_linestr);
s = swallow_bom((U8*)s);
incline(s);
} while (PL_doextract);

End of patch

I have not looked at this closely, but the indendation suggests you want

@​@​ -2519,6 +2519,9 @​@​
  }
  }
  if (bof)
+ {
+ PL_bufend = SvPVX(PL_linestr) + SvCUR(PL_linestr);
  s = swallow_bom((U8*)s);
+ }
  incline(s);
  } while (PL_doextract);

Robin

[p5pRT]

From [Unknown Contact. See original ticket]

Robin Barker <rmb1@​cise.npl.co.uk> wrote

I have not looked at this closely, but the indendation suggests you want

D'oh! Yes, of course I do.

The amazing thing is that "make test" didn't bat an eyelid.

Revised version of whole patch attached.

Mike Guy

Inline Patch

--- ./t/comp/require.t.orig	Tue Jul 25 12:31:43 2000
+++ ./t/comp/require.t	Tue Jul 25 11:01:17 2000
@@ -132,7 +132,8 @@
 do_require("$utf8\nprint \"ok $i\n\"; 1;\n");
 $i++;
 do_require("$utf16\n1;");
-print "ok $i\n" if $@ =~ /Unsupported script encoding/;
+print "not " unless $@ =~ /^Unrecognized character /;
+print "ok $i\n";
 
 END { 1 while unlink 'bleah.pm'; 1 while unlink 'bleah.do'; }
 
--- ./toke.c.orig	Tue Jul 25 12:31:43 2000
+++ ./toke.c	Tue Jul 25 12:32:38 2000
@@ -2519,7 +2519,10 @@
 		}
 	    } 
 	    if (bof)
+	    {
+		PL_bufend = SvPVX(PL_linestr) + SvCUR(PL_linestr);
 		s = swallow_bom((U8*)s);
+	    }
 	    incline(s);
 	} while (PL_doextract);
 	PL_oldoldbufptr = PL_oldbufptr = PL_bufptr = PL_linestart = s;

End of patch

[p5pRT]

From @jhi

On Tue, Jul 25, 2000 at 12​:52​:45PM +0100, M.J.T. Guy wrote​:

Robin Barker <rmb1@​cise.npl.co.uk> wrote

I have not looked at this closely, but the indendation suggests you want

D'oh! Yes, of course I do.

The amazing thing is that "make test" didn't bat an eyelid.

Revised version of whole patch attached.

Thanks. I'll also add/act on your notes from the earlier attempt.