goto out of a block resets the last successful match when it should not.

[jo-37]

Description
This bug was detected by @choroba and there is a related discussion at perlmonks.

The "empty pattern" // and $& provide the last successfully matched pattern and string. Both get lost when jumping out of a block in several ways. Examples are next and goto.

The example below uses goto for a convenient way to switch between "jump out of a block" and "no jump out of a block". next without an inner block produces similar results.

After the first successful match, the "empty pattern" and $& behave as expected. A "out-of-block-jump" then clears both, making // an always matching genuine empty pattern and undefining $&.

Steps to Reproduce

#!/usr/bin/perl

use v5.12;
use warnings;

for my $label ('inner', 'outer') {
    say "\$goto:\t$label";
    for ('a' .. 'd') {
        say "\$_:\t'$_'";
        say "/b/:\t",  /b/ ? "match" : "no match";
        say "\$&:\t", defined $& ? "'$&'" : "undef";
        my $match = //;
        say "//:\t", $match ? "match" : "no match";
        {
            goto $label;
            inner:
        }
        outer:
    }
    say '';
}

Expected behavior
Annotated output from above script, note "ERROR":

$goto:  inner       # no jump off a block, expected behaviour
$_:     'a'
/b/:    no match    # /b/ does not match 'a'
$&:     undef       # no matched string yet
//:     match       # always matching genuine empty pattern
$_:     'b'
/b/:    match       # /b/ matches 'b'
$&:     'b'         # 'b' has matched /b/
//:     match       # // is /b/ and matches 'b'
$_:     'c'
/b/:    no match    # /b/ does not match 'c'
$&:     'b'         # last successfully matched string
//:     no match    # // is /b/ and does not match 'c'
$_:     'd'
/b/:    no match    # /b/ does not match 'd'
$&:     'b'         # last successfully matched string
//:     no match    # // is /b/ and does not match 'd'

$goto:  outer       # jump off a block, unexpected behaviour
$_:     'a'         # same as 'goto inner'
/b/:    no match
$&:     undef
//:     match
$_:     'b'         # same as 'goto inner'
/b/:    match
$&:     'b'
//:     match
$_:     'c'         # Things go wrong from here
/b/:    no match    # /b/ does not match 'c'
$&:     undef       # ERROR: the last successfully matched string has gone
//:     match       # ERROR: // is the always matching genuine empty pattern
$_:     'd'         # Same as 'c'
/b/:    no match
$&:     undef
//:     match

Perl configuration
Should be reproducible with more recent perl versions, too.

Summary of my perl5 (revision 5 version 32 subversion 1) configuration:
   
  Platform:
    osname=linux
    osvers=4.19.0
    archname=x86_64-linux-gnu-thread-multi
    uname='linux localhost 4.19.0 #1 smp debian 4.19.0 x86_64 gnulinux '
    config_args='-Dmksymlinks -Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -ffile-prefix-map=/dummy/build/dir=. -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-z,relro -Dlddlflags=-shared -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.32 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.32 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.32 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.32.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.32.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Ui_xlocale -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.32.1'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=define
    usemultiplicity=define
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='x86_64-linux-gnu-gcc'
    ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2 -g'
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion=''
    gccversion='10.2.1 20210110'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='x86_64-linux-gnu-gcc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
    libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
    perllibs=-ldl -lm -lpthread -lc -lcrypt
    libc=libc-2.31.so
    so=so
    useshrplib=true
    libperl=libperl.so.5.32
    gnulibc_version='2.31'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -L/usr/local/lib -fstack-protector-strong'


Characteristics of this binary (from libperl): 
  Compile-time options:
    HAS_TIMES
    MULTIPLICITY
    PERLIO_LAYERS
    PERL_COPY_ON_WRITE
    PERL_DONT_CREATE_GVSV
    PERL_IMPLICIT_CONTEXT
    PERL_MALLOC_WRAP
    PERL_OP_PARENT
    PERL_PRESERVE_IVUV
    USE_64_BIT_ALL
    USE_64_BIT_INT
    USE_ITHREADS
    USE_LARGE_FILES
    USE_LOCALE
    USE_LOCALE_COLLATE
    USE_LOCALE_CTYPE
    USE_LOCALE_NUMERIC
    USE_LOCALE_TIME
    USE_PERLIO
    USE_PERL_ATOF
    USE_REENTRANT_API
    USE_THREAD_SAFE_LOCALE
  Locally applied patches:
    DEBPKG:debian/cpan_definstalldirs - Provide a sensible INSTALLDIRS default for modules installed from CPAN.
    DEBPKG:debian/db_file_ver - https://bugs.debian.org/340047 Remove overly restrictive DB_File version check.
    DEBPKG:debian/doc_info - Replace generic man(1) instructions with Debian-specific information.
    DEBPKG:debian/enc2xs_inc - https://bugs.debian.org/290336 Tweak enc2xs to follow symlinks and ignore missing @INC directories.
    DEBPKG:debian/errno_ver - https://bugs.debian.org/343351 Remove Errno version check due to upgrade problems with long-running processes.
    DEBPKG:debian/libperl_embed_doc - https://bugs.debian.org/186778 Note that libperl-dev package is required for embedded linking
    DEBPKG:fixes/respect_umask - Respect umask during installation
    DEBPKG:debian/writable_site_dirs - Set umask approproately for site install directories
    DEBPKG:debian/extutils_set_libperl_path - EU:MM: set location of libperl.a under /usr/lib
    DEBPKG:debian/no_packlist_perllocal - Don't install .packlist or perllocal.pod for perl or vendor
    DEBPKG:debian/fakeroot - Postpone LD_LIBRARY_PATH evaluation to the binary targets.
    DEBPKG:debian/instmodsh_doc - Debian policy doesn't install .packlist files for core or vendor.
    DEBPKG:debian/ld_run_path - Remove standard libs from LD_RUN_PATH as per Debian policy.
    DEBPKG:debian/libnet_config_path - Set location of libnet.cfg to /etc/perl/Net as /usr may not be writable.
    DEBPKG:debian/perlivp - https://bugs.debian.org/510895 Make perlivp skip include directories in /usr/local
    DEBPKG:debian/squelch-locale-warnings - https://bugs.debian.org/508764 Squelch locale warnings in Debian package maintainer scripts
    DEBPKG:debian/patchlevel - https://bugs.debian.org/567489 List packaged patches for 5.32.1-4+deb11u2 in patchlevel.h
    DEBPKG:fixes/document_makemaker_ccflags - https://bugs.debian.org/628522 [rt.cpan.org #68613] Document that CCFLAGS should include $Config{ccflags}
    DEBPKG:debian/find_html2text - https://bugs.debian.org/640479 Configure CPAN::Distribution with correct name of html2text
    DEBPKG:debian/perl5db-x-terminal-emulator.patch - https://bugs.debian.org/668490 Invoke x-terminal-emulator rather than xterm in perl5db.pl
    DEBPKG:debian/cpan-missing-site-dirs - https://bugs.debian.org/688842 Fix CPAN::FirstTime defaults with nonexisting site dirs if a parent is writable
    DEBPKG:fixes/memoize_storable_nstore - [rt.cpan.org #77790] https://bugs.debian.org/587650 Memoize::Storable: respect 'nstore' option not respected
    DEBPKG:debian/makemaker-pasthru - https://bugs.debian.org/758471 Pass LD settings through to subdirectories
    DEBPKG:debian/makemaker-manext - https://bugs.debian.org/247370 Make EU::MakeMaker honour MANnEXT settings in generated manpage headers
    DEBPKG:debian/kfreebsd-softupdates - https://bugs.debian.org/796798 Work around Debian Bug#796798
    DEBPKG:fixes/memoize-pod - [rt.cpan.org #89441] Fix POD errors in Memoize
    DEBPKG:debian/hurd-softupdates - https://bugs.debian.org/822735 Fix t/op/stat.t failures on hurd
    DEBPKG:fixes/math_complex_doc_great_circle - https://bugs.debian.org/697567 [rt.cpan.org #114104] Math::Trig: clarify definition of great_circle_midpoint
    DEBPKG:fixes/math_complex_doc_see_also - https://bugs.debian.org/697568 [rt.cpan.org #114105] Math::Trig: add missing SEE ALSO
    DEBPKG:fixes/math_complex_doc_angle_units - https://bugs.debian.org/731505 [rt.cpan.org #114106] Math::Trig: document angle units
    DEBPKG:fixes/cpan_web_link - https://bugs.debian.org/367291 CPAN: Add link to main CPAN web site
    DEBPKG:debian/hppa_op_optimize_workaround - https://bugs.debian.org/838613 Temporarily lower the optimization of op.c on hppa due to gcc-6 problems
    DEBPKG:debian/installman-utf8 - https://bugs.debian.org/840211 Generate man pages with UTF-8 characters
    DEBPKG:debian/hppa_opmini_optimize_workaround - https://bugs.debian.org/869122 Lower the optimization level of opmini.c on hppa
    DEBPKG:debian/sh4_op_optimize_workaround - https://bugs.debian.org/869373 Also lower the optimization level of op.c and opmini.c on sh4
    DEBPKG:debian/perldoc-pager - https://bugs.debian.org/870340 [rt.cpan.org #120229] Fix perldoc terminal escapes when sensible-pager is less
    DEBPKG:debian/prune_libs - https://bugs.debian.org/128355 Prune the list of libraries wanted to what we actually need.
    DEBPKG:debian/mod_paths - Tweak @INC ordering for Debian
    DEBPKG:debian/configure-regen - https://bugs.debian.org/762638 Regenerate Configure et al. after probe unit changes
    DEBPKG:debian/deprecate-with-apt - https://bugs.debian.org/747628 Point users to Debian packages of deprecated core modules
    DEBPKG:debian/disable-stack-check - https://bugs.debian.org/902779 [GH #16607] Disable debugperl stack extension checks for binary compatibility with perl
    DEBPKG:debian/perlbug-editor - https://bugs.debian.org/922609 Use "editor" as the default perlbug editor, as per Debian policy
    DEBPKG:debian/eu-mm-perl-base - https://bugs.debian.org/962138 Suppress an ExtUtils::MakeMaker warning about our non-default @INC
    DEBPKG:fixes/hurd-cachepropagate-test-fix - https://bugs.debian.org/963214 GNU/Hurd doesn't support SO_PROTOCOL
    DEBPKG:fixes/io_socket_ip_ipv6 - Disable getaddrinfo(3) AI_ADDRCONFIG for localhost and IPv4 numeric addresses
    DEBPKG:fixes/encode-CVE-2021-36770 - mitigate @INC pollution when loading ConfigLocal
    DEBPKG:fixes/regcomp-memleak - https://bugs.debian.org/994834 [GH #18604] [5f41fa4] regcomp.c: Remove memory leak
  Built under linux
  Compiled at Sep 24 2021 16:10:58
  %ENV:
    PERL_UNICODE="SDAL"
  @INC:
    /etc/perl
    /usr/local/lib/x86_64-linux-gnu/perl/5.32.1
    /usr/local/share/perl/5.32.1
    /usr/lib/x86_64-linux-gnu/perl5/5.32
    /usr/share/perl5
    /usr/lib/x86_64-linux-gnu/perl-base
    /usr/lib/x86_64-linux-gnu/perl/5.32
    /usr/share/perl/5.32
    /usr/local/lib/site_perl

[jkeenan]

Description This bug was detected by @choroba and there is a related discussion at perlmonks.

The "empty pattern" // and $& provide the last successfully matched pattern and string. Both get lost when jumping out of a block in several ways. Examples are next and goto.

Can you provide a useful definition of "get lost"?

[jo-37]

Sure: // does no longer match the "last successfully matching pattern" and $& becomes undefined.

[mauke]

I've tested it using:

$ perl -wle '"hi" =~ /./ or die; print $&; { goto out; } out: print $&;'
h
Use of uninitialized value $& in print at -e line 1.

$

The output is consistent going back to 5.8.9; i.e. this hasn't worked right since at least 5.8.9 (and possibly never).

(However, I can't reproduce it like this using next or last.)

[choroba]

Just for completeness and for those who are allergic to PerlMonks, here's the original example:

$ perl -le 'print for a .. z' | perl  -nle 'if (/d/ .. /h/) { next unless //; print }'
d
f
g
h

The question is why f and g are printed.

[jo-37]

@mauke: This is certainly the erroneous behavior I reported. $& should be "h", but is undefined instead. You need a loop around this to reproduce it with next:

echo hi | perl -wle 'while (<>) {/./ or die; print "innner:$&"; next;}  print "outer:$&";
innner:h
Use of uninitialized value $& in concatenation (.) or string at -e line 1, <> line 1.
outer:

[mauke]

@jo-37 That example matches my expectations since match variables are dynamically scoped, so leaving the scope of the successful match makes them disappear.

[jo-37]

@mauke: Please take a deeper look at my example script. Replacing the inner block containing the goto with a simple next does not leave any scope but fails, too.

use v5.12;
use warnings;

for ('a' .. 'd') {
    say "\$_:\t'$_'";
    say "/b/:\t",  /b/ ? "match" : "no match";
    say "\$&:\t", defined $& ? "'$&'" : "undef";
    my $match = //;
    say "//:\t", $match ? "match" : "no match";
    next;
}
__DATA__
$_:	'a'
/b/:	no match
$&:	undef
//:	match
$_:	'b'
/b/:	match
$&:	'b'
//:	match
$_:	'c'
/b/:	no match
$&:	undef        #ERROR
//:	match        #ERROR
$_:	'd'
/b/:	no match
$&:	undef
//:	match

[jo-37]

For completeness, here is a one-liner using next

$ perl -wle 'for (0, 1) {/0/; print $&; next}'
0
Use of uninitialized value $& in print at -e line 1.

[demerphq]

For completeness, here is a one-liner using next

$ perl -wle 'for (0, 1) {/0/; print $&; next}'
0
Use of uninitialized value $& in print at -e line 1.

Not a bug. When the loop exits the match state is reset.

Consider this reformulation of the code:

$ perl -wle '"x"=~/x/; for (0, 1) {/0/; print $&; next}'
0
x

It prints 'x' because $& is bound to the last successful match in scope.

A successful match behaves much like a local declaration. When the scope exits the localization is undone.

It is morally equivalent to how $s behaves in the following:

$ perl -wle 'our $s="x"; for (0, 1) {print $s; local $s = $_ if !$_; print $s; next;}'
x
0
x
x

[demerphq]

@choroba wrote:

The question is why f and g are printed.

Because when there is no last successful match in the current scope the pattern is treated as being the empty string.

In the most recent Perl release I added ${^LAST_SUCCESSFUL_PATTERN}, which you can use to check what is going on.

perl -le 'print for c .. i' | ./perl -Ilib -nle 'if (/d/ .. /h/) { use re "debug"; warn "LSP=",${^LAST_SUCCESSFUL_PATTERN} // "EMPTY PATTERN"; next unless //; warn $_ }'
Compiling REx ""
Final program:
   1: NOTHING (2)
   2: END (0)
minlen 0 
LSP=(?^:d) at -e line 1, <> line 2.
d at -e line 1, <> line 2.
LSP=(?^:d) at -e line 1, <> line 3.
LSP=EMPTY PATTERN at -e line 1, <> line 4.
Matching REx "" against "f"
   0 <> <f>                  |   0| 1:NOTHING(2)
   0 <> <f>                  |   0| 2:END(0)
Match successful!
f at -e line 1, <> line 4.
LSP=EMPTY PATTERN at -e line 1, <> line 5.
Matching REx "" against "g"
   0 <> <g>                  |   0| 1:NOTHING(2)
   0 <> <g>                  |   0| 2:END(0)
Match successful!
g at -e line 1, <> line 5.
LSP=(?^:h) at -e line 1, <> line 6.
h at -e line 1, <> line 6.
Freeing REx: ""

The only bug I see in this thread is the one reported by @mauke, rewritten here to use ${^LAST_SUCCESSFUL_PATTERN}.

./perl -wle '"hi" =~ /./ or die; print ${^LAST_SUCCESSFUL_PATTERN} // "undef"; { goto out; } out: print ${^LAST_SUCCESSFUL_PATTERN} // "undef";'
(?^:.)
undef

The goto out of the scope shouldn't clear the pattern, it should be the same as it would be if the goto was removed. But I think that deserves its own ticket.

I am closing this ticket, as not a bug.

[demerphq]

Actually i will leave it, and just rename it.