-e ignores the UTF8 flag

[p5pRT]

Migrated from rt.perl.org#77242 (status was 'open')

Searchable as RT77242$

[p5pRT]

From @cpansprout

#!perl -l
$fn = "\xe2\x80\x99";
`touch \Q$fn`;
print -e $fn; # prints 1
print -e substr "$fn\x{100}", -1; # prints nothing

This can happen easily if the file name comes from a URI (URIs always represent byte sequences) and the URI happened to come from a UTF-8 web page.

---

Flags​:
  category=core
  severity=medium

---

Site configuration information for perl 5.13.3​:

Configured by sprout at Thu Aug 12 17​:53​:37 PDT 2010.

Summary of my perl5 (revision 5 version 13 subversion 3 patch v5.13.3-193-g798ae1b) configuration​:
  Snapshot of​: 798ae1b
  Platform​:
  osname=darwin, osvers=10.4.0, archname=darwin-2level
  uname='darwin pint.local 10.4.0 darwin kernel version 10.4.0​: fri apr 23 18​:28​:53 pdt 2010; root​:xnu-1504.7.4~1release_i386 i386 '
  config_args='-de -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
  optimize='-O3',
  cppflags='-no-cpp-precomp -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.2.1 (Apple Inc. build 5664)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib
  libs=-ldbm -ldl -lm -lutil -lc
  perllibs=-ldl -lm -lutil -lc
  libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Locally applied patches​:
 

---

@​INC for perl 5.13.3​:
  /usr/local/lib/perl5/site_perl/5.13.3/darwin-2level
  /usr/local/lib/perl5/site_perl/5.13.3
  /usr/local/lib/perl5/5.13.3/darwin-2level
  /usr/local/lib/perl5/5.13.3
  /usr/local/lib/perl5/site_perl
  .

---

Environment for perl 5.13.3​:
  DYLD_LIBRARY_PATH (unset)
  HOME=/Users/sprout
  LANG=en_US.UTF-8
  LANGUAGE (unset)
  LD_LIBRARY_PATH (unset)
  LOGDIR (unset)
  PATH=/usr/bin​:/bin​:/usr/sbin​:/sbin​:/usr/local/bin​:/usr/X11/bin​:/usr/local/bin
  PERL_BADLANG (unset)
  SHELL=/bin/bash

[p5pRT]

From @ikegami

On Sun, Aug 15, 2010 at 5​:11 PM, Father Chrysostomos
<perlbug-followup@​perl.org> wrote​:

# New Ticket Created by  Father Chrysostomos
# Please include the string​:  [perl #77242]
# in the subject line of all future correspondence about this issue.
# <URL​: http​://rt.perl.org/rt3/Ticket/Display.html?id=77242 >

#!perl -l
$fn = "\xe2\x80\x99";
`touch \Q$fn`;
print -e $fn; # prints 1
print -e substr "$fn\x{100}", -1; # prints nothing

This can happen easily if the file name comes from a URI (URIs always represent byte sequences) and the URI happened to come from a UTF-8 web page.

Bad demo. The arguments for substr are wrong. The following
demonstrates the bug​:

/#!perl -l
$fn = "\xe2\x80\x99";
`touch \Q$fn`;
print -e $fn; # prints 1
print -e substr "$fn\x{100}", 0, -1; # prints nothing

So does this simpler code​:

#!perl -l
$fn = "\xe2\x80\x99";
`touch \Q$fn`;
print -e $fn; # prints 1
utf8​::upgrade($fn);
print -e $fn; # prints nothing

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From dbook@cpan.org

Created by dbook@cpan.org

File test operators like -e and -f appear to be using the
internal representation of the file path passed to them, or are
otherwise broken by upgrading the string when it contains
non-ASCII bytes. It seems unreasonable to expect users to
downgrade any file paths passed to these operators just in case
it was accidentally upgraded; the expected behavior would be to
use the perl-level representation of the string. Reproduction
case below, or at https://perlbot.pl/raw/l2vc6w .

use strict;
use warnings;

use Encode 'encode';
use File​::Spec​::Functions 'catfile';
use File​::Temp;
use Test​::More;

my $filename = "t\x{eb}st";

my $dir = File​::Temp->newdir;

my $filepath = catfile $dir, encode('UTF-8', $filename);
open my $fh, '>', $filepath or die "Failed to create file $filepath​: $!";
close $fh;

ok -e $filepath, "File $filepath exists";
utf8​::upgrade $filepath;
ok -e $filepath, "File $filepath still exists"; # this fails

done_testing;

__END__
ok 1 - File /tmp/GnS7A2ORvG/tëst exists
not ok 2 - File /tmp/GnS7A2ORvG/tëst still exists
# Failed test 'File /tmp/GnS7A2ORvG/tëst still exists'
# at test_filename.pl line 19.
1..2
# Looks like you failed 1 test of 2.

Perl Info

Flags:
    category=core
    severity=medium

Site configuration information for perl 5.26.0:

Configured by grinnz at Tue May 30 17:37:21 EDT 2017.

Summary of my perl5 (revision 5 version 26 subversion 0) configuration:
   
  Platform:
    osname=linux
    osvers=4.8.13-100.fc23.x86_64
    archname=x86_64-linux
    uname='linux home.grinnz.com 4.8.13-100.fc23.x86_64 #1 smp fri dec 9 14:51:40 utc 2016 x86_64 x86_64 x86_64 gnulinux '
    config_args='-Dprefix=/home/grinnz/.plenv/versions/5.26.0 -de -Dusedevel -A'eval:scriptdir=/home/grinnz/.plenv/versions/5.26.0/bin''
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2'
    optimize='-O2'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='5.3.1 20160406 (Red Hat 5.3.1-6)'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib /lib/../lib64 /usr/lib/../lib64 /lib /lib64 /usr/lib64 /usr/local/lib64
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.22.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.22'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Locally applied patches:
    Devel::PatchPerl 1.38


@INC for perl 5.26.0:
    /home/grinnz/.plenv/versions/5.26.0/lib/perl5/site_perl/5.26.0/x86_64-linux
    /home/grinnz/.plenv/versions/5.26.0/lib/perl5/site_perl/5.26.0
    /home/grinnz/.plenv/versions/5.26.0/lib/perl5/5.26.0/x86_64-linux
    /home/grinnz/.plenv/versions/5.26.0/lib/perl5/5.26.0


Environment for perl 5.26.0:
    HOME=/home/grinnz
    LANG=en_US.utf8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/home/grinnz/.plenv/versions/5.26.0/bin:/home/grinnz/.plenv/libexec:/home/grinnz/.plenv/plugins/perl-build/bin:/home/grinnz/.plenv/plugins/plenv-contrib/bin:/home/grinnz/.plenv/shims:/home/grinnz/.plenv/bin:/usr/local/cuda/bin:/home/grinnz/.plenv/bin:/usr/local/cuda/bin:/usr/local/bin:/usr/bin
    PERL_BADLANG (unset)
    SHELL=/usr/bin/fish

[p5pRT]

From @Grinnz

This is a duplicate of https://rt.perl.org/Public/Bug/Display.html?id=77242
.
-Dan

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'