-pi doesn't sanely handle write errors

[p5pRT]

Migrated from rt.perl.org#1154 (status was 'resolved')

Searchable as RT1154$

[p5pRT]

From morty@sled.gsfc.nasa.gov

perl 5.00502 and 5.00503 (probably earlier as well) don't check for write
errors when they do -pi. For example​:

morty@​frakir​:/spare/morty 12​:16​:11 1513$ df -k .
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/sda7 513147 513145 2 100% /spare
morty@​frakir​:/spare/morty 12​:16​:14 1514$ ls -l bar*
-rw-r--r-- 1 morty wheel 6923 Aug 2 19​:03 bar
morty@​frakir​:/spare/morty 12​:16​:24 1515$ perl -pi~ -e '' bar
morty@​frakir​:/spare/morty 12​:16​:28 1516$ echo $?
0
morty@​frakir​:/spare/morty 12​:16​:31 1517$ ls -l bar*
-rw-r--r-- 1 morty wheel 2048 Aug 3 12​:16 bar
-rw-r--r-- 1 morty wheel 6923 Aug 2 19​:03 bar~

IMHO, better behavior would be for perl to put the file back the way it was
if it encounters a write error.

I haven't tried the latest development version; my apologies if this is a
known bug and/or has already been fixed and/or is documented somewhere i
didn't think to look.

Perl Info

Site configuration information for perl 5.00503:

Configured by morty at Mon Aug  2 20:33:38 EDT 1999.

Summary of my perl5 (5.0 patchlevel 5 subversion 3) configuration:
  Platform:
    osname=linux, osvers=2.0.36, archname=i686-linux
    uname='linux frakir 2.0.36 #1 tue dec 29 13:11:13 est 1998 i686 unknown '
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef useperlio=undef d_sfio=undef
  Compiler:
    cc='cc', optimize='-O2', gccversion=2.7.2.3
    cppflags='-Dbool=char -DHAS_BOOL -I/usr/local/include'
    ccflags ='-Dbool=char -DHAS_BOOL -I/usr/local/include'
    stdchar='char', d_stdstdio=define, usevfork=false
    intsize=4, longsize=4, ptrsize=4, doublesize=8
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lc -lposix -lcrypt
    libc=, so=so, useshrplib=false, libperl=libperl.a
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynamic'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Locally applied patches:
    


@INC for perl 5.00503:
    /home/frakir/morty/lib/perl5
    /usr/local/lib/perl5/5.00503/i686-linux
    /usr/local/lib/perl5/5.00503
    /usr/local/lib/perl5/site_perl/5.005/i686-linux
    /usr/local/lib/perl5/site_perl/5.005
    .


Environment for perl 5.00503:
    HOME=/home/frakir/morty
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/local/bin:/usr/local/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/sbin:/usr/local/sbin:/sbin:/opt/sybase/bin:/home/frakir/morty/bin:/usr/local/pgsql/bin:/usr/local/ssl/bin
    PERL5LIB=/home/frakir/morty/lib/perl5
    PERL_BADLANG (unset)
    SHELL=/bin/bash

[p5pRT]

From @smpeters

[morty@​sled.gsfc.nasa.gov - Tue Aug 03 05​:24​:27 1999]​:

-----------------------------------------------------------------
[Please enter your report here]

perl 5.00502 and 5.00503 (probably earlier as well) don't check for
write
errors when they do -pi. For example​:

morty@​frakir​:/spare/morty 12​:16​:11 1513$ df -k .
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/sda7 513147 513145 2 100% /spare
morty@​frakir​:/spare/morty 12​:16​:14 1514$ ls -l bar*
-rw-r--r-- 1 morty wheel 6923 Aug 2 19​:03 bar
morty@​frakir​:/spare/morty 12​:16​:24 1515$ perl -pi~ -e '' bar
morty@​frakir​:/spare/morty 12​:16​:28 1516$ echo $?
0
morty@​frakir​:/spare/morty 12​:16​:31 1517$ ls -l bar*
-rw-r--r-- 1 morty wheel 2048 Aug 3 12​:16 bar
-rw-r--r-- 1 morty wheel 6923 Aug 2 19​:03 bar~

IMHO, better behavior would be for perl to put the file back the way
it was
if it encounters a write error.

I haven't tried the latest development version; my apologies if this
is a
known bug and/or has already been fixed and/or is documented somewhere
i
didn't think to look.

steve@​kirk pi_test $ perl -pi- -e'exit' test
Can't rename test to test-​: Permission denied, skipping file.

It seems to have been fixed somewhere prior to Perl 5.8.5.

Steve Peters

[p5pRT]

@smpeters - Status changed from 'open' to 'resolved'

[p5pRT]

From morty@sled.gsfc.nasa.gov

On Fri, Nov 12, 2004 at 03​:46​:36AM -0000, Steve Peters via RT wrote​:

steve@​kirk pi_test $ perl -pi- -e'exit' test
Can't rename test to test-​: Permission denied, skipping file.
It seems to have been fixed somewhere prior to Perl 5.8.5.

That didn't duplicate the steps that led to the original problem. The
problem is not when the initial rename fails or file create fails,
it's when a subsequent write fails.

I just tried it under 5.8.5, and it still happens. Example​:

root@​red-sonja​:/mnt 05​:30​:13 1061# dd if=/dev/zero of=bar bs=1024
count=700
700+0 records in
700+0 records out
root@​red-sonja​:/mnt 05​:30​:46 1062# df -k .
Filesystem 1k-blocks Used Available Use% Mounted on
/root/foo 1003 717 235 75% /mnt
root@​red-sonja​:/mnt 05​:30​:50 1063# ls -l
total 716
-rw-r--r-- 1 root root 716800 Nov 12 05​:30 bar
drwxr-xr-x 2 root root 12288 Nov 12 05​:29 lost+found
root@​red-sonja​:/mnt 05​:30​:55 1064# perl -pi~ -e 1 bar
root@​red-sonja​:/mnt 05​:31​:12 1065# echo $?
0
root@​red-sonja​:/mnt 05​:31​:16 1066# ls -l
total 999
-rw-r--r-- 1 root root 286720 Nov 12 05​:31 bar
-rw-r--r-- 1 root root 716800 Nov 12 05​:30 bar~
drwxr-xr-x 2 root root 12288 Nov 12 05​:29 lost+found

- Morty

[p5pRT]

From morty@frakir.org

Please reopen. Thanks!

A complete demo script is attached.

[p5pRT]

From morty@frakir.org

perlbug-perl-pi

[p5pRT]

From morty@sled.gsfc.nasa.gov

It occurs to me that I never included a demo of how to create an
artificial space problem for testing purposes. Here is such a
script; requires root on a modern linux system.

#!/bin/sh

# script to demo problem of perl not detecting write errors with -pi
# requires root on a linux system

IMG=/root/floppy.img
IMG_DIR=/root
IMG_SIZE=1440
BS=1024
SOURCE=/dev/zero
MOUNTPOINT=/mnt
MKFS_ARGS="-t ext2"
MOUNT_ARGS="-o loop"
FILE=bar
FILE_SIZE=900

cd / &&
mkdir -p $IMG_DIR &&
dd if=$SOURCE of=$IMG bs=$BS count=$IMG_SIZE &&
mkfs $MKFS_ARGS $IMG </dev/null &&
mount $MOUNT_ARGS $IMG $MOUNTPOINT &&
cd $MOUNTPOINT &&
dd if=$SOURCE of=$FILE bs=$BS count=$FILE_SIZE &&
echo before running perl​: && df -k . && ls -l bar* &&
echo Running perl. . . &&
perl -pi~ -e 1 $FILE

echo Done. Status is​: $? && df -k . && ls -l bar*

# clean up regardless of status
cd /; umount $MOUNTPOINT; rm $IMG

[p5pRT]

@smpeters - Status changed from 'resolved' to 'open'

[p5pRT]

From @chipdude

A first step in handling write errors would be dying if either print or
close returns false. This would be a significant change from the old
ways, but I think it's OK ... data loss is one of the primary evils,
after all.

As for restoring the partially read file back to its original name on
error​: I actually think that's a bad idea. There may be a LOT of
processing involved, and partial output is definitely better than no
output in some circumstances.

Finally, adding death to the behavior of existing code that has its own
read/print loop is far more questionable, and I'd rather not do it. In
other words, if you write this you should get death on disk full​:

  perl -pi~ 's/foo/bar/' *.c

But if you write this you should _not_ get automatic death​:

  #!/usr/bin/perl
  $^I = '~';
  while (<>) { ....; print }

Comments?

[p5pRT]

From @chipdude

{{ I filed this comment in RT and asked it to copy p5p, but over an hour
  later I don't see it. I wonder what's up with that? Meanwhile... }}

A first step in handling write errors in -pi would be dying if either print
or close returns false. This would be a significant change from the old
ways, but I think it's OK ... data loss is one of the primary evils, after
all.

As for restoring the partially read file back to its original name on
error​: I actually think that's a bad idea. There may be a LOT of
processing involved, and partial output is definitely better than no
output in some circumstances.

Finally, adding death to the behavior of existing code that has its own
read/print loop is far more questionable, and I'd rather not do it. In
other words, if you write this you should get death on disk full​:

  perl -pi~ 's/foo/bar/' *.c

But if you write this you should _not_ get automatic death​:

  #!/usr/bin/perl
  $^I = '~';
  while (<>) { ....; print }

Comments?
--
Chip Salzenberg <chip@​pobox.com>

[p5pRT]

From Mordechai.T.Abzug@nasa.gov

On Mon, Nov 17, 2008 at 10​:38​:05AM -0800, Chip Salzenberg via RT wrote​:

A first step in handling write errors would be dying if either print
or close returns false. This would be a significant change from the
old ways, but I think it's OK ... data loss is one of the primary
evils, after all.

Agreed. At the very least, I should get a non-zero exit status.

As for restoring the partially read file back to its original name
on error​: I actually think that's a bad idea. There may be a LOT of
processing involved, and partial output is definitely better than no
output in some circumstances.

In my use case, I was looking to use Perl to in-place edit system
configs. The processing is minimal. I would much rather that failure
automatically revert the config rather than leave a corrupt config
(neither the old nor the new) in place.

And what about the case where one has an error when using -i without a
backup, i.e. perl -pi -e $something $file? Perl knows that something
has gone horribly wrong, but it doesn't have a name to give a second
file. In that case, I'd say that the correct behavior is to revert
the original file name to the original contents.

Finally, adding death to the behavior of existing code that has its own
read/print loop is far more questionable, and I'd rather not do it. In
other words, if you write this you should get death on disk full​:

perl \-pi~ 's/foo/bar/' \*\.c

But if you write this you should _not_ get automatic death​:

\#\!/usr/bin/perl
$^I = '~';
while \(\<>\) \{ \.\.\.\.; print \}

Comments?

Agreed. The main problem here is that the -p implicit loop doesn't
have error handling. The programmer is relying on Perl's -p to be
properly implemented, so -p should do error handling and related
goodness. If the programmer chooses to do an explicit loop or uses -n
with explicit print, the onus is on the programmer to do error
handling on any print or close calls. [Is there error handling on the
implicit close that is part of <>?]

But what if the programmer's code does throw die while processing <>
with -i, with $^I enabled, or with -n -i? In that case, I would still
expect an atexit handler / END block to clean up the mess in the same
way as for -p -i. [This might be an argument in favor of not trying
to clean up the mess, as you suggested above. It's easy to tell the
programmer "use -i$ext $file", and if there is an error, your original
file will be in ${file}$ext". It's a lot harder to rely on Perl to
correctly determine when an error has occurred while processing $file,
and then make policy decisions on what constitutes safe cleanup.
Also, one can't really rely on Perl being able to safely recover --
what if Perl is running on a network mount, and the actual problem
here is that the network mount failed? What if the system reboots in
the middle of the script run?]

- Morty

[p5pRT]

From ben@morrow.me.uk

Quoth Mordechai.T.Abzug@​nasa.gov (Morty Abzug)​:

On Mon, Nov 17, 2008 at 10​:38​:05AM -0800, Chip Salzenberg via RT wrote​:

As for restoring the partially read file back to its original name
on error​: I actually think that's a bad idea. There may be a LOT of
processing involved, and partial output is definitely better than no
output in some circumstances.

In my use case, I was looking to use Perl to in-place edit system
configs. The processing is minimal. I would much rather that failure
automatically revert the config rather than leave a corrupt config
(neither the old nor the new) in place.

  perl -pi~ -e'... END { $? and rename "$ARGV~", $ARGV }'

And what about the case where one has an error when using -i without a
backup, i.e. perl -pi -e $something $file? Perl knows that something
has gone horribly wrong, but it doesn't have a name to give a second
file. In that case, I'd say that the correct behavior is to revert
the original file name to the original contents.

But that's not easy, since the original file has been unlinked and Unix
doesn't have a 'flink' syscall. The contents would have to be copied,
which could in turn incur errors (disk full, for instance), which would
then have to have their own handling... Just use a backup if you need to
recover from errors.

Ben

[p5pRT]

From tchrist@perl.com

<<<====================================================>>>

¶ SUMMARY​: Yes, Chip, it's a bug, one I've kvetched
  about for many a year (but see §x below).
  Yet patching's complex, as you'll read anon.

¶ METAPHOR​: There is nothing new under the sun. [Eccl 1​:9]

¶ IN LATIN​: Sicut erat in principio, et nunc, et semper.

<<<====================================================>>>

On Mon, 17 Nov 2008 10​:38​:07 PST, Chip Salzenberg wrote​:

A first step in handling write errors would be dying if either
print or close returns false. This would be a significant
change from the old ways, but I think it's OK ...

Indeed; see §xv below.

data loss is one of the primary evils, after all.

Agreed​: vehemently, vociferously, and vituperatively.

See §xiii below.

As for restoring the partially read file back to its original
name on error​: I actually think that's a bad idea. There may
be a LOT of processing involved, and partial output is
definitely better than no output in some circumstances.

Finally, adding death to the behavior of existing code that has
its own read/print loop is far more questionable, and I'd
rather not do it. In other words, if you write this you should
get death on disk full​:

perl -pi~ 's/foo/bar/' *.c

But if you write this you should _not_ get automatic death​:

#!/usr/bin/perl
$^I = '~';
while (<>) { ....; print }

Comments?

How nice of you to ask! I'm only too glad to provide a few
brief ones--which can be found following my signature. :-)

Chip, this bug is one I've long lamented, and the problem more
general than -i alone, but 'twas there I first felt its bite.

When a filehandle is *implicitly* closed (or even flushed) by
Perl, such as when there is a reopen on a handle that's already
open, when an autovivved *GLOB{IO} object gets DESTROYed (and,
I hope, a local *GLOB), or when the program itself exits, there
is usually no notification of this.

This makes for incorrect programs--and silently lost data!!

If the programmer cannot check, than Perl *must*; and doesn't.

If you look at pp_fork(), you'll see that it does, voidally,

  PERL_FLUSHALL_FOR_CHILD

I presume you recall why we must do that, and may even
recall how Sun's fflush(NULL) would accidentally also
fpurge() input streams, not just fflush() output streams.

The fflush NULL case is here in Perl_PerlIO_flush​:

  /*
  * Is it good API design to do flush-all on NULL, a
  * potentially errorneous input? Maybe some magical value
  * (PerlIO* PERLIO_FLUSH_ALL = (PerlIO*)-1;)? Yes, stdio does
  * similar things on fflush(NULL), but should we be bound by
  * their design decisions? --jhi
  */
  PerlIO **table = &PL_perlio;
  int code = 0;
  while ((f = *table)) {
  int i;
  table = (PerlIO **) (f++);
  for (i = 1; i < PERLIO_TABLE_SIZE; i++) {
  if (*f && PerlIO_flush(f) != 0)
  code = -1;
  f++;
  }
  }
  return code;

But again, people are voiding (erroneously disregarding) it.
Similarly, if you look at pp_flock(), whose issues, Chip, I
believe you will recall, again you see disreputable voiding​:

  if (fp) {
  (void)PerlIO_flush(fp);
  value = (I32)(PerlLIO_flock(PerlIO_fileno(fp), argtype) >= 0);
  }

Meanwhile in Perl_do_openn(), called by pp_open(), you find
many calls that eventually chase down to PerlIO__close, and
these in turn to Perl_PerlIO_flush with a specific handle.

That case is this​:

  const PerlIO_funcs *tab = PerlIOBase(f)->tab;

  if (tab && tab->Flush)
  return (*tab->Flush) (aTHX_ f);

I get lost in the longjump() in S_my_exit_jump, but you can
go to perl_destruct() and find

  /* Need to flush since END blocks can produce output */
  my_fflush_all();

which leads you to the following evil code​:

  if (open_max > 0) {
  long i;
  for (i = 0; i < open_max; i++)
  if (STDIO_STREAM_ARRAY[i]._file >= 0 &&
  STDIO_STREAM_ARRAY[i]._file < open_max &&
  STDIO_STREAM_ARRAY[i]._flag)
  PerlIO_flush(&STDIO_STREAM_ARRAY[i]);
  return 0;
  }

Notice again, we've neglecting reporting a flush failure.

If reading through the code isn't enough (to make your head spin),
I also have direct empirical evidence, well tested, that at program
exit time, the program exits silently and erroneously unerroneously
if the STDOUT flush fails.

I complain about this every 5 or 10 years. The last time was only
this past July. Here is part of it​:

http​://www.gossamer-threads.com/lists/perl/porters/230671?search_string=even%20my%20cat%20is%20smarter%20than;#230671

in which I prove that even my cat is smarter than perl. :-)

Pardon me, while I repeat myself​:

» Which brings me to my final complaint of the day. That code is
» wrong because Perl itself is wrong. No really, it is.

» Watch​:

» % df -h .
» Filesystem Size Used Avail Capacity Mounted on
» /dev/wd0a 124M 117M -5.0K 100% /
»
» % ls -l it*
» -rw-r--r-- 1 tchrist wheel 0 Jul 29 14​:06 it
»
» % perl -i.orig -pe 'print "this is more stuff\n"' it
»
» % echo $?
» 0
»
» % ls -l it*
» 0 -rw-r--r-- 1 tchrist wheel 0 Jul 29 15​:05 it
» 0 -rw-r--r-- 1 tchrist wheel 0 Jul 29 14​:06 it.orig

» To this day, Perl's implicit closing of files doesn't warn you
» of errors, let alone exit nonzero. This makes it do wrong
» thing and not even tell you it did them wrong. This is a
» *true* problem, because checking for the success of print() is
» neither necessary nor sufficient to detect the success of
» print(). Yes, you read that correctly. It's because of
» buffering, plus the persistence of the err flag on the file
» structure.

» I've never convinced anybody this is important. Since *every*
» program should do this for correctness, it has to be in the run-
» time system to avoid it ever being forgotten. Sure, there's
» stuff like this you can do​:

» END { close(STDOUT) || die "can't close stdout​: $!" }

See §vi below.

» But that sort of thing should happen on all implicitly closed
» things. And it really must. Even IO​::Handle​::close doesn't
» bother. Perl knows what handles it's flushing closing during
» global destruction, at least if you don't just blindly
» fflush(0).

» Watch again, starting from before the bogus, ill-reported
» rename attempt​:

» % df -h .
» Filesystem Size Used Avail Capacity Mounted on
» /dev/wd0a 124M 117M -5.0K 100% /
»
» % ls -l it
» -rw-r--r-- 1 tchrist wheel 0 Jul 29 14​:06 it
»
» % perl -e 'print "stuff\n"' >> it; echo $?
» 0
»
» % perl -e 'open(my $fh, "&gt;&gt;it") || die "open $!"; print $fh "stuff\n"; print STDOUT "ok now\n"'
» ok now
»
» % echo $?
» 0
»
» % ls -l it
» -rw-r--r-- 1 tchrist wheel 0 Jul 29 14​:06 it

» This is all incorrect behavior on Perl's part. Even cat knows better!

» % echo foo | cat >> it; echo $?
» cat​: stdout​: No space left on device
» 1

» +-------------------------------------------------------+
» | Notice how even my cat is smarter than your perl!? :( |
» +-------------------------------------------------------+

» What's that about, eh?

» But I've been saying this for years, just like everything else.
» Makes no difference. Depressing, eh?

» BTW, this proves my point about checking for print's status
» being a waste of time, neither necessary nor sufficient to
» catch failed prints!

» % perl -e 'open(my $fh, "&gt;&gt;it") || die "open $!"; print $fh "stuff\n" or die "print $!"; print STDOUT "ok now\n"'; echo exit status was $?
» ok now
» exit status was 0

» And you can't get the danged thing to detect its folly​:

» % perl -WE 'open(my $fh, "&gt;&gt;it") || die "open $!"; say $fh "stuff" or die "
» print $!"; say "ok now"' ; echo exit status was $?
» ok now
» exit status was 0

» I firmly believe that this needs to be a part of *EVERY* Perl
» program, which consequently means it should *not* be there, but
» in the core itself​:

» END { close(STDOUT) || die "can't close stdout​: $!" }

» And I believe this *much* more than some here believe <ARGV>
» needs to implicitly map {"< $_\0"} @​ARGV (since that breaks
» existing working code, whereas mine fixes existing broken
» code). Furthermore, I also believe all *IMPLICIT* closes
» that fail need to generate a mandatory io warning, but that
» of STDOUT should be a fatal. I've believed all this for a
» long time; said it often enough. Anything else is just plain
» wrong behavior. I'm not quite sure whether ARGVOUT failure
» should be a warning or a fatal, but it should be *something*
» suitably noisy.

And then in a grandchild followup, where I wrote​:

»> I am glad you mentioned this, because I also think it's a
»> pretty egregious bug, but I was rather imagining you'd come
»> out to defend it, point out that it has always been that way
»> and why change it now, and anyway is clearly mentioned as a
»> note to an addendum to a particular perlfaq answer. I am glad
»> we agree on this issue at least :-(.

[...]

»> Yup, and also for STDERR (if closing STDERR fails you can
»> attempt to print a warning message to it, which might or might
»> not get through, but you can at least exit with nonzero
»> status).

» Possibly. Probably. Maybe.

»>> Furthermore, I also believe all *IMPLICIT* closes that fail
»>> need to generate a mandatory io warning,

»> This also makes perfect sense.

» Easy for you to say. Others have said otherwise in the past,
» and I've not prevailed. I don't know if those people are
» still alive anymore though, for I don't recall who they were
» nor the thrust of their arguments against such sanity. I
» don't recall ever having been convinced of whatever
» "reasoning" they used, but absence of evidence is not to be
» confused with evidence of absence, so perhaps I was and that
» memory is now hidden from me.

»> Does anyone really object to this?

» They clearly did, or it would have been done. But maybe this
» is one of those rare periods in history where we can fix it
» when the PC police aren't breathing down our throats. One
» shall see.

»> In principle it is a backwards incompatible change (code that
»> previously chugged along without visible problems will now
»> start producing warnings or exiting with nonzero status) but I
»> can't see the point of taking compatibility to such extremes.

» A bug is a bug, and this is. Always has been.

Rafaël then kindly spoke up, saying​:

»>> To this day, Perl's implicit closing of files doesn't warn
»>> you of errors, let alone exit nonzero. This makes it do wrong
»>> thing and not even tell you it did them wrong. This is a
»>> *true* problem, because checking for the success of print()
»>> is neither necessary nor sufficient to detect the success of
»>> print(). Yes, you read that correctly. It's because of
»>> buffering, plus the persistence of the err flag on the file
»>> structure.

»>> I've never convinced anybody this is important.

»> It absolutely is. I had no idea, and as far as I'm
»> concerned it's broken obviously enough that it needs
»> no supporting argument.

» I concur. A patch would help, but I know that PerlIO is not
» the simplest thing to patch, and the problem is probably a
» bit complex now.

Indeed; see §xvi below.

» A proper bug report would help, too, because currently this one
» is deep buried in another thread.

I was glad to see that I'd finally convinced SOMEONE this to be
a genuine problem. And I *think* I even made such a bug report,
but don't recall for certain.

But I know I didn't submit a patch; alas for §ix below!

I did, however, look into the problem, as you see from the C code
cited above, where I found Rafaël to be quite on the mark when he
said that "PerlIO is not the simplest thing to patch." :-(

And as I believe I have shown, it's not just one place that
needs fixing, but many. And they may not require the same
treatment​: see §v below.

I believe that all voided flushes are wrong.

I also believe that the code in Perl_do_openn() that reads

  if (result == EOF && fd > PL_maxsysfd) {
  /* Why is this not Perl_warn*() call ? */
  PerlIO_printf(Perl_error_log,
  "Warning​: unable to close filehandle %s properly.\n",
  GvENAME(gv));
  }

is a Microsoftening of the true etiology behind the underlying
error, which we shall not now ever come to know--and must.

Where now the errno and strerror thereof? "Can't close file"
bereft of the cause is *excruciatingly* frustrating to those
it's perpetrated upon. It is EVIL.

My five rules for "proper" (=good) error messages are​:

  $0​: They name the program generating the message.
  $1​: They name the function that failed.
  $2​: They name the arguments to said function.
  $3​: They name the resulting errno, that is, the
  standard system error message, if appropriate.
  $4​: They go out STDERR, unbuffered, which is fileno 2.

It's hard to call one of those more important than another, but
insofar as I can discern, the "Warning" above (which doesn't seem
to be a proper warning, per the comment), fails at $3, occluding
the true problem from the user in a rather Redmundial lapse;
see §ii below.

--tom

--
§ i Help save the world!

  --Larry Wall in README

§ ii You tell it that it's indicative by appending $!. That's
  why we made $! such a short variable name, after all. :-)

  --Larry Wall in <199709081801.LAA20629@​wall.org>

§ iii And you can still put in all that cruft if you want to.
  You can even force yourself to have to do it. But to me,
  it feels a bit like slavery, so I'm still looking for a
  land flowing with milk and honey, even if there are a few
  giants in it.

  --Larry Wall

§ iv Be consistent.

  --Larry Wall in the perl man page

§ v The problem with being consistent is that there are lots of ways
  to be consistent, and they're all inconsistent with each other.

  --Larry Wall in-ID <20050516005559.GC26184@​wall.org>,
  to the perl6-language mailing-list

§ vi The computer should be doing the hard work.
  That's what it's paid to do, after all.

  --Larry Wall in <199709012312.QAA08121@​wall.org>

§ vii I think I'll side with the pissheads on this one.

  --Larry Wall

§ viii It seems like a sane thing to me,
  but that's a rather low standard.
  --Larry Wall in <20050419150023.GA19507@​wall.org>

§ ix This has been planned for some time. I guess we'll just have
  to find someone with an exceptionally round tuit.

  --Larry Wall in <199709302338.QAA17037@​wall.org>

§ x Er, Tom, I hate to be the one to point this out, but your fix
  list is starting to resemble a feature list. You must be
  human or something.

  --Larry Wall in <199801081824.KAA29602@​wall.org>

§ xi Down that path lies madness. On the other hand, the road to
  hell is paved with melting snowballs.
  --Larry Wall in <1992Jul2.222039.26476@​netlabs.com>

§ xii But maybe we could try to set some slushiness milestones on
  the road to hell freezing over...

  --Larry Wall in <20050314165932.GA12577@​wall.org>

§ xiii I have no opinion on its suitability for any particular task.
  I'm just the language designer--my job is to shoot you in the
  foot and make you think you did it to yourself. :-)

  --Larry Wall in <20050309170804.GA22973@​wall.org>

§ xiv I recommend not remaking my mistakes.
  Please make different mistakes. :-)

  --Larry Wall in <20040317192052.GA10645@​wall.org>

§ xv I suppose one could claim that an undocumented feature
  has no semantics. :-(

  --Larry Wall in <199710290036.QAA01818@​wall.org>

§ xvi It's not designed to make people happy who want to confuse
  those issues. We have macros for that.

  --Larry Wall in <20050419155213.GC19507@​wall.org>

§ xvii If you write something wrong enough, I'll be glad to make up
  a new witticism just for you.

  --Larry Wall in <199702221943.LAA20388@​wall.org>

§xviii ...sometimes collections of stupid utterances can be
  rather clever. If my writings are ever published
  posthumously, they should probably be called "A
  Collection of Stupid Utterances", or some such... :-)

  --Larry Wall in <20050303163144.GA5235@​wall.org>

§ xix : If I don't document something, it's usually either for
  : a good reason, or a bad reason. In this case it's a
  : good reason. :-)
  : --Larry Wall in <1992Jan17.005405.16806@​netlabs.com>

  Yeah, I keep saying that.

  The trouble with being quoted a lot is that it makes
  other people think you're quoting yourself when in fact
  you're merely repeating yourself.

  --Larry Wall

§ xx : 1. What is the possibility of this being added
  : in the future?

  In the near future, the probability is close to zero.
  In the distant future, I'll be dead, and posterity can
  do whatever they like... :-)

  --Larry Wall

§ xxi Well, you know, Hubbard had a bunch of people sworn
  to commit suicide when he died. So of course he never
  officially died...

  --Larry Wall in <199804141540.IAA05247@​wall.org>

[p5pRT]

From @chipdude

Aside​: the referenced message includes a romaniacally-numbered list of Larry
quotations. I've often joked about all Perl programmers owning a copy of The
Collected Sayings of Chairman Larry, but Tom's preferred metaphors run more
to the canon than the commie.

On Tue, Nov 18, 2008 at 04​:23​:41PM -0700, Tom Christiansen wrote​:

On Mon, 17 Nov 2008 10​:38​:07 PST, Chip Salzenberg wrote​:

A first step in handling write errors would be dying if either
print or close returns false. This would be a significant
change from the old ways, but I think it's OK ...
Indeed
data loss is one of the primary evils, after all.
Agreed​: vehemently, vociferously, and vituperatively.

Yay. Action item #1​: die on output failure in a -i loop.

You go on at length (usefully - I have much context to swap back in) about
Perl's cavalier disgregard for reporting flush failures in many
circumstances. I'll take that exposition as a guide, first, to help complete
action item #1. Afterwards, though, I'll be interested in revisiting the
broader question of reporting all such failures correctly. Of course I will
run afoul of §xxii, but c'est la vie.

§ xxii There's often more than one correct thing.
  There's often more than one right thing.
  There's often more than one obvious thing.

--
Chip Salzenberg twitter​:chipsalz
"UTOPIA PLANITIA IS IN ORBIT is the new HAN SHOT FIRST" - Crisper Than Thou

[p5pRT]

From @chipdude

On Tue, Nov 18, 2008 at 01​:11​:27PM +0000, Ben Morrow wrote​:

Quoth Mordechai.T.Abzug@​nasa.gov (Morty Abzug)​:

On Mon, Nov 17, 2008 at 10​:38​:05AM -0800, Chip Salzenberg via RT wrote​:

As for restoring the partially read file back to its original name
on error​: I actually think that's a bad idea. There may be a LOT of
processing involved, and partial output is definitely better than no
output in some circumstances.

In my use case, I was looking to use Perl to in-place edit system
configs. The processing is minimal. I would much rather that failure
automatically revert the config rather than leave a corrupt config
(neither the old nor the new) in place.

perl \-pi~ \-e'\.\.\. END \{ $? and rename "$ARGV~"\, $ARGV \}'

That's a clever idiom that will work in many cases. Kinda ugly, but oh
well. I wonder if we already ensure that $ARGV is undef after the entire
while (<>) {} loop is complete. If not, we should. Otherwise a die or
nonzero exit or setting of $? in another END block could trigger a spurious
rename. If we do appropriately unset $ARGV, then​:

  perl -pi~ -e'... END { $? and $ARGV and rename "$ARGV~", $ARGV }'

I wonder whether we could make this a standard but optional feature of
inplace editing. How about​:

  ${^INPLACE_SAFE}++; # nop in old Perls, forces rename on error in new Perls

And what about the case where one has an error when using -i without a
backup [...]

But that's not easy, since the original file has been unlinked and Unix
doesn't have a 'flink' syscall. [...] Just use a backup if you need to
recover from errors.

Indeed.
--
Chip Salzenberg twitter​:chipsalz
"UTOPIA PLANITIA IS IN ORBIT is the new HAN SHOT FIRST" - Crisper Than Thou