Consistent I/O function error return values

[ap]

We (PSC) have been thinking about the situation created by #20103, with problems reported in #22883 and #23026. A small improvement was made in #22907, and @tonycoz also suggested what he calls an alternative fix (I’ll come to that in a moment) in #23314.

Initially we had thoughts that I posted about, but we now think we were wrong at the time.

We now believe the situation is this:

• The core problem is the fact that I/O functions derive their return code from the filehandle error flag. Instead, they should signal an error only based on the I/O operation they themselves performed, ignoring whether the error flag was already set.
• Back when readline was clearing the error flag indiscriminately, the resulting overall behavior tended to incidentally match what would happen if this is how the I/O functions worked, even though they didn’t.
• When we changed readline to actually be able to signal failures, this charade broke down, and now we are dealing with the fallout from that. The design was broken but held together with chewing gum, and we removed the chewing gum.

As for the error flag, clearly the point of the flag is to preserve error states until they have been dealt with, which that proposal contradicts. Effectively, the error flag exists to prevent data loss in code that is somewhat sloppy (but not too sloppy) in dealing with I/O errors. And because it is a documented interface, we cannot remove it or alter its behavior. Therefore, I/O functions should continue to set and clear it exactly as before, so that user code that was written to examine the flag will continue to behave as it always has. But the functions should entirely ignore the state of the flag in their own operation.

This means we want both #22907 and #23314, plus of course a number of addtional yet-to-be-written changes.

The question now is whether the design of PerlIO even allows for implementing this.

As @tonycoz said in #22909,

Writing tests for this encountered a problem: read() uses the stream error flag to detect whether an error occurred, the underlying PerlIO_read() behaves like the stdio fread() in that you tell the difference between an error and a short read/eof by checking the stream error or eof flag.

So read() cannot return undef on error unless we do set the error flag. 🙁

I’ll ping @Leont here for another opinion, given he replied that

PerlIO was designed to be an drop-in stdio replacement, instead of being a sensible buffered IO implementation. This is the source of almost all issues in PerlIO.

(Not knowing anything much about the area, I wonder if the envisaged abolition of stdio is useful or even necessary or maybe irrelevant to a bid to do this.)

[ap]

@haarg In writing this down, it occurs to me that this proposal would amount to undoing the effect of the change to readline, since readline has no other way of signaling errors besides the error flag, and the error flag would no longer be paid any attention to by core functions.

I am still convinced this is the right general direction, but maybe further thought is still needed.

[Leont]

The core problem is the fact that I/O functions derive their return code from the filehandle error flag. Instead, they should signal an error only based on the I/O operation they themselves performed, ignoring whether the error flag was already set.

The problem is that a buffered I/O doesn't necessarily map to an actual system I/O. It may map to 0, 1, or more syscalls. We can't get rid of the concept of error flags, even if we can change some of the details.

[jkeenan]

This means we want both #22907 and #23314, plus of course a number of addtional yet-to-be-written changes.

Could I ask what the implications of this are for the scheduling of the release of perl-5.42.0?

[haarg]

We aren't intending on making any additional changes before 5.42.

The problem is that a buffered I/O doesn't necessarily map to an actual system I/O. It may map to 0, 1, or more syscalls. We can't get rid of the concept of error flags, even if we can change some of the details.

The functions may need to track an error flag, but that would only be for the lifetime of the single operation. I don't see why it would need to be exposed or persist between calls. This may be where the details of PerlIO make this impossible.

@haarg In writing this down, it occurs to me that this proposal would amount to undoing the effect of the change to readline, since readline has no other way of signaling errors besides the error flag, and the error flag would no longer be paid any attention to by core functions.

I'm not sure that I follow. Isn't our proposal here that readline should be able to signal failures without relying on a per-handle error flag?

[bulk88]

The core problem is the fact that I/O functions derive their return code from the filehandle error flag. Instead, they should signal an error only based on the I/O operation they themselves performed, ignoring whether the error flag was already set.

The problem is that a buffered I/O doesn't necessarily map to an actual system I/O. It may map to 0, 1, or more syscalls. We can't get rid of the concept of error flags, even if we can change some of the details.

PerlIO is bug compatible with LibC's stdio API falls apart super quick with PerlIO objects whose ultimate backend is $SCALAR or mmap().

What is the fileno() of a SV* of type SVt_PVNV with only NOK_on()?

Rewriting unit tests is not allowed.

[ap]

@haarg In writing this down, it occurs to me that this proposal would amount to undoing the effect of the change to readline, since readline has no other way of signaling errors besides the error flag, and the error flag would no longer be paid any attention to by core functions.

I'm not sure that I follow. Isn't our proposal here that readline should be able to signal failures without relying on a per-handle error flag?

I mean that #20103 made sloppy code more robust, in the sense that code which doesn’t carefully handle $! while calling readline will now at least still encounter any readline errors when it calls close (whose errors are checked by a lot more code than errors from readline).

On one hand, if we now make the return value of close ignore the error flag, then this improvement in robustness for such sloppy code is lost again.

OTOH, this is balanced against an improvement of the situation for careful code that was already trying to do everything correctly.

I’m not bringing this up as a definite argument against pursuing this plan, only a potential one – mainly as an observation to be conscious about as we make any further calls about what to do.

[bulk88]

How about adding @! to https://perldoc.perl.org/perlvar ?

or maybe local( $! ) = sub { $self = shift; foreach(@_) { warn $_; } }; # sub handler ?

or maybe $!->() = sub { $self = shift; foreach(@_) { warn $_; } }; # sub handler ?

or this, the CV* sticks around!

C:\sources>perl -E "$! = sub { return 'abc'; }; say( ${\$!}+0 );  say( ${\$!}+0 );"
30611444
30611444

or maybe

C:\sources>perl -E "*{!} = sub { return 'abc_err'; }; say( &! );"
abc_err