Expose raw std{in,out,err}

[SUPERCILEX]

Proposal

Problem statement/motivation

It is well known that people want more control over stdout and friends (1 2 3 4 5 6 etc etc). Some want to avoid locking, others want to disable line buffer, others still want to change the buffer sizes, and so on.

Solution sketches

The current std* fns are fundamentally incompatible with the flexibility desired above. For example, you cannot both use buffering and not require locks. Thus, I believe the most flexible and simple solution is to simply expose the raw std* Read/Write impls. Users can then:

• Add locking around their own data structure if they need it.
• Add buffering (of any kind) around their own data structure if they need it.
• Compose different variants of the std* fns as needed (maybe some bit of code wants buffering, but other parts don't)

In the future if we allow overriding the println and co streams, this idea will be complementary since you could build up the desired wrappers around stdout and replace the default stream with the one you just built.

Non-goals: this proposal does not suggest changing the current std* impls. rust-lang/rust#60673 is therefore not addressed by this proposal (but of course you can build your own version and use it in writeln).

Downsides: interaction between raw std* and normal std* may be a little weird. I view this as a non-issue in the same sense that files take an "it's your problem, figure it out" attitude. For example, I can open the same file path twice and it'll be my problem if my writes smash each other. Similarly, it's on the user to decide how they want to handle interleavings (and they might not even care b/c they know their program is single-threaded).

API

Expose the existing Std{in,out,err}Raw (and constructor functions) in std::io.

Links and related work

See links above.

[thomcc]

How should this work on Windows? We currently convert from utf8 to utf16 when you write to the console.

[SUPERCILEX]

Any reason why that's bad? I don't see raw as meaning raw bytes stream, but rather no locking or buffering. If you want a raw byte stream, then you can grab the FD and convert it to a File. As a sidenote, I considered proposing implementing AsFd on Stdout (instead of the current StdoutLock) so you could manually convert it to a File, but that API ends up being really annoying to use since you need to use unsafe and wrap the file in a ManuallyDrop. Hence proposing letting you access raw stdout, but not the raw byte stream.

[thomcc]

If we don't buffer at all, it becomes difficult to ensure the conversion succeeds, we'd have to panic if your write is a truncated UTF-8 sequence, even if you're later going to write the rest of the sequence. For example, "😀" is the bytes [240, 159, 152, 128].

But if you want to write [240, 159, 152] and then [128], that succeeds on Unix, but on Windows we'd need to either buffer or panic (and if we buffer, we pretty much have to lock).

[SUPERCILEX]

Interesting, I poked around and it turns out windows is double buffered: https://github.com/rust-lang/rust/blob/master/library/std/src/sys/windows/stdio.rs.

I don't know how to word this properly then, but the proposal is to do the minimal amount of processing to still be considered a stdio stream as opposed to a byte stream. So windows would continue to be buffered for UTF-8 writes to work.

[the8472]

None of the linked issues make a good argument for removing the mutex. The first one says that they're overkill for single-threaded programs. But on single-threaded programs a mutex lock-unlock is 1 uncontended atomic CAS and 1 uncontended atomic write, that's very cheap compared to the IO syscalls.

The rest should be covered by making buffering switchable between line, block and no buffering. rust-lang/rust#78515 (comment) outlines the libs team's preferred path forward on that.

[SUPERCILEX]

Sure, but I can easily see someone wanting to use the kernel's locking: just print stuff to stdout and as long as it's a single write under the pipe buffer size, you'll get atomic writes (on Linux at least IIRC).

Also to be clear, I'm not suggesting removing the lock on Stdout, I'm saying that Stdout won't ever be able to satisfy everyone because it's a tailored use case and therefore we should offer as unfettered access as possible while acting as stdout.

[ChrisDenton]

I don't think std will ever be able to satisfy everyone. Nor do I think it has to. And if it's independent of the normal std I/O then I wonder why it needs to be in std rather than a crate?

[m-ou-se]

should offer as unfettered access as possible while acting as stdout

Note that StdoutRaw on windows isn't as 'unfettered' / 'raw' as one might expect. It still checks if the output is a console or not, and if it is, has a bunch of logic for using WriteConsoleW after converting the [u8] stream to UTF-16.

[SUPERCILEX]

Right, hence why I'd like to have this be in the stdlib. A crate would have to replicate all the craziness in https://github.com/rust-lang/rust/blob/8327047b23dc1eb150fdfb42177939388661eb6d/library/std/src/sys/windows/stdio.rs. I don't know much about windows, but I presume the stdlib is doing that for a reason and a crate would have to mirror the stdlib implementation.

[the8472]

Can you describe use-cases, how the API should look like? How it should behave under concurrent access?
We debated some sort of take() method that would extract the handle/descriptor and leave the equivalent of /dev/null in its place, but we don't even know if that would solve your use-cases.

Generally it is unclear what a reasonable "raw access" API would look like in a portable way or if non-portable solutions making more platform-specific guarantees would be preferred or ...

[ChrisDenton]

As far as I understand this proposal, it's just like io::Stdout without the lock, right? It's not doing anything special.