errors and contradictions in docs for process.stdout/stderr

[sam-github]

This situation keeps changing around, but my understanding ATM is that process.stdout and process.stderr (fd 1 and 2) are set blocking when they are files, and on UNIX (OS X/Linux) also set blocking when they are TTYs. I'm not sure what is done if they are pipes or any of the other possible UNIX devices.

I think the standard way to describe this in node documentaion is that an API that "can block" is synchronous (it runs to completion, whether the OS blocks or not will depend on buffering, etc, but its the run-to-completion that is important), and to describe "non-blocking" APIs the word "asynchronous" is used.

Given that understanding, the docs have these problems:

In https://github.com/nodejs/node/blob/master/doc/api/process.md#processstderr

Writes can block when output is redirected to a file.

Should say that stdout/err is synchronous when connected to a file. I believe that use of the word "can" is trying to express that blocking won't occur unless the O/S chooses to block, but its not very helpful. It could also mean "node reserves the right to set the fd to blocking mode, or not to, so depend on nothing", but I don't think it means that.

Note that disks are fast and operating systems normally employ write-back caching so this is very uncommon.

Probably meant to be a note about previous statement, but because of list structure, reads as a note on the i. and ii. bullet points that follow.

If we wanted to, we could say "If data is written faster than the OS I/O systems can buffer it, blocking may occur, but that blocking is unusual even when process.stderr is synchronous due to modern OS I/O buffering strategies.".

Writes on UNIX will block by default if output is going to a TTY (a terminal).

Should say its sync. Whether it blocks or not depends on whether TTY buffers fill up. The file case used the weasel-word "can", but here we use the emphatic WILL, which is a contradictory, and its not true that writes to a TTY will block, even if the fd is set to blocking.

Windows functionality differs. Writes block except when output is going to a TTY.

Here its partially restating the "can block on files" from the earlier paragraph, except that the "can" is omitted. So, its a contradictory restating of previous docs. Should just say TTY writes are async on windows for historical reasons.

Nowhere in here does it say what happens when stdout/stderr is a pipe! An important case. Is it sync like files and TTYs? Async like TTY on Windows? Depends on system? I don't know, and the docs don't say.

To check if Node.js is being run in a TTY context, read the isTTY property on process.stderr, process.stdout, or process.stdin:

Has a typo, a trailing :.

isTTY does not have to have same value for stdin/out/err. This sentence should say "stderr is attached to a TTY if the isTTY property is set.", and should say nothing about stdin/stdout, leave that to their docs.

In https://github.com/nodejs/node/blob/master/doc/api/process.md#processstdout

Copy and paste of the stderr docs, all comments apply to it.

In https://github.com/nodejs/node/blob/master/doc/api/process.md#tty-terminals-and-processstdout

TTY Terminals and process.stdout

Wrongly named, should be "TTY Terminals and process.stdout and process.stderr" or "TTY Terminals and process output".

The process.stderr and process.stdout streams are blocking when outputting to TTYs (terminals) on OS X

Legally speaking, not wrong, since OS X is considered UNIX, and the above is true for UNIX, but is very misleading, since it implies they are not blocking for Windows and Linux (which is not true).

Should say stderr/out are synchronous when writing to TTYs on UNIX, to agree with previous docs.

as a workaround for the operating system's small, 1kb buffer size.

Not my understanding, don't know where that crept in. Its sync because users expect console.log('hello'); process.exit(0) to not exit until the I/O has been written, so the I/O has to be sync, not async. :-(

https://github.com/nodejs/node/blob/master/doc/api/console.md#asynchronous-vs-synchronous-consoles

The console functions are usually asynchronous unless the destination is a file.

That directly contradicts the docs for process.stdout and process.stderr, unless by usually "on windows" is meant, and "on OS X and Linux" is considered "unusual", but I think its just a misstatement.

Additionally, console functions are blocking when outputting to TTYs (terminals) on OS X as a workaround for the OS's very small, 1kb buffer size. This is to prevent interleaving between stdout and stderr.

Copied verbatim from https://github.com/nodejs/node/blob/master/doc/api/process.md#tty-terminals-and-processstdout, including with the problems.

Entire section should be replaced by a web link to https://github.com/nodejs/node/blob/master/doc/api/process.md#tty-terminals-and-processstdout

I'll PR improvements to this text after my understanding of how it currently works is confirmed.

[Fishrock123]

/wall of text comment/

I suppose I am best qualified to answer this.

First off, please look at https://nodejs.org/dist/latest-v7.x/docs/api/process.html#process_process_stdout rather than the .md, GitHub doesn't format it correctly.

Reading more...

[Fishrock123]

I suppose it could be clarified. Here is a summary to the best of my (confident) understanding:

• The default for streams is non-blocking asynchronous, as such anything not listed in non-blocking async.
• When interacting with files, operations are "blocking" but may or may not be instantaneous due to write-back caching.
• When interacting with TTYs, operations are blocking except on Windows.
• Windows Pipes are blocking.
• There may be other unknown Windows edge cases.

---

The process.stderr and process.stdout streams are blocking when outputting to TTYs (terminals) on OS X

I'm not sure where you got this from but that is an older version of the UNIX comment.

---

Also, TTYs definitely do not block on Windows.

---

as a workaround for the operating system's small, 1kb buffer size.

Not my understanding, don't know where that crept in. Its sync because users expect console.log('hello'); process.exit(0) to not exit until the I/O has been written, so the I/O has to be sync, not async. :-(

It may not be your understanding but it would be prudent to check the history. There is various information linked to from this issue: #6980.

---

Also to reiterate: Non-windows platforms block when interacting with TTYs by default. They do not block to Pipes. Windows is the opposite.

---

Edit: Is the behaviour confusing? Yes. Is there a better answer? ¯\_(ツ)_/¯

[sam-github]

@Fishrock123 re:

as a workaround for the operating system's small, 1kb buffer size.

Thanks for the historical references. I found via #6980 the PR #6895, and can see that you authored this text. From what I can see, it was accurate at the time you added it. It must have been some time after that, when TTYs became blocking on all UNIX systems, that your original text was not updated and thus became retroactively confusing, but my goal is to improve the current accuracy of the docs, not explore the history of how we got here.

I take it that you are suggesting that the OS X 1K buffer size, the interleaving of I/O, and the truncation of output on process.exit (which is mentioned repeatedly in #6980) be included in the docs to justify their unique sync nature on UNIXes? Isn't the 1K buffer size just something that made the output truncation and interleaving more common on OS X than Linux, rather than a sufficient reason in itself to make TTYs synchronous?

I quoted:

The process.stderr and process.stdout streams are blocking when outputting to TTYs (terminals) on OS X

You said:

I'm not sure where you got this from but that is an older version of the UNIX comment.

I preceeded every set of quotes with a link to its source, that one is from https://github.com/nodejs/node/blob/master/doc/api/process.md#tty-terminals-and-processstdout and is current. Or do you mean older as-in, it preceeded the changes and didn't get updated when it should have?

Windows Pipes are blocking.

... and UNIX Pipes are blocking (to confirm).

OK, with that info and your other confirmations I should be able to correct the docs, thanks.

[Fishrock123]

... and UNIX Pipes are blocking (to confirm).

Pipes are non-blocking by default except on Windows.

It must have been some time after that, when TTYs became blocking on all UNIX systems, that your original text was not updated and thus became retroactively confusing, but my goal is to improve the current accuracy of the docs, not explore the history of how we got here.

Oh I understand now, yes, sorry. It should probably clarified that this is mostly a workaround for OS X but other platforms are also adjusted because the difference in negligible otherwise and maintains better consistency.

[Fishrock123]

I take it that you are suggesting that the OS X 1K buffer size, the interleaving of I/O, and the truncation of output on process.exit (which is mentioned repeatedly in #6980) be included in the docs to justify their unique sync nature on UNIXes? Isn't the 1K buffer size just something that made the output truncation and interleaving more common on OS X than Linux, rather than a sufficient reason in itself to make TTYs synchronous?

It was the primary reason, but similar problems exist on all platforms above their buffer size.

[addaleax]

When interacting with files, operations are "blocking" but may or may not be instantaneous due to write-back caching.

How sure are you about that? Last I heard the libuv thread pool was used for that, and a quick look at the source code seems to confirm that.

Ignore me, I missed the context here.