Repository navigation
process.stdin example from docs no longer works in node 10 #20503
Description
Activity
Bisecting this turned into trisecting, because it seems like we introduced this issue gradually:
- 1e0f331 made a first change in behaviour for the test case, because now only the first chunk of data is being read before the process exits on its own
- 4383c0add89c9075a7c195d07f2c7dfe595f811b from src: clean up resources on Environment teardown #19377 already takes care of this part of the problem. Yay!
- 0778f79 made the change in behaviour to break this fully. I’m not quite sure (yet) why this is happening, though.
@nodejs/streams
- 1e0f331 made a first change in behaviour for the test case, because now only the first chunk of data is being read before the process exits on its own
I finally narrowed down the problem I was having with Node 10 and node-pg-query-stream to this and lo and behold someone else had already reported it! What I noticed was that I'd get the first batch of results out of the stream, but then it would abort prematurely without emitting
end.The commit message for 0778f79 looks at odds with the conditional
if (!state.destroyed && (state.length || state.ended))though -- if it's not supposed to emitreadablewhen the stream has ended, shouldn't it be negatingstate.endedat minimum? In the testcase accompanying the change it looks like the conditional might be satisfied bystate.length.I think the idea behind
(state.length || state.ended)was that we do emit thereadableevent both for incoming data, and for the end-of-data marker. Soreadableis supposed to be emitted even if the stream has ended, just not after it has been destroyed./cc @nodejs/streams @mafintosh
- My commit message there ended up a bit confusing after some back and fourths in the PR sorry. The intent there in the if to add a generic conditional of when a readable event is okay to emit (stream not destroyed, and has data or is ended). Can someone share a concrete example that is failing now? Then I'll help take a look. /m…On Fri, May 4, 2018, 09:35 Anna Henningsen ***@***.***> wrote: I think the idea behind (state.length || state.ended) was that we do emit the readable event both for incoming data, and for the end-of-data marker. So readable is supposed to be emitted even if the stream has ended, just not after it has been destroyed. /cc @nodejs/streams <https://github.com/orgs/nodejs/teams/streams> @mafintosh <https://github.com/mafintosh> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#20503 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAW_VeW1n2TbpOixo-9h8fs-ZT374Y5_ks5tvATZgaJpZM4Txle2> .
@mafintosh The PR description has a standalone test case, if you want
@mafintosh this might be more involved than you're looking for but the problem I was having is reducible to the "fast-reader" test case in https://github.com/brianc/node-pg-query-stream if you check out that repository and run the tests.
Any update on that? It breaks some of our tools in development
I'd like to work on the issue, I seem to have found the underlying cause and hopefully a correct solution =). Will submit a PR soon.
Reacted by Gireesh Punathil and Dian Fay26 remaining items
- added a commit that references this issue
on Jan 14, 2019 - added 2 commits that reference this issue
on Jan 16, 2019 The docs for this feature still aren't very clear...
Has anyone tried piping in a file longer than 65536 bytes via stdin?
The example from the docs shows this code:
process.stdin.on('readable', () => { let chunk; // Use a loop to make sure we read all available data. while ((chunk = process.stdin.read()) !== null) { process.stdout.write(`data: ${chunk}`); } });
Note the code comment:
Use a loop to make sure we read all available data.
Perhaps it's stupid of me, but I assumed this meant the while loop there would cause it to read all of stdin, so I made some code like:
process.stdin.on('readable', () => { let chunk, chunks, content; chunks = []; // Use a loop to make sure we read all available data. while ((chunk = process.stdin.read()) !== null) { chunks.push(chunk); } content = chunks.join(''); console.log(content.length); render(content); });
What I see from this naïve code though is that my
content.lengthis always65536, but I know the file is longer than that - what I've got incontentis truncated.Unfortunately because the code was trying to
render(content), exceptions prevented me immediately noticing what actually happens: which is that theon('readable' ...)event handler gets called multiple times... then eventually theendevent when there is no more to read.So I have to structure the code more like this:
var chunk, chunks, content; chunks = []; process.stdin.on('end', () => { content = chunks.join(''); console.error(content.length); // 88106 - yay! return content; }); process.stdin.on('readable', function() { console.error('READABLE') while ((chunk = process.stdin.read()) !== null) { chunks.push(chunk); console.error('CHUNK') } });
This makes perfect sense, but I think something of this behaviour ought to be mentioned in the docs.
https://nodejs.org/api/process.html#process_process_stdin
Just says:The
process.stdinproperty returns a stream connected tostdin(fd0). It is a net.Socket (which is a Duplex stream) unless fd0refers to a file, in which case it is a Readable stream.and the docs for
Readablestreams waffle on quite a bit but I don't see anything about getting multiplereadableevents:
https://nodejs.org/api/stream.html#stream_readable_streamsI understand from the long discussion in the thread above that what the while loop is really for is to ensure that the stream doesn't end early, if there is initially nothing to read?
Perhaps this could all be made a bit clearer somehow.
@anentropic I think a PR to improve the docs would be welcomed!
@mcollina ok no prob, give me a day or two
- added 2 commits that reference this issue
on Apr 28, 2019 - added 2 commits that reference this issue
on May 10, 2019 - added 2 commits that reference this issue
on May 16, 2019 - added a commit that references this issue
on Apr 16, 2025 - added 2 commits that reference this issue
on Jul 27, 2026
I am running
process.stdinexample from docs:I expect the same results as in node v9 and below: command line should wait for my input and return it prefixed with
data:. Instead, process is closed right away. I believe this is regression as example works fine in node v9 and v8.(edited by @addaleax: syntax highlighting)