Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Readable stream pipe made with {end: false} does not emit an 'unpipe' event on the writable stream. #11837

Closed
zaide-chris opened this issue Mar 14, 2017 · 3 comments
Labels
stream Issues and PRs related to the stream subsystem.

Comments

@zaide-chris
Copy link
Contributor

zaide-chris commented Mar 14, 2017

  • Version: v7.7.2
  • Platform: Windows 10 64bit
  • Subsystem: stream

After a readable stream emits an end event would it not be natural for the stream to completely unpipe it's self from all the destinations even if the pipe was made with pipeOpts = {end: false}?

The pipe does partially clean it's self up but it fails to emit an unpipe event on the writable stream and it ends up leaving a reference to the writable stream in the internal state.pipes of the readable stream.

Looking at _stream_readable.js it appear like it would be easy to fix this by changing:
line 512 from var endFn = doEnd ? onend : cleanup; to var endFn = doEnd ? onend : unpipe;
and
line 548 from src.removeListener('end', cleanup); to src.removeListener('end', unpipe);

the unpipe in this scope calls src.unpipe(dest)
that ends with calling dest.emit('unpipe', src)
that triggers the dest.on('unpipe', onunpipe) listener
onunpipe then calls cleanup

So replacing directly calling cleanup with unpipe there shouldn't result in any side effects other then an 'unpipe' event emitted on the writable stream and readable stream losing an reference to an writable stream that it's never going to use again.

Examples:

const stream = require('stream');
function makeEampleSource(name){
	let source = new stream.Readable({read: () => {
		source.push(`I'm ${name}`)
		source.push(null)
	}})
	source.name = name
	return source
}
let nullOut = new stream.Writable({write: (chunk, encoding, callback) => callback()});
nullOut.on('unpipe', (source)=>{
	console.log(`Unpiped ${source.name}`);
})
nullOut.on('pipe', (source)=>{
	console.log(`Piped ${source.name}`);
})

let sourceA = makeEampleSource('sourceA')
sourceA.pipe(nullOut, {end: false})
sourceA.on('end', ()=>{
	setImmediate(() => {
		console.log('I expected this to be 0:', sourceA._readableState.pipesCount)
	})
})


setTimeout(() => {
	let sourceB = makeEampleSource('sourceB')
	sourceB.pipe(nullOut, {end: false})
	//This next line works around partial unpipe issue
	sourceB.on('end', ()=>{sourceB.unpipe()})
	sourceB.on('end', ()=>{
		setImmediate(() => {
			console.log('This time it is 0:', sourceB._readableState.pipesCount)
		})
	})
}, 10)


setTimeout(() => {
	let sourceC = makeEampleSource('sourceC')
	sourceC.pipe(nullOut)
	sourceC.on('end', ()=>{
		setImmediate(() => {
			console.log('This time it is still 0 as the target\'s \'finish\' ended up trigering unpipe:', sourceC._readableState.pipesCount)
		})
	})
}, 20)

Note the line 'Unpiped sourceA' was never logged but sourceA did end and will never send data to the nullOut again.

@mscdex mscdex added the stream Issues and PRs related to the stream subsystem. label Mar 14, 2017
@Fishrock123
Copy link
Contributor

cc @nodejs/streams

@mcollina
Copy link
Member

I will check asap.

@addaleax
Copy link
Member

@zaide-chris I haven’t taken an in-depth look but since you basically lined out a solution to this problem which seems to make sense, do you think you could open a PR with that? If so, we’re here for questions (or in #node-dev on freenode) if you have any.

zaide-chris added a commit to zaide-chris/node that referenced this issue Apr 7, 2017
  Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
  When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.

Fixes: nodejs#11837
mcollina pushed a commit to mcollina/node that referenced this issue May 2, 2017
Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.

Fixes: nodejs#11837
PR-URL: nodejs#11876
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
mcollina pushed a commit to mcollina/node that referenced this issue May 2, 2017
Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.

Fixes: nodejs#11837
PR-URL: nodejs#11876
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
evanlucas pushed a commit that referenced this issue May 3, 2017
Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.

Fixes: #11837
PR-URL: #11876
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
evanlucas pushed a commit that referenced this issue May 3, 2017
Currently when the destination emits an 'error', 'finish' or 'close'
event the pipe calls unpipe to emit 'unpipe' and trigger the clean up
of all it's listeners.
When the source emits an 'end' event without {end: false} it calls
end() on the destination leading it to emit a 'close', this will again
lead to the pipe calling unpipe. However the source emitting an 'end'
event along side {end: false} is the only time the cleanup gets ran
directly without unpipe being called. This fixes that so the 'unpipe'
event does get emitted and cleanup in turn gets ran by that event.

Fixes: #11837
PR-URL: #11876
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stream Issues and PRs related to the stream subsystem.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants