Writable#end does not trigger 'finish' event: why?

[fbiville]

• Node.js Version: v12.8.0
• OS: Mac OS Mojave (10.14.6)
• Scope (install, code, runtime, meta, other?): runtime
• Module (and version) (if relevant): stream

Quoting the stream documentation:

The 'finish' event is emitted after the stream.end() method has been called, and all data has been flushed to the underlying system.

In the setup explained below, it seems that the 'finish' is not triggered or received despite calling end on the stream.

I have the following Write "pipeline" stream, whose main responsibility is to dispatch data to the appropriate Transform "input" stream among the n available. These n Transform all write to a destination stream.

The "pipeline" stream is implemented as follows (and hardcoded to 3 "input" streams here to keep things concise):

class Pipeline extends Writable {
    constructor(options, destinationStream) {
        super(options);
        this.destinationStream = destinationStream;
        this.inputStreams = [
            new InputTransform(options),
            new InputTransform(options),
            new InputTransform(options)
        ];
        this.inputStreams.forEach((it) => {
            it.on('error', (err) => {
                this.emit('error', err);
            });
            it.pipe(this.destinationStream);
        });
        this.on('finish', () => {
            this.inputStreams.forEach((it) => it.emit('end'));
        });
        this.on('error', () => {
            this.end();
        })
    }

    _write(chunk, _, callback) {
        this.inputStreams[chunk['index']].write(chunk['data']);
    }
}

You can see that the "pipeline" intercepts errors coming from any of the "input" streams and emits the intercepted error itself.
When an 'error' event occurs, the "pipeline" ends itself and I expect that the pipeline receives a "finish" event afterwards to end all the "input" streams.

However, as you see if you clone and run this experiment, the 'finish' event is never triggered or never received.

What is going on?

[igoradamenko]

@fbiville, hey.

I guess that the reason is an unfired callback in _write. Here is the doc about implementing Writable Streams:

https://nodejs.org/api/stream.html#stream_implementing_a_writable_stream

In the description of _write (https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback_1) there is a line:

The callback method must be called to signal either that the write completed successfully or failed with an error. The first argument passed to the callback must be the Error object if the call failed or null if the write succeeded.

You're not firing this callback.

This is why you should do it. There is an internal function called doWrite:

https://github.com/nodejs/node/blob/master/lib/_stream_writable.js#L408-L420

As you see, it fires your _write and passes state.onwrite as a callback. If you check the code of the callback:

https://github.com/nodejs/node/blob/master/lib/_stream_writable.js#L446-L478

you will see that it fires a function called afterWrite:

https://github.com/nodejs/node/blob/master/lib/_stream_writable.js#L480-L488

which actually tries to finish your stream in by firing finishMaybe:

https://github.com/nodejs/node/blob/master/lib/_stream_writable.js#L631-L650

That's the place where 'finish' event is emitted.

But you're not firing the callback in your code, so it doesn't work properly. If you fire it, you will get this:

❯ nts

> pipeline-event-not-triggered@1.0.0 test ~/Downloads/nodejs_help_issue2112-master         
> yarn run jasmine

yarn run v1.12.3
$ jasmine
Randomized with seed 01932
Started
.


1 spec, 0 failures
Finished in 0.01 seconds
Randomized with seed 01932 (jasmine --random=true --seed=01932)
✨  Done in 0.25s.

Hope it helps.

[fbiville]

Thanks, it does help!
The mistake I was doing in the actual codebase was calling this.emit('error', err) instead of callback(err) for some edge cases (the initial intent was to avoid the "multiple callback called" error but this was misguided there).

[fbiville]

And now I know where to look in the sources, I thought I had to learn C++ but it appears not :D