Skip to content

Node 16.14.2 highWaterMark:0 doesn't handle backpressure #42457

Closed
Closed
Node 16.14.2 highWaterMark:0 doesn't handle backpressure#42457
Labels
staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.streamIssues and PRs related to Node.js streams.

Description

@paulrutter
paulrutter
opened on Mar 24, 2022

Version

16.14.2

Platform

x64

Subsystem

No response

What steps will reproduce the bug?

async function start() {
	const { Transform } = require('stream');
	const transformStream = new Transform({
		objectMode: true,
		highWaterMark: 0,

		transform(item, enc, callback) {
	  	  console.error("writing", item);
	  	  this.push(item);
	  	  callback();
	    }
	});

	transformStream.write('hello1');
	transformStream.write('hello2');
	transformStream.write('hello3');
	transformStream.write('hello4');
	transformStream.write('hello5');
	transformStream.write('hello6');
	transformStream.write('hello7');
	transformStream.write('hello8');
	transformStream.write('hello9');
        transformStream.end();

	for await (const text of transformStream) {
	  await wait();
	  console.error(text);
	}
};

async function wait() {
  return new Promise(resolve => {
	setTimeout(() => resolve(), 1000);
  });
}

start();

I wouldn't expect the transform method to be called, while the highWaterMark is already reached.

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior?

Node 14 and older, backpressure is handled properly (even with highWaterMark: 0):

Expected output:

writing hello1
writing hello2
hello1
hello2
writing hello3
hello3
writing hello4
hello4
writing hello5
hello5
writing hello6
hello6
writing hello7
hello7
writing hello8
hello8
writing hello9
hello9

What do you see instead?

PS > node .\backpressure.js
writing hello1
writing hello2
writing hello3
writing hello4
writing hello5
writing hello6
writing hello7
writing hello8
writing hello9
hello1
hello2
hello3
hello4
hello5
hello6
hello7
hello8
hello9

Additional information

When using highWaterMark: 1 in Node 16.x, backpressure works as expected again.
Did anything change around using highWaterMark:0 in Node16? It doesn't seem backwards compatible.

Activity

  1. paulrutter
    changed the title [-]Node 16.14.2 highWaterMark:0 does handle backpressure[/-] [+]Node 16.14.2 highWaterMark:0 doesn't handle backpressure[/+] on Mar 24, 2022
  2. VoltrexKeyva
    added
    streamIssues and PRs related to Node.js streams.
    on Mar 24, 2022

  3. paulrutter commented on Mar 25, 2022

    @paulrutter
    paulrutter
    on Mar 25, 2022
    Author

    Probably related changes: #40935 and #40947

  4. kanongil commented on Jun 27, 2022

    @kanongil
    kanongil
    on Jun 27, 2022
    Contributor

    Indeed, just noticed this issue with a test case that is now failing on node 16.14.0+. Does anyone care about this regression? @ronag?

  5. targos commented on Jun 27, 2022

    @targos
    targos
    on Jun 27, 2022
    Member

    @nodejs/streams

  6. kanongil
    added a commit that references this issue on Jun 27, 2022

    Node v16.14+ doesn't work

    dab78e6

  7. mcollina commented on Jun 27, 2022

    @mcollina
    mcollina
    on Jun 27, 2022
    SponsorMember

    It seems #40947 should not have been backported at least.

    @kanongil @paulrutter what behavior were you expecting for highWaterMark: 0? Was this documented anywhere at all?

  8. paulrutter commented on Jun 27, 2022

    @paulrutter
    paulrutter
    on Jun 27, 2022
    Author

    My thought with using highwatermark 0 was to not buffer any chunks, instead just wait until the current chunk is done and then proceed to the next chunk.
    I don't think the docs describe using 0 specifically, but it worked for my usecase so i didn’t give it more thought until it broke in node16.

  9. mcollina commented on Jun 27, 2022

    @mcollina
    mcollina
    on Jun 27, 2022
    SponsorMember

    Essentially you assumed that it would be the same behavior of highWaterMark: 1?

    In my mind a value of 0 would be "false", so no buffering at all. The current behavior matches that description.

  10. paulrutter commented on Jun 27, 2022

    @paulrutter
    paulrutter
    on Jun 27, 2022
    Author

    Yes, true. In hindsight, using 1 would have made more sense. Still i think it's not great that the change is not backwards compatible anymore.

  11. mcollina commented on Jun 27, 2022

    @mcollina
    mcollina
    on Jun 27, 2022
    SponsorMember

    In that I agree.

    @nodejs/tsc @nodejs/releasers, what should we do?

  12. kanongil commented on Jun 28, 2022

    @kanongil
    kanongil
    on Jun 28, 2022
    Contributor

    I use highWaterMark: 0 to enable an object mode transform stream to be pull-based.

    This means a call to stream.read() will initially return null, and once 'readable' triggers, I can stream.read() a single element out of it. Then I have the option to do another stream.read() to trigger a new pull, or just wait until it suits me.

    When applied to a Transform stream that is piped into, it means that it internally only stores 1 incoming object (in the write end), before applying backpressure.

    writableHighWaterMark: 0, readableHighWaterMark: 1 doesn't work, since it will cause 2 objects to be internally buffered. 1 on the write side, and 1 in the read side.

  13. mcollina commented on Jun 30, 2022

    @mcollina
    mcollina
    on Jun 30, 2022
    SponsorMember

    @ronag you should chime in on this one, because it looks like we might want to revert to the original behavior.

  14. ronag commented on Jun 30, 2022

    @ronag
    ronag
    on Jun 30, 2022
    Member

    I'll try to have a look next week.

  15. ronag commented on Jun 30, 2022

    @ronag
    ronag
    on Jun 30, 2022
    Member

    I'm not sure hwm of 0 makes any sense. Maybe the simple solution is to simply automatically change 0 to 1?

  16. 4 remaining items

  17. kanongil commented on Jul 1, 2022

    @kanongil
    kanongil
    on Jul 1, 2022
    Contributor

    @mcollina Yes, looking into making a PR right now.

  18. kanongil commented on Jul 1, 2022

    @kanongil
    kanongil
    on Jul 1, 2022
    Contributor

    Hmm, it appears that my issue it not the same as this (at least the reproduction code). The reproduction code relies on calling write() into a stream that returns false. This makes the case fundamentally incompatible with the new transform-on-write model which was already present in v16.0.

    As such my fix doesn't handle this exact issue, though it might fix the actual issue this is based on.

  19. kanongil commented on Jul 1, 2022

    @kanongil
    kanongil
    on Jul 1, 2022
    Contributor

    PR up in #43648. As per my previous comment, it doesn't fix the exact reproduction code from this issue, but it fixes the regression in my transform stream.

  20. kanongil
    added a commit that references this issue on Jul 1, 2022

    stream: fix Transform with hwm 0 regression

    b8a1698
  21. aduh95
    added a commit that references this issue on Jul 24, 2022

    stream: fix 0 transform hwm backpressure

    31d9edc
  22. danielleadams
    added a commit that references this issue on Jul 26, 2022

    stream: fix 0 transform hwm backpressure

    bf3991b
  23. guangwong
    added a commit that references this issue on Oct 10, 2022

    stream: fix 0 transform hwm backpressure

    1029988

  24. github-actions commented on Jun 25, 2026

    @github-actions
    github-actionsbot
    on Jun 25, 2026 – with GitHub Actions
    Contributor

    This issue has been marked as stale due to 210 days of inactivity.
    It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

  25. github-actions
    added
    staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.
    on Jun 25, 2026
  26. mcollina
    closed this as completedon Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleIssues and PRs marked stale due to inactivity and scheduled for automatic closure.streamIssues and PRs related to Node.js streams.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions