Memory overflow in Readable.toWeb due to the default strategy

@debadree25

Version

v20.0.0-pre

Platform

Linux Mars 5.19.0-35-generic #36~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 17 15:17:25 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

stream

What steps will reproduce the bug?

To reproduce, run the following script with the debugger enabled, i.e. node inspect mytest.mjs

// mytest.mjs

import fs from 'node:fs'
import process from 'node:process'
import { Readable } from 'node:stream'

const reportMemoryUsage = (maxMb, randomWebStream) => {
	const { arrayBuffers } = process.memoryUsage()

	if (arrayBuffers > maxMb * 1024 * 1024) {
		debugger
		process.exit(1)
	}
}

async function main() {
	const randomNodeStream = fs.createReadStream('/dev/urandom', { highWaterMark: 5556 })
	const randomWebStream = Readable.toWeb(randomNodeStream)

	setInterval(() => reportMemoryUsage(16, randomWebStream), 1e3)
}

main()

When the debugger stops at the breakpoint, open up the repl, and print out the contents of randomWebStream, i.e. console.dir(randomWebStream, { depth: 32 }).
Inspect the output of the queue in randomWebStream[Symbol(kState)].controller[Symbol(kState)].queue.

How often does it reproduce? Is there a required condition?

Always. However, the underlying stream that's being turned into the web stream must not be in the object mode.

What is the expected behavior? Why is that the expected behavior?

Because the underlying stream was not in the object mode and had its highWaterMark set at 5556 bytes, the expected behavior is for the web stream's underlying queue to also have the total size circa 5556 bytes.

What do you see instead?

In my case I see the queue has the length of 5556 elements, with each element being an instance of Uint8Array of size 5556 bytes by itself - thus producing a grand total of 30869136 bytes, - as opposed to the 5556 bytes which the highWaterMark of the underlying stream randomNodeStream had.

Additional information

This may be related to #46347.

The source of this behavior is likely this line:

node/lib/internal/webstreams/adapters.js

Lines 420 to 424 in fe514bf

    
           // When not running in objectMode explicitly, we just fall 
        
           // back to a minimal strategy that just specifies the highWaterMark 
        
           // and no size algorithm. Using a ByteLengthQueuingStrategy here 
        
           // is unnecessary. 
        
           return { highWaterMark };

This should probably use the ByteLengthQueuingStrategy instead, as previously suggested by @debadree25 #46347 (comment)

My output:

< ReadableStream {
<   [Symbol(kType)]: 'ReadableStream',
<   [Symbol(kState)]: <ref *1> {
<     disturbed: false,
<     reader: undefined,
<     state: 'readable',
<     storedError: undefined,
<     stream: undefined,
<     transfer: {
<       writable: undefined,
<       port1: undefined,
<       port2: undefined,
<       promise: undefined
<     },
<     controller: ReadableStreamDefaultController {
<       [Symbol(kType)]: 'ReadableStreamDefaultController',
<       [Symbol(kState)]: {
<         cancelAlgorithm: [Function: bound cancel],
<         closeRequested: false,
<         highWaterMark: 5556,
<         pullAgain: false,
<         pullAlgorithm: [Function: bound pull],
<         pulling: false,
<         queue: [
<           {
<             value: Uint8Array(5556) [
<                68,  58,  34, 241,  86,  28, 225,  57, 134, 151,  97, 210,
<               202,  53, 112,  41, 211, 174, 206,  99, 122, 169,   2, 151,
<                60, 189, 167, 183, 100, 204, 138,  91, 114,  68,  40,  54,
<               208,  26, 200, 105,  34, 101, 236, 249, 197, 131,  59,  96,
<               221, 162, 243, 119,   4,  55, 228, 212, 232, 226, 153, 153,
<                73, 137, 103, 109,  43, 165,  61, 168,  89, 251, 153,   1,
<                59, 174, 150,  60,  42,  10,  13, 114, 169, 222, 144,  46,
<               176, 177,  48, 146,  87, 176,  38, 135, 149, 175, 152,  55,
<               230,  85, 157, 128,
<               ... 5456 more items
<             ],
<             size: 1
<           },

[TRUNCATED]

<           {
<             value: Uint8Array(5556) [
<               105,  82,  92, 129, 142,  78,  16, 182,  94, 173, 208,  48,
<               250, 180,  81,  99, 240,  92, 187,   1,  26, 138,  41, 175,
<                60,  95,  91, 236,  70, 189, 130, 146,  48, 103, 159,  40,
<                18, 251, 183, 204,  96,  98, 225,  75,   2,  74,  81, 126,
<               200, 137, 151, 171,  81, 197, 161, 118, 181, 189,  23,  86,
<               154,  55,  38,  80,  35, 131, 163,  43,  80, 200, 211, 205,
<                70,  25,  67, 212, 196, 203,  19, 248, 232,  44,  57, 143,
<               203,  21, 165, 242, 100, 200,   8, 187, 252, 221, 229,  20,
<               232, 101,  44, 109,
<               ... 5456 more items
<             ],
<             size: 1
<           },
<           {
<             value: Uint8Array(5556) [
<               189,   7, 147, 159, 128,  93,  34, 247, 167,  28, 202,  89,
<               197,  76, 189,  37, 189, 244, 242, 166, 232,  16,  82, 102,
<                69, 247, 211, 149, 149, 210,  88, 110,  12, 103,  69, 184,
<                88, 162, 181, 123, 167, 244, 170, 165,  89, 197,  68,  72,
<                70,  48, 194, 207,  11, 255, 126,  67,  32, 148, 181,  14,
<               182,  78, 187, 110, 155, 232,  10,  76,  12, 132,  45, 168,
<               119, 193,  73, 229, 203, 253,  50,  90, 195, 128, 149, 240,
<                51, 179, 243, 139, 235, 141,  65,   5,  77, 120, 178,  43,
<                86,  98, 187, 212,
<               ... 5456 more items
<             ],
<             size: 1
<           }
<         ],
<         queueTotalSize: 5556,
<         started: true,
<         sizeAlgorithm: [Function (anonymous)],
<         stream: ReadableStream {
<           [Symbol(kType)]: 'ReadableStream',
<           [Symbol(kState)]: [Circular *1],
<           [Symbol(nodejs.webstream.isClosedPromise)]: [Object]
<         }
<       }
<     }
<   },
<   [Symbol(nodejs.webstream.isClosedPromise)]: {
<     promise: Promise { <pending> },
<     resolve: [Function (anonymous)],
<     reject: [Function (anonymous)]
<   }
< }
<

Suggested fix: #47129

This issue has been marked as stale due to 210 days of inactivity.
It will be automatically closed in 30 days if no further activity occurs. If this is still relevant, please leave a comment or update it to keep it open.

This issue has been automatically closed after 30 days of inactivity following its stale status (no activity for a total of 240 days).
If this is still relevant, feel free to reopen it or leave a comment with additional details so we can continue the discussion.

Navigation Menu

Repository navigation

Memory overflow in `Readable.toWeb` due to the default strategy #47128

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// When not running in objectMode explicitly, we just fall
	// back to a minimal strategy that just specifies the highWaterMark
	// and no size algorithm. Using a ByteLengthQueuingStrategy here
	// is unnecessary.
	return { highWaterMark };

Memory overflow in Readable.toWeb due to the default strategy #47128

Description

Version

Platform

Subsystem

What steps will reproduce the bug?

How often does it reproduce? Is there a required condition?

What is the expected behavior? Why is that the expected behavior?

What do you see instead?

Additional information

Activity

lilsweetcaligula commented on Mar 16, 2023

lilsweetcaligula commented on Mar 16, 2023

github-actions commented on Jun 19, 2026

github-actions commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Memory overflow in `Readable.toWeb` due to the default strategy #47128