`Readable.toWeb` seems to load file contents to memory

[Dzieni]

Version

v19.5.0

Platform

Darwin dzieni 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:35 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T8101 arm64

Subsystem

stream

What steps will reproduce the bug?

• run REPL
• create a file stream: const nodeStream = fs.createReadStream('/some/large/file')
• check process.memoryUsage()
• convert it to a web stream: const webStream = (require('stream').Readable).toWeb(nodeStream)
• check process.memoryUsage()

How often does it reproduce? Is there a required condition?

At all times

What is the expected behavior?

Memory usage does not grow significantly.

What do you see instead?

Memory usage (precisely arrayBuffers section) grows by a few orders of magnitude. It seems to be correlated with the file size.

Additional information

No response

[bnoordhuis]

Can you post a complete, ready-to-run test case? Maybe you've hit a bug, maybe you haven't, but it's impossible to tell right now.

[Dzieni]

@bnoordhuis

// since it's ESM, save it as .mjs

import fs from 'node:fs'
import process from 'node:process'
import {Readable} from 'node:stream'

// we initialize a stream, but not start consuming it
const randomNodeStream = fs.createReadStream('/dev/urandom')
// after 10 seconds, it'll get converted to web stream
let randomWebStream

// we check memory usage every second
// since it's a stream, it shouldn't be higher than the chunk size
const reportMemoryUsage = () => {
	const {arrayBuffers} = process.memoryUsage()
	console.log(
		`Array buffers memory usage is ${Math.round(
			arrayBuffers / 1024 / 1024
		)} MiB`
	)
	if (arrayBuffers > 256 * 1024 * 1024) {
		// streaming should not lead to such a memory increase
		// therefore, if it happens => bail
		console.log('Over 256 MiB taken, exiting')
		process.exit(0)
	}
}
setInterval(reportMemoryUsage, 1000)

// after 10 seconds we use Readable.toWeb
// memory usage should stay pretty much the same since it's still a stream
setTimeout(() => {
	console.log('converting node stream to web stream')
	randomWebStream = Readable.toWeb(randomNodeStream)
}, 10000)

// after 15 seconds we start consuming the stream
// memory usage will grow, but the old chunks should be garbage-collected pretty quickly
setTimeout(async () => {
	console.log('reading the chunks')
	for await (const chunk of randomWebStream) {
		// do nothing, just let the stream flow
	}
}, 15000)

This produces the same behavior on macOS and Linux:

michal@dzieni ~ % node test.mjs
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
converting node stream to web stream
Array buffers memory usage is 0 MiB
Array buffers memory usage is 617 MiB
Over 256 MiB taken, exiting

Immediately after using Readable.toWeb the memory usage spikes. Since we use /dev/urandom (so the file that never ends), the usage will grow indefinitely.

You can compare it to a third-party library readable-stream-node-to-web, where it works properly:

// since it's ESM, save it as .mjs

import fs from 'node:fs'
import process from 'node:process'
import nodeToWebStream from 'readable-stream-node-to-web'

// we initialize a stream, but not start consuming it
const randomNodeStream = fs.createReadStream('/dev/urandom')
// after 10 seconds, it'll get converted to web stream
let randomWebStream

// we check memory usage every second
// since it's a stream, it shouldn't be higher than the chunk size
const reportMemoryUsage = () => {
	const {arrayBuffers} = process.memoryUsage()
	console.log(
		`Array buffers memory usage is ${Math.round(
			arrayBuffers / 1024 / 1024
		)} MiB`
	)
	if (arrayBuffers > 256 * 1024 * 1024) {
		// streaming should not lead to such a memory increase
		// therefore, if it happens => bail
		console.log('Over 256 MiB taken, exiting')
		process.exit(0)
	}
}
setInterval(reportMemoryUsage, 1000)

// after 10 seconds we use nodeToWebStream
// memory usage should stay pretty much the same since it's still a stream
setTimeout(() => {
	console.log('converting node stream to web stream')
	randomWebStream = nodeToWebStream(randomNodeStream)
}, 10000)

// after 15 seconds we start consuming the stream
// memory usage will grow, but the old chunks should be garbage-collected pretty quickly
setTimeout(async () => {
	console.log('reading the chunks')
	for await (const chunk of randomWebStream) {
		// do nothing, just let the stream flow
	}
}, 15000)

In that case, the memory usage is fine:

michal@dzieni ~ % node src/test.mjs 
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
converting node stream to web stream
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
reading the chunks
Array buffers memory usage is 1 MiB
Array buffers memory usage is 6 MiB
Array buffers memory usage is 9 MiB
Array buffers memory usage is 15 MiB
Array buffers memory usage is 5 MiB
Array buffers memory usage is 9 MiB
Array buffers memory usage is 11 MiB
Array buffers memory usage is 4 MiB
Array buffers memory usage is 19 MiB
Array buffers memory usage is 16 MiB
Array buffers memory usage is 1 MiB
Array buffers memory usage is 30 MiB
Array buffers memory usage is 24 MiB
Array buffers memory usage is 6 MiB
Array buffers memory usage is 4 MiB
Array buffers memory usage is 2 MiB
Array buffers memory usage is 1 MiB

[bnoordhuis]

Thanks, I see what you mean. The stream starts reading before there's something consuming it.

@nodejs/whatwg-stream this is a legitimate bug. You can see it even more clearly when you switch from /dev/urandom to /dev/zero.

edit: bug also exists in v19.6.0.

[debadree25]

Hello, @Dzieni I think this issue is solved if you pass a strategy while converting the stream from node stream something like:

randomWebStream = Readable.toWeb(randomNodeStream, {
    strategy: new CountQueuingStrategy({ highWaterMark: 100 }),
  });

Tried this on my local machine and memory doesn't seem to be overflowing

I think the behavior here is somewhat expected when the readable stream is created, the pull function as described here https://nodejs.org/api/webstreams.html#new-readablestreamunderlyingsource--strategy would be called continuously as soon as the ReadableStream is created and given how adapters from webstreams are defined here

node/lib/internal/webstreams/adapters.js

Line 462 in 23effb2

pull() { streamReadable.resume(); },

this would mean if there is no strategy defined streamReadable.resume() will try to consume the whole file causing memory overflow, but when we pass a strategy it ensures backpressure is applied

Maybe the docs here could be updated to include this scenario https://nodejs.org/api/stream.html#streamreadabletowebstreamreadable-options @bnoordhuis

[mcollina]

I think this is a bug. The whole point of streams is to manage the flow of data.

[debadree25]

I think this is a bug. The whole point of streams is to manage the flow of data.

So something like a default highWatermark while doing toWeb()?

[mcollina]

No I think there is an actual bug somewhere. Instead of resume, this should call read()

[Dzieni]

@debadree25
Sounds like a workaround that is good enough for me, thanks!

[lilsweetcaligula]

Hey guys and gals, could you please confirm my findings. So I've looked into this briefly, and I believe I have traced this to /lib/internal/webstreams/readablestream.js.

Now, I am not entirely sure if it's by design but neither readableByteStreamControllerShouldCallPull nor readableStreamDefaultControllerShouldCallPull seem to check for presence of a reader:
https://github.com/nodejs/node/blob/main/lib/internal/webstreams/readablestream.js#L2458-L2481
https://github.com/nodejs/node/blob/main/lib/internal/webstreams/readablestream.js#L2223-L2240

Refusing to pull when a stream has no reader seems to alleviate the issue. Something like this seems to do the trick:
1f0e249 (#46643)

Testing against @Dzieni's benchmark - slightly modified to convert to the web stream sooner, - here's before the change is applied:

Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
converting node stream to web stream
Array buffers memory usage is 0 MiB
Array buffers memory usage is 109 MiB
Array buffers memory usage is 206 MiB
Array buffers memory usage is 336 MiB
Over 256 MiB taken, exiting

Here's after the change is applied:

Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
converting node stream to web stream
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB
Array buffers memory usage is 0 MiB

For completeness, I'm attaching @Dzieni's benchmark with the slight modification that I mentioned along with this message.

// since it's ESM, save it as .mjs

import fs from 'node:fs'
import process from 'node:process'
import {Readable} from 'node:stream'

// we initialize a stream, but not start consuming it
const randomNodeStream = fs.createReadStream('/dev/urandom')
// in a few seconds, it'll get converted to web stream
let randomWebStream

// we check memory usage every second
// since it's a stream, it shouldn't be higher than the chunk size
const reportMemoryUsage = () => {
	const {arrayBuffers} = process.memoryUsage()
	console.log(
		`Array buffers memory usage is ${Math.round(
			arrayBuffers / 1024 / 1024
		)} MiB`
	)
	if (arrayBuffers > 256 * 1024 * 1024) {
		// streaming should not lead to such a memory increase
		// therefore, if it happens => bail
		console.log('Over 256 MiB taken, exiting')
		process.exit(0)
	}
}
setInterval(reportMemoryUsage, 1000)

// after 3 seconds we use Readable.toWeb
// memory usage should stay pretty much the same since it's still a stream
setTimeout(() => {
	console.log('converting node stream to web stream')
	randomWebStream = Readable.toWeb(randomNodeStream)
}, 3000)

// after 30 seconds we start consuming the stream
// memory usage will grow, but the old chunks should be garbage-collected pretty quickly
setTimeout(async () => {
	console.log('reading the chunks')
	for await (const chunk of randomWebStream) {
		// do nothing, just let the stream flow
	}
}, 30000)

[Edit, Feb 14]: updated the draft solution link to point to a specific commit.

[mcollina]

@jasnell @KhafraDev could you take a look?