attempting to use a readstream to read a specific size of bytes no longer works properly with Node 10

[BrianFreedman]

Here's my OS info:
Edition: Windows 10
Version: 1903
OS Build: 18362.356

Here's the output from Node 10.16.3 x64

c:\code\temp>node -v
v10.16.3

c:\code\temp>node readBytes.js
size > bufferSize : true
null segment: true
null segment: true

Here's the output from Node 8.15.1 x64

c:\code\temp>node -v
v8.15.1

c:\code\temp>node readBytes.js
size > bufferSize : true
null segment: true
null segment: true
null segment: false
null segment: true
null segment: false
null segment: true
null segment: false
null segment: true
done! 3 chunks

here's the contents of readBytes.js which illustrates the problem

const fs = require('fs');
const filepath = 'C:/code/temp/21_MB_File';
const bufferSize = 10 * 1024 * 1024;

const chunks = [];
const stats = fs.statSync(filepath);
console.log(`size > bufferSize : ${stats.size > bufferSize}`);

const readableStream = fs.createReadStream(filepath);
let segmentIndex = 1;
readableStream.on('readable', () => {
	let segment = readableStream.read(bufferSize);
	console.log(`null segment: ${segment === null}`);
	while (segment !== null) {
		chunks.push({ index: segmentIndex++, body: segment });
		segment = readableStream.read(bufferSize);
		console.log(`null segment: ${segment === null}`);
	}
});
readableStream.on('end', () => {
	console.log(`done! ${chunks.length} chunks`);
});

[addaleax]

/cc @ronag

[ronag]

@mcollina There is a broken invariant here:

• The docs say that read(n) will always return n bytes or null.
• Readable will increase the highWaterMark to match n in order to buffer enough data to make a buffer of size n.
• However, if n > MAX_HWM it won't raise the highWaterMark and thus read(n) can never complete.

I can see only two solutions to this:

1. Break the read(n) invariant to read Math.min(n, MAX_HWM) bytes.
2. Error if n > MAX_HW.

I'm guessing 1. is the least breaking option?

The reason this specific issue fails is because the bufferSize = 10 * 1024 * 1024 > MAX_HWM.

[ronag]

Fixed in #29938