Repository navigation
Buffer.from() output breaks after optimizing in nodejs 22.7 #54521
Description
Activity
I managed to reproduce using your code snippet, and the slightly modified one below (for readability)
let i = 0; while (i < 1_000_000) { const asHex = Buffer.from("\x80").toString("hex"); if (asHex === '80') { break; } else if (asHex !== 'c280') { console.log("Unexpected return value:", asHex); } i++; } if (i < 1_000_000) { console.log("FAILED after %d iterations", i); process.exit(1); } else { console.log("PASSED after %d iterations", i); }
$ node repro.js FAILED after 8827 iterations $ node repro.js FAILED after 5509 iterations $ node repro.js FAILED after 48741 iterations $ node repro.js FAILED after 7487 iterations $ node repro.js FAILED after 9189 iterations
bisecting gives me, as first bad commit:
commit c9dabe2
Author: Robert Nagy ronagy@icloud.com
Date: Thu Aug 15 03:01:05 2024 +0200buffer: use fast API for writing one-byte stringshowever, that one's even worse and shows a lot of incorrect 'asHex' values:
Unexpected return value afdead0b Unexpected return value 00dead0b Unexpected return value 00000000 Unexpected return value 00dead0b Unexpected return value 01dead0b Unexpected return value 00000000 Unexpected return value 905e0150 Unexpected return value 00738148so it looks like the more extreme bug got fixed (probably in 7800893) but then issues still remained for (eg) \x80
CC @ronag
I'll have a look.
Reacted by Aviv Keller and Shubhanshu Sonilet i = 0; while (i < 1_000_000) { const buf = Buffer.from("\x80") if (buf[0] !== 194 || buf[1] !== 128) { console.log("Unexpected return value:", buf, buf[0], buf[1]); break } i++; } if (i < 1_000_000) { console.log("FAILED after %d iterations", i); process.exit(1); } else { console.log("PASSED after %d iterations", i); }
This seems to be something in V8.
In
uint32_t FastWriteString(Local<Value> receiver, const v8::FastApiTypedArray<uint8_t>& dst, const v8::FastOneByteString& src, uint32_t offset, uint32_t max_length) { }
src.length === 1i.e.FastOneByteStringthinks thatBuffer.from(['\x80'])has a length of 1.@targos @joyeecheung @nodejs/buffer
Which is kind of correct... is it the slow path that is broken?
Is the problem here that we are not handling incomplete utf8 sequences?
I'm not sure where the
c2from the expectedc280is coming from?29 remaining items
- added a commit that references this issue
on Sep 4, 2024 - added a commit that references this issue
on Sep 12, 2024
Version
v22.7.0
Platform
Subsystem
Buffer
What steps will reproduce the bug?
How often does it reproduce? Is there a required condition?
For me, it will consistently fail somewhere between 7000 to 1200 iterations.
What is the expected behavior? Why is that the expected behavior?
In node 20 this code does not fail , even after 1_000_000 iterations.
Buffer.from("\x80").toString("hex")always returnsc280on node 20What do you see instead?
Buffer.from("\x80").toString("hex")incorrectly returns80after sufficient iterationsAdditional information
No response