Symptom
On the production perry-hub (a long-running Fastify server compiled with perry 0.5.1159, auto-optimize), large base64 artifact strings written to disk get their first 12 bytes overwritten with 00 00 40 00 00 00 00 00 00 00 00 00 — i.e. the little-endian word 0x00400000 (= 4194304 = the 4 MB write-chunk size) followed by eight zero bytes — clobbering the first 12 characters of the string. Everything after byte 12 is intact.
The writer is a chunked append (a single fs.writeFileSync of a >16 MB string was itself unreliable, hence chunking):
const B64_WRITE_CHUNK = 4 * 1024 * 1024; // 0x00400000
function writeB64File(path: string, data: string): void {
fs.writeFileSync(path, '');
for (let i = 0; i < data.length; i += B64_WRITE_CHUNK) {
fs.appendFileSync(path, data.slice(i, i + B64_WRITE_CHUNK));
}
}
The value written into the corrupted head is exactly the chunk constant 0x00400000, which strongly suggests the slice length/offset metadata is leaking into the string's backing buffer when the large source string is relocated mid-loop.
Evidence
- Confirmed on the live hub's stored file:
head -c 24 of the on-disk .b64 is 00 00 40 00 00 00 00 00 00 00 00 00 66 30 56 4d 52 67 ... (f0VMRg = the real base64, starting at offset 12). So the corruption is at write time, baked into the file — not a read-back artifact.
- The source
data string originates from a large HTTP request body (request.rawBody, ~32 MB) then .trim().
NOT reproducible on an unloaded process
I rebuilt the exact upload→write path on a worker (same perry 0.5.1159, auto-optimize, same 32 MB body delivered over HTTP, same writeB64File) and it produced a clean file every time — both chunked and one-shot writes, sha-correct. It only manifests on the live server, which runs under sustained memory/GC pressure (known: the hub leaks ~1.4 GB/h and auto-GC has collected live objects before — see the GC unsafe-zone history). That points at GC relocation of the large string during the slice/append loop as the trigger.
Hypothesis
When a multi-MB string is sliced in a loop and the GC relocates/compacts the backing store mid-iteration, the first slice's destination buffer gets a header/metadata word (the chunk length 0x00400000) written over the payload. Likely the same family as the documented fs.readFileSync Buffer 4-byte offset bug and the auto-optimize request-property-loss bug (#5037) — string/buffer representation not surviving a boundary (here: GC, not a function call).
Workaround in the affected app
perry-hub now stores a raw Buffer copy (single fs.writeFileSync(path, buffer) — no string slicing) and serves that, sha-gated. Buffer writes don't exhibit this. So the bug is specific to large string slice/append under load.
Repro materials
Not isolated-reproducible (load-dependent), but the live evidence + the negative isolated repro are both above. Happy to add instrumentation to the hub to capture more if useful.
Symptom
On the production perry-hub (a long-running Fastify server compiled with perry 0.5.1159, auto-optimize), large base64 artifact strings written to disk get their first 12 bytes overwritten with
00 00 40 00 00 00 00 00 00 00 00 00— i.e. the little-endian word0x00400000(= 4194304 = the 4 MB write-chunk size) followed by eight zero bytes — clobbering the first 12 characters of the string. Everything after byte 12 is intact.The writer is a chunked append (a single
fs.writeFileSyncof a >16 MB string was itself unreliable, hence chunking):The value written into the corrupted head is exactly the chunk constant
0x00400000, which strongly suggests the slice length/offset metadata is leaking into the string's backing buffer when the large source string is relocated mid-loop.Evidence
head -c 24of the on-disk.b64is00 00 40 00 00 00 00 00 00 00 00 00 66 30 56 4d 52 67 ...(f0VMRg= the real base64, starting at offset 12). So the corruption is at write time, baked into the file — not a read-back artifact.datastring originates from a large HTTP request body (request.rawBody, ~32 MB) then.trim().NOT reproducible on an unloaded process
I rebuilt the exact upload→write path on a worker (same perry 0.5.1159, auto-optimize, same 32 MB body delivered over HTTP, same
writeB64File) and it produced a clean file every time — both chunked and one-shot writes, sha-correct. It only manifests on the live server, which runs under sustained memory/GC pressure (known: the hub leaks ~1.4 GB/h and auto-GC has collected live objects before — see the GC unsafe-zone history). That points at GC relocation of the large string during the slice/append loop as the trigger.Hypothesis
When a multi-MB string is sliced in a loop and the GC relocates/compacts the backing store mid-iteration, the first slice's destination buffer gets a header/metadata word (the chunk length
0x00400000) written over the payload. Likely the same family as the documentedfs.readFileSyncBuffer 4-byte offset bug and the auto-optimize request-property-loss bug (#5037) — string/buffer representation not surviving a boundary (here: GC, not a function call).Workaround in the affected app
perry-hub now stores a raw
Buffercopy (singlefs.writeFileSync(path, buffer)— no string slicing) and serves that, sha-gated. Buffer writes don't exhibit this. So the bug is specific to large string slice/append under load.Repro materials
Not isolated-reproducible (load-dependent), but the live evidence + the negative isolated repro are both above. Happy to add instrumentation to the hub to capture more if useful.