Skip to content

Large string corruption under server load: chunked slice/append writes a 4MB chunk-size word into the buffer head #5062

Description

@proggeramlug

Symptom

On the production perry-hub (a long-running Fastify server compiled with perry 0.5.1159, auto-optimize), large base64 artifact strings written to disk get their first 12 bytes overwritten with 00 00 40 00 00 00 00 00 00 00 00 00 — i.e. the little-endian word 0x00400000 (= 4194304 = the 4 MB write-chunk size) followed by eight zero bytes — clobbering the first 12 characters of the string. Everything after byte 12 is intact.

The writer is a chunked append (a single fs.writeFileSync of a >16 MB string was itself unreliable, hence chunking):

const B64_WRITE_CHUNK = 4 * 1024 * 1024; // 0x00400000
function writeB64File(path: string, data: string): void {
  fs.writeFileSync(path, '');
  for (let i = 0; i < data.length; i += B64_WRITE_CHUNK) {
    fs.appendFileSync(path, data.slice(i, i + B64_WRITE_CHUNK));
  }
}

The value written into the corrupted head is exactly the chunk constant 0x00400000, which strongly suggests the slice length/offset metadata is leaking into the string's backing buffer when the large source string is relocated mid-loop.

Evidence

  • Confirmed on the live hub's stored file: head -c 24 of the on-disk .b64 is 00 00 40 00 00 00 00 00 00 00 00 00 66 30 56 4d 52 67 ... (f0VMRg = the real base64, starting at offset 12). So the corruption is at write time, baked into the file — not a read-back artifact.
  • The source data string originates from a large HTTP request body (request.rawBody, ~32 MB) then .trim().

NOT reproducible on an unloaded process

I rebuilt the exact upload→write path on a worker (same perry 0.5.1159, auto-optimize, same 32 MB body delivered over HTTP, same writeB64File) and it produced a clean file every time — both chunked and one-shot writes, sha-correct. It only manifests on the live server, which runs under sustained memory/GC pressure (known: the hub leaks ~1.4 GB/h and auto-GC has collected live objects before — see the GC unsafe-zone history). That points at GC relocation of the large string during the slice/append loop as the trigger.

Hypothesis

When a multi-MB string is sliced in a loop and the GC relocates/compacts the backing store mid-iteration, the first slice's destination buffer gets a header/metadata word (the chunk length 0x00400000) written over the payload. Likely the same family as the documented fs.readFileSync Buffer 4-byte offset bug and the auto-optimize request-property-loss bug (#5037) — string/buffer representation not surviving a boundary (here: GC, not a function call).

Workaround in the affected app

perry-hub now stores a raw Buffer copy (single fs.writeFileSync(path, buffer) — no string slicing) and serves that, sha-gated. Buffer writes don't exhibit this. So the bug is specific to large string slice/append under load.

Repro materials

Not isolated-reproducible (load-dependent), but the live evidence + the negative isolated repro are both above. Happy to add instrumentation to the hub to capture more if useful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions