Verification environment

Reproducible checks for every measurable claim in the article "Processing a 2GB CSV in Node Without Running Out of Memory."

Nothing here is mocked. It generates a real CSV, runs both the naive load-everything approach and the generator pipeline in separate processes, measures peak memory, and asserts the totals are correct against an independently computed expected sum.

Requirements

• Node.js 18 or newer (tested on Node 22).
• Core demo (generate.js, naive.js, pipeline.js, verify.js) has no dependencies.
• Optional: npm install adds csv-parse for a third streaming approach (pipeline-csv-parse.js).

Quick start

# Optional — npm streaming comparison (csv-parse)
npm install

# 1. Run the full check at the article's size (2,000,000 rows / ~45 MB)
node verify.js
# or: npm run verify

# 2. Prove the headline claim: naive dies, pipeline survives the same heap cap
./stress.sh
# or: npm run stress

That's it. verify.js exits 0 if all checks pass. stress.sh exits 0 if the naive approach crashes while the pipeline succeeds and returns the correct total.

What each claim maps to

Claim in the article	How it's verified	File
Both approaches produce the same total	Both totals asserted equal to a closed-form expected sum computed separately	verify.js
Naive holds ~5x the file size in RAM	Peak RSS sampled during processing in an isolated process	naive.js
Pipeline stays around 89-90 MB	Peak RSS sampled during processing in an isolated process	pipeline.js
Pipeline memory stays flat as the file grows	Pipeline RSS at ~45 MB and ~390 MB file sizes compared within 50 MB	verify.js
"You can process a file bigger than your RAM"	Pipeline succeeds under a heap cap smaller than the file; naive crashes (OOM or string limit)	./stress.sh

Why separate processes

If you ran the naive and pipeline approaches in one process, the naive run's allocations would still be sitting in memory (or mid-collection) when the pipeline ran, polluting its RSS reading. verify.js spawns each as its own node process so each peak-memory number reflects only that approach.

Why RSS and not heapUsed

RSS (Resident Set Size) is the total physical RAM the process holds: the V8 heap plus the file read buffers, which live in C++ "external" memory, not the JS heap. Reporting only heapUsed would hide the read-buffer cost and flatter the streaming approach unfairly. RSS is the honest "what does this cost the box" number, and it's the one that determines whether you OOM.

Peak RSS is sampled every 10 ms during async work, with extra samples after synchronous allocations (the naive read-and-split path). See peak-rss.js.

Why naive fails differently at different file sizes

The naive approach uses readFileSync + split('\n'), which requires the entire file in one JavaScript string. Node caps any single string at ~512 MB (0x1fffffe8 bytes). Above that you get ERR_STRING_TOO_LONG before heap limits even matter.

File size	Typical naive failure
~390 MB (default stress test)	Heap OOM under a 128 MB cap
~2 GB (./stress.sh 91000000 128)	String size limit (~512 MB max)

Both are fatal to the naive approach. The pipeline handles either size under the same heap cap. naive.js checks file size up front and prints a clear message instead of a stack trace.

Can npm fix naive? No. Any approach that loads the whole file into one JavaScript string hits the same ~512 MB V8 limit — including wrappers like neat-csv. The npm fix is streaming, not a bigger string. After npm install, pipeline-csv-parse.js uses csv-parse to stream row-by-row (same idea as the hand-rolled generator pipeline, production-grade parser).

npm install
node pipeline-csv-parse.js data/big.csv   # works on 2 GB+ files
npm run stress:2gb                        # naive crashes, pipeline + csv-parse succeed

Tuning the stress test

The default stress test uses a 128 MB heap cap against a ~390 MB file. If your machine happens to give Node enough room that the naive version doesn't crash, lower the cap or raise the row count:

./stress.sh 16000000 96     # smaller heap cap
./stress.sh 30000000 128    # bigger file (~730 MB)
./stress.sh 91000000 128    # ~2 GB file (article headline size)

verify.js accepts an optional second argument for the large-file flat-memory check (default 16,000,000 rows / ~390 MB):

node verify.js 2000000 16000000

Expected output (representative)

naive       total=   999000000  peakRSS= 246 MB
pipeline    total=   999000000  peakRSS=  90 MB
...
[PASS]  naive total == expected sum
[PASS]  pipeline total == expected sum
[PASS]  naive total == pipeline total
[PASS]  pipeline peak RSS < naive peak RSS
        90 MB vs 246 MB (2.7x less)
[PASS]  pipeline peak RSS flat within 50 MB (45.35 MB → 390 MB file)
ALL CHECKS PASSED

RSS values wobble a few MB run to run (GC timing, OS), which is normal. The ratio and the pass/fail outcomes are stable.

Project layout

File	Role
generate.js	Writes a deterministic CSV
naive.js	Loads the entire file into memory
pipeline.js	Streaming generator pipeline
pipeline-csv-parse.js	Streaming pipeline via csv-parse (requires npm install)
verify.js	Orchestrates checks and assertions
stress.sh	Naive crash vs pipeline survival under a heap cap
peak-rss.js	Peak RSS sampling helper
expected-sum.js	Closed-form expected total

Cleanup

Generated CSVs land in data/ and can get large. Delete them anytime:

rm -rf data/

Get the Free Ebook

Get it here