Skip to content

Revert to Node 22 - #2057

Merged
sagzy merged 1 commit into
mainfrom
revert-to-node22
Jul 28, 2026
Merged

Revert to Node 22#2057
sagzy merged 1 commit into
mainfrom
revert-to-node22

Conversation

@sagzy

@sagzy sagzy commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

ref #2013
ref #2038
ref #2035
ref nodejs/node#63574

Problem

Since the Node 22 → 24.18.0 upgrade went out on July 22, every service has been leaking memory until it crashes — external roughly every 20 minutes (~180 heap-OOM SIGSEGVs/day at peak), queue/api via slower container OOM kills — returning 503s for whatever federation requests are in flight. Raising memory limits and capping the V8 heap (#2038) reduced the crash rate ~3× but couldn't stop it.

The root cause is a Node core regression, not our code: Node ≥ 24.16.0 never releases the reader-wakeup handle registered by Blob.prototype.stream(), so any HTTP body that passes through a Blob is pinned forever by a strong GC root (nodejs/node#63574). Our request pipeline (x-forwarded-fetch → undici) sends every inbound POST body through exactly that path — one permanently leaked body per federation delivery and per Pub/Sub push, until the heap cap kills the instance. Heap-snapshot diffs of a loaded instance show the retainer chain verbatim: GC roots → Global handles → Blob → DataQueue → InMemoryEntry → BackingStore, 8,400 pinned bodies after 4,000 activities.

Solution

Revert to Node 22.23.1, which predates the regression and still receives security updates. The same application code and dependency set ran leak-free on Node 22 right up to the upgrade, and load-testing this branch's image shows flat RSS and zero retained Blobs under federation traffic.

Beyond the base image, this reverts the pieces that only exist because of Node 24:

  • drops --max-old-space-size-percentage=75 from the container CMD — the flag was added in Node 24 and would fail to boot on Node 22 (the heap-OOM it mitigated is what this revert fixes properly);
  • restores python3 g++ make during pnpm install for native builds on the Node 22 alpine image;
  • reverts .nvmrc, engines, @types/node, and the Renovate allowedVersions cap so nothing walks us back onto Node 24 automatically.

Exit criteria: the upstream fix (nodejs/node#63577) merged on July 9 but hasn't shipped in any 24.x release. Once a 24.x release contains it, the upgrade can be re-attempted — validated beforehand against the same load-test that isolated this leak. Until then, staying on 24.x in any form means either leaking (24.16–24.18) or dropping the June security patches (24.15).

@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The project is aligned with Node.js 22: .nvmrc, package.json, and Renovate constraints are updated, and @types/node is downgraded to the Node 22 line. The Dockerfile now uses a pinned Node 22 Alpine image, installs and removes native build dependencies, and starts the application without the previous heap tuning option.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly summarizes the main change: reverting the runtime back to Node 22.
Description check ✅ Passed The description is directly related to the Node 22 rollback and explains the motivation and changes.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch revert-to-node22

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
Dockerfile (1)

3-3: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Use --no-cache when installing build dependencies.

This apk add leaves package indexes in the image layer. Use apk add --no-cache ... (or combine installation, build, and cleanup in one layer) to avoid unnecessary image contents.

Proposed fix
-RUN apk add python3 g++ make
+RUN apk add --no-cache python3 g++ make
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@Dockerfile` at line 3, Update the Dockerfile’s apk installation command to
include the --no-cache option while preserving the existing python3, g++, and
make dependencies.

Source: Linters/SAST tools

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Line 1: Add an explicit non-root runtime user in the Dockerfile before CMD,
and update ownership of every path the application must write at runtime so that
user can access them. Keep the existing build flow intact and ensure CMD runs
under the new user.

---

Nitpick comments:
In `@Dockerfile`:
- Line 3: Update the Dockerfile’s apk installation command to include the
--no-cache option while preserving the existing python3, g++, and make
dependencies.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6022a213-42a1-48cd-9260-aafeb1e4c64d

📥 Commits

Reviewing files that changed from the base of the PR and between fef4d02 and 77aa321.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (4)
  • .github/renovate.json5
  • .nvmrc
  • Dockerfile
  • package.json

Comment thread Dockerfile

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 77aa321f90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread package.json
ref #2013
ref #2038
ref #2035

We are noticing higher RAM usage after the node 24 upgrade. This reverts back
to Node 22 whilst we investigate further

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Dockerfile`:
- Line 3: Update the Dockerfile’s apk add command to include the --no-cache
option when installing python3, g++, and make, ensuring the Alpine package index
is not retained.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d01e4809-0316-466f-882c-23f1ffa7a72e

📥 Commits

Reviewing files that changed from the base of the PR and between 77aa321 and 511ccec.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (4)
  • .github/renovate.json5
  • .nvmrc
  • Dockerfile
  • package.json
🚧 Files skipped from review as they are similar to previous changes (3)
  • .github/renovate.json5
  • .nvmrc
  • package.json

Comment thread Dockerfile
@sagzy sagzy changed the title Revert Node 24 upgrade Revert to Node 22 Jul 28, 2026
@sagzy
sagzy merged commit 3b7983f into main Jul 28, 2026
15 checks passed
@sagzy
sagzy deleted the revert-to-node22 branch July 28, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants