[Bug]: t3 CLI fails to exec on Oracle UEK8 kernels (ENOEXEC) — Node SEA PT_NOTE blob exceeds 4 MB kernel limit #12628

Activity

1. juliusmarminge commented on Sep 19, 2026

   

   juliusmarminge

   on Sep 19, 2026

   Member

   More actions

   Triage

   Confirmed against current main. This is a real exec failure on Oracle UEK8, not an arch/launcher bug, and not a duplicate.

   The Linux CLI is a Node single-executable. apps/server/vite.config.ts packs it with --build-sea / postject, so the JS bundle lives in a NODE_SEA_BLOB ELF note. UEK8’s load_elf_binary() parses every PT_NOTE at exec time and returns -ENOEXEC when p_filesz exceeds 4 MB (MAX_FILE_NOTE_SIZE). That matches oracle/linux-uek#46 (closed by Oracle, still present on 6.12.0-204.92.4.4.3). The reporter’s readelf / phdr bisection is consistent with that: cap or drop the NOTE header and the same aarch64 file runs.

   What we do today

   npx t3 is bin/t3.js generated from NPM_LAUNCHER_SCRIPT in scripts/build-npm-platform-packages.ts. After require.resolve finds @t3code/t3-<platform>-<arch>, it spawnSyncs the SEA and on any error prints only:

   t3: failed to start <path>: <error.message>

   packages/shared/src/legacyCliLauncher.ts (dist/bin.mjs) is the same. scripts/install.sh runs "$staging/t3" --version and fails with the downloaded executable does not run. There is no UEK / ENOEXEC hint in those paths or in docs/user/install.md.

   This is every Linux CLI distribution, not just npm: release archives, t3 update, the boot service, and SSH remotes all exec the same binary. The desktop AppImage is Electron and should be unaffected.

   A non-SEA npm package would be a product change in the opposite direction of #11510 (runtimes are archives only). Intel Macs already document the supported escape hatch: build from source and run node apps/server/dist/bin.mjs.

   Workarounds

   1. Boot Oracle RHCK (or any mainline-based kernel) instead of UEK8.
   2. From-source server, same as Intel Macs in docs/user/install.md.
   3. Locally capping the installed NOTE p_filesz to 4 MB works (Node looks up the blob via p_memsz) but must be redone after every update. We should not ship that as the official binary.

   Suggested fix

   • In bin/t3.js and the legacy launcher, if error.code === 'ENOEXEC', print that UEK8 rejects Node SEA notes larger than 4 MB, and point at RHCK / from-source. ENOEXEC can also mean a corrupt or wrong-arch file; keep the wording as the likely cause on Linux when the file exists.
   • Give scripts/install.sh the same hint when the smoke test cannot exec.
   • Add an Oracle Linux / UEK8 note next to the Intel Mac section in docs/user/install.md.

   Do not treat a second npm/Node distribution or an in-tree ELF patch as the first fix. Unstripped debug_info (~160 MB) is a separate download-size issue and does not change the ~11 MB note.

   Accepting as a bug. Root cause is upstream UEK; we can still make this diagnosable.
2. juliusmarminge

   added

   acceptedfeature request acceptedfeature request accepted

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   upstream

   via-triageFiled through npx t3 triageFiled through npx t3 triage

   on Sep 19, 2026
3. Geczy commented on Sep 27, 2026

   

   Geczy

   on Sep 27, 2026

   More actions

   Hit this on Oracle Linux 10.1 aarch64 (6.12.0-109.67.6.el10uek.aarch64) with 0.0.43-nightly.20260927.2344, so el10uek is affected too. For a server still on 0.0.41, the last Node-based npm build, the in-app update fails with Could not prepare t3@… because the staged preflight exits 126.

   A detail that might change the fix options: UEK8 only size-checks the last PT_NOTE, not every one. In load_elf_binary() the phdr loop assigns elf_notes_phdata = elf_ppnt for each PT_NOTE segment (binfmt_elf.c#L1099-L1100). Only that final one reaches get_elf_notes() and the 4 MB check (L1164, L832). A function_graph trace of the failing exec agrees. load_elf_binary returns -8 right after parse_elf_properties, before the loader maps anything.

   So a binary runs on UEK8 with the SEA note header left exactly as postject wrote it, as long as a small PT_NOTE comes after it in the program header table. I tested that on an installed binary by turning PT_GNU_RELRO (the last phdr) into a PT_NOTE over .note.gnu.build-id + .note.ABI-tag (0x44 bytes). The whole server runs this way: service launcher, __service-preflight returns ready, T3 Connect relay, remote clients. The cost is losing RELRO. Setting the SEA PT_NOTE to PT_NULL also gets past the kernel, but then Node segfaults because it can no longer find the blob.

   If a build-side fix is ever considered, appending a small trailing PT_NOTE after injection would let the stock binary run on UEK8 without capping or rewriting the SEA note.

   To keep in-app updates working on a UEK host in the meantime, point the updater at a local mirror that serves patched releases. The updater already honors T3CODE_RELEASE_BASE_URL. It fetches SHA256SUMS and the linux-arm64 tarball for the requested version from GitHub, checks the upstream checksum, patches <top>/t3 in place (the file size doesn't change), repacks, and serves the archive with a SHA256SUMS for the patched file. A t3code.service.d drop-in with Environment=T3CODE_RELEASE_BASE_URL=http://127.0.0.1:47733 reaches both the launcher and the server. I've run the mirror's output through the updater's steps by hand (checksum, tar --strip-components=1, --version, preflight). A real in-app update will first go through it when the next release comes out.

   Patch script (adds a small trailing PT_NOTE)

   import struct,sys
   # UEK8 rejects executables whose *last* PT_NOTE is >4 MiB (fs/binfmt_elf.c MAX_FILE_NOTE_SIZE).
   # T3 is a Node SEA whose blob lives in a ~11.7 MB PT_NOTE. Turn PT_GNU_RELRO into a small
   # trailing PT_NOTE over the build-id/ABI-tag notes so the kernel checks that one instead.
   PT_NOTE,PT_GNU_RELRO=4,0x6474e552
   p=sys.argv[1]; f=open(p,"r+b"); h=f.read(64)
   assert h[:4]==b"\x7fELF" and h[4]==2 and h[5]==1
   phoff,=struct.unpack_from("<Q",h,0x20); phentsize,phnum=struct.unpack_from("<HH",h,0x36)
   ph=[]
   for i in range(phnum):
       f.seek(phoff+i*phentsize); ph.append(list(struct.unpack("<IIQQQQQQ",f.read(56))))
   notes=[i for i,x in enumerate(ph) if x[0]==PT_NOTE]
   if notes and ph[notes[-1]][5] <= 4<<20: print("already OK: last PT_NOTE is small"); sys.exit(0)
   big=ph[notes[-1]]; relro=[i for i,x in enumerate(ph) if x[0]==PT_GNU_RELRO]
   assert relro and relro[-1] > notes[-1], "no GNU_RELRO after the big note"
   # first two notes of the big segment: .note.gnu.build-id (0x24) + .note.ABI-tag (0x20)
   f.seek(big[2]); size=0
   for _ in range(2):
       namesz,descsz,_t=struct.unpack("<III",f.read(12)); n=12+((namesz+3)&~3)+((descsz+3)&~3); size+=n; f.seek(big[2]+size)
   i=relro[-1]; new=[PT_NOTE,4,big[2],big[3],big[4],size,size,4]
   f.seek(phoff+i*phentsize); f.write(struct.pack("<IIQQQQQQ",*new)); f.close()
   print(f"phdr {i}: GNU_RELRO -> PT_NOTE off={big[2]:#x} size={size:#x}")

   Local release mirror for T3CODE_RELEASE_BASE_URL

   #!/usr/bin/env python3
   """Local T3 release mirror for Oracle Linux UEK kernels.
   
   T3's updater downloads <base>/v<version>/SHA256SUMS and <base>/v<version>/t3-<version>-linux-arm64.tar.gz.
   UEK8 refuses to exec the T3 binary (its >4 MiB PT_NOTE), so this serves the upstream archive with the
   binary patched by t3-uek-note-patch.py, plus a SHA256SUMS that matches the patched archive.
   T3 uses it via T3CODE_RELEASE_BASE_URL (see t3code.service.d/10-release-mirror.conf).
   """
   import hashlib, http.server, os, re, shutil, subprocess, tarfile, tempfile, threading, urllib.request
   
   UPSTREAM = "https://github.com/pingdotgg/t3code/releases/download"
   PLATFORM = "linux-arm64"
   PATCHER = os.path.expanduser("~/.local/libexec/t3-uek-note-patch.py")
   CACHE = os.path.expanduser("~/.cache/t3-release-mirror")
   KEEP = 3
   VERSION = re.compile(r"^\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?$")
   build_lock = threading.Lock()
   
   
   def fetch(url, dest):
       req = urllib.request.Request(url, headers={"User-Agent": "t3-release-mirror"})
       with urllib.request.urlopen(req, timeout=300) as r, open(dest, "wb") as f:
           shutil.copyfileobj(r, f, 1 << 20)
   
   
   def sha256(path):
       h = hashlib.sha256()
       with open(path, "rb") as f:
           for chunk in iter(lambda: f.read(1 << 20), b""):
               h.update(chunk)
       return h.hexdigest()
   
   
   def build(version):
       """Return the cache dir holding the patched archive and its SHA256SUMS, building it on first use."""
       name = f"t3-{version}-{PLATFORM}.tar.gz"
       out = os.path.join(CACHE, "v" + version)
       with build_lock:
           if os.path.exists(os.path.join(out, "SHA256SUMS")):
               return out
           os.makedirs(CACHE, exist_ok=True)
           with tempfile.TemporaryDirectory(dir=CACHE, prefix=".build-") as tmp:
               sums, src = os.path.join(tmp, "SHA256SUMS"), os.path.join(tmp, name)
               fetch(f"{UPSTREAM}/v{version}/SHA256SUMS", sums)
               fetch(f"{UPSTREAM}/v{version}/{name}", src)
               expected = next((l.split()[0].lower() for l in open(sums) if l.split() and l.split()[-1].lstrip("*") == name), None)
               if expected is None or expected != sha256(src):
                   raise RuntimeError(f"upstream checksum mismatch for {name}")
               dst = os.path.join(tmp, "out")
               os.mkdir(dst)
               patched, exe, found = os.path.join(dst, name), os.path.join(tmp, "t3"), False
               with tarfile.open(src, "r:gz") as tin, tarfile.open(patched, "w:gz") as tout:
                   for m in tin:
                       parts = [p for p in m.name.split("/") if p not in ("", ".")]
                       if m.isfile() and parts == [parts[0], "t3"]:  # <top>/t3, the executable
                           with open(exe, "wb") as f:
                               shutil.copyfileobj(tin.extractfile(m), f, 1 << 20)
                           subprocess.run(["python3", PATCHER, exe], check=True)
                           assert os.path.getsize(exe) == m.size  # the patch rewrites one header in place
                           with open(exe, "rb") as f:
                               tout.addfile(m, f)
                           found = True
                       else:
                           tout.addfile(m, tin.extractfile(m) if m.isfile() else None)
               if not found:
                   raise RuntimeError(f"no t3 executable in {name}")
               with open(os.path.join(dst, "SHA256SUMS"), "w") as f:
                   f.write(f"{sha256(patched)}  {name}\n")
               os.rename(dst, out)
           for old in sorted((d for d in os.listdir(CACHE) if d.startswith("v")),
                             key=lambda d: os.path.getmtime(os.path.join(CACHE, d)))[:-KEEP]:
               shutil.rmtree(os.path.join(CACHE, old), ignore_errors=True)
           return out
   
   
   class Handler(http.server.BaseHTTPRequestHandler):
       def do_GET(self):
           m = re.fullmatch(r"/v([^/]+)/([^/]+)", self.path)
           if not m or not VERSION.match(m[1]) or m[2] not in ("SHA256SUMS", f"t3-{m[1]}-{PLATFORM}.tar.gz"):
               return self.send_error(404)
           try:
               path = os.path.join(build(m[1]), m[2])
           except Exception as e:
               self.log_error("building %s failed: %s", m[1], e)
               return self.send_error(502, str(e))
           self.send_response(200)
           self.send_header("Content-Type", "application/octet-stream")
           self.send_header("Content-Length", str(os.path.getsize(path)))
           self.end_headers()
           with open(path, "rb") as f:
               shutil.copyfileobj(f, self.wfile, 1 << 20)
   
   
   if __name__ == "__main__":
       http.server.ThreadingHTTPServer(("127.0.0.1", 47733), Handler).serve_forever()

   👍React with 👍2Reacted by merowinger92 and 4shain