Skip to content

[Bug]: Service unit stays pinned to the runtime that installed it, so deleting old runtimes breaks the next restart #17490

Description

@SvejoDev

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

  1. On Linux, run t3 service install with version A. The systemd unit gets ExecStart=~/.t3/runtime/versions/A/t3 __service-launcher.
  2. Update the server several times with Update server from a client (A → B → C → D). Each update hands off to the running launcher. service-state.json ends up with activeVersion: D and update.fromVersion: C.
  3. Delete old runtime directories, keeping only those service-state.json refers to (active plus fromVersion), the retention rule proposed in [Bug]: Remote server updates accumulate every immutable runtime and consume disk space #8345 and the workaround people describe there. Version A is deleted.
  4. Wait for the service process to exit. For us, an OOM kill. systemctl --user restart t3code.service does the same.

Expected behavior

The service restarts on the active runtime. The unit should not depend on a runtime directory that nothing in service-state.json records.

Actual behavior

systemd cannot run the unit's ExecStart binary, so every restart fails with 203/EXEC until the start limit is hit. The server stays down until someone edits the unit or runs t3 service install with the active version.

The unit still pointed at 0.0.43-nightly.20260917.1851 three weeks and several remote updates later. The launcher had been running from that directory the whole time, so nothing showed the problem until the process restarted.

Cause on main @ ec80933:

  • bootService.ts#L607-L630: the unit's program is pinnedRuntimePaths(..., input.cliVersion), the runtime of the CLI that ran install.
  • selfUpdate.ts#L313-L332: a remote update calls launcher.requestUpdate and never re-renders the unit. Only t3 update from the CLI does (cli/update.ts, target.install(...)).
  • service-state.json records activeVersion and update.{fromVersion,targetVersion} but not the version the unit executes, so any cleanup based on that file, manual or automatic, can delete it.

Possible fixes, for maintainers to choose: re-render the unit (without restarting) after a remote update commits, or record the unit's runtime version in service-state.json so retention keeps it. Either should be settled before #8345's pruning ships, since its proposed rule would delete this runtime.

Impact

Major degradation or frequent failure

Version or commit

0.0.46-nightly.20261008.2819 (active); unit pinned to 0.0.43-nightly.20260917.1851. Code references are to main @ ec80933.

Environment

Ubuntu 24.04.4 LTS, x86_64, kernel 6.8.0, systemd 255, systemd user service, self-contained runtime. Connected from the macOS desktop app over Tailscale Serve.

Logs or stack traces

$ systemctl --user cat t3code.service | grep ExecStart
ExecStart=~/.t3/runtime/versions/0.0.43-nightly.20260917.1851/t3 __service-launcher

$ ls ~/.t3/runtime/versions
0.0.46-nightly.20261005.2676  0.0.46-nightly.20261005.2702  0.0.46-nightly.20261007.2787  0.0.46-nightly.20261008.2819

$ cat ~/.t3/runtime/service-state.json
{ "protocol": 3, "activeVersion": "0.0.46-nightly.20261008.2819",
  "update": { "fromVersion": "0.0.46-nightly.20261005.2702", "targetVersion": "0.0.46-nightly.20261008.2819", "status": "committed" } }

$ journalctl --user -u t3code.service
14:41:39 t3code.service: Failed with result 'oom-kill'.
14:41:44 Started t3code.service - T3 Code server.
14:41:44 t3code.service: Main process exited, code=exited, status=203/EXEC
...
14:42:10 t3code.service: Start request repeated too quickly.
14:42:10 Failed to start t3code.service - T3 Code server.

Workaround

Point ExecStart at the active runtime and restart:

sed -i 's#versions/0.0.43-nightly.20260917.1851/t3#versions/0.0.46-nightly.20261008.2819/t3#' ~/.config/systemd/user/t3code.service
systemctl --user daemon-reload && systemctl --user reset-failed t3code.service && systemctl --user start t3code.service

Running t3 service install with the active runtime's CLI should also re-render the unit. Before deleting old runtimes, check which one the unit's ExecStart uses.

Related: #8345 (runtime accumulation and the retention rule in step 3); #14609 (remote self-update doesn't update the CLI symlink; same pattern of the remote path skipping what t3 update does, but a different artifact).

Investigated and filed by Claude Code (claude-opus-5-5).

Activity

  1. spiky02plateau commented on Oct 11, 2026

    @spiky02plateau
    Contributor

    Same on macOS with the launchd service.

    The LaunchAgent ProgramArguments still points at the runtime that ran t3 service install on Sep 25, after several remote updates since:

    $ /usr/libexec/PlistBuddy -c "Print :ProgramArguments:0" ~/Library/LaunchAgents/com.t3tools.t3code.service.plist
    ~/.t3/runtime/versions/0.0.43-nightly.20260925.2251/t3
    
    $ jq '{activeVersion, update: {fromVersion: .update.fromVersion, status: .update.status}}' ~/.t3/runtime/service-state.json
    { "activeVersion": "0.0.46-nightly.20261011.2967",
      "update": { "fromVersion": "0.0.46-nightly.20261008.2813", "status": "committed" } }

    The launcher process runs from 0.0.43-nightly.20260925.2251, the server from 0.0.46-nightly.20261011.2967.
    ~/.t3/runtime/versions holds 20 runtimes, 6.8 GiB.
    Pruning by the #8345 rule (active plus fromVersion) would delete the runtime the LaunchAgent executes, so the next launchctl kickstart or reboot would fail the same way.

    Version: 0.0.46-nightly.20261011.2967, macOS 26.5.2, Apple Silicon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions