Skip to content

Have ephemeral JIT runners started enforcing an idle minutes limit? #4726

Description

@enescakir

Recently, we've been observing an increased number of ephemeral JIT runners being removed from the server after listening for jobs for 2 to 6 minutes, mostly close to 6 minutes.

Has GitHub started enforcing an idle minutes limit for ephemeral JIT runners? If so, what is the limit, and is there any documentation about it?

Example 1

Sep 16 15:14:45: √ Connected to GitHub
Sep 16 15:14:46: Current runner version: '2.337.0'
Sep 16 15:14:46: 2026-09-16 15:14:46Z: Listening for Jobs
Sep 16 15:16:47: The runner no longer exists on the server. Cleaning up local configuration.
Sep 16 15:16:47: √ Removed .credentials
Sep 16 15:16:47: √ Removed .runner
Sep 16 15:16:47: Runner listener exit with 0 return code, stop the service, no retry needed.
Sep 16 15:16:47: Exiting runner...

Example 2

Sep 16 17:26:15: √ Connected to GitHub
Sep 16 17:26:16: Current runner version: '2.337.0'
Sep 16 17:26:16: 2026-09-16 17:26:16Z: Listening for Jobs
Sep 16 17:32:17: The runner no longer exists on the server. Cleaning up local configuration.
Sep 16 17:32:17: √ Removed .credentials
Sep 16 17:32:17: √ Removed .runner
Sep 16 17:32:17: Runner listener exit with 0 return code, stop the service, no retry needed.
Sep 16 17:32:17: Exiting runner...

Activity

  1. luketomlinson commented on Sep 16, 2026

    @luketomlinson
    Contributor

    @enescakir do you have a sessionId for your runner we could look at?

  2. enescakir commented on Sep 16, 2026

    @enescakir
    ContributorAuthor

    @enescakir do you have a sessionId for your runner we could look at?

    @luketomlinson How can I find the sessionId? I couldn't find it in the _diag logs.

    It's also hard to access _diag logs for previous runners since it doesn't happen all the time. The occurrence increased after Sep 13-14. Is there anything else I can share to help identify it for older runners other than sessionId?

  3. luketomlinson commented on Sep 16, 2026

    @luketomlinson
    Contributor

    @enescakir it'd be in the _diag logs. It'll be a query param to the messages endpoint ?sessionId=<your_id>

    If you don't have a session id, you can also let us know the org/repo/enterprise where the runner is registered, along with the integer runner id (available in the UI/API).

    The .runner file on disk also has the identifying info.

    If you don't want to drop it in this public issue you can email it to my handle @github.com

  4. joergstreeck commented on Oct 9, 2026

    @joergstreeck

    We hit the same pattern on 2026-10-09, but as a burst: from 20:43 to ~21:10 UTC almost no ephemeral JIT runner on our repository received a job.

    Setup: repository-level JIT runners via RunsOn v3.2.2 (one EC2 instance per job), runner version 2.337.0.

    What we saw:

    • The instance starts, the runner registers via JIT config, logs Connected to GitHub / Listening for Jobs.
    • About 1–2 minutes later: The runner no longer exists on the server. The runner exits, RunsOn launches a new one, and the same thing repeats.
    • After several attempts, GitHub cancels the job: The job was not acquired by Runner of type self-hosted even after multiple attempts.
    • Over that window, 318 jobs were queued and only 67 started.
    • GitHub-hosted jobs in the same workflow runs started normally.
    • Recovery happened on its own at ~21:10 UTC, with no change on our side.

    Occurrences of "assigned runner disappeared before the job was claimed", per 10 minutes (UTC):

    Window Count
    20:40 74
    20:50 112
    21:00 37
    21:10 4
    after 21:20 0

    In the 6 hours before 20:43 there were none.

    Identifiers (repository-level runners):

    • Repository: joergstreeck/freshplan-sales-tool (repo id 991565390)
    • Example: run 37988621368, job 114020298411. Runner names runs-on--i-0be7aa99ca0fb1c26--… (JIT config generated 20:50:40 UTC, removed by ~20:52) and runs-on--i-0c9331092f8eaf218--…
    • Example: run 37989246642, same symptom.

    Unlike the reports above, this was not an occasional idle timeout. It was a window in which jobs were not dispatched to freshly registered JIT runners at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions