Skip to content

teardown: the workspaces NFS server stops before the node's mounts are released, leaving the kind node unkillable #595

Closed

Description

@teemow

Problem

The lab's workspaces NFS server is a ClusterIP service inside the kind cluster. A teardown (down, recreate, restore) that stops the server before the node's mounts of it are released leaves the node's NFS client in an unkillable wait: docker exec into the node fails with setns errors and the node cannot be stopped or removed without a host-level lazy unmount. Observed twice on one lab host in one evening.

#592 covers only detecting such a broken lab and failing fast; this issue is the cause.

Proposed solution

Teardown releases the workspace volumes first (drain or delete the pods that mount them, remove the PersistentVolumeClaims and PersistentVolumes, or unmount inside the node) and only then stops the NFS server and the cluster; the order is the same on every path that removes the server (down, recreate, restore).

Acceptance criteria

  • A teardown of a lab with workspaces mounted by running pods leaves no node in an unkillable state.
  • A test or proof covers the teardown with a mounted workspace volume.

Written by an agent.

Activity

  1. teemow commented on Oct 12, 2026

    @teemow
    MemberAuthor

    Fixed by #599, released in v0.97.1-rc.1. Live proof on a lab host: agentlab down with a workspace volume mounted by a running pod ended with exit 0 in 42 s, the node gone and docker ps -a clean (#599 (comment)). Written by an agent.

  2. moved this from Validation ☑️ to Done ✅ in Roadmapon Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      Done ✅

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions