nfsd: validate open state and persist client lifetimes - #146
Draft
stevephodgson wants to merge 4 commits into
Draft
nfsd: validate open state and persist client lifetimes#146stevephodgson wants to merge 4 commits into
stevephodgson wants to merge 4 commits into
Conversation
pynfs showed that CLOSE and OPEN_CONFIRM accepted bad owner seqids and invalid stateid generations. READ accepted stale and old stateids, while SETATTR(SIZE) resized through a read-only open. Track open owners and stateid generations in each nfsd process. Require confirmation and validate READ, WRITE, CLOSE, OPEN_CONFIRM, and SETATTR(SIZE), while preserving OPEN/CLOSE replay and the protocol-defined anonymous stateids. Recover CLOSE from the local staging sidecar after an nfsd restart, and discard staging if a racing OPEN loses owner-seqid validation. This addresses CLOSE3/5/6, OPCF1/4/5/6, RD10/11, and SATT4. Share reservations remain unsupported. They do not match TernFS last-closer-wins semantics and could not be enforced consistently by process-local state.
pynfs requires new clientids for replaced registrations and verifier changes, while callback updates must retain the confirmed clientid. OPEN and RENEW must reject unconfirmed or stale clients. Active state must also remain owned by its original RPC principal across the nfsd fleet. Store each client identity under /.nfs/clients and represent registrations as distinct incarnation directories whose inode IDs are the clientids returned on the wire. Atomic confirmed and pending pointers select the live incarnation. Immutable records retain confirmation verifiers, normalized RPC principals, and callback addresses. Track active state with per-stateid markers and per-nfsd lease slots. OPEN and RENEW refresh the lease; READ, WRITE, CLOSE, OPEN_CONFIRM, and SETATTR(SIZE) require the incarnation and state marker to remain live. Confirming a reboot switches the confirmed pointer before retryable open-state cleanup, so every nfsd rejects the old incarnation immediately. This addresses CID1/1b/2a/4/4a-e/6, CIDCF1/3, and RENEW1/2.
Repeated SETCLIENTID calls leave superseded incarnation directories behind. Removing them inline is unsafe: one nfsd may replace the pending pointer while another is still confirming the previous incarnation, and cleanup failures would make an otherwise successful registration difficult to replay. After SETCLIENTID and SETCLIENTID_CONFIRM, queue bounded collection for that identity. Protect in-flight confirmation with per-nfsd claims, retain confirmed, pending, and reboot roots, and age the observation that an incarnation is unreachable for one lease period. Recheck roots and claims before soft-unlinking the directory. Collection is opportunistic and asynchronous. Failures are logged and retried by a later registration, so client operations do not depend on cleanup succeeding.
Describe local write staging, process-local open state, and the persistent client incarnation and lease hierarchy shared across the nfsd fleet. Cover registration, restart recovery, asynchronous collection, and intentionally unsupported NFS state.
stevephodgson
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
pynfs exposed several related problems in nfsd's handling of NFSv4 open state and client lifetimes.
nfsd did not fully track open-owner sequence IDs or stateid generations. As a result, operations such as CLOSE, OPEN_CONFIRM and READ could accept invalid or obsolete state. Client registration and lease state also needed to be shared across the nfsd fleet so that a reboot confirmed by one server invalidates the old client state everywhere.
This PR adds that state model while retaining TernFS's existing write semantics: new files are staged locally until CLOSE and become immutable once published.
Open state
Each nfsd now tracks open owners, stateid generations, access modes and replay state locally.
READ, WRITE, CLOSE, OPEN_CONFIRM and size-changing SETATTR operations validate that state before acting. This includes detecting bad owner sequence IDs, old or stale stateids, and attempts to resize through a read-only open.
OPEN and CLOSE replay remain supported. The protocol-defined anonymous stateids are also preserved where NFS permits them. A staged write can still complete CLOSE after an nfsd restart using its local sidecar state.
Persistent client state
Client registration, open ownership and leases are stored in TernFS so that every nfsd serving the namespace sees the same client lifetime.
Each client identity has separate incarnation directories (an incarnation is one lifetime of an nfs client between reboots). Their inode IDs are the NFS clientids returned to clients. Confirmed and pending pointers select the active registration, while open markers and per-nfsd lease records identify live state.
This provides the required behavior for callback updates, client reboots, unconfirmed registrations, conflicting RPC principals and RENEW. Reboot confirmation publishes the new incarnation before removing old state, so the old clientid is rejected across the fleet before cleanup begins.
Superseded incarnation directories are collected asynchronously after registration. Confirmation claims protect concurrent confirmation, and collection failures do not affect the client operation.
The local staging model, the division between process-local and persistent state, and the complete
/.nfs/clientshierarchy are documented in docs/nfsd.md.pynfs coverage
The open-state changes address:
CLOSE3/5/6: bad owner sequence IDs and old or stale stateids.OPCF1/4/5/6: repeated confirmation, bad sequence IDs, and bad or stale stateids.RD10/11: stale and old stateids passed to READ.SATT4: resizing a file through a read-only open.The persistent client-state changes address:
CID1/1b: client reboot invalidates old state, while a callback update preserves the confirmed clientid and its opens.CID2a: another RPC principal cannot claim an identity with live state.CID4/4a-e: confirmed, pending and replaced registrations receive the correct clientids and confirmation verifiers.CID6: an unconfirmed clientid cannot be used by OPEN.CIDCF1/3: unknown and superseded confirmations are rejected.RENEW1/2: confirmed clients can renew their lease and stale clientids are rejected.Deliberate limits
Share-deny reservations remain unsupported. TernFS does not store them, and process-local reservations would not be consistent across the nfsd fleet. Byte-range locking, delegations and grace-period reclaim also remain outside the supported state model.
Testing
make -C go/nfsd test-pynfs-clusterpasses after applying this PR (in series).Attestation
This PR was mostly written by AI agents. I used a combination of Sol, Opus and Fable in an adversarial loop, then rewrote the history at the end into a reviewable state. I was involved in the design and implementation throughout