Skip to content

Implement the idempotent-claims and host API ADRs #105

Description

@phoban01

Tracking issue for implementing the two ADRs merged in #100 and #101:

Order

Idempotent claims goes first. It is the smaller change, and it introduces the store's migration mechanism (PRAGMA user_version plus numbered migrations). Both ADRs say whichever lands second reuses that mechanism, so the host API then adds its TLS columns as migration 2.

The work is split into six phases. Each phase is one commit that builds, passes go test -race ./..., golangci-lint run, and buf lint, and leaves main usable. Phases 1–2 ship as one PR (closes #98) and phases 3–6 as a second PR (closes #97).

Part 1: Idempotent claims (#98)

Phase 1: Store migrations and request_id persistence

  • store.Open reads PRAGMA user_version and applies a numbered list of migrations in order, each in a transaction. schema.sql remains the version-0 baseline.
  • Migration 1: ALTER TABLE leases ADD COLUMN request_id TEXT and a partial UNIQUE index on request_id WHERE request_id IS NOT NULL.
  • Proto: ClaimVMRequest.request_id = 2 and LeaseRecord.request_id = 8, then regenerate.
  • Store: CreateLease writes request_id (empty string stored as NULL) and returns ErrDuplicateRequestID on a unique-index violation. Add GetLeaseByRequestID.
  • Tests: fresh DB, a pre-existing version-0 DB upgraded in place, reopening a migrated DB is a no-op, duplicate detection, and lookup.
  • poolmgrctl lease claim --request-id, and a request ID in lease output (AGENTS.md).

Phase 2: ClaimVM replay semantics

  • A request_id over 255 bytes returns INVALID_ARGUMENT.
  • Look up by request_id before claiming. If the pool matches, rebuild the response from the lease and VM, fetch interfaces, and leave expiry alone. If the pool differs, return INVALID_ARGUMENT. If there is no match, claim as today and write request_id.
  • Concurrent duplicate: on ErrDuplicateRequestID, set the loser's VM back to AVAILABLE and return the winner's lease.
  • Add a replayed="true|false" label to poolmgr_vm_claims_total.
  • Tests for each semantic rule in the ADR (1–6), including the race.
  • Set the ADR status to Accepted and document retries in the README or design doc.

Part 2: Host API (#97)

Phase 3: Host TLS in the store and protos

  • Proto: HostTLS, Host.tls = 7, HostStatus.flintlock_version = 3.
  • Migration 2: tls_insecure, ca_file, cert_file, key_file columns on hosts.
  • Store: CreateHost (ErrHostExists), UpdateHost (address and TLS only, keeps cordon state), and DeleteHost, all round-tripping TLS. seedHosts also writes TLS for now.
  • Tests, including an upgrade from a version-1 DB.

Phase 4: Mutable flintlockclient.Pool

  • sync.RWMutex over the maps. Client, ExecClient, SSHProxyClient, Address, and Hosts take the read lock.
  • New takes a host list, not *config.Config. Add Add, Update (close the old connection, dial the new one, clear the version cache entry), and Remove.
  • Expose the last version CheckVersion saw, for HostStatus.flintlock_version.
  • Update callers, the e2etest fake, and reconciler test doubles. Add race tests for concurrent reads and mutations.

Phase 5: HostAdmin RPCs, pool referential checks, and CLI

  • AddHost: validates the spec with the same rules as config.Validate today; dials and runs CheckVersion unless skip_validation is set; returns FAILED_PRECONDITION or ALREADY_EXISTS; stores the host, then calls Pool.Add.
  • UpdateHost: returns NOT_FOUND for an unknown host; re-validates; calls Pool.Update; never renames and never changes cordon state.
  • RemoveHost: returns FAILED_PRECONDITION while any pool spec names the host or vm_count > 0; otherwise deletes the host and calls Pool.Remove.
  • GetHost: returns HostStatus with vm_count and flintlock_version. ListHosts also fills in flintlock_version.
  • CreatePool and UpdatePool return INVALID_ARGUMENT for an unknown flintlock_hosts entry.
  • Log host changes at Info.
  • poolmgrctl host add|update|remove|get, with --address, --insecure, --ca-file, --cert-file, --key-file, --skip-validation, and -o table|json.

Phase 6: The store becomes the only source of hosts

  • Remove hosts from the config. A config that still has a hosts key fails to load with an error that names poolmgrctl host add. api_server becomes required.
  • Remove seedHosts and UpsertHostIfMissing. At startup, build the Pool from store.ListHosts. With zero hosts, log a warning and let provisioning fail with ErrNoEligibleHost.
  • Update the e2e test, the Helm chart and deploy/, the README, and the design doc's "static config" section.
  • Set the ADR status to Accepted.

Activity

  1. phoban01 commented on Sep 25, 2026

    @phoban01
    ContributorAuthor

    All six phases are done. Part 1 is in #106. Part 2 is in #107, which is built on #106, so please merge #106 first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions