Skip to content

Extend #600 credential refresher to all expiring credential sources (instance roles, Pod Identity) via a CanExpire gate #601

Description

@xe-nvdk

Summary

The #600 fix gives IRSA deployments an Arc-managed credential refresher: credentials resolved via the Go AWS SDK, handed to DuckDB as session credentials, re-issued before expiry (internal/database/s3refresh.go). Its detection gate is deliberately narrow — no static keys and both AWS_ROLE_ARN + AWS_WEB_IDENTITY_TOKEN_FILE present — because IRSA is the case we could verify against live AWS STS.

Two credential sources with the same underlying DuckDB expiry bug are NOT covered:

  1. EC2 instance roles — IMDS sessions last ~6h and DuckDB's PROVIDER CREDENTIAL_CHAIN resolves them once at CREATE SECRET time. Same failure as IRSA: DuckDB query path fails with ExpiredToken ~1h after start (credential_chain never refreshes) #600, just a longer fuse. Arc currently logs a startup Warn (credential_mode=credential_chain) in this configuration.
  2. EKS Pod Identity — injects AWS_CONTAINER_CREDENTIALS_FULL_URI / AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE; neither IRSA env var is set, so the gate never fires. DuckDB has no matching chain either, so those deployments have no working path at all today.

Proposal

Broaden the gate from "IRSA env detected" to "the SDK chain resolved credentials with CanExpire=true":

  • No static keys configured (config or env) → attempt a Go-SDK resolve at startup.
  • Resolve succeeds with CanExpire=true → start the refresher (covers IRSA, instance roles, Pod Identity, SSO — anything the SDK chain handles).
  • Resolve succeeds with CanExpire=false → emit once, no loop.
  • Resolve fails (keyless MinIO / anonymous) → fall back to today's plain CREDENTIAL_CHAIN emission.

The refresher itself already handles all of this (CanExpire branches exist); only the gate in configureS3Access / ConfigureS3 / New changes.

Why it was not shipped in #600

The broadened gate's new cells (IMDS, Pod Identity) could not be live-tested in the time available, and shipping untested credential-path coverage is how #600's class of bug happens. The IRSA cell was verified against real STS end-to-end; the others deserve the same before the gate widens (an IMDS test needs a real EC2 instance; Pod Identity needs an EKS cluster).

Notes for implementation

  • The startup Warn in configureS3Access (credential_mode=credential_chain → "temporary credentials will NOT auto-refresh") is the operator-facing marker of the uncovered cell; remove it when this ships.
  • useWebIdentityChain's doc comment names this follow-up.
  • Release notes 26.09.1 state the limitation publicly ("Not covered yet, stated plainly").

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions