v2 fetch: zero-request frozen paths (#414), Go frozen re-download, TS lock platforms and error class, Python index duplicates (#487)

docs/guides/fetch-v1.md

@@ -172,6 +172,17 @@ Every one of the checks above runs **in this order**, and a failure at any step
 
 - `fetch --lock` resolves normally (§3–§4) and then writes the lock.
 - `--frozen` **skips resolution entirely**: it fetches the platform manifest, bundle and layer by digest from any configured base, verifies them exactly as a normal fetch would (§4–§5), and refuses (`CHTYPES_ARTIFACT_PINNED`) if the lock names no entry for the requested platform, or the lock itself fails schema validation. A tree with **no tags, no referrers API and no `tags/list`** still succeeds under `--frozen` — every request it makes is by digest (the `frozen-no-discovery` case).
+- **A build already installed under exactly the pin is used as it is, with zero requests** (public issues #414 and #487). Before any request, `--frozen` looks for `unpacked/sha256/<pinned manifest>/verified.json`, in the cache and then each system directory, and answers from it when the record is for the requested platform and records exactly the pinned manifest, layer and bundle digests. A missing record, or one that differs in any of the three, falls through to the by-digest fetch and verification above. The `frozen-warm-cache-zero-requests` case asserts `requests.max: 0`.
+- **`--frozen --offline` verifies the installed build against the lock with zero network.** It makes the same lookup and never falls through to a fetch. It never answers with a different installed build that merely satisfies the request. The outcomes:
+
+  | situation                                                                                                                      | code                       |
+  | ------------------------------------------------------------------------------------------------------------------------------ | -------------------------- |
+  | the lock file is missing or fails schema validation, or names no entry for the request                                         | `CHTYPES_ARTIFACT_PINNED`  |
+  | the pinned build is installed and its record matches the pin                                                                   | answered, zero requests    |
+  | the pinned build is not installed (no record under the pinned manifest digest), the offline nothing-installed outcome          | `CHTYPES_ARTIFACT_MISSING` |
+  | a record is installed under the pinned manifest digest but its platform, manifest, layer or bundle digest differs from the pin | `CHTYPES_ARTIFACT_PINNED`  |
+
+  The `frozen-offline-lock-match` case asserts the match with `requests.max: 0`, `frozen-offline-lock-not-installed` asserts `CHTYPES_ARTIFACT_MISSING`, and `frozen-offline-lock-mismatch` asserts `CHTYPES_ARTIFACT_PINNED`, each with `requests.max: 0`. `--offline` without `--frozen` is unchanged: it reads the cache only and does not consult the lock for its answer.
 - `update` re-resolves every locked request against the current index and rewrites the lock; it never merges a stale entry with a fresh one.
 - `--offline` reads the cache only (§1): the **source of truth is the immutable `unpacked/sha256/*/verified.json` records**, never `index.json`. A pre-seeded `index.json` entry with no corresponding `unpacked/` directory is the one case `--offline` still verifies and unpacks before answering, because it has not yet produced a `verified.json` record of its own. A directory whose record a reader cannot accept (§1) counts as having none.
 - **A lookup creates nothing.** `resolve_installed`, `list_installed`, `verify_installed` and `ensure` with `offline` never create a directory or a file: the layout is made when an install begins, the pre-seed install above included, so a read-only mount reads cleanly and a mistyped `CHTYPES_CACHE` is not created by a lookup. A `chtypes verify` that verified no build says so on stderr, `chtypes: verified 0 builds under <root>`, so an empty pass never looks like a good one (public issue #486).

docs/reference/ts.md

@@ -15,7 +15,7 @@ A bad **row** is never an exception: the verdict is `RowResult#outcome`, one of
 - `UsageError`: caller misuse, whichever side caught it (a closed object, a refused version spelling, a zone given twice).
 - `InternalError`: a library bug, or a document that does not decode.
 
-The four are **peers** under the abstract `CallError`: a decline never satisfies `instanceof SchemaError`. Loader and fetch errors are `ArtifactError`s (`ArtifactIncompatibleError`, `ArtifactCorruptError`, `ArtifactMissingError` and the rest of the fetch family); `isChtypesError` is true for every error this package throws on purpose.
+The four are **peers** under the abstract `CallError`: a decline never satisfies `instanceof SchemaError`. Loader and fetch errors are `ArtifactError`s (`ArtifactIncompatibleError`, `ArtifactCorruptError`, `ArtifactMissingError` and the rest of the fetch family); `ArtifactError` extends `ChtypesError`, as the other bindings' hierarchies have it, so catching `ChtypesError` takes the fetch and loader errors along with the call errors, and `isChtypesError` is true for every error this package throws on purpose.
 
 ## Setup, registry and library
 

go/CHANGELOG.md

@@ -51,6 +51,7 @@ The four bindings in this repository are released together and give one answer,
 
 ### Fixed
 
+- `fetch --frozen` no longer re-downloads the layer on a warm cache (#487): the frozen path checked no installed record, so every run made three requests and fetched about 42.5 MB (measured on the published 1.0.2 package). It now looks the pin up in the cache first, as Python does. `--frozen` with a warm, verified cache makes zero requests, and `--frozen --offline` verifies the installed build against the lock with zero network (#414). An installed build whose `verified.json` is for the platform and records exactly the lock's pinned manifest, layer and bundle digests is used as it is. `--frozen --offline` is `CHTYPES_ARTIFACT_MISSING` when the pinned build is not installed, and `CHTYPES_ARTIFACT_PINNED` when there is no lock entry or the installed record does not match the pin; `--offline` without `--frozen` is unchanged. The `frozen-warm-cache-zero-requests`, `frozen-offline-lock-match`, `frozen-offline-lock-mismatch` and `frozen-offline-lock-not-installed` conformance cases hold all four bindings to it, under both corpora. The CLI no longer refuses `--frozen --offline` as a usage error.
 - A 2.0.0-dev SDK whose own-fingerprint alias has expired, with a cached build of that fingerprint for the request, now says so in the `CHTYPES_ARTIFACT_UNPUBLISHED` message of `fetch` (#581): `; a cached build for this fingerprint (build <id>) is still usable by open; upgrade the SDK to get newer builds` is appended. The code, the exit status (4) and the cache are unchanged, and all four bindings give the same message (`fp-ahead-cached-own-fp`).
 - A 2.0.0-dev SDK whose fingerprint no published build carries yet now fails closed with a precise message (#578). When every base answers its own alias 404 and the tag names a build signed for another fingerprint, `fetch` is `CHTYPES_ARTIFACT_UNPUBLISHED` (exit status 4): `no published build for this SDK's fingerprint <own fp>; newest published on this channel is <fp> (build <id>)`. It used to install that build, or report it already installed, and the lookup at open then answered `CHTYPES_ARTIFACT_MISSING` with advice to run `fetch`: a loop with no pointer to the cause. Nothing is downloaded or installed and the cache is unchanged. An alias that answers, a tag at the SDK's own fingerprint and the v1 contract are unchanged. All four bindings run the same fetch conformance cases, `fp-ahead-*`, and the rule is `docs/guides/fetch-v1.md` section 3.
 - Opening a line no longer waits for another line's fetch: the registry never holds its lock across network I/O, and concurrent opens of one request share one fetch (#491). `ForContext` returns `ctx.Err()` when its context is done before the open has an answer; the fetch goes on for any other open of the same request, and is canceled once no open waits on it.

go/cmd/chtypes/main.go

@@ -282,8 +282,6 @@ func cmdFetch(ctx context.Context, args []string, stdout, stderr io.Writer) erro
 		return &usageError{fmt.Sprintf("--all fetches every line; drop the version arguments (%s)", strings.Join(spellings, " "))}
 	case !all && !update && len(spellings) == 0:
 		return &usageError{"a ClickHouse version spelling is required (or --all)"}
-	case frozen && cf.offline:
-		return &usageError{"--frozen fetches by digest and --offline forbids the network; pass one"}
 	}
 	opts := cf.options()
 	opts.Frozen, opts.LockPath = frozen, lock

go/cmd/chtypes/main_test.go

@@ -33,7 +33,6 @@ func TestUsageErrorsExitTwo(t *testing.T) {
 		{"fetch"},
 		{"fetch", "26.8", "--all"},
 		{"fetch", "26.8", "--platform", "plan9-mips"},
-		{"fetch", "26.8", "--frozen", "--offline"},
 		{"fetch", "v26.8", "--offline"},
 		{"verify", "extra"},
 		{"list", "extra"},

go/internal/ocifetch/ensure.go

@@ -390,6 +390,9 @@ func ensure(ctx context.Context, ro resolvedOptions, req Request) (*Resolved, er
 	}
 
 	if ro.offline {
+		if ro.frozen {
+			return resolveFrozenOffline(ro, req, platform)
+		}
 		return resolveOffline(ro, req, platform)
 	}
 	if ro.strict {
@@ -439,6 +442,68 @@ func resolveOffline(ro resolvedOptions, req Request, platform Platform) (*Resolv
 		"%s", WithNotes(fmt.Sprintf("no installed artifact for %s/%s, and --offline forbids a network fetch", req.Spelling, platform.Key), missingNotes(ro)))
 }
 
+// installedByPin finds the installed build a lock pin names: an unpacked
+// directory named by the pinned manifest digest, in the cache or a system
+// directory, whose verified.json is for the requested platform and records
+// exactly the pinned manifest, layer and bundle digests. It touches no
+// network and creates nothing. nil, nil means no installed build matches.
+func installedByPin(ro resolvedOptions, req Request, platform Platform, pin LockPin) (*Resolved, error) {
+	res, _, err := lookupByPin(ro, req, platform, pin)
+	return res, err
+}
+
+// lookupByPin is installedByPin that also reports whether a record exists
+// under the pinned manifest's directory but disagrees with the pin.
+func lookupByPin(ro resolvedOptions, req Request, platform Platform, pin LockPin) (res *Resolved, mismatch bool, err error) {
+	warnings, err := probeRoots(ro)
+	if err != nil {
+		return nil, false, err
+	}
+	for i, dir := range ro.searchDirs() {
+		udir := newLayout(dir, true).unpackedDir(pin.Manifest)
+		rec, err := readVerifiedRecord(udir)
+		if err != nil {
+			continue
+		}
+		if rec.Platform != platform.Key || rec.Digests.Manifest != pin.Manifest ||
+			rec.Digests.Layer != pin.Layer || rec.Digests.Bundle != pin.Bundle {
+			mismatch = true
+			continue
+		}
+		return recordToResolved(rec, udir, platform.Key, req.Spelling, "", true, rootSource(i, dir), warnings), false, nil
+	}
+	return nil, mismatch, nil
+}
+
+// resolveFrozenOffline implements --frozen --offline: verify the installed
+// build against the lock with zero network. A lock with no entry for the
+// request, or an installed record that does not match the pin, is
+// CHTYPES_ARTIFACT_PINNED; a pinned build that is not installed is CHTYPES_ARTIFACT_MISSING.
+func resolveFrozenOffline(ro resolvedOptions, req Request, platform Platform) (*Resolved, error) {
+	lock, err := readLock(ro.lockPath)
+	if err != nil {
+		return nil, err
+	}
+	pin, ok := lock.Pin(req.Spelling, platform.Key)
+	if !ok {
+		return nil, newError(CodeArtifactPinned, req.Spelling, platform.Key, "", nil,
+			"lock file %s names no entry for %s/%s; re-lock with `fetch --lock`", ro.lockPath, req.Spelling, platform.Key)
+	}
+	res, mismatch, err := lookupByPin(ro, req, platform, pin)
+	if err != nil {
+		return nil, err
+	}
+	if res == nil {
+		if mismatch {
+			return nil, newError(CodeArtifactPinned, req.Spelling, platform.Key, "", nil,
+				"the installed build for %s/%s does not match the lock's pin %s (platform, manifest, layer or bundle digest differs)", req.Spelling, platform.Key, pin.Manifest)
+		}
+		return nil, newError(CodeArtifactMissing, req.Spelling, platform.Key, "", nil,
+			"%s", WithNotes(fmt.Sprintf("the build the lock pins (%s) is not installed for %s/%s, and --frozen --offline forbids a network fetch", pin.Manifest, req.Spelling, platform.Key), missingNotes(ro)))
+	}
+	return res, nil
+}
+
 // MissingNotes is what a CHTYPES_ARTIFACT_MISSING answer from the cache opts
 // names adds to its message, each a complete sentence: the default mode's
 // warning for every unusable root or unreadable entry, then the 0.x hint when
@@ -1312,6 +1377,15 @@ func (s *session) ensureFrozen(ctx context.Context, ro resolvedOptions, l *layou
 			"lock file %s names no entry for %s/%s; re-lock with `fetch --lock`", ro.lockPath, req.Spelling, platform.Key)
 	}
 
+	// A build already installed under exactly the pinned manifest, layer and
+	// bundle digests is used as it is: zero requests (§6, public issues #414
+	// and #487).
+	if res, err := installedByPin(ro, req, platform, pin); err != nil {
+		return nil, err
+	} else if res != nil {
+		return res, nil
+	}
+
 	manifestResult, base, err := s.fetchAcrossBases(ctx, ro.bases, "manifests/"+string(pin.Manifest), notFoundRetryOnLast, requestOptions{maxBytes: ManifestMaxBytes, accept: manifestAccept})
 	if err != nil {
 		return nil, err

python/CHANGELOG.md

@@ -51,6 +51,8 @@ The four bindings in this repository are released together and give one answer,
 
 ### Fixed
 
+- The layout's `index.json` never gains a second entry for one manifest (#487): concurrent installs of one build each appended an entry (7 or 8 entries for the same digest after 8 concurrent installs, measured on 1.0.2). The merge now deduplicates by digest, as Go and TypeScript do.
+- `--frozen` with a warm, verified cache makes zero requests, and `--frozen --offline` verifies the installed build against the lock with zero network (#414). An installed build whose `verified.json` is for the platform and records exactly the lock's pinned manifest, layer and bundle digests is used as it is. `--frozen --offline` is `CHTYPES_ARTIFACT_MISSING` when the pinned build is not installed, and `CHTYPES_ARTIFACT_PINNED` when there is no lock entry or the installed record does not match the pin; `--offline` without `--frozen` is unchanged. The `frozen-warm-cache-zero-requests`, `frozen-offline-lock-match`, `frozen-offline-lock-mismatch` and `frozen-offline-lock-not-installed` conformance cases hold all four bindings to it, under both corpora. The Python frozen path also compares the record's layer and bundle digests to the pin, and looks in the system directories too.
 - A 2.0.0-dev SDK whose own-fingerprint alias has expired, with a cached build of that fingerprint for the request, now says so in the `CHTYPES_ARTIFACT_UNPUBLISHED` message of `fetch` (#581): `; a cached build for this fingerprint (build <id>) is still usable by open; upgrade the SDK to get newer builds` is appended. The code, the exit status (4) and the cache are unchanged, and all four bindings give the same message (`fp-ahead-cached-own-fp`).
 - A 2.0.0-dev SDK whose fingerprint no published build carries yet now fails closed with a precise message (#578). When every base answers its own alias 404 and the tag names a build signed for another fingerprint, `fetch` is `CHTYPES_ARTIFACT_UNPUBLISHED` (exit status 4): `no published build for this SDK's fingerprint <own fp>; newest published on this channel is <fp> (build <id>)`. It used to install that build, or report it already installed, and the lookup at open then answered `CHTYPES_ARTIFACT_MISSING` with advice to run `fetch`: a loop with no pointer to the cause. Nothing is downloaded or installed and the cache is unchanged. An alias that answers, a tag at the SDK's own fingerprint and the v1 contract are unchanged. All four bindings run the same fetch conformance cases, `fp-ahead-*`, and the rule is `docs/guides/fetch-v1.md` section 3.
 - Opening a line no longer waits for another line's fetch: the registry never holds its lock across network I/O, and concurrent opens of one request share one fetch (#491).