Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The other half is the sibling core repository — private: the wrapper, the buil
- **The header is owned here, and it is generated.** `include/chtypes.h` is generated with every binding's declaration layer from `spec/abi-v1/abi.json` by `scripts/abi-v1/gen.py`; an ABI change edits the description and regenerates, never a header or a declaration by hand (`gen.py --check` and `scripts/abi-v1/check-no-hand-decls.py` enforce it, in the required `v1-abi-gen` check). The core repository then pulls the header and relinks every artifact.
- **Never trust exit codes or self-reports.** Every test here skips LOUDLY without a library and refuses a zero-run; `scripts/check-standalone.sh` reads its verdict off a `go test -json` census and `scripts/check-suite.sh` off each runner's own summary line, color stripped. Before either suite runs, both print, via the shared `scripts/lib/provenance.py`, the ABI version and fingerprint this binding was built against; the loaded library's own `build_info` names the artifact.
- **The Go package is dlopen-only by default.** The linked path is behind `-tags chtypes_linked` and needs a core build tree via `CGO_LDFLAGS`. **Only LINKING a binary needs that tree — type-checking the tagged path needs nothing but a C compiler and the in-repo header**, so `scripts/abi-v1/check-linked.sh` builds it in the required `v1-abi-linked` check on every PR, selftest first.
- **Libraries live in one per-user cache**, an OCI image layout at `~/.cache/chtypes/v1/` (`$CHTYPES_CACHE` overrides), with each verified library unpacked beside it. Every binding fetches for itself over OCI and verifies a signed statement before it unpacks: there is no shell fetcher, and `CHTYPES_REGISTRY` is retired. The contract is `docs/guides/fetch-v1.md`; a CLI named `chtypes` with `fetch`, `verify`, `list` and `where` ships in every binding.
- **Libraries live in one per-user cache**, an OCI image layout at `~/.cache/chtypes/v1/` (`$CHTYPES_CACHE` overrides), with each verified library unpacked beside it. Every binding fetches for itself over OCI and verifies a signed statement before it unpacks: there is no shell fetcher, and `CHTYPES_REGISTRY` is retired. The contract is `docs/guides/fetch-v1.md`; a CLI named `chtypes` with `fetch`, `verify`, `list`, `where`, `resolve` and `prune` ships in every binding.
- darwin artifacts match ClickHouse on macOS; on 26.7+ every platform agrees bit for bit. On 26.3 ClickHouse's own float text parse is OS- and arch-dependent, so 26.3 float expectations come from a server on the same platform (docs/limitations.md).

Gates: `scripts/check-standalone.sh` (Go, from a bare copy) and `scripts/check-suite.sh python|ts|rust`; per language `go test ./...`, `uv run pytest -q`, `pnpm test`, `cargo test`. With a fetched library they run everything; without one every artifact test skips by name and the rest still runs, and a suite that ran nothing fails. `.github/workflows/ci.yml` runs on hosted runners with no secret and no repository variable: each binding builds, lints and runs its unit suite with no library (`--no-artifacts`). The ABI and fetch halves run in their own workflows (`v1-abi.yml`, `v1.yml`): every binding passes every ABI case against a generated stub and every fetch case against a registry the tests start themselves, with no network beyond loopback and no published artifact. The server-truth suites for these SDKs live in the core repository and run from there against this tree as a sibling.
Expand Down
42 changes: 41 additions & 1 deletion docs/guides/fetch-v1.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ cargo install chtypes && chtypes fetch 26.8 # Rust

A spelling is two, three or four parts (`26.8`, `26.8.15`, `26.8.15.10`), with no `v` prefix and no channel suffix. The command installs into the per-user OCI cache `${XDG_CACHE_HOME:-~/.cache}/chtypes/v1/`, which every binding reads by default, so **one machine set up once serves all four**. `CHTYPES_CACHE` overrides the cache directory, and `chtypes where` prints the one in use. `CHTYPES_REGISTRY`, which the previous generation used, is retired: set, it warns once and is otherwise ignored.

Before anything lands, the command verifies a Sigstore bundle signed under the release key and the digest of every byte. `fetch --all` takes every line the registry publishes for this platform; `verify`, `list` and `where` are the other three subcommands. [`guides/artifacts.md`](guides/artifacts.md) has the whole story, including pinning for CI with `--lock` and `--frozen`.
Before anything lands, the command verifies a Sigstore bundle signed under the release key and the digest of every byte. `fetch --all` takes every line the registry publishes for this platform; `verify`, `list`, `where`, `resolve` (what a line resolves to, installing nothing) and `prune` (remove the builds newer ones supersede) are the other five subcommands. [`guides/artifacts.md`](guides/artifacts.md) has the whole story, including pinning for CI with `--lock` and `--frozen`.

Pick a line you actually need. Each artifact is 160–300 MB on disk and about 120 MB resident once loaded, so fetch the versions your deployments run rather than all of them.

Expand Down
11 changes: 9 additions & 2 deletions docs/reference/bindings-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,16 @@ One row per public operation. `request` is a version spelling under [`fetch-v1.m
| construct a registry | none: it opens nothing | `chtypes.NewRegistry(opts ...RegistryOption) (*Registry, error)` | `chtypes.Registry(*, fetch: Optional[FetchOptions] = None, autofetch: Optional[bool] = None, preload: Sequence[str] = ())` | `await Registry.open(options?: RegistryOptions): Promise<Registry>` | `Registry::new(options: RegistryOptions) -> Result<Registry>` | FIRM |
| registry options | none | `WithFetchOptions(FetchOptions)`, `WithAutoFetch(bool)`, `WithPreload(requests ...string)` | the keywords above | `RegistryOptions { fetch?, autofetch?, preload? }` | `RegistryOptions { fetch, autofetch, preload }`, `Default` | FIRM |
| open a version | the load steps 1 to 7 | `(r *Registry) For(request string) (*Library, error)`; `ForContext(ctx context.Context, request string) (*Library, error)` | `Registry.for_version(request: str) -> Library` | `registry.for(request: string): Promise<Library>` | `Registry::for_version(&self, request: &str) -> Result<Arc<Library>>` | FIRM |
| fetch without opening | none: the fetch layer's `ensure` | `(r *Registry) Fetch(ctx context.Context, request string) (Resolved, error)` | `Registry.fetch(request: str) -> Resolved` | `registry.fetch(request: string): Promise<Resolved>` | `Registry::fetch(&self, request: &str) -> Result<Resolved>` | FIRM |
| what is installed | none: the fetch layer's `list_installed` | `(r *Registry) Installed() ([]Resolved, error)` | `Registry.installed() -> tuple[Resolved, ...]` | `registry.installed(): Promise<readonly Resolved[]>` | `Registry::installed(&self) -> Result<Vec<Resolved>>` | FIRM |
| what is open | none | `(r *Registry) Libraries() []*Library` | `Registry.libraries() -> tuple[Library, ...]` | `registry.libraries(): readonly Library[]` | `Registry::libraries(&self) -> Vec<Arc<Library>>` | FIRM |
| the cache root | none: the fetch layer's `cache_root` | `chtypes.CacheRoot(o FetchOptions) (string, error)` | `chtypes.cache_root(options: Optional[FetchOptions] = None) -> Path` | `cacheRoot(options?: FetchOptions): string` | `chtypes::cache_root(options: &FetchOptions) -> Result<PathBuf>` | FIRM |
| the directories searched | none: the fetch layer's search order | `chtypes.SearchDirs(o FetchOptions) ([]string, error)` | `chtypes.search_dirs(options: Optional[FetchOptions] = None) -> tuple[Path, ...]` | `searchDirs(options?: FetchOptions): readonly string[]` | `chtypes::search_dirs(options: &FetchOptions) -> Result<Vec<PathBuf>>` | FIRM |
| open a local build | the load steps without steps 1 and 5 | `chtypes.OpenUnverified(path string, allow bool) (*Library, error)` | `chtypes.open_unverified(path: str, *, allow: bool = False) -> Library` | `openUnverified(path: string, options: { allow: boolean }): Library` | `Library::open_unverified(path: impl AsRef<Path>, allow: bool) -> Result<Arc<Library>>` | FIRM |
| open the linked image (Go) | the linked table filler; steps 3, 4, 6 and 7 | `chtypes.OpenLinked() (*Library, error)`, under `-tags chtypes_linked` | none | none | none | FIRM; that it stays is §8 question 9 |

`fetch` (public issue #492) is the library counterpart of `chtypes fetch`: it resolves, fetches, verifies and installs the build a request names into the cache, for this host, and opens nothing. No library is loaded, the process setup is untouched (it needs no `setup`), nothing enters the memo and `libraries()` is unchanged; the `Resolved` it returns names the installed version, build, digests and library path. It runs the fetch layer's `ensure` with the registry's fetch options, so offline, the cache, the system directories and, under the production channel, the bases, the trust and the lock or frozen pin apply exactly as for an open's fetch, and its errors are the fetch codes' classes (§4). It shares the open's single flight (§6, "The sequence"): a concurrent `fetch` and open of one request, or two `fetch`es, make one fetch and each gets its answer, and a failed fetch is never remembered. Go's `Fetch` takes a context, and returns `ctx.Err()` when the context is done before the fetch has an answer; the fetch goes on for any other caller waiting on it. The build it returns is held by the process until it exits (§6), so `chtypes prune` never removes it meanwhile. A consumer warms a cache with it at start-up, or in a sidecar, before any open.

`cache_root` and `search_dirs` (public issue #530) answer where a lookup reads, for a caller that logs the cache, checks a directory it was given, probes it for writability or names the searched directories in an error, without copying the precedence. They run the fetch layer's own resolution (the explicit directory, else `CHTYPES_CACHE`, else `${XDG_CACHE_HOME:-~/.cache}/chtypes/v1`; [`fetch-v1.md` §1](../guides/fetch-v1.md#1-the-cache)) and take the same `FetchOptions` as `fetch`, `list` and `chtypes where`, whose `system_dirs` field replaces the built-in list (an empty list searches none). `search_dirs` returns the cache root first, then each system directory, in the order the fetch layer reads them, so its first entry is `cache_root`'s answer. Both return absolute paths (a relative explicit directory or `CHTYPES_CACHE` is resolved against the working directory) and are read-only: they create nothing, read no cache and make no network call. `chtypes where` prints the root alone; `chtypes where --all` prints `search_dirs`' answer, one directory per line. In a 2.0.0-dev SDK the answers carry the major's `v2-dev` subroot, for an explicit directory from the option as well as from `CHTYPES_CACHE` (`spec/abi-v2/docs.md`, rule r5; public issue #527).

### The library
Expand Down Expand Up @@ -511,7 +514,9 @@ The per-call zone needs no setup: it is the `session_timezone` key of a call's s
registry.for(request)
1. the registry's memo: a request it has opened before returns the same Library, for the registry's life
2. resolved = fetch.resolve_installed(request, host platform, fetch options) never the network
3. on a miss: autofetch on → resolved = fetch.ensure(request, fetch options) may use the network
then hold(resolved.dir); gone before it was held: step 2 once more, then a miss
3. on a miss: autofetch on → resolved = the request's one fetch.ensure(request, fetch options)
then hold(resolved.dir); gone: fetch once more may use the network
autofetch off → ArtifactMissing, naming the request and the platform
4. input = LoadInput{ library_path: resolved.library_path,
predicate: resolved.predicate, verbatim, never re-encoded
Expand All @@ -528,6 +533,8 @@ The load-time assertion (step 7) is `fetch-v1.md` §9's: an exact request needs
- **Construction opens nothing.** As in v0, constructing a registry dlopens nothing, and nothing opens an artifact except a request for a version or `preload`. TypeScript's `Registry.open` is asynchronous only because its fetch layer is (§8 question 5).
- **The memo keeps v0's "a line request never moves mid-process" rule** (v0 §Version selection, rule 4): `26.8` resolves once per registry, and a newer `26.8` patch installed later is picked up by a new registry, not by the old one. Everything else about resolution is the fetch layer's: which installed build a floating request means, and the spelling rules. The binding orders and matches no versions itself.
- **An open never waits for another request's fetch** (public issue #491). The registry's lock guards its memo and the attempts in progress, and is never held across steps 2 to 8, so an open the cache answers needs neither the network nor another open's fetch. The setup guard is held across a load (steps 5 and 6), which is local, and never across the network. Concurrent opens of one request share one attempt, and so one fetch: each gets that attempt's `Library` or its error, so two racing opens of one request get the same `Library`, and a failed attempt is never remembered, so the next open starts a new one. Go's `ForContext` returns `ctx.Err()` when its context is done before the open has an answer; the fetch goes on for the other opens waiting on it, and is canceled once none does. Python, TypeScript and Rust have no cancellation of an open.
- **One fetch per request, shared with `fetch`** (public issue #492). The fetch of step 3 runs through a table of the fetches in progress, by request, which `registry.fetch` uses too: an open that misses the cache and a `fetch` of the same request wait on one `ensure` call and each gets its answer.
- **What an open uses is held** (public issue #494). Before anything reads the library, the registry takes the fetch layer's `hold` on the build: a shared advisory lock (`flock`) on the entry's `verified.json`, kept until the process exits ([`fetch-v1.md` §1, "In use"](../guides/fetch-v1.md#1-the-cache)). `chtypes prune` removes a build only after taking the exclusive lock without waiting, so it never removes one a live process opened or fetched. A build a prune removed between the lookup and the hold is looked up (or fetched) once more; removed twice, the lookup is a miss and the fetch is `ArtifactMissing`.
- **`preload`** opens each listed request through steps 1 to 7 at construction, in list order, and never fetches, even with autofetch on. A request no installed build answers is the ordinary `ArtifactMissing`, raised at construction.
- **`autofetch`** defaults to the fetch constants' `CHTYPES_AUTOFETCH`, as in v0, and is off when that is unset.
- **The fetch options** carry everything [`fetch-v1.md`](../guides/fetch-v1.md) configures (bases, cache directory, system directories, trusted keys, token, allow-unsigned, offline, frozen, the lock path, strict cache mode) under the fetch module's own names: Go `FetchOptions` (a public struct the binding converts, because the fetch package is internal and its options include test hooks), Python `FetchOptions`, TypeScript `FetchOptions`, Rust `FetchOptions`. `Resolved` is the fetch layer's record, re-exported unchanged (Go `type Resolved = ocifetch.Resolved`). The trusted keys are raw 32-byte ed25519 public keys, each as 64 hex digits, the spelling `CHTYPES_TRUSTED_KEYS` uses, in all four, and each key's id is derived from it (`sha256-first16hex`). No fetch option names a platform: a registry opens libraries for this host, and `chtypes fetch --platform` is the command's own flag. No option sets a timeout: the fetch layer's connect and idle-read timeouts are the fetch constants'.
Expand All @@ -545,7 +552,7 @@ One conformance case per binding drives the real derivation, as the v1 FFI plan'

### What the public API does not wire

`fetch_signed` (goldens and fixtures) and `verify_installed` stay the fetch module's own surface, used by the test harnesses and the CLIs. The CLIs, the `examples/` tours and the 0.x shell fetcher's retirement belong to the switch, not to this page.
`fetch_signed` (goldens and fixtures) and `verify_installed` stay the fetch module's own surface, used by the test harnesses and the CLIs, and so do `resolve` and `prune`, behind `chtypes resolve` and `chtypes prune` (public issues #493 and #494; [`fetch-v1.md`, "The command line"](../guides/fetch-v1.md#the-command-line)), and `hold`, which the registry calls. Every CLI ships `fetch`, `verify`, `list`, `where`, `resolve` and `prune`, with one output, held to it by `scripts/check-cli-parity.sh`. The CLIs, the `examples/` tours and the 0.x shell fetcher's retirement belong to the switch, not to this page.

## 7. What v0 API is deleted, and why

Expand Down
Loading
Loading