Skip to content

Plan/RFC: registry protocol v1 — self-hosted registries, private deps on zpkg.net, shared-auth login #45

Open
Open
Plan/RFC: registry protocol v1 — self-hosted registries, private deps on zpkg.net, shared-auth login#45

Description

@ORESoftware
ORESoftware
opened on Aug 7, 2026

Planning anchor — full plan lives in the Linear project zpkg Self-Hosted Registries and Private Deps. This repo hosts the protocol RFC (spec-first).

Goals

  1. Registry protocol + zed-cli support for multiple registries, including self-hosted ones.
  2. Private packages on the global registry (zpkg.net).
  3. A turnkey "host your own registry" story + written guide.
  4. zpkg.net signup/login and CLI auth via the shared-auth org's server(s). (zpkg.tech stays parked.)

Protocol v1 sketch (RFC to land here)

  • Sparse HTTP index: GET /index/{org}/{name} → NDJSON per version (deps, cksum, yanked).
  • Content-addressed tarballs: GET /pkgs/{org}/{name}/{version}.tar.zst.
  • Discovery: GET /.well-known/zpkg-registry.json (schema_version, endpoints, auth_modes, publish_supported`) for client feature-detection.
  • Publish (PUT, idempotent, versions immutable) + yank/unyank; bearer auth.
  • Hard constraint: the public read path is plain files — servable from R2/S3/nginx/GitHub Pages. Only publish/yank and private authz need an active server. This is what makes self-hosting nearly free.
  • Org-segment normalization rules (lowercase canonical form, client-validated at parse time) — kills the known wrong-org-segment bug class that resolves never and is invisible to cargo build.

Client & server split

zed-cli gains named registries (~/.zpkg/config.toml), per-dep registry = "...", lockfile (registry, sha256) pinning, env-only ZPKG_TOKEN_<NAME> creds, strict source replacement. Server-side, registry routes extract into zed-registry-core, consumed by zed-api-server.rs (global) and a new standalone zpkg-registryd (fs + S3/R2 backends; docker/compose/helm; plus a static read-only export mode).

Private packages: visibility flags, org roles, scoped tokens, short-lived signed R2 URLs for tarballs, audit log. GA depends on cluster promotion (DEN-534/DEN-535) — not on the transitional tunnel registry.

Phases (Linear)

DEN-2854 protocol RFC (this repo) → DEN-2855 zed-cli multi-registry → DEN-2856 registry core + registryd → DEN-2857 static export + guide → DEN-2858 private packages → DEN-2859 shared-auth integration → DEN-2860 global policy → DEN-2861 zed-e2e → DEN-2862 external (ChatGPT bridge) review.

Review/improvement feedback from the ai-agent-bridge review will be posted here and on DEN-2862.

Activity

  1. ORESoftware commented on Aug 7, 2026

    @ORESoftware
    ORESoftware
    on Aug 7, 2026
    ContributorAuthor

    ChatGPT bridge review dispatched 2026-08-07 (workflow 72dac60a, agent chatgpt-reviewer/gpt-5.5) — provider call failed on exhausted OpenAI credits; Codex subscription limited until 2026-08-09. Retry staged; status tracked on DEN-2862. Review will be posted here once it lands.

  2. ORESoftware commented on Aug 7, 2026

    @ORESoftware
    ORESoftware
    on Aug 7, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    ChatGPT fallback architecture review applied — 2026-08-07

    The original provider-backed AI-agent-bridge workflow remains preserved as 72dac60a-807a-4700-91de-ab361e0cdc40; it reached chatgpt-reviewer but failed at the provider boundary with credit_balance_exhausted. This update does not claim that bridge run succeeded.

    A current-session GPT-5.6 Pro adversarial review was completed and the authoritative Linear project was rewritten with:

    • immutable registry_id, canonical origin/trust-root enrollment, and lockfile origin/checkpoint pinning distinct from local aliases;
    • signed monotonic metadata/checkpoints to detect rollback/freeze while retaining static sparse reads;
    • NDJSON as v1 authority, with any SQLite snapshot derivative and signed only;
    • explicit source replacement/mirroring—no transparent namespace fallback—and canonical safe archives with server-side digest validation;
    • human keychain/credential-helper login versus short-lived CI/workload identity, with shared-auth identity exchanged for registry-audience scoped credentials;
    • one shared-auth identity boundary for canonical zpkg.net and parked/redirected zpkg.tech;
    • GitHub App/DNS organization verification, explicit active/yanked/security-revoked/legal-tombstoned states, and stronger self-hosting backup/key-rotation guidance;
    • E2E tests for alias remap, dependency confusion, mirror rollback, revocation, and tombstoned locked packages.

    Rejected/deferred decisions are recorded in the Linear plan and DEN-2862. DEN-2862 remains In Progress pending a successful provider-backed transcript or an explicit waiver.

  3. ORESoftware commented on Aug 7, 2026

    @ORESoftware
    ORESoftware
    on Aug 7, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    Fallback ChatGPT architecture/security review applied

    The provider-backed bridge workflow 72dac60a-807a-4700-91de-ab361e0cdc40 remains retryable but failed at the OpenAI provider boundary with credit_balance_exhausted; DEN-2862 remains open for the real transcript.

    A current-session GPT-5.6 Pro review has been folded into the Linear plan:

    • immutable registry_id, canonical origin and trust root are distinct from local config aliases; lockfiles pin origin/signing/checkpoint identity so alias remaps cannot reinterpret dependencies;
    • sparse per-package NDJSON remains authoritative, with signed monotonic freshness/checkpoint metadata for rollback/freeze detection, expiry, rotation and recovery;
    • mirrors/source replacement are explicit and preserve upstream identity/signatures/digests; transparent namespace fallback is rejected;
    • package archives have a canonical deterministic/safe extraction contract and bounded file/size/compression behavior;
    • human credentials use shared-auth plus OS keychain/credential helpers, while CI uses short-lived scoped secrets or workload OIDC—never device flow, flags or plaintext config;
    • zpkg.net and any zpkg.tech entry point share one shared-auth tenant/account model; no parallel identity or cross-domain cookie shortcut;
    • private index/archive paths are auth-gated and never publicly cacheable; signed URLs are short-lived and log-redacted;
    • organization claims require GitHub App/DNS proof plus reserved/confusable/dispute/transfer/recovery policy;
    • lifecycle distinguishes active, yanked, security-revoked and legal-tombstoned versions;
    • private-package GA is blocked on production cluster/TLS/object storage/backups/key recovery/multi-replica and rollback/freeze/dependency-confusion E2E gates.

    Authoritative disposition: Linear project zpkg Self-Hosted Registries and Private Deps and DEN-2862. Durable provider retry queue: ORESoftware/ai-agent-bridge.rs#104.

  4. ORESoftware commented on Aug 7, 2026

    @ORESoftware
    ORESoftware
    on Aug 7, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    GitHub ProjectV2 routing status

    The registry plan and dedicated Linear project are complete. The canonical zed-pkg routing registry names the organization-owned board zed-pkg-project, but currently records it as permission-blocked with project number 0 and no URL; the connected GitHub app does not expose ProjectV2 mutation.

    Tracking blocker: DEN-2875. It requires restoring organization Project permission, creating or resolving the one canonical board, attaching this issue exactly once, and updating the routing registry with the real project URL/number/item/status through protected-secret or ephemeral encrypted credential handling.

  5. ORESoftware commented on Aug 8, 2026

    @ORESoftware
    ORESoftware
    on Aug 8, 2026
    ContributorAuthor

    Attached to org GitHub ProjectV2 zed-pkg-project (#1): https://github.com/orgs/zed-pkg/projects/1 — item PVTI_lADOEmIPx84BfYhgzg1v4fY, status Todo. Routing-registry metadata update (permission-blocked → #1 + URL) remains with the bridge-repo thread; tracked on DEN-2875.

  6. ORESoftware commented on Aug 8, 2026

    @ORESoftware
    ORESoftware
    on Aug 8, 2026
    ContributorAuthor

    Registry follow-up: PROJECT_TRACKING.md now records the verified board (org project #1, node PVT_kwDOEmIPx84BfYhg) via zed-pkg/.github#35. After the Status-option revamp this issue's board status is Backlog (was Todo), with Linear/Package layer/Target branch fields populated.

  7. ORESoftware commented on Aug 8, 2026

    @ORESoftware
    ORESoftware
    on Aug 8, 2026 – with ChatGPT Codex Connector
    ContributorAuthor

    Registry protocol v1 implementation: draft PR #48

    The contract-first implementation is now open at #48.

    It adds immutable registry identity/discovery/trust metadata, sparse NDJSON records with explicit dependency origins, signed monotonic checkpoints, canonical archive manifests, immutable publication requests, explicit lifecycle/error states, generated schemas, golden fixtures, adversarial validation tests, and the self-hosted/private-package RFC.

    A one-use exact-head workflow successfully formatted the source, ran the focused protocol tests, generated all schemas, validated the materialized diff, committed it, and deleted itself. The branch then incorporated current main as a real two-parent merge, preserving the independent registry.zpkg.net default change.

    Current immutable head: b7d027f48488a3c42c5e6ba68d0b33b471e1380f. It is zero commits behind main. Normal PR checks (ci, environment-lock, Nix, agents policy, native-registry contract) are now queued/pending, so #48 remains draft.

    Coordination note: #43 targets a future polyglot layout and touches the future Rust export/schema-generator paths. If that baseline lands first, #48's module and generator entries must be moved semantically into src/rust/**; do not accept textual conflict markers or duplicate generated schemas.

    No user-supplied credential or production Cloudflare/R2/shared-auth secret is present.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions