PJ Cloud Connector (pj-mcap-server)

Self-hosted Go server + C++ client SDK + PlotJuggler 4 toolbox plugin that serves MCAP recordings from an S3-compatible or GCS bucket to PlotJuggler on demand: browse a queryable catalog (time/topic/tag filters), select files + topics + a time range, and stream exactly that subset — including server-side stitching of consecutive recordings into one continuous session — with reconnect-resume and a repeat-fetch cache.

Layout

Path	What
proto/pj_cloud.proto	Canonical wire schema (WS + Protobuf envelope)
server/	Go server: read-only catalog reader, session streaming, tag-edit IPC forwarding
mcap_catalog/	Python mcap_catalog builder (vendored in-repo) — the SOLE catalog writer + tag-edit IPC server
plugin/toolbox_mcap_cloud/	Cloud connector toolbox plugin + mcap-cloud-cli (builds standalone)
infra/minio/	Local S3 (Minio) — development storage endpoint
scripts/smoke.sh	make smoke — end-to-end regression gate
scripts/e2e-layout-import.sh	make e2e-layout — cross-repo layout-import gate (real PlotJuggler + real plugins + live server)
arch/	Design spec and implementation plans

The backend is a two-process system: the Python builder (mcap_catalog/) scans the bucket and owns the SQLite catalog + tag-edit IPC; the Go server (server/) opens that catalog read-only and streams sessions — it has no writer path of its own and will not start until the builder has published a catalog. run.sh (below) starts both, in order, for local development.

flowchart LR
    Client["PlotJuggler plugin<br/>or mcap-cloud-cli"]
    Server["Go server"]
    Builder["Python catalog builder"]
    Catalog[("SQLite catalog")]
    Storage[("S3 / GCS / Minio")]

    Client <-->|"WebSocket + Protobuf<br/>browse, stream, resume"| Server
    Server -->|"read-only queries"| Catalog
    Server -->|"range reads for selected data"| Storage
    Server -->|"tag edits over Unix socket"| Builder
    Builder -->|"scan MCAP summaries"| Storage
    Builder -->|"atomic catalog publish"| Catalog

Loading

Object-store recordings must use the Hive key layout customer=<c>/customer_site=<site>/robot=<r>/source=<s>/date=<d>/<file>.mcap, or they are reported in catalog_failures and omitted from the catalog. The builder's local-filesystem mode additionally accepts an intended Hive key from an MCAP s3_key metadata record.

Quick start — local, no cloud account needed

Prerequisites: Docker, Go 1.23+ at $HOME/.local/go, a C++20 compiler, Conan 2, CMake ≥ 3.22, and a Python 3 venv for the catalog builder at ~/.venvs/pj-catalog (bootstrap once; pins match CI/scripts/smoke.sh): python3 -m venv ~/.venvs/pj-catalog && ~/.venvs/pj-catalog/bin/pip install boto3==1.43.40 google-cloud-storage==3.12.0 mcap==1.4.0 watchdog==6.0.0. The plotjuggler_sdk Conan package must be in your cache before building the plugin (see plugin/SDK_VERSION for the required version).

# 1. Build the Go server + CLI plugin
./build.sh

# 2. Start the local backend: Minio + synthetic recordings + the catalog
#    builder + the server on :8080
./run.sh

# 3. Check the catalog via the CLI
plugin/toolbox_mcap_cloud/build/bin/mcap-cloud-cli \
  --url ws://localhost:8080 list

Stop everything: make server-stop && (cd infra/minio && docker compose down) (server-stop reaps BOTH the Go server and the catalog builder daemon).

Targets

run.sh takes a named target:

Command	What
./run.sh or ./run.sh --local	Local Minio + synthetic data. No credentials. :8080
./run.sh --aws	S3 staging bucket on AWS S3. Fill in server/deploy/config.aws-staging.yaml; creds: AWS_PROFILE (defaults to aws-staging). :8084
./run.sh --gcs	GCS staging bucket. Creds: Application Default Credentials. (fill in server/deploy/config.gcs-staging.yaml first) :8085
./run.sh <path/to.yaml>	Any S3/GCS server config file.

One backend (builder + server) runs at a time — make server-stop to switch targets.

Port already in use? run.sh hard-errors (rather than silently misbehaving) if the target port is held by another process — :8080 is commonly taken on dev boxes. Use ./run.sh <path/to.yaml> with a config whose listen: is a free port.

Auth: run.sh always passes -allow-anonymous, so local development needs no token. If you also set PJ_CLOUD_TOKEN, the server still enforces it (the flag only disables auth when no token is configured). Outside the launcher the server is fail-closed: it refuses to start without a token unless -allow-anonymous / PJ_CLOUD_ALLOW_ANONYMOUS=1 explicitly permits open access. See server/deploy/README.md and docs/ec2-deploy.md.

Connecting to a remote server (wss://)

Against a server behind a TLS front-end (nginx/ALB, Tailscale Serve, Cloudflare) with a publicly-trusted certificate, a client needs only the URL and the token — no certificate configuration:

export MCAP_CLOUD_API_KEY=<token>
plugin/toolbox_mcap_cloud/build/bin/mcap-cloud-cli \
  --url wss://recordings.example.com list

• Give the host only — no port and no path. The client appends the WebSocket path (/api/ws) itself, and a TLS front-end normally serves on 443.
• The system CA bundle is auto-detected. Override with SSL_CERT_FILE, or pass --cert FILE / MCAP_CLOUD_CACERT for a private CA.
• For a self-signed cert, prefer --cert with the CA over --insecure.
• In the GUI plugin the same settings live behind the connect row's "Cert / API Key…" button; leave the certificate field EMPTY for auto-detect.

After connecting, the PlotJuggler plugin asks for a customer and site before loading any recordings -- the catalog can hold tens of thousands of files. The selection is remembered per server, so reconnecting lands back on the last site. mcap-cloud-cli list stays unfiltered.

See plugin/toolbox_mcap_cloud/README.md for the full flag/environment table, the certificate-discovery order, and the gated browse-flow details.

Shareable layouts that re-download their data

A PlotJuggler layout saved from a cloud fetch can embed how to obtain that session again — server, MCAP keys, topic subset, time window — so opening the layout (in the GUI or with plotjuggler --layout, no prompts) re-creates the exact session: instantly from a local cache artifact when present, otherwise by re-downloading while the plots grow. No credentials are ever embedded; each machine resolves its own, and an origin must be connected to once interactively before a layout may import from it.

• docs/layout-sharing-runbook.md — the user/ops guide: what a shared layout does and does not contain (including the metadata/path-leakage warning read this before sharing), the one-time trust bootstrap, the cache and how to purge it, the headless flow with its exit codes and diagnostic-id table, and how to run the gate.
• docs/layout-import-architecture.md — the engineering reference: cross-repo component map, runtime flows, and the invariants any change must preserve.

Regression gates

Run the full regression gate: make smoke. It generates and seeds its own synthetic corpus; see scripts/RUNBOOK.md for its additional tooling prerequisites. make e2e-layout additionally proves the layout-import stack cross-repo against a real PlotJuggler build (see the runbook above for its prerequisites).