Infra setup — hotpath schema on Lakebase

Sets up the hotpath_abcdef schema in the rct-demo Lakebase Postgres project (production branch, databricks_postgres database) and loads simulated "current" content for the voice agent's hot-path topics.

Tables

Table	Content
hotpath_abcdef.hotpath_bollywood	Current Bollywood news
hotpath_abcdef.hotpath_astrology	Today's outlook for all 12 star signs
hotpath_abcdef.hotpath_cricket	Current cricket news
hotpath_abcdef.hotpath_devotion	Devotional / spiritual updates
hotpath_abcdef.hotpath_entertainment	Entertainment news (OTT, music, TV)
hotpath_abcdef.hotpath_finance	Markets and personal finance news

News tables share one shape (headline, summary, tags, source, published_at) with an index on published_at DESC; astrology is one row per sign per outlook_date (unique on sign, outlook_date).

All content is simulated (see simulated_data.py) — fictional people, films, matches, and market levels for demo purposes only.

Run

uv run infra/setup_hotpath.py

• Dependencies (psycopg, python-dotenv) are declared inline (PEP 723); uv resolves them in an isolated env — nothing is added to the project's deps.
• Auth comes from .env.local at the repo root (DATABRICKS_HOST, DATABRICKS_TOKEN); the script uses the databricks CLI to mint a short-lived Postgres OAuth credential.
• Idempotent: DDL is IF NOT EXISTS; data is truncated and re-seeded on each run, so re-running refreshes published_at / outlook_date to "now".

Override targets via env vars if needed: LAKEBASE_ENDPOINT (default projects/rct-demo/branches/production/endpoints/primary) and LAKEBASE_DATABASE (default databricks_postgres).

Query it

databricks psql --project rct-demo -- \
  -c "SELECT headline FROM hotpath_abcdef.hotpath_bollywood ORDER BY published_at DESC LIMIT 5;"

Search on the hot path

The 5 news tables carry a generated search_tsv column (Postgres computes the stemmed token list on every insert) and a Lakebase Search BM25 index ({table}_bm25_idx, lakebase_bm25 type from the lakebase_text extension). Query pattern (lower score = more relevant):

SELECT headline, search_tsv <@> to_bm25query(
         to_tsvector('english', 'RBI repo rate'), 'hotpath_abcdef.hotpath_finance_bm25_idx') AS score
FROM hotpath_abcdef.hotpath_finance ORDER BY score ASC NULLS LAST LIMIT 5;

BM25 corpus statistics are computed at index build time, so this script rebuilds the indexes after every reseed. lakebase_vector is installed but unused — it's the hybrid (semantic) upgrade path.

Why this scales with concurrency (talk track)

1. Storage/compute separation — data + search indexes live once in shared storage; endpoints scale horizontally without copying data or rebuilding indexes.
2. Read/write split — content pipelines write to the primary; the agent fleet reads from read-only endpoints (each autoscaling 0.5–32 CU) or HA -ro secondaries.
3. Connection math — per-endpoint limits scale with CU (104 @ 0.5 CU → 4,000 cap @ 32 CU); stateless agent workers hold 2–4-connection pools; when a fleet outgrows one endpoint, add read replicas — capacity multiplies linearly.
4. Search inside Postgres — no separate search cluster to shard or sync; search QPS rides the same read replicas; lakebase_ann handles 1B+ vectors per index and survives scale-to-zero without warmup.
5. BM25 rebuild cadence — build-time corpus stats mean streaming content needs periodic REINDEX; this refresh script models that.
6. Data API — PostgREST HTTP for connectionless fan-out at extreme client counts.
7. Always-warm hot path — the production branch has scale-to-zero disabled.
8. Token lifecycle — long-lived services refresh the 1h OAuth credential every 30–40 min via a pool creator callback (short voice sessions don't need it).