Skip to content

feat: Create Notes DB schema, indexes, and RLS migration - #169

Merged
FrkAk merged 17 commits into
mainfrom
feat/pyz-248-create-notes-db-schema-indexes-and-rls
Jul 3, 2026
Merged

feat: Create Notes DB schema, indexes, and RLS migration#169
FrkAk merged 17 commits into
mainfrom
feat/pyz-248-create-notes-db-schema-indexes-and-rls

Conversation

@FrkAk

@FrkAk FrkAk commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Summary

Task Reference: [PYZ-248]

Adds the Notes relational foundation: four tables in lib/db/schema.ts (notes, note_task_links, note_links, note_revisions), their TypeScript enums in lib/types.ts, and the RLS surface in docker/rls-functions.sql, docker/rls-policies.sql, and docker/grants.sql.

  • notes carries the §4.1 column set (visibility, agent_writable/locked, feed_mode/feed_categories/feed_tags/feed_task_ids, version, embedding_status, share_requested_by, soft-delete deleted_at) plus a weighted FTS generated column search_tsv (setweight(left(title, 2000), 'A') || setweight(left(body, 131072), 'B'), coalesced, STORED). The length caps are exported schema constants inlined via sql.raw (NOTE_TITLE_MAX_BYTES=2000, NOTE_BODY_MAX_CHARS=200000, NOTE_SEARCH_INDEXED_CHARS=131072) so the notes CHECKs, the tsvector bounds, and the note_revisions CHECKs cannot desync. The search bound is 131072, not 200000, because a char-count CHECK cannot bound tsvector bytes: worst-case content (space-separated distinct hyphenated multibyte compounds, three lexemes per token) costs ~7.5 tsvector bytes per char and crosses Postgres's 1,048,575-byte tsvector ceiling at ~163k chars (measured on postgres:18); 131072 body chars + a max title lands at ~998 KB worst case. Bodies keep the full 200k contract — search ignores chars past the bound. A pending share request is share_requested_by IS NOT NULL; there is no separate boolean to desync, and the FK's ON DELETE SET NULL auto-cancels a request whose requester was deleted. CHECK constraints pin visibility, type, feed_mode, and embedding_status to their value sets and bound title/slug to ≤ 2000 bytes (btree-safe) and body to ≤ 200000 chars; note_task_links.kind and note_links (no self-link) are likewise constrained.
  • Indexes: GIN notes_search_idx / notes_tags_idx; partial notes_feed_idx (feed_mode <> 'none'); partial slug-unique and title indexes (deleted_at IS NULL); embedding-queue partial index; ETag index (project_id, updated_at). Link and revision tables index the FK directions a cascade or backlink query needs, without indexes a unique constraint already covers.
  • RLS: notes_member_access (2-hop via projects, plus per-note visibility='team' OR created_by = (SELECT current_app_user_id()) — the caller lookup is (SELECT ...)-wrapped so it evaluates once per statement as an InitPlan, matching the file's current_user_org_ids() discipline). The 3-hop link/revision policies check their endpoints with correlated EXISTS pkey probes instead of IN (SELECT id FROM public.notes): every policy clause carrying an IN-sublist plans its own hashed SubPlan over the caller's entire visible-notes set, so a single-row link write evaluated that set up to six times (EXPLAIN-measured 327 vs 33 buffer hits at 5k notes); EXISTS keeps identical RLS-filtered semantics at one index probe per check. Both endpoints checked for note_task_links and note_links; RESTRICTIVE write floors on both link tables; a notes_insert_author_only floor pinning created_by (strict) and updated_by/share_requested_by (self-or-NULL) on INSERT; and a note_revisions_insert_author_only floor re-pinning snapshot authorship. New current_app_user_id() helper (plpgsql STABLE SECURITY INVOKER). Defense-in-depth triggers mirroring the tasks/task_edges suite: project_id and created_by immutability on notes (the created_by trigger rejects reassignment and non-author null-erasure while allowing the ON DELETE SET NULL cascade), an updated_by/share_requested_by attribution pin, and SECURITY INVOKER cross-project rejection on both link tables (an invisible note reads back like a nonexistent one, so no SQLSTATE existence oracle). The three notes triggers are AFTER-row, not BEFORE: any BEFORE UPDATE row trigger on notes forces the STORED search_tsv recompute on every metadata-only UPDATE, while AFTER keeps the executor's skip (verified on postgres:18: 316ms vs 7ms). note_revisions is append-only for app_user (UPDATE revoked in grants.sql, table-existence-guarded so container-init stays safe).
  • Deploy sequencing is orderable and machine-verified. The notes-family trigger DDL in rls-functions.sql is guarded on to_regclass, so db:rls:owner (owner-managed, applied out-of-band — the CI migrator role has no piyaz_auth access) can run before migration 0003 and installs current_app_user_id() ahead of the deploy's db:rls:ci. apply-public-rls.ts preflights every function rls-policies.sql references and fails with the owner-apply runbook instead of a bare 42883 mid-transaction rollback. verify-rls.ts now also asserts every trigger declared in rls-functions.sql exists on the deploy DB and that note_revisions UPDATE stays revoked, so a guarded trigger skip or grant drift blocks the deploy with an actionable message.

The Drizzle migration drizzle/0003_amusing_micromacro.sql is committed and applies cleanly across the 0000 to 0003 chain, verified on a throwaway DB in the real deploy order (0000–0002 → owner apply → 0003 → owner apply → grants/policies/storage), with all five notes triggers installed and the UPDATE revoke in place. The reviewable source is lib/db/schema.ts and docker/*.sql.

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation

Testing

  • Tested locally with bun run dev
  • Linting passes (bun run lint)
  • Typecheck passes (bun run typecheck)

tests/data/notes-rls.test.ts covers private/team visibility, cross-org isolation, GIN and partial index presence, the search_tsv STORED generated column, FK cascade on note and task delete (both link directions), the cross-project rejection triggers (asserted by SQLSTATE 23514, not RLS), created_by immutability against privatize-and-steal and against null-erasure by a non-author (42501) with the author-delete FK cascade still nulling it, the updated_by/share_requested_by attribution pin (forge, erase, and forged INSERT rejected as 42501; self-claims and share-request clears allowed), the note_revisions UPDATE revoke (42501), project_id immutability, a same-project link insert under FORCE RLS, trashed-slug reuse, the title/slug/body length CHECKs (oversize writes fail as 23514, not opaque btree/tsvector errors), and a tsvector overflow regression test: a full 200k-char body of distinct hyphenated CJK compounds — content that overflows an unbounded tsvector at ~163k chars — saves under the left(body, 131072) bound. tests/db/rls-coverage.test.ts auto-covers the four tables. bun run test:db: 501 pass.

Notes for reviewer

  • Deploy ordering: db:rls:owner is safe to run before this PR's migration (the notes trigger DDL self-guards on table existence) and should be run once before or right after merge; the deploy's db:rls:ci preflight and db:rls:verify trigger/grant assertions turn any missed step into an actionable failure instead of a raw SQLSTATE.
  • lz4 TOAST compression on body and search_tsv lives in docker/storage.sql (Drizzle has no compression API), with STORAGE EXTENDED; it is applied post-migrate by apply-public-rls.ts and mirrored into the push-based test container by the test harness, and a deploy-time pg_attribute.attcompression check in scripts/verify-rls.ts guards it.
  • Access level (agent_writable / locked) and visibility transitions are enforced at the call surface (PYZ-249, PYZ-252), not RLS. The append-only DELETE retention cap for note_revisions lands with PYZ-249.
  • search_tsv is server-side only (GIN-matched, up to hundreds of KB per row): the PYZ-249 data-access layer must read notes through explicit column projections that exclude it — a bare select() ships it over the wire on every row (flagged in a comment on the column).
  • PYZ-251 contract change: request_share sets only share_requested_by (the pending_share_request boolean was dropped as derivable, desync-prone state).
  • task_edges keeps the inherited IN (SELECT ...) policy shape; converting it to the EXISTS form is a follow-up outside this PR's scope.

Docs impact

none

@FrkAk FrkAk self-assigned this Jun 30, 2026
@FrkAk
FrkAk force-pushed the feat/pyz-248-create-notes-db-schema-indexes-and-rls branch from d586230 to 1ee018d Compare June 30, 2026 20:03
@FrkAk
FrkAk merged commit ec14217 into main Jul 3, 2026
5 checks passed
@FrkAk
FrkAk deleted the feat/pyz-248-create-notes-db-schema-indexes-and-rls branch July 3, 2026 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants