Feedback Aggregation & Clustering System #2534

Description/Context

Free-text feedback about our platforms and courses is scattered across systems that never talk to each other: Zendesk support tickets, edX discussion-forum posts, Learn AI tutor conversations, Open edX ORA peer-assessment feedback, and a forthcoming edX feedback plugin. Nobody can answer "what are the recurring problems people hit this month?" without reading tickets one at a time, and nothing distinguishes a one-off complaint from a systemic issue.

Four audiences feel this gap at three altitudes:

Design review: RFC #12210 (Draft — supersedes #10793)
Spec set: #2422 — README_feedback_aggregation.md is the index

Plan/Design

Five decisions are settled and captured in the RFC and spec set:

1. Ingress — contract-first (RFC Option 3). Land feedback in the Learn AI Postgres DB as an interim path using the existing Airbyte → dbt scaffolding, but define a stable common event contract now and key the warehouse tables off stable source business identifiers, so the later cut over to the ol-analytics-api/StarRocks data bus is source-swap + backfill + parity rather than a rebuild. Building the data bus first was rejected on schedule risk — its write path does not exist and its sink topology is an unsettled platform-wide question.

2. A fact pair, not one table.

   • tfact_feedback — the ingested record, one row per human free-text utterance at turn grain (Zendesk public requester-authored comment, forum *.created post, tutor human turn, ORA submission, plugin event). Sparse-nullable conformed FKs are the source-flexibility mechanism. Insert-only — no post-insert write path exists.
   • afact_feedback_conversation — the analysis unit, one row per conversation, carrying lifecycle attributes and everything model-generated: the LLM summary, the embedding vector, category_fk, sentiment_fk, cluster membership, with version stamps. Fully rebuildable without touching the turn fact.

   The conversation is the analysis unit because a complaint usually emerges over several turns; embedding turns independently splits one issue into several weak cluster members and scores sentiment off a fragment.

3. Conformance rule — an attribute earns a column on the fact or a conformed dimension only if two or more sources can populate it. Everything else rides in a source_metadata variant and is promoted when a second source needs it. This is what stops Zendesk's shape from becoming the model's shape.

4. ML — engine-external compute via Fenic in a Dagster asset (portable across Trino→StarRocks, no Galaxy-only AI SQL functions), our own sklearn UMAP+HDBSCAN for clustering (its noise class is the one-off bucket), embedding model chosen by measured task effectiveness on a labeled Zendesk sample. Presidio redaction mandatory pre-embed.

5. Consumption — Superset for MVP dashboards (RLS already solved as code), a deployed Marimo notebook-as-webapp for cluster exploration and the category-curation loop, a net-new app only if write-back justifies it.

Phasing. Phase 1 = Zendesk-only batch MVP for support + engineering (~198K conversations). Phase 2 = forum/tutor/ORA as additive CTEs, the cross-source daily rollup for leadership, instructor course views. Phase 3 = migrate ingress to the data bus.

Known cost exposure. The per-conversation LLM summary is the one per-record model call in the design — roughly low hundreds of dollars one-time for Zendesk against $2–16 for the entire embedding backfill. It is skipped for single-turn and short conversations, and the multi-turn share must be sample-measured before the backfill.

Gate. The RFC is still Draft and the spec PR is unmerged. Those reviews are deciding the fact's grain and business key, so the build should not start until both land.

Build order

The sub-issue panel does not convey dependencies, so they are stated here. The principle is that the warehouse layer ships and is useful before any ML — every model-generated column is nullable, and the ML asset never writes to tfact_feedback.

Prerequisites — independent of each other, can run in parallel:

• Carry comment_author_user_id through int__zendesk__ticket_comment #2535 comment_author_user_id — hard blocker for Build the dbt warehouse layer: tfact_feedback + afact_feedback_conversation + dims #2540; nothing about the turn grain works without it
• Measure Zendesk requester-comment volume and multi-turn share #2536 volume + multi-turn measurement — blocks Conversation summarization asset + skip rule (the one per-record LLM cost) #2542 (sizes the LLM budget)
• Run the PII profiler over comment_plain_body before the first embed run #2537 PII profiling of comment_plain_body — blocks the first embed/summarize run
• Confirm the conformed channel_slug value set against each source's real channel values #2538 channel_slug value set — needed before seeding dim_feedback_channel
• Add the ticket_metrics stream to the Zendesk Airbyte connector #2539 ticket_metrics stream — non-blocking; unblocks the conversation duration measures whenever it lands

Warehouse — #2540, blocked by #2535. Ships a queryable fact pair with tag-seeded categories and CSAT-derived sentiment, no ML required.

Redaction — #2541 stands up the ML code location and builds the Presidio feedback_redacted asset. It is Phase 1 and not blocked by #2540: the fact cannot carry text until redaction runs. #2540 ships text-less until it lands.

ML pipeline — #2542 (summarize, also blocked by #2536 and #2537) → #2543 (embed) → #2544 (cluster, categorize, score sentiment). Blocked by #2540 and by the redaction asset in #2541.

Consumption and governance — #2545 and #2546, both blocked by #2540. #2546 should land before real users get access, not after.

Phase 3 — #2547 is parked on platform work that does not exist yet. It is listed for completeness; do not schedule it.