Skip to content

feat(docs): add Slack reporting for interesting docs bot prompts#1091

Open
sarinali wants to merge 3 commits intotrycua:mainfrom
sarinali:slack-docs-prompts
Open

feat(docs): add Slack reporting for interesting docs bot prompts#1091
sarinali wants to merge 3 commits intotrycua:mainfrom
sarinali:slack-docs-prompts

Conversation

@sarinali
Copy link
Contributor

@sarinali sarinali commented Feb 17, 2026

Summary

  • Analyzes each docs bot prompt with Haiku to score on actionability, traction signal, and use case insight
  • Posts only interesting prompts (score >= 7/10 on any criterion) to a Slack channel via incoming webhook
  • Fire-and-forget async pattern — zero impact on user response latency
  • Also fixes the pre-commit tsc hook that was failing due to lockfile rewrites

Changes

  • docs/src/lib/prompt-analyzer.ts — Haiku classification call with threshold filtering
  • docs/src/lib/slack-reporter.ts — Slack Block Kit message formatting and webhook posting
  • docs/src/app/api/copilotkit/route.ts — async hook after PostHog capture, hoisted categorization
  • docs/.env.example — added SLACK_WEBHOOK_URL
  • scripts/typescript-typecheck.js — restore lockfile after install to fix pre-commit

Test plan

  • Set SLACK_WEBHOOK_URL env var in Vercel/local
  • Send a specific prompt (e.g., "How do I set up Cua for automated QA testing in our fintech pipeline?") and verify Slack post appears
  • Send generic prompts ("what is cua?", "hello") and verify they do NOT post to Slack
  • Verify bot response latency is unaffected
  • Verify no errors when SLACK_WEBHOOK_URL is unset

🤖 Generated with Claude Code

Analyze each user prompt with Haiku to score on actionability,
traction signal, and use case insight. Post interesting prompts
(score >= 7/10) to a Slack channel via incoming webhook.

- prompt-analyzer.ts: Haiku classification, threshold filtering
- slack-reporter.ts: Slack Block Kit formatting, webhook posting
- route.ts: fire-and-forget async hook, hoisted categorization
- .env.example: added SLACK_WEBHOOK_URL
- typescript-typecheck.js: fix lockfile restore to prevent
  pre-commit "files were modified" false failure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link

coderabbitai bot commented Feb 17, 2026

Important

Review skipped

Auto reviews are limited based on label configuration.

🏷️ Required labels (at least one) (1)
  • rabbit

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

@vercel
Copy link
Contributor

vercel bot commented Feb 17, 2026

@sarinali is attempting to deploy a commit to the Cua Team on Vercel.

A member of the Team first needs to authorize it.

@vercel
Copy link
Contributor

vercel bot commented Feb 17, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Feb 17, 2026 5:52pm

Request Review

sarinali and others added 2 commits February 18, 2026 01:29
Replace the inline fire-and-forget approach with a Vercel cron job
that queries PostHog as a derived data source. This decouples
prompt reporting from the request path and makes the data source
swappable.

- Revert inline hook from copilotkit route.ts
- Refactor prompt-analyzer.ts for batch analysis (one Haiku call)
- Refactor slack-reporter.ts for batch digest format
- Add posthog-query.ts to fetch events via PostHog API
- Add cron route at /api/cron/prompt-digest (hourly)
- Add vercel.json with cron schedule
- Add POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID, CRON_SECRET
  to .env.example

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-derive API host from NEXT_PUBLIC_POSTHOG_HOST by stripping
  the ingestion subdomain (eu.i. → eu., us.i. → us.)
- Add ?hours=N query param to cron endpoint for testing
- Add debug logging for PostHog query URL and error bodies

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant