feat(platform): add brain dump STT quality gate and eval corpus - #13792
Conversation
github-project-automation
Bot
moved this to 🆕 Needs initial review
in AutoGPT development kanban
|
/review |
github-actions
Bot
added
platform/frontend
github-actions
Bot
added
cla: pending
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR adds transcript-quality validation, evaluation-corpus tooling, insufficient-content recovery, shared response parsing, and onboarding analytics. It also adds quality-gate fixtures, reports, tests, and documentation. ChangesOnboarding brain-dump flow
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant BrainDumpUI
participant OnboardingAPI
participant QualityGate
participant ExtractionPipeline
User->>BrainDumpUI: submit voice or typed brain dump
BrainDumpUI->>OnboardingAPI: finalize transcript and audio
OnboardingAPI->>QualityGate: check transcript quality
QualityGate-->>OnboardingAPI: pass or quality error
OnboardingAPI->>ExtractionPipeline: queue extraction when quality passes
OnboardingAPI-->>BrainDumpUI: accepted or insufficient result
BrainDumpUI-->>User: continue, retry, type, or skip
Possibly related PRs
Suggested labels: Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🔍 PR Overlap DetectionThis check compares your PR against all other open PRs targeting the same branch to detect potential merge conflicts early. 🟢 Low Risk — File Overlap OnlyThese PRs touch the same files but different sections (click to expand)
Summary: 0 conflict(s), 0 medium risk, 4 low risk (out of 4 PRs with file overlap) Auto-generated on push. Ignores: |
github-actions
Bot
added
cla: signed
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (3)
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx (1)
6-11: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the component-directory props declaration.
Replace
interface Propswith a non-exportedtype Props = { ... }.As per coding guidelines,
autogpt_platform/frontend/src/**/components/**/*.{tsx,jsx}requirestype Props = { ... }for component props in component directories.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@autogpt_platform/frontend/src/app/`(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx around lines 6 - 11, Update the InsufficientState component’s Props declaration from an interface to a non-exported type alias, preserving the existing canRecord, onRecordAgain, onTypeInstead, and onSkip fields unchanged.Source: Coding guidelines
autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py (1)
209-330: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoffSplit media processing from corpus planning.
This file is 334 lines. Extract the FFmpeg encoding, probing, RMS, and mixing helpers into a dedicated module. Keep corpus download and clip planning in this file.
As per coding guidelines, “Keep files under ~300 lines; if a file grows beyond this, split by responsibility.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py` around lines 209 - 330, Extract the FFmpeg-specific helpers—encode_opus, concat_utterances, mix_noise, build_garbage_clip, probe_duration, measure_rms_db, _ffmpeg, and _ffmpeg_stderr—into a dedicated media-processing module. Update build_clip and any other callers in eval_corpus.py to import and use those helpers, leaving corpus download and clip-planning logic in eval_corpus.py.Source: Coding guidelines
autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx (1)
240-247: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse named handler functions for both new click paths.
Both changed handlers use multi-statement arrow callbacks. Extract named function declarations or child components and pass named handlers to the buttons.
autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx#L240-L247: extract the prompt selection handler.autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/OnboardingWelcomeDialog.tsx#L251-L256: extract the connect-tools handler.As per coding guidelines, use function declarations for React handlers and reserve arrow functions for small inline lambdas such as
mapandfilter.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@autogpt_platform/frontend/src/app/`(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx around lines 240 - 247, Replace the inline multi-statement click callbacks with named function-declaration handlers: extract the prompt selection logic around the IntroPromptCard onClick in autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx#L240-L247, and the connect-tools logic around the button onClick in autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/OnboardingWelcomeDialog.tsx#L251-L256. Pass these named handlers to the buttons while preserving their existing tracking, state updates, and action calls.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval.py`:
- Around line 109-114: Update evaluate_gate_only() to catch only expected
transcription and I/O failures when obtaining the transcript, preserving the
existing warning and empty-transcript fallback for those cases. Remove the broad
Exception catch so evaluator faults such as unexpected programming errors or
unsupported transcription failures propagate instead of being passed to
evaluate_transcript as an empty transcript.
- Around line 52-61: Update load_manifest to validate the parsed manifest with a
strict Pydantic dictionary validator, accepting only the exact “pass” and
“reject” values and rejecting invalid entries such as uppercase “REJECT”. Do not
filter invalid items; allow the validator’s validation errors to propagate to
the caller.
In
`@autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py`:
- Around line 326-330: Update the subprocess.run call in _ffmpeg_stderr to
explicitly pass check=False, preserving the helper’s behavior of returning
FFmpeg stderr even when the command fails.
- Around line 143-144: Replace the extractall call in the tarfile-open block
with explicit member validation and extraction: reject symlinks, hard links,
device files, FIFOs, absolute paths, and paths escaping root; extract only
regular files after validating their resolved destinations. Do not rely on the
filter argument, preserving compatibility with Python 3.10 and older 3.11
versions.
In `@autogpt_platform/backend/backend/api/features/onboarding_dump/quality.py`:
- Around line 164-168: Update the quality-check function around response parsing
to safely handle empty or malformed response.choices entries, including
choices=[], without raising; return INSUFFICIENT_CONTENT for these structural
failures while preserving the existing malformed-verdict handling. Add a test
covering an empty choices list.
In `@autogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.ts`:
- Around line 78-87: Update registerBrainDumpContext to use PostHog’s
register_for_session() API instead of register(), while preserving the existing
properties argument and silent error handling.
In `@autogpt_platform/frontend/src/services/onboarding/brain-dump-handoff.ts`:
- Around line 42-48: Update the shared setFlag and takeFlag helpers used by
setIntroPromptClicked and takeIntroPromptClicked to catch sessionStorage setItem
and getItem failures, returning the normal no-flag fallback so prompt submission
continues through the typed path. Add coverage that makes both storage methods
throw and verifies onSelectPrompt still submits the prompt.
---
Nitpick comments:
In
`@autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py`:
- Around line 209-330: Extract the FFmpeg-specific helpers—encode_opus,
concat_utterances, mix_noise, build_garbage_clip, probe_duration,
measure_rms_db, _ffmpeg, and _ffmpeg_stderr—into a dedicated media-processing
module. Update build_clip and any other callers in eval_corpus.py to import and
use those helpers, leaving corpus download and clip-planning logic in
eval_corpus.py.
In
`@autogpt_platform/frontend/src/app/`(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx:
- Around line 6-11: Update the InsufficientState component’s Props declaration
from an interface to a non-exported type alias, preserving the existing
canRecord, onRecordAgain, onTypeInstead, and onSkip fields unchanged.
In
`@autogpt_platform/frontend/src/app/`(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx:
- Around line 240-247: Replace the inline multi-statement click callbacks with
named function-declaration handlers: extract the prompt selection logic around
the IntroPromptCard onClick in
autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsx#L240-L247,
and the connect-tools logic around the button onClick in
autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/OnboardingWelcomeDialog.tsx#L251-L256.
Pass these named handlers to the buttons while preserving their existing
tracking, state updates, and action calls.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: caa77d8c-6624-429a-ac2a-da280fb04477
⛔ Files ignored due to path filters (29)
autogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/gate-reject-brown-noise.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/gate-reject-pink-noise.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/gate-reject-silence.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/gate-reject-white-noise.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-01.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-02.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-03.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-04.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-05.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-06.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-07.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-08.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-09.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-10.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-11.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-12.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-13.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-01.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-02.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-03.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-04.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-05.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-06.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-01.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-02.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-03.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-04.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-05.oggis excluded by!**/*.oggautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-06.oggis excluded by!**/*.ogg
📒 Files selected for processing (58)
autogpt_platform/backend/backend/api/features/onboarding_dump/EVAL.mdautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_eval.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/gate_manifest.jsonautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-01.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-02.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-03.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-04.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-05.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-06.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-07.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-08.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-09.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-10.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-11.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-12.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-clean-13.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-01.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-02.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-03.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-04.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-05.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-loud-noise-5db-06.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-01.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-02.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-03.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-04.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-05.txtautogpt_platform/backend/backend/api/features/onboarding_dump/eval_data/stt-wer-moderate-noise-15db-06.txtautogpt_platform/backend/backend/api/features/onboarding_dump/quality.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/routes_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/service.pyautogpt_platform/backend/backend/api/features/onboarding_dump/service_test.pyautogpt_platform/frontend/src/app/(no-navbar)/onboarding/__tests__/brain-dump.test.tsxautogpt_platform/frontend/src/app/(no-navbar)/onboarding/helpers.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/BrainDumpStep.tsxautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/__tests__/helpers.test.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/__tests__/useBrainDumpStep.test.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsxautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/helpers.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpRecorder.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/useOnboardingPage.tsautogpt_platform/frontend/src/app/(platform)/copilot/components/ChatInput/useOnboardingMicGlow.tsautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/OnboardingIntroCard.tsxautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/__tests__/useOnboardingIntroCard.test.tsxautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/useOnboardingIntroCard.tsautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/OnboardingWelcomeDialog.tsxautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/__tests__/welcome-dialog.test.tsxautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingWelcomeDialog/useConnectToolsPanel.tsautogpt_platform/frontend/src/app/(platform)/copilot/useCopilotPage.tsautogpt_platform/frontend/src/providers/posthog/posthog-provider.tsxautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.tsautogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.tsautogpt_platform/frontend/src/services/onboarding/brain-dump-handoff.ts
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #13792 +/- ##
==========================================
+ Coverage 77.64% 77.73% +0.08%
==========================================
Files 2859 2876 +17
Lines 217035 217614 +579
Branches 20673 20736 +63
==========================================
+ Hits 168513 169156 +643
+ Misses 44011 43936 -75
- Partials 4511 4522 +11
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…val harnesses from coverage
github-actions
Bot
added
cla: pending
…g, eval tooling, and analytics scoping
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@autogpt_platform/backend/backend/api/features/onboarding_dump/quality.py`:
- Around line 161-163: Update the exception handling in the quality-check
function around the existing logger.warning fallback: replace the broad
Exception catch with only the expected timeout, response-shape, and permitted
OpenAI-style provider exception types used by this client path. Preserve the
warning and INSUFFICIENT_CONTENT return for those handled exceptions, while
allowing unexpected application errors to propagate.
In
`@autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.ts`:
- Around line 194-196: Update the storage spies in the onboarding handoff tests
to target window.sessionStorage.setItem and window.sessionStorage.getItem
directly instead of Storage.prototype, including the related setup at the other
referenced case. Keep the assertions scoped to sessionStorage so localStorage
usage cannot satisfy the tests.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 42a46cea-5d89-4df5-8ad3-b7a154c159d2
📒 Files selected for processing (10)
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.pyautogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/__tests__/useOnboardingIntroCard.test.tsxautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.tsautogpt_platform/frontend/src/services/onboarding/brain-dump-handoff.ts
🚧 Files skipped from review as they are similar to previous changes (5)
- autogpt_platform/frontend/src/app/(platform)/copilot/components/OnboardingIntroCard/tests/useOnboardingIntroCard.test.tsx
- autogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.ts
- autogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
- autogpt_platform/frontend/src/services/onboarding/brain-dump-handoff.ts
- autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval.py
📜 Review details
⏰ Context from checks skipped due to timeout. (15)
- GitHub Check: check API types
- GitHub Check: integration_test
- GitHub Check: lint
- GitHub Check: end-to-end tests
- GitHub Check: type-check (3.12)
- GitHub Check: lint
- GitHub Check: type-check (3.11)
- GitHub Check: test (3.12)
- GitHub Check: test (3.13)
- GitHub Check: test (3.11)
- GitHub Check: type-check (3.13)
- GitHub Check: Seer Code Review
- GitHub Check: Check PR Status
- GitHub Check: Analyze (typescript)
- GitHub Check: Analyze (python)
🧰 Additional context used
📓 Path-based instructions (15)
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}: Use Node.js 21+ with pnpm package manager for frontend development
Always run 'pnpm format' for formatting and linting code in frontend developmentFormat frontend code using
pnpm format
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}: Fully capitalize acronyms in symbols, e.g.graphID,useBackendAPI
No linter suppressors (//@ts-ignore``,// eslint-disable) — fix the actual issue
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/**/*.{tsx,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{tsx,ts}: Use function declarations for components and handlers (not arrow functions) in React components
Only use arrow functions for small inline lambdas (map, filter, etc.) in React components
Use PascalCase for component names and camelCase with 'use' prefix for hook names in React
Use Tailwind CSS utilities only for styling in frontend components
Use design system components from 'src/components/' (atoms, molecules, organisms) in frontend development
Never use 'src/components/legacy/' in frontend code
Only use Phosphor Icons (@phosphor-icons/react) for icons in frontend components
Use generated API hooks from '@/app/api/__generated__/endpoints/' instead of deprecated 'BackendAPI' or 'src/lib/autogpt-server-api/'
Use React Query for server state (via generated hooks) in frontend development
Default to client components ('use client') in Next.js; only use server components for SEO or extreme TTFB needs
Use '' component for rendering errors in frontend UI; use toast notifications for mutation errors; use 'Sentry.captureException()' for manual exceptions
Separate render logic from data/behavior in React components; keep comments minimal (code should be self-documenting)
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{ts,tsx}: No barrel files or 'index.ts' re-exports in frontend code
Regenerate API hooks with 'pnpm generate:api' after backend OpenAPI spec changes in frontend development
autogpt_platform/frontend/**/*.{ts,tsx}: Use function declarations (not arrow functions) for components/handlers
Noanytypes unless the value genuinely can be anything
Keep render functions and hooks under ~50 lines; extract named helpers or sub-components when they grow longer
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
autogpt_platform/frontend/src/**/*.{ts,tsx}: Use generated API hooks from@/app/api/__generated__/endpoints/following the patternuse{Method}{Version}{OperationName}, and regenerate withpnpm generate:api
Separate render logic from business logic using component.tsx + useComponent.ts + helpers.ts pattern, colocate state when possible and avoid creating large components, use sub-components in local/componentsfolder
Use function declarations for components and handlers, use arrow functions only for callbacks
Do not useuseCallbackoruseMemounless asked to optimise a given function
autogpt_platform/frontend/src/**/*.{ts,tsx}: Keep files under ~200 lines; extract sub-components or hooks into their own files when a file grows beyond this
Use generated API hooks from@/app/api/__generated__/endpoints/with patternuse{Method}{Version}{OperationName}
Always import the-Icon-suffixed alias from@phosphor-icons/react(e.g.TrashIcon,PlusIcon,SquareIcon) — bare exports are deprecated
Do not useuseCallbackoruseMemounless asked to optimize a given function
Never usesrc/components/__legacy__/*— use design system components fromsrc/components/
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
No barrel files or
index.tsre-exports in the frontend
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Do not type hook returns, let Typescript infer as much as possible
autogpt_platform/frontend/src/**/*.ts: Extract component logic into custom hooks grouped by concern, not by component, with each hook in its own.tsfile
Do not type hook returns; let TypeScript infer as much as possible
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never type with
any, if no types available useunknown
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
autogpt_platform/frontend/**/*.{test,spec}.{ts,tsx}: Use Vitest + RTL + MSW for integration tests as the primary testing approach (~90%, page-level), use Playwright for E2E critical flows, and use Storybook for design system components
Run frontend integration tests withpnpm test:unit(Vitest + RTL + MSW)
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/src/**/__tests__/**/*.{test,spec}.{ts,tsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
Use Orval-generated MSW handlers from
@/app/api/__generated__/endpoints/{tag}/{tag}.msw.tsfor API mocking
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/frontend/src/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
Avoid index and barrel files
Files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
autogpt_platform/backend/**/*.py: Usepoetry run ...command for executing Python package dependencies
Use top-level imports only — avoid local/inner imports except for lazy imports of heavy optional dependencies likeopenpyxl
Use absolute imports withfrom backend.module import ...for cross-package imports; single-dot relative imports are acceptable for sibling modules within the same package; avoid double-dot relative imports
Do not use duck typing — avoidhasattr/getattr/isinstancefor type dispatch; use typed interfaces/unions/protocols instead
Use Pydantic models over dataclass/namedtuple/dict for structured data
Do not use linter suppressors — no# type: ignore,# noqa,# pyright: ignore; fix the type/code instead
Prefer list comprehensions over manual loop-and-append patterns
Use early return with guard clauses first to avoid deep nesting
Use%sfor deferred interpolation indebuglog statements for efficiency; use f-strings elsewhere for readability (e.g.,logger.debug("Processing %s items", count)vslogger.info(f"Processing {count} items"))
Sanitize error paths by usingos.path.basename()in error messages to avoid leaking directory structure
Be aware of TOCTOU (Time-Of-Check-Time-Of-Use) issues — avoid check-then-act patterns for file access and credit charging
Usetransaction=Truefor Redis pipelines to ensure atomicity on multi-step operations
Usemax(0, value)guards for computed values that should never be negative
Keep files under ~300 lines; if a file grows beyond this, split by responsibility (extract helpers, models, or a sub-module into a new file)
Keep functions under ~40 lines; extract named helpers when a function grows longer
...
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
autogpt_platform/backend/backend/api/features/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Update routes in '/backend/backend/api/features/' and add/update Pydantic models in the same directory for API development
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
autogpt_platform/{backend,autogpt_libs}/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
autogpt_platform/backend/**/api/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/api/**/*.py: UseSecurity()instead ofDepends()for authentication dependencies to get proper OpenAPI security specification
Follow SSE (Server-Sent Events) protocol: usedata:lines for frontend-parsed events (must match Zod schema) and: commentlines for heartbeats/status
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/*_test.py: Use pytest with snapshot testing for API responses
Colocate test files with source files using*_test.pynaming convention
Mock at boundaries — mock where the symbol is used, not where it's defined; after refactoring, update mock targets to match new module paths
UseAsyncMockfromunittest.mockfor async functions in tests
When writing tests, use Test-Driven Development (TDD): write failing tests marked with@pytest.mark.xfailbefore implementation, then remove the marker once the implementation is complete
When creating snapshots in tests, usepoetry run pytest path/to/test.py --snapshot-update; always review snapshot changes withgit diffbefore committing
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.py
🧠 Learnings (17)
📚 Learning: 2026-04-01T18:54:16.035Z
Learnt from: Bentlybro
Repo: Significant-Gravitas/AutoGPT PR: 12633
File: autogpt_platform/frontend/src/app/(platform)/library/components/AgentFilterMenu/AgentFilterMenu.tsx:3-10
Timestamp: 2026-04-01T18:54:16.035Z
Learning: In the frontend, the legacy Select component at `@/components/__legacy__/ui/select` is an intentional, codebase-wide visual-consistency pattern. During code reviews, do not flag or block PRs merely for continuing to use this legacy Select. If a migration to the newer design-system Select is desired, bundle it into a single dedicated cleanup/migration PR that updates all Select usages together (e.g., avoid piecemeal replacements).
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-04-07T09:24:16.582Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 12686
File: autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/__tests__/PainPointsStep.test.tsx:1-19
Timestamp: 2026-04-07T09:24:16.582Z
Learning: In Significant-Gravitas/AutoGPT’s `autogpt_platform/frontend` (Vite + `vitejs/plugin-react` with the automatic JSX transform), do not flag usages of React types/components (e.g., `React.ReactNode`) in `.ts`/`.tsx` files as missing `React` imports. Since the React namespace is made available by the project’s TS/Vite setup, an explicit `import React from 'react'` or `import type { ReactNode } ...` is not required; only treat it as missing if typechecking (e.g., `pnpm types`) would actually fail.
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-07-28T15:32:54.931Z
Learnt from: Abhi1992002
Repo: Significant-Gravitas/AutoGPT PR: 13699
File: autogpt_platform/frontend/src/components/layout/Navbar/components/Wallet/components/WalletFullPanel.tsx:0-0
Timestamp: 2026-07-28T15:32:54.931Z
Learning: In AutoGPT's frontend (autogpt_platform/frontend), prefer importing the non-legacy ScrollArea component from `@/components/ui/scroll-area` over `@/components/__legacy__/ui/scroll-area` for new or migrated code. The non-legacy component is a drop-in superset: it preserves the legacy component’s props and additionally supports the optional `showScrollToTop` prop—so reviewers should flag new legacy imports unless there’s a specific, documented reason they can’t use the non-legacy version.
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-04-02T05:43:49.128Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 12640
File: autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/WelcomeStep.tsx:13-13
Timestamp: 2026-04-02T05:43:49.128Z
Learning: Do not flag `import { Question } from "phosphor-icons/react"` as an invalid import. `Question` is a valid named export from `phosphor-icons/react` (as reflected in the package’s generated `.d.ts` files and re-exports via `dist/index.d.ts`), so it should be treated as a supported named export during code reviews.
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-04-20T20:07:22.981Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 11235
File: autogpt_platform/frontend/src/app/(platform)/admin/diagnostics/__tests__/ExecutionsTable.test.tsx:27-76
Timestamp: 2026-04-20T20:07:22.981Z
Learning: In this codebase, Orval-generated API modules under `src/app/api/__generated__/` are not committed to git and must be generated via `pnpm generate:api` (requires a running backend). In integration tests, it’s acceptable—and expected—to stub generated hooks/modules by mocking them with `vi.mock("`@/app/api/__generated__/endpoints/`{tag}/{tag}")`. Do not treat `vi.mock` of these generated hook modules as a violation of the MSW handler guideline, since the corresponding MSW handlers cannot be imported at test time when generated files are absent.
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-08-06T15:47:58.674Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 13787
File: autogpt_platform/frontend/src/app/(platform)/team/[expertId]/components/ExpertAboutSection.tsx:5-5
Timestamp: 2026-08-06T15:47:58.674Z
Learning: Within autogpt_platform/frontend, use Hugeicons through the shared Icon atom at src/components/atoms/Icon/Icon.tsx. Pass Hugeicons-compatible IconSvgElement values because the atom renders HugeiconsIcon. Do not follow the stale root AGENTS.md Phosphor icon quick-reference guidance for this frontend.
Applied to files:
autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-03-05T15:42:08.207Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12297
File: .claude/skills/backend-check/SKILL.md:14-16
Timestamp: 2026-03-05T15:42:08.207Z
Learning: In Python files under autogpt_platform/backend (recursively), rely on poetry run format to perform formatting (Black + isort) and linting (ruff). Do not run poetry run lint as a separate step after poetry run format, since format already includes linting checks.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-03-16T16:35:40.236Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12440
File: autogpt_platform/backend/backend/api/features/workflow_import.py:54-63
Timestamp: 2026-03-16T16:35:40.236Z
Learning: Avoid using the word 'competitor' in public-facing identifiers and text. Use neutral naming for API paths, model names, function names, and UI text. Examples: rename 'CompetitorFormat' to 'SourcePlatform', 'convert_competitor_workflow' to 'convert_workflow', '/competitor-workflow' to '/workflow'. Apply this guideline to files under autogpt_platform/backend and autogpt_platform/frontend.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-03-31T15:37:38.626Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12623
File: autogpt_platform/backend/backend/copilot/tools/agent_generator/fixer.py:37-47
Timestamp: 2026-03-31T15:37:38.626Z
Learning: When validating/constructing Anthropic API model IDs in Significant-Gravitas/AutoGPT, allow the hyphen-separated Claude Opus 4.6 model ID `claude-opus-4-6` (it corresponds to `LlmModel.CLAUDE_4_6_OPUS` in `autogpt_platform/backend/backend/blocks/llm.py`). Do NOT require the dot-separated form in Anthropic contexts. Only OpenRouter routing variants should use the dot separator (e.g., `anthropic/claude-opus-4.6`); `claude-opus-4-6` should be treated as correct when passed to Anthropic, and flagged only if it’s used in the OpenRouter path where the dot form is expected.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-04-15T02:43:36.890Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12780
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:0-0
Timestamp: 2026-04-15T02:43:36.890Z
Learning: When reviewing Python exception handlers, do not flag `isinstance(e, X)` checks as dead/unreachable if the caught exception `X` is a subclass of the exception type being handled. For example, if `X` (e.g., `VirusScanError`) inherits from `ValueError` (directly or via an intermediate class) and it can be raised within an `except ValueError:` block, then `isinstance(e, X)` inside that handler is reachable and should not be treated as dead code.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-05-23T05:29:43.085Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 13200
File: autogpt_platform/backend/backend/executor/scheduler.py:590-593
Timestamp: 2026-05-23T05:29:43.085Z
Learning: When reviewing Python code that uses Pydantic discriminated/tagged unions (e.g., `Annotated[Union[...], Field(discriminator="kind")]`), recognize that using `isinstance(x, SomeVariantInfo)` to narrow the union is an intentional and correct runtime guard and should also enable static type narrowing in tools like Pyright. Do not recommend replacing such `isinstance`-based narrowing with `cast(...)` when the check already proves the variant at runtime.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-04-22T11:46:04.431Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/config.py:0-0
Timestamp: 2026-04-22T11:46:04.431Z
Learning: Do not flag the Claude Sonnet 4.6 model ID as incorrect when it uses the project’s established hyphenated convention: `anthropic/claude-sonnet-4-6`. This hyphen form is the intentional, production convention and should be treated as valid (including in files like llm.py, blocks tests, reasoning.py, `_is_anthropic_model` tests, and config defaults). Note that OpenRouter also accepts the dot variant `anthropic/claude-sonnet-4.6`, so either form may be tolerated, but `anthropic/claude-sonnet-4-6` should be considered the standard to match project usage.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-04-22T11:46:12.892Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/baseline/service.py:322-332
Timestamp: 2026-04-22T11:46:12.892Z
Learning: In this codebase (Significant-Gravitas/AutoGPT), OpenRouter-routed Anthropic model IDs should use the hyphen-separated convention (e.g., `anthropic/claude-sonnet-4-6`, `anthropic/claude-opus-4-6`). Although OpenRouter may accept both hyphen and dot variants, treat the hyphen-separated form as the intended, correct codebase-wide convention and do not flag it as an error. Only flag the dot-separated variant (e.g., `anthropic/claude-sonnet-4.6`) as incorrect when reviewing/validating model ID strings for OpenRouter-routed Anthropic models.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-05-07T18:48:14.242Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 13040
File: autogpt_platform/backend/backend/blocks/llm.py:0-0
Timestamp: 2026-05-07T18:48:14.242Z
Learning: In this repository, isort may split imports from the same module into separate blocks when some imports are aliased (e.g., `from module import X as Y`) and others are not. Preserve the two-block layout when it results from isort (such as keeping `from openai.types.chat import ChatCompletion as OpenAIChatCompletion` separate from non-aliased imports from `openai.types.chat`). Do not treat that split as a style issue during review; merging them into a single block can fail CI with `Imports are incorrectly sorted and/or formatted`.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-05-26T14:24:34.866Z
Learnt from: Abhi1992002
Repo: Significant-Gravitas/AutoGPT PR: 13217
File: autogpt_platform/backend/backend/api/features/search/service.py:137-137
Timestamp: 2026-05-26T14:24:34.866Z
Learning: In the Significant-Gravitas/AutoGPT backend, treat `user_id` (an opaque UUID used only for correlation/tracing) as non-PII. Do not flag direct logging of `user_id` in `logger.warning`/`logger.info` statements as a PII exposure issue, as the established convention is to log `user_id` for tracing while reserving PII for fields like email or display name.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
📚 Learning: 2026-06-11T19:39:10.493Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 13337
File: autogpt_platform/backend/backend/copilot/graphiti/reranker.py:0-0
Timestamp: 2026-06-11T19:39:10.493Z
Learning: In the Significant-Gravitas/AutoGPT Python backend, when calling the OpenAI Python client `chat.completions.create`, construct the `messages` payload using the concrete typed-dict variants from `openai.types.chat` (e.g., `ChatCompletionSystemMessageParam`, `ChatCompletionUserMessageParam`, etc.) rather than trying to instantiate `ChatCompletionMessageParam` directly. `ChatCompletionMessageParam` is a `Union` alias and is not constructible, so `ChatCompletionMessageParam(role=..., content=...)` should fail type checking. Build each message element with the appropriate concrete typed dict and then annotate the resulting list as `list[ChatCompletionMessageParam]` (e.g., `messages: list[ChatCompletionMessageParam] = [ChatCompletionSystemMessageParam(...), ...]`).
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
🪛 ast-grep (0.45.0)
autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py
[error] 345-350: Command coming from incoming request
Context: subprocess.run(
["ffmpeg", "-hide_banner", *args],
check=False,
capture_output=True,
text=True,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
🪛 Ruff (0.16.1)
autogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
[warning] 161-161: Do not catch blind exception: Exception
(BLE001)
🔇 Additional comments (7)
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.py (1)
14-14: LGTM!Also applies to: 51-58, 176-178
autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py (2)
144-145: LGTM!Also applies to: 148-164
347-350: LGTM!autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-analytics.test.ts (3)
4-17: LGTM!
57-71: LGTM!
20-54: 🗄️ Data Integrity & IntegrationVerify the brain-dump event-name migration on analytics consumers.
trackBrainDumpemits title-cased event names to PostHog. Check each consumer/dashboard that parses these names so it does not still filter by the old snake_case form.autogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.ts (1)
1-1: LGTM!Also applies to: 12-18
![autogpt-pr-reviewer[bot]](https://avatars.githubusercontent.com/u/130738209?s=60&v=4){kind=small}
There was a problem hiding this comment.
📋 Automated Review — PR #13792
PR #13792 — feat(platform): add brain dump STT quality gate and eval corpus
Author: Abhi1992002 | Files: 87
🎯 Verdict: APPROVE
PR Description Quality
✅ Has Why + What + How — the description explains the "transcription succeeding ≠ usable speech" motivation, the layered gate design, the recovery UI, and the eval harness. One checklist item ("Gate eval run against eval_data corpus") is unchecked because it requires LLM credentials — acceptable, but the committed corpus has not been demonstrated to run end-to-end.
What This PR Does
It inserts a quality gate between speech-to-text transcription and the personalized onboarding generation. Previously any transcript — silence, background noise, STT hallucinations, or filler — would flow straight into a personalized greeting/recommendations. The gate now rejects unusable transcripts with a deterministic fast-path (empty/looping/spaceless → no_usable_speech) or an LLM judge for the ambiguous middle (insufficient_content), retains the transcript+audio so the user can retry/type/skip, and falls the intro card back to a generic "Path B" greeting instead of one built from garbage. It also ships an offline eval harness (STT WER + gate accuracy) with a committed 29-file audio corpus.
Specialist Findings
🛡️ Security ✅ — Defensively designed: fail-closed gate (every judge failure → recoverable reject, quality.py:135-168), no injection into persistence (hardcoded error-code constants via Prisma), no new auth surface, no XSS (rejected transcript is never reflected into the recovery UI), and safe tar extraction (extractall(filter="data")).
🔵 Prompt injection can steer the LLM half of the gate (quality.py:154) — contained, since the gate is a soft quality filter, not a security boundary.
🏗️ Architecture ✅ — Clean separation (gate decides, service persists), sound layered fast-path, fail-safe defaults, no cycles.
🟠 Private cross-module import of intro._parse_response_json (quality.py:32) and a synchronous LLM call added to the finalize request budget (service.py:186).
⚡ Performance _TIMEOUT_SECONDS=15 ceiling (quality.py:47), on the same request that already blocked on transcription — the code defers extraction/greeting to background tasks precisely to dodge the 30s proxy timeout.
🟠 Inline gate latency vs. 30s proxy budget (service.py:186, service.py:273).
🧪 Testing ✅ — Genuinely thorough: layered gate, both service reject paths, route error-code surfacing, and the full frontend recovery flow all have meaningful assertions (deterministic tests even assert client_factory.assert_not_called() to prove the model was bypassed). 21 backend gate tests + 10 gate-eval tests pass.
🟠 No boundary-value tests at the threshold seams; no credential-free corpus-consistency check.
📖 Quality ✅ — Unusually clean: descriptive names, intent-explaining comments, follows frontend repo conventions. Minor duplication (rejection block, GATE_MANIFEST_NAME constant) and style nits.
📦 Product ✅ — All four claimed behaviors delivered and wired into both voice and typed finalize paths. Recovery flow is complete with a skip escape hatch.
🟠 The shared recovery screen is voice-centric copy/actions even for typed rejections, and lacks a screen-reader live-region announcement (InsufficientState.tsx:22).
📬 Discussion response.choices[0] is indexed outside the try/except (quality.py:164).
🔎 QA ✅ — Exercised the gate live via the typed finalize path (same check_transcript_quality as voice). Verified: substantial dump passes with no LLM call; looping/symbol transcripts get deterministic no_usable_speech with transcript retained (not deleted); intro card falls to Path B after rejection (generic "No rush…" greeting, no rejected content reflected) and Path A after accept; 401/422 negative cases all correct. Caveat: no LLM key in the sandbox, so the ambiguous-middle judge decision hit the documented no-client fail-safe rather than a real model verdict — that band and the InsufficientState screen (onboarding already complete) are covered only by the PR's own mocked tests.
QA Screenshots
| Screenshot | Description |
|---|---|
 |
Accepted dump → personalized "Hey, …" greeting + suggested prompts ✅ |
 |
Rejected dump → generic "No rush…" greeting, transcript-copy button correctly absent ✅ |
🟠 Should Fix
- Unguarded LLM response index breaks the fail-safe contract (
quality.py:164) —response.choices[0].message.contentis accessed outside the try/except. An OpenAI-compatible provider returning an emptychoiceslist would raise, turning a quality-gate check into a 500 instead of the module's documented recoverableinsufficient_contentreject. Move the access inside the try/except (or guard for emptychoices) and add achoices=[]test. (Flagged by: discussion/CodeRabbit — 1) - Inline LLM judge latency on the user-blocking finalize path (
service.py:186,service.py:273,quality.py:47) — the gate is awaited beforeFinalizeResponsewith a 15s timeout, on the same request that already blocked on transcription; the surrounding code defers other LLM work to background tasks specifically to avoid the 30s proxy timeout. Tighten_TIMEOUT_SECONDSto ~5-8s (ample for amax_tokens=100,temperature=0Haiku verdict) and confirm transcribe+gate stays comfortably under 30s. (Flagged by: performance, architect, security — 3 specialists) - Private cross-module import couples the gate to
introinternals (quality.py:32) — importing_parse_response_json, an underscore-private helper, breaks silently ifintrorefactors it. Promote it to a public shared helper consumed by both. (Flagged by: architect, quality — 2 specialists) - Missing boundary-value and corpus-consistency tests (
quality_test.py,eval_data/gate_manifest.json) — every gate test uses inputs far from the threshold edges, so a>=/>off-by-one (e.g. 39 vs 40 words routing wrong) would pass CI; and nothing in CI validates the committed corpus's internal consistency without LLM credentials. Add seam tests (39 vs 40 words, unique ratio straddling 0.3) and a pure-filesystem manifest↔file consistency check. (Flagged by: testing — 1) - Recovery screen is voice-centric and not announced to screen readers (
InsufficientState.tsx:22-25,BrainDumpStep.tsx:23) — "We didn't catch enough of that" / "We heard you" read as a mishearing to a user who typed, and "Record again" is primary even for typed rejections; the message also renders in a plaindivwith norole="alert"/aria-live. Pass the input mode through and add a live region. (Flagged by: product — 1)
🟡 Nice to Have
- Typed-retry re-runs the gate (
service.py:272) — retrying a rejected typed dump re-runs the gate becausestart_dumpresets status, unlike the voice path's claim-guard short-circuit. Add an idempotency check or note why it's intentional. (discussion/Seer) - Duplicated rejection block / constant (
service.py:190&:277,brain_dump_gate_eval.py:26) — extract a_quality_rejection(...)helper and single-sourceGATE_MANIFEST_NAME. (quality) - Analytics conflates STT failures with gate rejections (
useBrainDumpStep.ts:207) — quality rejections fire under "Brain Dump Transcription Failed"; a dedicated event would keep the funnel readable. (product) - Config read at import time (
quality.py:44) — routeBRAIN_DUMP_QUALITY_MODELthrough the platform settings mechanism instead ofos.environ. (architect)
🔵 Nits
- Superfluous parentheses (
quality.py:115) — inline as_compression_ratio(text) > MAX_COMPRESSION_RATIO. - Log interpolation style (
service.py:191,quality.py) — backend convention prefers f-strings for info/warning,%sonly for debug.
Human Review Needed
NO — This adds a quality gate inside an already-authenticated onboarding path; it introduces no new auth/authz surface, no credential/secret handling, and no cross-service trust boundary change. The security specialist explicitly confirmed no new auth surface. Standard domain work that does not require extra human eyes on the security boundary.
Risk Assessment
Merge risk: LOW | Rollback: EASY — net-new, self-contained feature package; the gate is the only production wiring and reverts cleanly. Blast radius of a gate bypass is a single user's onboarding greeting — no cross-tenant data, no privilege escalation, no persistence corruption.
CI Status
GitHub CI (per discussion specialist): 26/32 checks green (lint, type-check 3.11/3.12/3.13, CodeQL, Snyk, etc.); test/integration/e2e suites still pending, none failing; mergeable, no conflicts.
Local harness (review sandbox, NOT GitHub CI): frontend pnpm lint and pnpm test:unit failed locally; backend lint, frontend typecheck, and frontend build passed. The frontend lint failure is environment skew — GitHub's own lint check ran green on this head SHA (rule 13), so it is reported as a warning, not a blocker. Backend gate tests (21) and gate-eval tests (10) were confirmed green by the QA specialist.
UI Testing — Variant Results
✅ local: Quality gate works as specified: deterministic reject/pass, transcript retention, error codes, and Path B intro fallback all verified live; only the LLM-judge band and InsufficientState UI were unverifiable due to a missing LLM key and completed onboarding in this env (both covered by the PR's own tests).
- low: The ambiguous-middle LLM judge path could not be exercised end-to-end because this environment has no LLM key configured (OPENAI_API_KEY and OPEN_ROUTER_API_KEY both empty). All ambiguous inputs — including a genuinely usable short request 'Automate my Shopify refund emails.' which the PR docstring/test says MUST pass — resolved to insufficient_content via the no-client fail-safe rather than a real model verdict. The fail-safe itself is verified correct, but the actual judge accept/reject decisions are unconfirmed in a running system.
✅ hosted: Quality gate works end-to-end: deterministic reject/pass, DB retention, Path B intro fallback, and the full frontend InsufficientState reject→recovery→accept flow all verified live; only the live LLM-accept path is unverifiable due to no LLM client in this sandbox (intended fail-safe).
github-project-automation
Bot
moved this from 🆕 Needs initial review
to 👍🏼 Mergeable
in AutoGPT development kanban
…ate timeout, seam tests, mode-aware recovery UI
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.py`:
- Around line 221-231: Update the manifest/audio validation around the audio and
references sets to assert that references is a subset of audio before iterating
over sorted(audio), ensuring every .txt reference has matching audio. Preserve
the existing manifest equality check and per-stem validation.
In
`@autogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py`:
- Around line 138-145: Replace the _DISTINCT_WORDS string construction and its
split() call with an explicit list literal containing the same words in the same
order, preserving the fixture’s values while resolving Ruff SIM905.
In
`@autogpt_platform/frontend/src/app/`(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.ts:
- Around line 216-224: Update useBrainDumpStep.ts at lines 216-224 so the “Brain
Dump Quality Rejected” voice telemetry includes input_mode: "voice". At lines
319-329, update the typed non-quality failure tracking to emit the appropriate
failure event with input_mode: "typed" and an error_code for non-200 responses.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4b338305-dc42-457b-a873-fade638884cb
📒 Files selected for processing (15)
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/routes_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/service.pyautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/BrainDumpStep.tsxautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/__tests__/useBrainDumpStep.test.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsxautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/services/onboarding/__tests__/brain-dump-handoff.test.tsautogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.ts
🚧 Files skipped from review as they are similar to previous changes (8)
- autogpt_platform/backend/backend/api/features/onboarding_dump/service.py
- autogpt_platform/frontend/src/services/onboarding/brain-dump-analytics.ts
- autogpt_platform/frontend/src/services/onboarding/tests/brain-dump-handoff.test.ts
- autogpt_platform/backend/backend/api/features/onboarding_dump/routes_test.py
- autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/tests/useBrainDumpStep.test.ts
- autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/BrainDumpStep.tsx
- autogpt_platform/backend/backend/api/features/onboarding_dump/quality.py
- autogpt_platform/backend/backend/api/features/onboarding_dump/eval_corpus.py
📜 Review details
⏰ Context from checks skipped due to timeout. (17)
- GitHub Check: integration_test
- GitHub Check: lint
- GitHub Check: check API types
- GitHub Check: Seer Code Review
- GitHub Check: lint
- GitHub Check: types
- GitHub Check: type-check (3.11)
- GitHub Check: test (3.13)
- GitHub Check: type-check (3.13)
- GitHub Check: type-check (3.12)
- GitHub Check: end-to-end tests
- GitHub Check: test (3.12)
- GitHub Check: test (3.11)
- GitHub Check: lint
- GitHub Check: Analyze (typescript)
- GitHub Check: Analyze (python)
- GitHub Check: Check PR Status
🧰 Additional context used
📓 Path-based instructions (18)
autogpt_platform/backend/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/backend/**/*.py: Use Python 3.11 (required; managed by Poetry via pyproject.toml) for backend development
Always run 'poetry run format' (Black + isort) before linting in backend development
Always run 'poetry run lint' (ruff) after formatting in backend development
autogpt_platform/backend/**/*.py: Usepoetry run ...command for executing Python package dependencies
Use top-level imports only — avoid local/inner imports except for lazy imports of heavy optional dependencies likeopenpyxl
Use absolute imports withfrom backend.module import ...for cross-package imports; single-dot relative imports are acceptable for sibling modules within the same package; avoid double-dot relative imports
Do not use duck typing — avoidhasattr/getattr/isinstancefor type dispatch; use typed interfaces/unions/protocols instead
Use Pydantic models over dataclass/namedtuple/dict for structured data
Do not use linter suppressors — no# type: ignore,# noqa,# pyright: ignore; fix the type/code instead
Prefer list comprehensions over manual loop-and-append patterns
Use early return with guard clauses first to avoid deep nesting
Use%sfor deferred interpolation indebuglog statements for efficiency; use f-strings elsewhere for readability (e.g.,logger.debug("Processing %s items", count)vslogger.info(f"Processing {count} items"))
Sanitize error paths by usingos.path.basename()in error messages to avoid leaking directory structure
Be aware of TOCTOU (Time-Of-Check-Time-Of-Use) issues — avoid check-then-act patterns for file access and credit charging
Usetransaction=Truefor Redis pipelines to ensure atomicity on multi-step operations
Usemax(0, value)guards for computed values that should never be negative
Keep files under ~300 lines; if a file grows beyond this, split by responsibility (extract helpers, models, or a sub-module into a new file)
Keep functions under ~40 lines; extract named helpers when a function grows longer
...
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
autogpt_platform/backend/backend/api/features/**/*.py
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Update routes in '/backend/backend/api/features/' and add/update Pydantic models in the same directory for API development
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
autogpt_platform/{backend,autogpt_libs}/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Format Python code with
poetry run format
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
autogpt_platform/backend/**/api/**/*.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/api/**/*.py: UseSecurity()instead ofDepends()for authentication dependencies to get proper OpenAPI security specification
Follow SSE (Server-Sent Events) protocol: usedata:lines for frontend-parsed events (must match Zod schema) and: commentlines for heartbeats/status
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}: Use Node.js 21+ with pnpm package manager for frontend development
Always run 'pnpm format' for formatting and linting code in frontend developmentFormat frontend code using
pnpm format
autogpt_platform/frontend/**/*.{ts,tsx,js,jsx}: Fully capitalize acronyms in symbols, e.g.graphID,useBackendAPI
No linter suppressors (//@ts-ignore``,// eslint-disable) — fix the actual issue
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/**/*.{tsx,ts}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{tsx,ts}: Use function declarations for components and handlers (not arrow functions) in React components
Only use arrow functions for small inline lambdas (map, filter, etc.) in React components
Use PascalCase for component names and camelCase with 'use' prefix for hook names in React
Use Tailwind CSS utilities only for styling in frontend components
Use design system components from 'src/components/' (atoms, molecules, organisms) in frontend development
Never use 'src/components/legacy/' in frontend code
Only use Phosphor Icons (@phosphor-icons/react) for icons in frontend components
Use generated API hooks from '@/app/api/__generated__/endpoints/' instead of deprecated 'BackendAPI' or 'src/lib/autogpt-server-api/'
Use React Query for server state (via generated hooks) in frontend development
Default to client components ('use client') in Next.js; only use server components for SEO or extreme TTFB needs
Use '' component for rendering errors in frontend UI; use toast notifications for mutation errors; use 'Sentry.captureException()' for manual exceptions
Separate render logic from data/behavior in React components; keep comments minimal (code should be self-documenting)
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
autogpt_platform/frontend/**/*.{ts,tsx}: No barrel files or 'index.ts' re-exports in frontend code
Regenerate API hooks with 'pnpm generate:api' after backend OpenAPI spec changes in frontend development
autogpt_platform/frontend/**/*.{ts,tsx}: Use function declarations (not arrow functions) for components/handlers
Noanytypes unless the value genuinely can be anything
Keep render functions and hooks under ~50 lines; extract named helpers or sub-components when they grow longer
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
autogpt_platform/frontend/src/**/*.{ts,tsx}: Use generated API hooks from@/app/api/__generated__/endpoints/following the patternuse{Method}{Version}{OperationName}, and regenerate withpnpm generate:api
Separate render logic from business logic using component.tsx + useComponent.ts + helpers.ts pattern, colocate state when possible and avoid creating large components, use sub-components in local/componentsfolder
Use function declarations for components and handlers, use arrow functions only for callbacks
Do not useuseCallbackoruseMemounless asked to optimise a given function
autogpt_platform/frontend/src/**/*.{ts,tsx}: Keep files under ~200 lines; extract sub-components or hooks into their own files when a file grows beyond this
Use generated API hooks from@/app/api/__generated__/endpoints/with patternuse{Method}{Version}{OperationName}
Always import the-Icon-suffixed alias from@phosphor-icons/react(e.g.TrashIcon,PlusIcon,SquareIcon) — bare exports are deprecated
Do not useuseCallbackoruseMemounless asked to optimize a given function
Never usesrc/components/__legacy__/*— use design system components fromsrc/components/
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
No barrel files or
index.tsre-exports in the frontend
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.ts
autogpt_platform/frontend/src/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Do not type hook returns, let Typescript infer as much as possible
autogpt_platform/frontend/src/**/*.ts: Extract component logic into custom hooks grouped by concern, not by component, with each hook in its own.tsfile
Do not type hook returns; let TypeScript infer as much as possible
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.ts
autogpt_platform/**/*.{ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
Never type with
any, if no types available useunknown
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/src/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
Avoid index and barrel files
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/backend/**/*_test.py
📄 CodeRabbit inference engine (autogpt_platform/backend/AGENTS.md)
autogpt_platform/backend/**/*_test.py: Use pytest with snapshot testing for API responses
Colocate test files with source files using*_test.pynaming convention
Mock at boundaries — mock where the symbol is used, not where it's defined; after refactoring, update mock targets to match new module paths
UseAsyncMockfromunittest.mockfor async functions in tests
When writing tests, use Test-Driven Development (TDD): write failing tests marked with@pytest.mark.xfailbefore implementation, then remove the marker once the implementation is complete
When creating snapshots in tests, usepoetry run pytest path/to/test.py --snapshot-update; always review snapshot changes withgit diffbefore committing
Files:
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
autogpt_platform/frontend/**/*.{tsx,css}
📄 CodeRabbit inference engine (AGENTS.md)
Use Tailwind CSS only for styling, use design tokens, and use Phosphor Icons only
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/src/**/*.tsx
📄 CodeRabbit inference engine (AGENTS.md)
Component props should use
interface Props { ... }(not exported) unless the interface needs to be used outside the component
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/**/*.{tsx,jsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
autogpt_platform/frontend/**/*.{tsx,jsx}: Nodark:Tailwind classes — the design system handles dark mode
Use Next.js<Link>for internal navigation — never raw<a>tags
Use Tailwind CSS only for styling with design tokens and Phosphor Icons only
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/src/**/components/**/*.{tsx,jsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
Put sub-components in local
components/folder; component props should betype Props = { ... }(not exported) unless used outside the component
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
autogpt_platform/frontend/src/**/components/**/*.{ts,tsx}
📄 CodeRabbit inference engine (autogpt_platform/frontend/AGENTS.md)
Structure components as
ComponentName/ComponentName.tsx+useComponentName.ts+helpers.ts
Files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
🧠 Learnings (21)
📚 Learning: 2026-02-26T17:02:22.448Z
Learnt from: Pwuts
Repo: Significant-Gravitas/AutoGPT PR: 12211
File: .pre-commit-config.yaml:160-179
Timestamp: 2026-02-26T17:02:22.448Z
Learning: Keep the pre-commit hook pattern broad for autogpt_platform/backend to ensure OpenAPI schema changes are captured. Do not narrow to backend/api/ alone, since the generated schema depends on Pydantic models across multiple directories (backend/data/, backend/blocks/, backend/copilot/, backend/integrations/, backend/util/). Narrowing could miss schema changes and cause frontend type desynchronization.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-03-05T15:42:08.207Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12297
File: .claude/skills/backend-check/SKILL.md:14-16
Timestamp: 2026-03-05T15:42:08.207Z
Learning: In Python files under autogpt_platform/backend (recursively), rely on poetry run format to perform formatting (Black + isort) and linting (ruff). Do not run poetry run lint as a separate step after poetry run format, since format already includes linting checks.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-03-16T16:35:40.236Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12440
File: autogpt_platform/backend/backend/api/features/workflow_import.py:54-63
Timestamp: 2026-03-16T16:35:40.236Z
Learning: Avoid using the word 'competitor' in public-facing identifiers and text. Use neutral naming for API paths, model names, function names, and UI text. Examples: rename 'CompetitorFormat' to 'SourcePlatform', 'convert_competitor_workflow' to 'convert_workflow', '/competitor-workflow' to '/workflow'. Apply this guideline to files under autogpt_platform/backend and autogpt_platform/frontend.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-03-31T15:37:38.626Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12623
File: autogpt_platform/backend/backend/copilot/tools/agent_generator/fixer.py:37-47
Timestamp: 2026-03-31T15:37:38.626Z
Learning: When validating/constructing Anthropic API model IDs in Significant-Gravitas/AutoGPT, allow the hyphen-separated Claude Opus 4.6 model ID `claude-opus-4-6` (it corresponds to `LlmModel.CLAUDE_4_6_OPUS` in `autogpt_platform/backend/backend/blocks/llm.py`). Do NOT require the dot-separated form in Anthropic contexts. Only OpenRouter routing variants should use the dot separator (e.g., `anthropic/claude-opus-4.6`); `claude-opus-4-6` should be treated as correct when passed to Anthropic, and flagged only if it’s used in the OpenRouter path where the dot form is expected.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-04-15T02:43:36.890Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 12780
File: autogpt_platform/backend/backend/copilot/tools/workspace_files.py:0-0
Timestamp: 2026-04-15T02:43:36.890Z
Learning: When reviewing Python exception handlers, do not flag `isinstance(e, X)` checks as dead/unreachable if the caught exception `X` is a subclass of the exception type being handled. For example, if `X` (e.g., `VirusScanError`) inherits from `ValueError` (directly or via an intermediate class) and it can be raised within an `except ValueError:` block, then `isinstance(e, X)` inside that handler is reachable and should not be treated as dead code.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-05-23T05:29:43.085Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 13200
File: autogpt_platform/backend/backend/executor/scheduler.py:590-593
Timestamp: 2026-05-23T05:29:43.085Z
Learning: When reviewing Python code that uses Pydantic discriminated/tagged unions (e.g., `Annotated[Union[...], Field(discriminator="kind")]`), recognize that using `isinstance(x, SomeVariantInfo)` to narrow the union is an intentional and correct runtime guard and should also enable static type narrowing in tools like Pyright. Do not recommend replacing such `isinstance`-based narrowing with `cast(...)` when the check already proves the variant at runtime.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-04-22T11:46:04.431Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/config.py:0-0
Timestamp: 2026-04-22T11:46:04.431Z
Learning: Do not flag the Claude Sonnet 4.6 model ID as incorrect when it uses the project’s established hyphenated convention: `anthropic/claude-sonnet-4-6`. This hyphen form is the intentional, production convention and should be treated as valid (including in files like llm.py, blocks tests, reasoning.py, `_is_anthropic_model` tests, and config defaults). Note that OpenRouter also accepts the dot variant `anthropic/claude-sonnet-4.6`, so either form may be tolerated, but `anthropic/claude-sonnet-4-6` should be considered the standard to match project usage.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-04-22T11:46:12.892Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12881
File: autogpt_platform/backend/backend/copilot/baseline/service.py:322-332
Timestamp: 2026-04-22T11:46:12.892Z
Learning: In this codebase (Significant-Gravitas/AutoGPT), OpenRouter-routed Anthropic model IDs should use the hyphen-separated convention (e.g., `anthropic/claude-sonnet-4-6`, `anthropic/claude-opus-4-6`). Although OpenRouter may accept both hyphen and dot variants, treat the hyphen-separated form as the intended, correct codebase-wide convention and do not flag it as an error. Only flag the dot-separated variant (e.g., `anthropic/claude-sonnet-4.6`) as incorrect when reviewing/validating model ID strings for OpenRouter-routed Anthropic models.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-05-07T18:48:14.242Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 13040
File: autogpt_platform/backend/backend/blocks/llm.py:0-0
Timestamp: 2026-05-07T18:48:14.242Z
Learning: In this repository, isort may split imports from the same module into separate blocks when some imports are aliased (e.g., `from module import X as Y`) and others are not. Preserve the two-block layout when it results from isort (such as keeping `from openai.types.chat import ChatCompletion as OpenAIChatCompletion` separate from non-aliased imports from `openai.types.chat`). Do not treat that split as a style issue during review; merging them into a single block can fail CI with `Imports are incorrectly sorted and/or formatted`.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-05-26T14:24:34.866Z
Learnt from: Abhi1992002
Repo: Significant-Gravitas/AutoGPT PR: 13217
File: autogpt_platform/backend/backend/api/features/search/service.py:137-137
Timestamp: 2026-05-26T14:24:34.866Z
Learning: In the Significant-Gravitas/AutoGPT backend, treat `user_id` (an opaque UUID used only for correlation/tracing) as non-PII. Do not flag direct logging of `user_id` in `logger.warning`/`logger.info` statements as a PII exposure issue, as the established convention is to log `user_id` for tracing while reserving PII for fields like email or display name.
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-06-11T19:39:10.493Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 13337
File: autogpt_platform/backend/backend/copilot/graphiti/reranker.py:0-0
Timestamp: 2026-06-11T19:39:10.493Z
Learning: In the Significant-Gravitas/AutoGPT Python backend, when calling the OpenAI Python client `chat.completions.create`, construct the `messages` payload using the concrete typed-dict variants from `openai.types.chat` (e.g., `ChatCompletionSystemMessageParam`, `ChatCompletionUserMessageParam`, etc.) rather than trying to instantiate `ChatCompletionMessageParam` directly. `ChatCompletionMessageParam` is a `Union` alias and is not constructible, so `ChatCompletionMessageParam(role=..., content=...)` should fail type checking. Build each message element with the appropriate concrete typed dict and then annotate the resulting list as `list[ChatCompletionMessageParam]` (e.g., `messages: list[ChatCompletionMessageParam] = [ChatCompletionSystemMessageParam(...), ...]`).
Applied to files:
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.pyautogpt_platform/backend/backend/api/features/onboarding_dump/intro.pyautogpt_platform/backend/backend/api/features/onboarding_dump/parsing.pyautogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.pyautogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
📚 Learning: 2026-04-01T18:54:16.035Z
Learnt from: Bentlybro
Repo: Significant-Gravitas/AutoGPT PR: 12633
File: autogpt_platform/frontend/src/app/(platform)/library/components/AgentFilterMenu/AgentFilterMenu.tsx:3-10
Timestamp: 2026-04-01T18:54:16.035Z
Learning: In the frontend, the legacy Select component at `@/components/__legacy__/ui/select` is an intentional, codebase-wide visual-consistency pattern. During code reviews, do not flag or block PRs merely for continuing to use this legacy Select. If a migration to the newer design-system Select is desired, bundle it into a single dedicated cleanup/migration PR that updates all Select usages together (e.g., avoid piecemeal replacements).
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-04-07T09:24:16.582Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 12686
File: autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/__tests__/PainPointsStep.test.tsx:1-19
Timestamp: 2026-04-07T09:24:16.582Z
Learning: In Significant-Gravitas/AutoGPT’s `autogpt_platform/frontend` (Vite + `vitejs/plugin-react` with the automatic JSX transform), do not flag usages of React types/components (e.g., `React.ReactNode`) in `.ts`/`.tsx` files as missing `React` imports. Since the React namespace is made available by the project’s TS/Vite setup, an explicit `import React from 'react'` or `import type { ReactNode } ...` is not required; only treat it as missing if typechecking (e.g., `pnpm types`) would actually fail.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-07-28T15:32:54.931Z
Learnt from: Abhi1992002
Repo: Significant-Gravitas/AutoGPT PR: 13699
File: autogpt_platform/frontend/src/components/layout/Navbar/components/Wallet/components/WalletFullPanel.tsx:0-0
Timestamp: 2026-07-28T15:32:54.931Z
Learning: In AutoGPT's frontend (autogpt_platform/frontend), prefer importing the non-legacy ScrollArea component from `@/components/ui/scroll-area` over `@/components/__legacy__/ui/scroll-area` for new or migrated code. The non-legacy component is a drop-in superset: it preserves the legacy component’s props and additionally supports the optional `showScrollToTop` prop—so reviewers should flag new legacy imports unless there’s a specific, documented reason they can’t use the non-legacy version.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-04-02T05:43:49.128Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 12640
File: autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/WelcomeStep.tsx:13-13
Timestamp: 2026-04-02T05:43:49.128Z
Learning: Do not flag `import { Question } from "phosphor-icons/react"` as an invalid import. `Question` is a valid named export from `phosphor-icons/react` (as reflected in the package’s generated `.d.ts` files and re-exports via `dist/index.d.ts`), so it should be treated as a supported named export during code reviews.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-08-06T15:47:58.674Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 13787
File: autogpt_platform/frontend/src/app/(platform)/team/[expertId]/components/ExpertAboutSection.tsx:5-5
Timestamp: 2026-08-06T15:47:58.674Z
Learning: Within autogpt_platform/frontend, use Hugeicons through the shared Icon atom at src/components/atoms/Icon/Icon.tsx. Pass Hugeicons-compatible IconSvgElement values because the atom renders HugeiconsIcon. Do not follow the stale root AGENTS.md Phosphor icon quick-reference guidance for this frontend.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.tsautogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-02-27T10:45:49.499Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12213
File: autogpt_platform/frontend/src/app/(platform)/copilot/tools/RunMCPTool/helpers.tsx:23-24
Timestamp: 2026-02-27T10:45:49.499Z
Learning: Prefer using generated OpenAPI types from '`@/app/api/__generated__/`' for payloads defined in openapi.json (e.g., MCPToolsDiscoveredResponse, MCPToolOutputResponse). Use inline TypeScript interfaces only for payloads that are SSE-stream-only and not exposed via OpenAPI. Apply this pattern to frontend tool components (e.g., RunMCPTool) and related areas where similar SSE/openapi-discrepancies occur; avoid re-implementing types when a generated type is available.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-03-24T02:05:04.672Z
Learnt from: majdyz
Repo: Significant-Gravitas/AutoGPT PR: 12526
File: autogpt_platform/frontend/src/app/(platform)/copilot/CopilotPage.tsx:0-0
Timestamp: 2026-03-24T02:05:04.672Z
Learning: When gating React component logic on a React Query result (e.g., hooks like `useQuery` / `useGetV2GetCopilotUsage`), prefer destructuring and checking `isSuccess` (or aliasing it to a meaningful boolean like `isSuccess: hasUsage`) instead of relying on `!isLoading`. Reason: `isLoading` can be `false` in error/idle states where `data` may still be `undefined`, while `isSuccess` indicates the query completed successfully and `data` is populated.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-04-13T13:11:07.445Z
Learnt from: 0ubbe
Repo: Significant-Gravitas/AutoGPT PR: 12764
File: autogpt_platform/frontend/src/app/(platform)/library/components/SitrepItem/SitrepItem.tsx:143-145
Timestamp: 2026-04-13T13:11:07.445Z
Learning: In `autogpt_platform/frontend`, do not flag direct interpolation of `executionID` UUID strings into URL query parameters (e.g., `activeItem=${executionID}` in JSX/Next links). If the value is a UUID string matching `[0-9a-f-]`, it contains no reserved URL characters, so additional `encodeURIComponent` or Next.js object-based `href` encoding is unnecessary. Only treat it as an encoding issue if the query-param value is not guaranteed to be UUID-formatted (i.e., may include characters outside `[0-9a-f-]`).
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-04-15T22:49:06.896Z
Learnt from: ntindle
Repo: Significant-Gravitas/AutoGPT PR: 11235
File: autogpt_platform/frontend/src/app/(platform)/admin/diagnostics/components/ExecutionsTable.tsx:0-0
Timestamp: 2026-04-15T22:49:06.896Z
Learning: In the AutoGPT frontend (React Query + toast/ErrorCard patterns), do not require `Sentry.captureException` in React Query mutation `catch` blocks. React Query handles error propagation for mutation paths, so follow the established pattern: show toast notifications for mutation errors and use `ErrorCard` for render/fetch errors. Only add `Sentry.captureException` for truly manual/unexpected exception paths that are outside React Query’s control (e.g., standalone async utilities or event handlers not wired through React Query).
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
📚 Learning: 2026-07-03T04:19:11.799Z
Learnt from: Abhi1992002
Repo: Significant-Gravitas/AutoGPT PR: 13474
File: autogpt_platform/frontend/src/app/(platform)/PlatformChrome/PlatformChrome.tsx:38-38
Timestamp: 2026-07-03T04:19:11.799Z
Learning: When reviewing Tailwind usage in .tsx components, allow intentional raw hex color values if they exactly match the design-spec and there is no equivalent Tailwind design token/utility class available (e.g., a utility like `bg-zinc-50` may be a different shade than the required `#f9f9f9`). Do not flag these as "design-token violations" as long as the reviewer can confirm that an appropriate Tailwind token does not exist or would not match the exact color.
Applied to files:
autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx
🪛 Ruff (0.16.1)
autogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py
[warning] 138-145: Consider using a list literal instead of str.split
Replace with list literal
(SIM905)
🔇 Additional comments (7)
autogpt_platform/backend/backend/api/features/onboarding_dump/quality_test.py (1)
148-153: LGTM!Also applies to: 156-160, 163-178, 181-193
autogpt_platform/backend/backend/api/features/onboarding_dump/parsing.py (1)
1-29: LGTM!autogpt_platform/backend/backend/api/features/onboarding_dump/intro.py (1)
19-19: LGTM!Also applies to: 158-158
autogpt_platform/backend/backend/api/features/onboarding_dump/recommend.py (1)
19-19: LGTM!Also applies to: 92-92
autogpt_platform/backend/backend/api/features/onboarding_dump/brain_dump_gate_eval_test.py (1)
211-224: LGTM!autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/components/InsufficientState.tsx (1)
6-64: LGTM!autogpt_platform/frontend/src/app/(no-navbar)/onboarding/steps/BrainDumpStep/useBrainDumpStep.ts (1)
10-10: LGTM!Also applies to: 25-26, 35-39, 72-72, 85-85, 139-139, 204-211, 225-230, 254-254, 275-275, 286-286, 347-347, 402-402, 476-476
…eral fixture, symmetric finalize telemetry
|
/review |
…quality-gate UX Analytics direction moved to datafast + DB (team decision pending), so this PR no longer grows the posthog surface: - revert event renames and new events in brain-dump-analytics to dev state - drop onboarding step funnel, connect-tools, intro-prompt and session-context tracking; gate rejections report through dev's existing transcription_failed - restore autocapture in the shared posthog provider - keep the insufficient-content recovery screen and all quality-gate behavior; keep the sessionStorage hardening in brain-dump-handoff
|
/review |
github-actions
Bot
added
cla: pending
github-actions
Bot
added
cla: signed
|
There was a problem hiding this comment.
📋 Automated Review — PR #13792
PR #13792 — feat(platform): add brain dump STT quality gate and eval corpus
Author: Abhi1992002 | Files: 93
🎯 Verdict: APPROVE
PR Description Quality
✅ Has Why + What + How — the description explains the quality-gate motivation, the layered design, the recovery UI, and the eval harness. brain-dump-analytics.ts, OnboardingIntroCard, posthog-provider.tsx) that is not present in the current tree — the latest commit 20ce15564 "drop this PR's posthog telemetry additions" removed it. The description should be reconciled with the shipped scope.
What This PR Does
Inserts a fail-closed quality gate between speech-to-text transcription and onboarding personalization, so that empty, looping, or content-free "brain dumps" are rejected into a recoverable UI instead of personalizing a user's onboarding from junk. The gate uses a three-layer cascade — deterministic reject → deterministic pass → a cheap LLM judge for the ambiguous middle — and ships an offline eval corpus + harness to measure its accuracy. Every judge failure resolves to a recoverable reject, never a silent pass.
Specialist Findings
🛡️ Security ✅ — Small, well-handled surface. Analytics carry only counts/durations (no transcript content), tar extraction is path-validated against traversal/symlink attacks (eval_corpus.py:_safe_members), subprocess calls avoid shell=True, and error codes are server-set constants.
🟡 Transcript is embedded into the LLM judge prompt (quality.py:160) — prompt-injection is possible but contained (the gate is a quality control, not a security boundary, and fails closed).
🟡 Corpus tarball downloaded without checksum pinning (eval_corpus.py:200) — dev/eval-only tooling.
🏗️ Architecture ✅ — Clean layering, single responsibility, fails closed into a recoverable state, and respects the existing DB-claim idempotency (gate runs only for the claim winner). Notably reduces existing coupling by extracting _parse_response_json into parsing.py, removing a cross-module private import.
🟠 Synchronous LLM call added to the finalize request path (service.py:189, :268) — the same 30s-budget concern that pushed extraction/greeting to background tasks.
⚡ Performance ✅ — Layering keeps the model call off the common path (deterministic pass/reject are free); the LLM call is bounded (max_tokens=100, temperature=0, 8s timeout) and runs once per user. alnum_chars full scan at quality.py:126 is computed unconditionally but only needed for the space-less-script branch — a free early-return win.
🧪 Testing client_factory.assert_not_called() proves deterministic layers avoid the LLM, and negative-path assertions (row failed, transcript retained, background.tasks == []) are strong. Gaps: the real-model gate accuracy was never measured (the eval harness the corpus was built for is unrun), the CLEAR_PASS_CHARS spaceless seam and the typed ordinary-failure branch lack tests.
📖 Quality ✅ — Readability A-. Named threshold constants with rationale, excellent docstrings, thorough type hints. Minor magic-truncation-width inconsistencies ([:40], [:60], [:80]) across eval renderers.
📦 Product ✅ — All four claimed behaviors are wired end-to-end. Recovery UX is calm, non-alarming, preserves typed text, and never dead-ends (retry/type/skip always available).
🟡 InsufficientState wraps a heading + 3 buttons in role="alert" (InsufficientState.tsx:29) — inconsistent with sibling FailureState and odd for screen readers.
📬 Discussion reviewDecision: REVIEW_REQUIRED), the only bot APPROVE is dismissed/stale against 6 newer commits, CodeRabbit is paused, and the telemetry-removal commit 20ce15564 landed post-review and is uninspected.
🔎 QA ✅ — Backend gate verified end-to-end via the typed-dump path (same check_transcript_quality code as voice). All three layers behave as specified: deterministic pass/reject (no model call confirmed via logs), transcript+audio retention on reject, correct error codes, LLM fail-safe when no client configured, and negatives (401/422/422). No PR-specific defects. Frontend InsufficientState could not be exercised live — an out-of-scope pricing/paywall overlay blocked the onboarding funnel — but it is covered by the PR's own integration tests.
🟠 Should Fix
- Synchronous LLM judge on the finalize request path (
service.py:189,:268) — For ambiguous transcripts,check_transcript_qualitymakes a blocking LLM call (up to_TIMEOUT_SECONDS=8, already reduced from the 15s in the prior review) before the finalize response, stacked on transcription latency inside the frontend proxy's 30s budget. It's self-limiting (long audio hits the deterministic-pass path) and the verdict is genuinely needed synchronously to returnerror_code, so this is a defensible tradeoff that needs a conscious sign-off — confirm the worst-case (slow transcription + ambiguous verdict) stays under the proxy budget, or defer the verdict behind the existing/statuspoll. (Flagged by: architect, performance — 2 specialists) - Run the gate eval and record a baseline — Every
quality.pyunit test mocks the Haiku judge, so there is no evidence the real prompt +claude-haiku-4-5makes correct pass/reject calls. For a gate that can false-reject a legitimate user's onboarding, real-model accuracy is the single most important thing to verify; the corpus + harness exist precisely to measure it. (Flagged by: testing) - Reconcile the mislabeled quality-rejection analytics / stale description (
useBrainDumpStep.ts:212) — A quality rejection currently firestrackBrainDump("transcription_failed", …), conflating quality rejections with STT failures, anduseBrainDumpStep.test.ts:2507enshrines that string. Note this may be partly moot after commit20ce15564removed telemetry — verify what analytics actually remain in the tree, and either emit a distinct quality-rejection event or remove the mislabeled tracking, then align the description. (Flagged by: testing, discussion) - Close the two test gaps — Add a
CLEAR_PASS_CHARSspaceless-script seam test (at 100 and just below, plus a spaceless-but-insufficient case that falls through to the LLM) atquality.py:131, and a typed ordinary-failure test assertingscreen === "failed"for a non-quality 500 atuseBrainDumpStep.ts:310. (Flagged by: testing)
🟡 Nice to Have
- Early-return before
alnum_charsscan (quality.py:126) — Return onlen(words) >= CLEAR_PASS_WORDSbefore computing the full O(n) alnum scan; the common path never needs it. (performance) - Contract test / shared source for error codes (
helpers.ts:111,quality.py:39) — The recoverable error-code strings are duplicated across codebases with no drift guard; surface them through the generated OpenAPI schema. (architect) - Pin corpus tarball SHA-256 (
eval_corpus.py:200) and delimit/structure the judge prompt (quality.py:160). (security) - Relocate
render_tableto a neutral eval helper (brain_dump_gate_eval.py:165) to fix the inverted dependency direction. (architect)
🔵 Nits
- Magic truncation widths (
brain_dump_gate_eval.py:150,brain_dump_eval.py:356) — unify[:40]/[:60]/80under named constants. (quality) role="alert"on interactive cluster (InsufficientState.tsx:29) — preferrole="status"and mirrorFailureState. (product)- Comment narrates a rejected past metric name (
brain-dump-analytics.ts:38, if still present) — keep only the standing constraint. (architect) _MODELenv captured at import (quality.py:44) — read at call time for test overridability. (quality)
Resolved Since Prior Review
- ✅
response.choices[0]unguarded → 500 (quality.py:164) — Addressed in commitad6b4d659: moved inside the try/except, withtest_empty_choices_rejectsadded. Judge failures now all resolve to a recoverableINSUFFICIENT_CONTENTreject as documented. - ✅ LLM judge timeout — reduced from the prior review's 15s to
_TIMEOUT_SECONDS=8, narrowing (though not eliminating) the request-budget concern now tracked as Should Fix #1.
QA Screenshots
| Screenshot | Description |
|---|---|
 |
Out-of-scope pricing/paywall overlay that blocked live access to the brain-dump onboarding step |
Human Review Needed
NO — This is onboarding personalization logic with no changes to authentication, authorization, secret handling, or service trust boundaries. The backend gate was QA-verified end-to-end and the change fails closed. The outstanding items are process (human/CodeRabbit re-review of the post-telemetry-removal head) and test-baseline gaps, not security-boundary concerns.
Risk Assessment
Merge risk: LOW | Rollback: EASY — The gate is confined to the onboarding path, fails closed into a recoverable state, and can be reverted cleanly. The main residual risk is real-model gate accuracy, which is unmeasured (Should Fix #2).
CI Status
GitHub CI: ✅ Per the discussion specialist, 41/41 required checks green on head (backend test 3.11/3.12/3.13, type-check, lint, e2e, integration, API-types, CodeQL, Snyk, codecov). Note: no human approval on record; the sole bot APPROVE is dismissed against 6 newer commits and CodeRabbit is paused — a fresh review pass on the current head is advisable before merge.
Local harness: lint (frontend + backend), typecheck, and build all passed. pnpm test:unit (frontend) failed locally — treated as environment skew, not a blocker, since GitHub CI ran the suite green on this head. Investigate only if the discrepancy reproduces in CI.
UI Testing — Variant Results
✅ local: Backend STT quality gate verified end-to-end via the typed-dump path — deterministic reject/pass, LLM fail-safe, transcript retention, and negatives all behave as designed; frontend recovery UI couldn't be exercised live due to an out-of-scope env paywall but is covered by the PR's integration tests.
✅ hosted: Quality gate works end-to-end — deterministic reject/pass, safe fail-open-to-reject on judge outage, transcript retention, and error_code surfacing all verified; only the live LLM-judge accept path was unverifiable due to no LLM credentials in the sandbox.
github-project-automation
Bot
moved this from 👍🏼 Mergeable
to ✅ Done
in AutoGPT development kanban
1 participant
Why / What / How
Why: A transcription request succeeding is not the same as the user saying anything. Silence, background noise, STT hallucinations, or a few seconds of filler all come back as a "successful" transcript — and the whole personalized pipeline (greeting, suggested prompts, provider picks) would confidently personalize from garbage.
What: Adds a quality gate between transcription and personalized generation, a recovery UI when the gate rejects, and an offline eval harness (STT WER corpus + gate accuracy corpus) to measure both the transcription and the gate.
How: The gate is layered so the common cases stay free:
Rejected dumps keep their audio + transcript on the row (
errorCode=no_usable_speech/insufficient_content, plain string column — no migration). The frontend shows an insufficient-content recovery state offering retry / type instead / skip, and the intro card falls back to Path B instead of greeting the user with content the gate refused.Changes 🏗️
Backend
quality.py— new layered quality gate (deterministic reject → deterministic pass → LLM judge)service.py— gate wired into both voice (finalize_voice_dump) and typed (finalize_typed_dump) paths; intro card treats quality-rejected dumps as Path Bbrain_dump_gate_eval.py+eval_corpus.py+eval_data/— offline evals: STT word-error-rate corpus (clean / 15dB / 5dB noise) and gate accuracy corpus (noise/silence fixtures + text cases)EVAL.md— how to run the evals and current baselinesquality_test.py,brain_dump_gate_eval_test.py, plus service/route coverage for the reject pathsFrontend
InsufficientState.tsx— recovery screen for gate rejections (retry / type / skip)useBrainDumpStep.ts/helpers.ts— handleno_usable_speech/insufficient_contenterror codes as a recoverable state instead of a generic failurebrain-dump-analytics.ts)Checklist 📋
For code changes:
error_codeeval_datacorpus (requires LLM credentials — see EVAL.md)For configuration changes:
.env.defaultis updated or already compatible with my changes (optionalBRAIN_DUMP_QUALITY_MODELenv override, defaults toanthropic/claude-haiku-4-5)docker-compose.ymlis updated or already compatible with my changes