One-page overview of how the bot is structured and how a review flows through it.
ThrillhouseBot is a Quarkus application that runs as a GitHub App. A webhook arrives when a pull request changes, the bot builds a review with an OpenAI-compatible model, and it posts the result back as a PR review plus a check run. A dashboard streams what is happening live.
flowchart TB
subgraph GH[GitHub.com]
IN[PR push, /review, @mention]
OUT[PR reviews · check runs · comments]
end
subgraph BOT[ThrillhouseBot · Quarkus]
WH[webhook/<br/>WebhookController]
RO[review/<br/>ReviewOrchestrator]
AI[review/ai/<br/>AiReviewService]
GHC[github/<br/>REST clients]
DB[dashboard/ + frontend/]
WH --> RO --> AI
RO --> GHC
AI -.->|live tokens / review.batch| DB
end
IN -->|POST /api/webhook<br/>HMAC-verified| WH
GHC --> OUT
The github/ clients wrap the GitHub REST surface the bot uses: installation
tokens, pull diffs and prior reviews, check runs, PR reviews with inline
comments, issue comments, and the instructions-file fallback chain
(.github/thrillhousebot.md, .github/copilot-instructions.md, CLAUDE.md,
AGENTS.md, AGENT.md).
flowchart TD
GH[GitHub: PR opened / synced / comment] -->|webhook| WH[webhook/]
WH -->|verify HMAC, filters, rate limit, 👀 ack| RO[review/ ReviewOrchestrator]
RO -->|fetch diff, instructions, prior findings| GHC[github/ API clients]
RO -->|budget plan · stream or batch| AI[review/ai/ LangChain4j]
AI -->|parse findings| RO
RO -->|verify findings — 2nd AI call per batch, on by default| AI
RO -->|post review + check run| GHC --> GH
AI -.->|live tokens or review.batch| DB[dashboard/ broadcaster]
DB -->|WebSocket| FE[frontend/ Next.js UI]
RO -->|persist session, cost, tokens| PG[(H2 / PostgreSQL)]
AI -->|traces, token & cost metrics| OT[(OpenTelemetry)]
Automatic triggers (pull_request opened / synchronize, and similar) are subject
to AUTO_REVIEW_MIN_INTERVAL: if the same PR was auto-reviewed too recently,
the webhook path skips the review silently. Manual /review always bypasses that
window. Slash and mention commands get a best-effort 👀 reaction before
pause/authorization; conversational @thrillhousebot mentions (no command word)
are answered without a reaction.
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
participant AI as AI Provider
Dev->>GH: git push (PR opened)
GH->>TB: POST /api/webhook (pull_request: opened)
Note over TB: Verify HMAC → JWT → install token
Note over TB: Auto-review rate limit (skip if within AUTO_REVIEW_MIN_INTERVAL)
TB->>GH: POST check-runs → status: queued
TB->>GH: PATCH check-run → status: in_progress
par Fetch context
TB->>GH: GET /pulls/{pr}/files (diff)
TB->>GH: GET /compare/{base}...{head} (regression context)
end
TB->>GH: GET /pulls/{pr}/reviews (check if already reviewed this SHA)
GH-->>TB: no prior reviews → first run
Note over TB: DiffBudgetPlanner — single-call or token-budgeted batches
alt Diff fits one call
TB->>AI: POST chat (diff + base comparison + review prompt) — live tokens to dashboard
AI-->>TB: findings + risk levels + suggestions
opt Findings found and REVIEW_VERIFIER_ENABLED (default)
TB->>AI: POST chat (re-check each finding against the diff)
AI-->>TB: confirmed / downgraded / dropped findings
end
else Large diff — multi-call
loop Each batch in parallel (up to REVIEW_MAX_AI_CALLS − 1)
TB-->>TB: review.batch progress (no per-token stream)
TB->>AI: POST chat (batch diff)
AI-->>TB: batch findings
opt Findings and verifier on
TB->>AI: POST chat (verify batch findings)
end
end
TB->>AI: POST chat (summary rollup of aggregated findings)
AI-->>TB: PR-level summary + previous-findings status
Note over TB: Any files that still won't fit are disclosed by name
end
alt AI fails
TB->>GH: PATCH check-run → conclusion: failure
TB->>GH: POST comment: retry hint (no internal details)
else AI succeeds + issues found
TB->>GH: POST PR review (REQUEST_CHANGES or COMMENT) with inline suggestions
TB->>GH: PATCH check-run → conclusion: failure (critical/high) or neutral
TB->>GH: POST comment: PR summary (risk table + key findings)
else AI succeeds + zero issues
TB->>GH: POST PR review → APPROVE (no body)
TB->>GH: PATCH check-run → conclusion: success
TB->>GH: POST comment: PR summary (celebration inside)
end
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
participant AI as AI Provider
Dev->>GH: git push (PR synchronize)
GH->>TB: POST /api/webhook (pull_request: synchronize)
Note over TB: Verify → auth → rate limit → create check run (in_progress)
par Fetch context
TB->>GH: GET /pulls/{pr}/files (diff)
TB->>GH: GET /compare/{base}...{head}
and Fetch prior review
TB->>GH: GET /pulls/{pr}/reviews (find ThrillhouseBot's last review)
GH-->>TB: previous findings + thread status
end
Note over TB: Prompt includes diff + prior findings + "check if each was addressed"
Note over TB: Same single-call or map-reduce path as first review
TB->>AI: POST chat (one or more review calls ± verifier ± summary)
AI-->>TB: resolved / unresolved / new findings
alt AI fails
Note over TB: Same sanitized error path as first review
else AI succeeds
TB->>GH: POST PR review (suggestions for unresolved + new issues)
TB->>GH: PATCH check-run → conclusion based on risk
Note over TB: No summary comment on follow-up (only on first run)
end
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
Dev->>GH: Comments "/review"
GH->>TB: POST /api/webhook (issue_comment: created)
Note over TB: Verify → parse trigger → 👀 ack (bounded wait) → auth
Note over TB: Manual /review bypasses AUTO_REVIEW_MIN_INTERVAL
Note over TB: Fetch diff, compare, and prior reviews
Note over TB: Full re-review even if this SHA was already reviewed
sequenceDiagram
actor Dev
participant GH as GitHub
participant TB as ThrillhouseBot
Dev->>GH: Mentions @thrillhousebot (in a PR thread or finding reply)
GH->>TB: POST /api/webhook (pull_request_review_comment or issue_comment: created)
Note over TB: Bot-loop guard → mention detected (no command) → ACK 200
Note over TB: No 👀 reaction — conversational mentions are answered, not reacted to
Note over TB: Async on review executor: authorize (write access)
Note over TB: Build threaded prompt (finding + diff hunk + thread)
TB->>GH: POST reply in the review thread (or a PR comment)
| Package | Responsibility | Notable classes |
|---|---|---|
webhook/ |
Receives GitHub events, verifies the HMAC signature, decides whether an event triggers a review (trigger filters, per-PR pause state, auto-review rate limit), acks slash/mention commands with 👀, runs the comment commands (/help, /summary, /describe, /changelog, /add-docs, /improve, /generate-tests, /resolve, /pause, /resume), and schedules finding-feedback capture on review-thread replies |
WebhookController, WebhookVerifier, TriggerDetector, ReviewTriggerFilter, AckReactionService, CommentCommandService, PrPauseService |
review/ |
Orchestrates a review: plans the token budget and the per-review spend ceiling, calls the AI layer (single-call or map-reduce), maps findings to a risk level and review state, re-checks a maintainer's decline against the reviewed code, writes the summary comment, optionally labels the PR, answers maintainer replies/mentions in PR threads, and persists maintainer finding feedback (👍/👎 / reply heuristics) for a future learnings pipeline | ReviewOrchestrator, ReviewDispatcher, DiffBudgetPlanner, FindingPipeline, AutoReviewRateLimiter, ReviewDiffFormatter, FollowUpAnalyzer, FindingFeedbackCaptureService, FindingFeedbackService, PrSummaryGenerator, PrLabeler, MaintainerReplyService, MaintainerReplyDispatcher, PrImprovementService, PatchCoverage, ConfigKeyContextResolver, RebuttalContradiction, SummarySurfaceDeduplicator, VerdictBuilder |
review/ai/ |
The LangChain4j layer: streams or batches model responses, parses findings, runs a second pass to verify them, applies generation/reasoning customizers, and writes conversational replies | PrReviewer, AiReviewService, ChatModelCustomizers, FindingVerifier, FindingVerificationService, ReviewResponseParser, ReplyAssistant, TruncatedResponseSalvager, FindingVerifierPrompts |
github/ |
Talks to the GitHub REST and GraphQL APIs: app auth, pull requests, reviews, check runs, comments, labels, reactions (create + list), and reading the repo instructions file | GitHubAuthClient, GitHubReviewClient, GitHubCheckRunClient, GitHubLabelClient, GitHubReactionClient, InstructionsResolver, GitHubWriteRetry |
dashboard/ |
The live UI backend: OAuth login (in-memory sessions), WebSocket broadcaster (review.stream / review.batch), review session persistence, and finding-feedback aggregates |
AuthResource, DashboardSessionStore, SessionEventBroadcaster, ReviewSessionRepository, DashboardResource |
config/ |
Wiring: the outbound HTTP client, the review thread pool, typed config, active-model settings (caps, generation params), fail-fast startup validation, and the shared bot-identity used to recognize the bot's own activity | HttpClientProducer, ReviewExecutorProducer, ThrillhouseConfig, ActiveModelSettings, StartupConfigValidator, BotIdentity |
frontend/ |
The Next.js dashboard, built to a static export and served by Quarkus | — |
PR reviews carry inline comments and suggestions; check runs carry pass/fail status for branch protection (no inline annotations on the check run itself).
AI call budget — a review that reports findings makes two model calls
by default: the review call plus a skeptical verification pass
(FindingVerifier) that re-sends the diff and each candidate finding, dropping
or downgrading what it can't confirm. It fails open — a verifier error keeps
the original findings, so a broken verifier can never block a review. Under
token-aware budgeting on large PRs this becomes N batch review calls + N
per-batch verification calls + one summary call. REVIEW_VERIFIER_ENABLED=false
skips only the AI pass (a deterministic hedging-language guard still runs) and
trades cost for more false positives. Expect two model spans per flagged
single-call review (or N+N+1 under budgeting) in the traces and in the
dashboard's session totals. Multi-call reviews do not stream tokens to the
dashboard; they emit review.batch progress events instead. Batches run
concurrently on virtual threads; a failed batch is retried once after the
parallel pass completes.
Cost ceiling — REVIEW_MAX_TOKENS_PER_REVIEW bounds the tokens one review may
spend across every call it makes, counting retries, the verifier and the summary.
Once reached, remaining batches are disclosed as not reviewed by name and the
summary degrades to counts rather than making further calls. 0, the default,
leaves it unbounded. The summary, the verifier and maintainer replies run on a
separate concise model binding with its own response cap
(REVIEW_CONCISE_MAX_OUTPUT_TOKENS) and its own reasoning effort, so they never
share a cap sized for batch review output.
Coverage honesty — a file the review never read does not pass silently. A file GitHub reported with changes but no patch text, a file that did not fit any batch, and a file whose batch call failed are each disclosed by name and withhold APPROVE. A response the model cut at its length cap keeps the findings that completed before the cut rather than being discarded whole.
Patch coverage as review context — when a PR's CI publishes a coverage
report, PatchCoverage reads the changed lines it does not cover and gives them
to the review, so new code with no test behind it can be named as such. Off
unless REVIEW_PATCH_COVERAGE_ENABLED is set.
Repository-supplied configuration — .github/thrillhousebot.yml carries a
repository's own ignore globs and path-scoped review instructions, read from the
default branch and cached for five minutes. Ignore globs are additive to the
deployment list; a repository can narrow its own review scope but cannot restore
a file the deployment excludes. Every failure mode (missing file, invalid YAML,
unexpected shape, uncompilable glob) is logged and skipped, leaving the
deployment configuration in force.
Write pacing — content-creating GitHub calls are spaced process-wide by
GITHUB_WRITE_MIN_INTERVAL so a burst of comments never reaches the secondary
rate limit in the first place, with GITHUB_WRITE_MAX_WAIT capping how long any
one caller waits.
Each AI call is bounded by AI_TIMEOUT (LangChain4j) and
thrillhousebot.review.ai-timeout-seconds. Cost and token metrics come from
OpenTelemetry. OAuth login sessions are opaque IDs in cookies with tokens kept
server-side; review history persists in the database. Maintainer finding
feedback (reactions and reply heuristics) is documented in
Finding feedback
(source: docs/FEEDBACK.md). See
SECURITY.md
for the reporting process.
There is no provider-specific code. The model is reached through LangChain4j's
OpenAI-compatible client, so a new provider is configuration: point AI_BASE_URL
and AI_MODEL at it. Add a thrillhousebot.ai.pricing.<model>.* pair for cost
tracking (without it the bot warns once and flags sessions as "no pricing"
instead of $0). Optionally set thrillhousebot.ai.models.<model>.* for the
model's input cap and generation parameters, and AI_REASONING_ENABLED /
AI_REASONING_EFFORT when the model supports reasoning. See the
provider table and
the configuration reference.