Skip to content

Conversation

@KaiserRuben
Copy link

Problem

The Gadget service was hallucinating technologies and missing actual dependencies, despite an
extensive 500+ line prompt. This led to inaccurate and unreliable outputs.

Solution

This pull request replaces the monolithic prompt with a structured 4-step verification workflow.
This new approach makes it structurally difficult for the model to claim a technology without
verifying its existence in the codebase.

The new workflow is as follows:

  1. Map project structure: The model starts by listing the directory contents to understand the
    project layout.
  2. Verify technologies: It then reads specific files to verify the presence of each technology.
  3. Extract configuration: All relevant configuration is extracted from the verified files.
  4. Validate build requirements: Finally, it validates the build requirements based on the
    extracted configuration.

Key Changes

  • Reduced the prompt from ~500 to ~100 lines.
  • Introduced verification tables to structure the process.
  • Made file reading mandatory before a technology can be claimed.
  • Implemented a sequential workflow where each step depends on the previous one.
  • Increased the default reasoning effort to ensure higher quality analysis.

This change significantly improves the reliability of the Gadget service by grounding its
analysis in the actual content of the repository.

PLEASE TEST BEFORE MERGING!

KaiserRuben and others added 2 commits September 30, 2025 13:53
… verification workflow

Problem: Gadget service was hallucinating technologies and missing actual dependencies
despite 500+ line prompt with extensive instructions.

Solution: Replace monolithic prompt with structured 4-step verification workflow:
- Step 1: Map project structure via list_dir
- Step 2: Verify each technology by reading specific files
- Step 3: Extract all configuration from verified files
- Step 4: Validate build requirements

Key changes:
- Reduce prompt from 500 to ~100 lines by removing redundant examples
- Introduce verification tables
- Make file reading mandatory before technology claims
- Use sequential steps where each depends on previous results
- Higher default reasoning effort (Medium minimum)

Result: Technologies can only be included if verified by actual file content,
making hallucination structurally difficult rather than behaviorally discouraged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant