🔍 Claude Code User Documentation Review - 2026-01-31

[github-actions[bot]]

Executive Summary

As a Claude Code user reviewing the gh-aw documentation, I found that Claude Code users CAN successfully adopt gh-aw, but the documentation has a moderate Copilot-first bias that creates unnecessary friction during onboarding. The core functionality is fully engine-agnostic, authentication is clearly documented, and there are sufficient Claude examples (30 workflows). However, the user journey assumes Copilot familiarity, which may cause Claude users to question whether gh-aw is right for them.

Key Finding: The product is excellent for Claude Code users, but the documentation's presentation order and emphasis make it feel Copilot-centric when it's actually engine-agnostic. This is primarily a documentation UX issue, not a technical limitation.

---

Persona Context

I reviewed this documentation as a developer who:

• ✅ Uses GitHub for version control
• ✅ Uses Claude Code as primary AI assistant
• ❌ Does NOT use GitHub Copilot
• ❌ Does NOT use Copilot CLI
• ❌ Does NOT have Copilot subscription

---

Question 1: Onboarding Experience

Can a Claude Code user understand and get started with gh-aw?

Answer: Yes, but with moderate friction.

The Quick Start guide IS engine-agnostic and includes an interactive mode that prompts users to select their engine. However, the journey to discovering this is not obvious:

Positive aspects:

• Step 2 of Quick Start explicitly states: "Select an AI Engine - You'll be prompted to choose between Claude, Copilot, or Codex"
• Interactive mode guides users through engine selection
• Prerequisites list both Copilot and alternatives: "A GitHub Copilot subscription, or a Anthropic Claude or OpenAI Codex API key"
• The gh aw init command supports --engine claude for non-interactive setup

Friction points:

1. README.md (lines 8-11) mentions "Copilot, Claude, Codex, ..." but Copilot is listed first and most prominently
2. Quick Start Prerequisites lists Copilot first with a clickable link, while Claude and Codex are mentioned after "or" with less prominence
3. CLI documentation (cli.md:166-172) shows Copilot in examples first: gh aw init --tokens --engine copilot
4. No upfront clarity on which engine is default or recommended for different use cases

Specific Issues Found:

Issue 1: README doesn't clarify engine choice upfront

• Location: README.md lines 8-11
• Problem: "The gh aw cli converts this into a GitHub Actions Workflow (.yml) that runs an AI agent (Copilot, Claude, Codex, ...) in a containerized environment"
• Impact: Claude users may assume Copilot is required or primary
• Recommended fix: Reorder to alphabetical or add clarifying text: "runs an AI agent (choose from: Claude, Codex, Copilot, or custom engines)"

Issue 2: Quick Start Prerequisites emphasize Copilot

• Location: docs/src/content/docs/setup/quick-start.md line 8
• Current: "✅ AI Account - A GitHub Copilot subscription, or a Anthropic Claude or [OpenAI Codex]((openai.com/redacted) API key"
• Problem: Copilot gets clickable link while alternatives are plain text
• Recommended fix: Equal treatment for all engines with links and clear "OR" separation

Recommended Fixes:

1. Add a prominent "Choose Your Engine" section early in the README
2. Make engine selection equally visible in Prerequisites (all links, clear alternatives)
3. Add a comparison table showing when to use each engine (cost, features, availability)

---

Question 2: Inaccessible Features for Non-Copilot Users

What features or steps don't work without Copilot?

Answer: Minimal - only Copilot-specific features require Copilot. Core functionality is fully accessible.

Features That Require Copilot:

1. engine: copilot - Obviously requires Copilot subscription
2. Copilot-specific safe outputs:
   • create-agent-session: - Creates Copilot coding agent sessions
   • assign-to-agent: with name: copilot - Assigns Copilot bot to issues
3. Copilot CLI model selection (model: gpt-5 or claude-sonnet-4) - Only works with Copilot engine

Features That Work Without Copilot (Engine-Agnostic):

1. ✅ All tools (github:, playwright:, web-fetch:, web-search:, bash:, edit:)
2. ✅ All safe outputs except Copilot-specific ones (create-issue:, create-pull-request:, add-comment:, etc.)
3. ✅ Network firewall and security features
4. ✅ MCP server integration (fully engine-agnostic)
5. ✅ All triggers (schedule, issues, pull_requests, etc.)
6. ✅ Compilation, testing, monitoring tools (gh aw compile, gh aw run, gh aw logs)
7. ✅ Safe inputs, safe outputs, threat detection
8. ✅ Campaign functionality
9. ✅ Memory tools (cache-memory, repo-memory)

Missing Documentation:

Gap 1: No feature parity table

• Missing: Clear documentation showing which features work with which engines
• Impact: Claude users may assume features are Copilot-only when they're not
• Recommended: Add "Engine Compatibility" badges or table in feature documentation

Gap 2: Default engine behavior not stated

• Missing: What happens if you don't specify an engine in frontmatter?
• Impact: Unclear whether Copilot is hardcoded as default
• Recommended: Document default engine selection behavior

---

Question 3: Documentation Gaps and Assumptions

Where does the documentation assume Copilot usage?

Answer: Several areas have Copilot-first language, but alternatives are documented.

Copilot-Centric Language Found In:

File: README.md

• Line 11: Lists "Copilot, Claude, Codex" with Copilot first
• No explanation of why you'd choose one over another

File: docs/src/content/docs/setup/quick-start.md

• Line 8: Copilot listed first with prominent link
• Line 28: "Select your preferred coding agent" - good, but after Copilot mention

File: docs/src/content/docs/reference/engines.md

• Lines 10-13: "GitHub Copilot CLI is the default and recommended AI coding agent engine"
• Issue: Explicitly states Copilot as "default and recommended" without qualifications
• Impact: Claude users may feel they're choosing a second-class option

File: docs/src/content/docs/setup/cli.md

• Line 166: gh aw init --tokens --engine copilot shown as example
• Line 246: gh aw secrets bootstrap --engine copilot shown first

Missing Alternative Instructions:

1. No "Why choose Claude?" section

   • Missing guidance on when Claude might be preferable
   • No comparison of strengths (Claude: better reasoning, Copilot: GitHub integration)

2. No engine-specific troubleshooting

   • Each engine may have unique failure modes
   • Claude-specific rate limits, model versions not mentioned

3. No cost comparison

   • Users need to understand pricing differences
   • Copilot subscription vs. Claude API costs

---

Severity-Categorized Findings

🚫 Critical Blockers (Score: 0/10 - None Found)

No critical blockers identified. All necessary information for Claude Code users to successfully adopt gh-aw is present in the documentation.

⚠️ Major Obstacles (Score: 3/10)

Obstacle 1: "Default and Recommended" Language Creates Perception of Second-Class Support

Impact: Claude users may hesitate to adopt gh-aw, feeling they're not using the "intended" engine

Current State: docs/src/content/docs/reference/engines.md lines 10-13:

GitHub Copilot CLI is the default and recommended AI coding agent engine.

Why It's Problematic:

• Implies Claude is inferior or not fully supported
• No context for WHY Copilot is "recommended" (organizational access? specific features?)
• Creates doubt for Claude users about whether they'll get full functionality

Suggested Fix:

## Choosing an Engine

GitHub Agentic Workflows supports multiple AI engines. Choose based on your needs:

- **GitHub Copilot CLI**: Best for teams with existing Copilot subscriptions and GitHub-centric workflows. Default engine.
- **Anthropic Claude**: Excellent for advanced reasoning tasks, full MCP support, flexible API access.
- **OpenAI Codex**: Alternative API-based option with OpenAI integration.

All engines provide full gh-aw functionality including tools, safe outputs, and security features.

Affected Files: docs/src/content/docs/reference/engines.md

Obstacle 2: Prerequisites Section Emphasizes Copilot Disproportionately

Impact: Significant friction in getting started - Claude users may skip the Quick Start thinking it's Copilot-only

Current State: docs/src/content/docs/setup/quick-start.md line 8:

- ✅ **AI Account** - A [GitHub Copilot](https://github.com/features/copilot) subscription, or a [Anthropic Claude](https://www.anthropic.com/) or [OpenAI Codex]((openai.com/redacted) API key

Why It's Problematic:

• Visual hierarchy favors Copilot (it's a clickable link, mentioned first)
• "or a" de-emphasizes alternatives
• Claude and Codex feel like afterthoughts

Suggested Fix:

- ✅ **AI Account** - Choose one:
  - [GitHub Copilot](https://github.com/features/copilot) subscription
  - [Anthropic Claude](https://www.anthropic.com/) API key
  - [OpenAI Codex]((openai.com/redacted) API key
```

**Affected Files:** `docs/src/content/docs/setup/quick-start.md`

</details>

<details>
<summary><b>Obstacle 3: No Engine Comparison or Selection Guide</b></summary>

**Impact:** Users don't know which engine to choose, leading to trial-and-error or default to Copilot

**Current State:** No comparison table or decision guide exists

**Why It's Problematic:**
- Users may choose Copilot by default even if Claude better fits their needs
- No guidance on cost implications (subscription vs. API usage)
- Missing feature parity information (do all engines support all tools?)

**Suggested Fix:** Add a comparison section to `docs/src/content/docs/reference/engines.md`:

| Feature | GitHub Copilot | Anthropic Claude | OpenAI Codex |
|---------|---------------|------------------|--------------|
| **Pricing** | Subscription ($10-19/mo) | API usage (pay-as-you-go) | API usage |
| **Authentication** | PAT with Copilot Requests | API key | API key |
| **GitHub Integration** | Native | Via MCP | Via MCP |
| **MCP Support** | Yes | Yes | Yes |
| **Model Selection** | gpt-5, claude-sonnet-4 | claude-sonnet-4-5, opus | gpt-4, codex |
| **Web Search** | Via MCP | Via MCP | Via MCP |
| **Best For** | Teams with Copilot | API users, reasoning tasks | OpenAI users |

**Affected Files:** `docs/src/content/docs/reference/engines.md` (needs new section)

</details>

### 💡 Minor Confusion Points (Score: 5/10)

- **Issue 1:** README lists engines as "Copilot, Claude, Codex, ..." - Alphabetical ordering would be more neutral - File: `README.md` line 11
- **Issue 2:** CLI examples show `--engine copilot` first in most commands - File: `docs/src/content/docs/setup/cli.md` lines 166, 246
- **Issue 3:** No explanation of what happens if `engine:` field is omitted from workflow frontmatter - File: `docs/src/content/docs/reference/engines.md`
- **Issue 4:** "Copilot CLI" terminology used without explaining it's just one engine option - File: `README.md` line 11
- **Issue 5:** Quick Start guide doesn't show a Claude workflow example alongside the generic example - File: `docs/src/content/docs/setup/quick-start.md`

---

## Engine Comparison Analysis

### Available Engines

Based on my review, gh-aw supports these engines with the following documentation quality:

- `engine: copilot` - **Excellent documentation**, positioned as default, clear setup instructions, multiple examples
- `engine: claude` - **Good documentation**, clear setup, good quick example (issue-analyzer), adequate examples (30 workflows)
- `engine: codex` - **Basic documentation**, setup instructions present, fewer examples (9 workflows)
- `engine: custom` - **Advanced documentation**, mentioned in custom engines reference

### Documentation Quality by Engine

| Engine | Setup Docs | Examples | Auth Docs | Overall Score |
|--------|-----------|----------|-----------|---------------|
| Copilot | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ (74 workflows) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Claude | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ (30 workflows) | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ |
| Codex | ⭐⭐⭐⭐ | ⭐⭐⭐ (9 workflows) | ⭐⭐⭐⭐ | ⭐⭐⭐ |
| Custom | ⭐⭐⭐ | ⭐ (undocumented) | ⭐⭐⭐ | ⭐⭐⭐ |

**Analysis:**
- Copilot has 2.5x more example workflows than Claude (74 vs 30)
- However, 30 Claude examples is sufficient for understanding patterns
- Authentication documentation is equally good for all engines
- Setup documentation is clear for all engines

---

## Tool Availability Analysis

### Tools Review

Analyzed tool compatibility across engines - **all tools are engine-agnostic**:

**Engine-Agnostic Tools:**
- ✅ `github:` - GitHub API operations (all engines)
- ✅ `playwright:` - Browser automation (all engines)
- ✅ `web-fetch:` - Fetch web content (all engines)
- ✅ `web-search:` - Web search (all engines, may require MCP servers)
- ✅ `bash:` - Shell commands (all engines)
- ✅ `edit:` - File editing (all engines)
- ✅ `agentic-workflows:` - Workflow introspection (all engines)
- ✅ `cache-memory:` - Persistent memory (all engines)
- ✅ `repo-memory:` - Repository memory (all engines)
- ✅ All custom MCP servers (all engines)

**Engine-Specific Tools:**
- ❌ None - there are NO engine-specific tools (except Copilot-specific safe outputs)

**Unclear/Undocumented:**
- ⚠️ Web search implementation details - Some engines may require third-party MCP servers for web search
- Note: Documentation states "Some engines require third-party Model Context Protocol (MCP) servers for web search" but doesn't specify which

**Key Finding:** The tools system is fully engine-agnostic. This is excellent for Claude Code users but should be stated more prominently.

---

## Authentication Requirements

### Current Documentation

Quick Start guide covers authentication for:
- ✅ Copilot (detailed instructions with PAT setup, organization requirements)
- ✅ Claude (status: **found and clear**) - `ANTHROPIC_API_KEY` documented in engines.md lines 72-76
- ✅ Codex (status: **found and clear**) - `OPENAI_API_KEY` documented in engines.md lines 140-144
- ⚠️ Custom (status: **mentioned but not detailed**)

### Secret Names

Documentation is clear about secret names needed:

**Copilot:**
- `COPILOT_GITHUB_TOKEN` - PAT with "Copilot Requests" permission
- For org-owned repos: Also needs Members (read-only) and GitHub Copilot Business permissions
- Clearly documented in engines.md lines 32-47 and tokens.md lines 264-323

**Claude:**
- `ANTHROPIC_API_KEY` - API key from Anthropic Console
- Setup command: `gh aw secrets set ANTHROPIC_API_KEY --value "(your-anthropic-api-key)"`
- Clearly documented in engines.md lines 72-76

**Codex:**
- `OPENAI_API_KEY` - API key from OpenAI Platform
- Setup command: `gh aw secrets set OPENAI_API_KEY --value "(your-openai-api-key)"`
- Clearly documented in engines.md lines 140-144

**Additional tokens (engine-agnostic):**
- `GH_AW_GITHUB_TOKEN` - Enhanced PAT for cross-repo operations (optional, works with all engines)
- `GH_AW_AGENT_TOKEN` - For agent assignments (optional)
- `GH_AW_PROJECT_GITHUB_TOKEN` - For GitHub Projects v2 (optional)

**Assessment:** ✅ Authentication documentation is excellent for all engines. Claude users will have no trouble setting up credentials.

---

## Example Workflow Analysis

### Workflow Count by Engine

```
Engine: copilot - 74 workflows found
Engine: claude - 30 workflows found  
Engine: codex - 9 workflows found
Total analyzed: 113 workflows

Distribution Analysis:

• Copilot: 65% of workflows
• Claude: 27% of workflows
• Codex: 8% of workflows

Quality of Examples

Copilot Examples:

• ⭐⭐⭐⭐⭐ Excellent quantity and variety
• Cover all major use cases (CI/CD, triage, reports, security)
• Many production-ready workflows

Claude Examples:

• ⭐⭐⭐⭐ Good quantity and sufficient variety
• 30 examples is enough to understand all patterns
• Includes: audit-workflows, blog-auditor, cli-version-checker, copilot-agent-analysis, daily-choice-test, and 25 more
• Demonstrates all major features (safe outputs, tools, triggers)
• Notable: Even the current workflow (claude-code-user-docs-review.md) uses engine: claude - excellent dogfooding!

Assessment: Claude has sufficient examples. While Copilot has 2.5x more, the 30 Claude examples cover all necessary patterns. More important: the examples show that Claude is a first-class citizen, not an afterthought.

---

Recommended Actions

Priority 1: Critical Documentation Fixes

None needed - No critical blockers found.

Priority 2: Major Improvements

1. Rewrite "Default and Recommended" Language - Add context about why Copilot is default (GitHub integration) and clarify that all engines are equally supported - File: docs/src/content/docs/reference/engines.md lines 10-13

2. Reformat Prerequisites for Equal Emphasis - Give all engines equal visual weight with bullet points and links - File: docs/src/content/docs/setup/quick-start.md line 8

3. Add Engine Comparison Table - Create a decision matrix showing features, pricing, and use cases for each engine - File: docs/src/content/docs/reference/engines.md (new section)

Priority 3: Nice-to-Have Enhancements

1. Add "Why Choose Claude?" Section - Highlight Claude's strengths (reasoning, cost model, API flexibility) - Would help in engines.md

2. Alphabetize Engine Mentions - Use alphabetical order in lists to avoid implied preference - Affects README.md, multiple docs

3. Add Claude Workflow Example to Quick Start - Show a complete Claude workflow alongside generic instructions - Would help in quick-start.md

4. Clarify Default Engine Behavior - Document what happens when engine: is omitted - Would help in engines.md or frontmatter.md

5. Add Engine-Specific Troubleshooting - Common issues for each engine (rate limits, model availability) - Would help in troubleshooting docs

---

Positive Findings

What Works Well for Claude Code Users

• ✅ Excellent Claude authentication documentation - ANTHROPIC_API_KEY setup is clear and concise
• ✅ Interactive engine selection - gh aw init guides users through choosing Claude
• ✅ Quick Example workflow - issue-analyzer.md shows complete Claude workflow (engines.md lines 82-118)
• ✅ Sufficient example workflows - 30 Claude workflows demonstrate all patterns
• ✅ Engine-agnostic tools - All tools work with Claude (no Copilot-specific tools except safe outputs)
• ✅ Clear secret names - No ambiguity about what credentials to configure
• ✅ Engine-neutral architecture - Security, compilation, execution docs don't favor Copilot
• ✅ MCP support - Full Model Context Protocol support works equally well with Claude
• ✅ Safe outputs - All safe outputs (except Copilot-specific ones) work with Claude
• ✅ Dogfooding - This very workflow uses engine: claude, showing confidence in Claude support
• ✅ No hidden Copilot dependencies - Nothing in the core system requires Copilot
• ✅ CLI is engine-agnostic - All gh aw commands work equally well with any engine

---

Conclusion

Can Claude Code Users Successfully Adopt gh-aw?

Answer: Yes, with moderate effort.

Reasoning:

The technical foundation is excellent - gh-aw is genuinely engine-agnostic, all tools work with Claude, authentication is straightforward, and there are 30 quality example workflows. A Claude Code user can successfully install gh-aw, configure credentials, and run workflows without encountering any technical blockers.

The documentation friction is moderate - the Copilot-first presentation creates an initial impression that gh-aw is "for Copilot users" when it's actually "for all AI engine users." This is primarily a documentation UX issue, not a technical limitation. The information is all there, but the journey to finding it assumes some Copilot familiarity.

Specific pain points:

1. Quick Start mentions Copilot prominently in Prerequisites (though alternatives are listed)
2. engines.md calls Copilot "default and recommended" without explaining why or that Claude is equally capable
3. No comparison guide to help users choose engines
4. Examples are 65% Copilot vs. 27% Claude (but 30 Claude examples is still sufficient)

What makes it work:

• Interactive mode guides engine selection
• Claude authentication is clearly documented
• All tools and features work with Claude
• Example workflows demonstrate all patterns
• No hidden dependencies on Copilot

For a Claude Code user to succeed: Read past the Copilot mentions in the first few paragraphs, use the interactive gh aw init mode, follow the Claude authentication instructions in engines.md, and review the 30 Claude example workflows. The path is clear, just not prominently signposted.

Overall Assessment Score: 7.5/10

Breakdown:

• Clarity for non-Copilot users: 7/10 - Information is present but requires careful reading
• Claude engine documentation: 8/10 - Authentication and setup are excellent, needs comparison context
• Alternative approaches provided: 6/10 - Alternatives exist but aren't emphasized equally
• Engine parity: 8/10 - Technical parity is excellent, documentation could be clearer

Comparison to ideal:

• Technical implementation: 9/10 - Truly engine-agnostic design
• Documentation UX: 6/10 - Copilot-first presentation creates friction
• Example coverage: 8/10 - 30 Claude workflows is good, could use a few more showcases

Next Steps

For gh-aw maintainers:

1. Implement Priority 2 improvements (engine comparison table, neutral language, equal emphasis)
2. Consider Priority 3 enhancements (Claude-specific guidance, alphabetization)
3. Continue expanding Claude example workflows (target: 40-50 for better parity)

For Claude Code users evaluating gh-aw:

1. Start with the Quick Start guide and use interactive mode (gh aw init)
2. Read the engines.md documentation for Claude-specific setup
3. Review the 30 Claude example workflows for patterns
4. Don't be discouraged by Copilot mentions - Claude is fully supported

For future reviews:

• Track ratio of Copilot vs. Claude examples over time
• Monitor whether language becomes more engine-neutral
• Check if comparison tables are added
• Assess whether Quick Start UX improves for non-Copilot users

---

Appendix: Files Reviewed

Complete List of Documentation Files Analyzed

Core Documentation:

• README.md - Main repository README
• docs/src/content/docs/setup/quick-start.md - Quick Start guide
• docs/src/content/docs/introduction/how-they-work.mdx - How It Works overview
• docs/src/content/docs/introduction/architecture.mdx - Security Architecture
• docs/src/content/docs/reference/tools.md - Tools reference
• docs/src/content/docs/setup/cli.md - CLI commands reference
• docs/src/content/docs/reference/engines.md - Engines documentation
• docs/src/content/docs/reference/tokens.md - GitHub tokens reference
• docs/src/content/docs/reference/faq.md - Frequently Asked Questions

Example Workflows Analyzed:

• .github/workflows/*.md - 113 total workflows examined for engine distribution

Searches Performed:

• Engine usage patterns (copilot: 74, claude: 30, codex: 9)
• Authentication documentation for all engines
• Tool compatibility across engines
• Safe outputs compatibility

---

Report Generated: 2026-01-31
Workflow Run: 21545414833
Workflow: claude-code-user-docs-review
Engine Used: claude (eating our own dog food! 🐕)
Documentation Version: Analyzed at commit 1d7b78b

---

Metadata

Review Methodology:

• Systematic reading of 9 core documentation files
• Analysis of 113 example workflows for engine distribution
• Grep searches for engine-specific mentions
• Authentication flow verification for each engine
• Tool compatibility matrix verification

Confidence Level: High - Based on comprehensive file reading and analysis

Reviewer Bias Declaration: This review was conducted by an AI agent configured to use Claude as its engine, reviewing documentation for a product that supports multiple engines. The review attempts to be objective but may have blind spots about Copilot-specific features or terminology that Copilot users would find clear.

AI generated by Claude Code User Documentation Review

• expires on Feb 7, 2026, 1:53 PM UTC