📚 Documentation Noob Test Report - 2026-01-23

[github-actions[bot]]

Summary

Date of test: January 23, 2026
Test approach: Analyzed HTML and source markdown documentation as a complete beginner
Pages analyzed:

• Home page (index.html)
• Quick Start (/setup/quick-start/)
• CLI Commands (/setup/cli/)
• Creating Workflows (/setup/agentic-authoring/)
• About Workflows (/introduction/overview/)

Overall impression as a new user: The documentation is comprehensive and well-structured, but contains several areas that could confuse newcomers, particularly around terminology, prerequisites, and the relationship between tools.

---

🔴 Critical Issues (Block getting started)

1. Confusing Tool Relationships: gh-aw vs GitHub Copilot

Location: Multiple pages (Quick Start, Agentic Authoring)

Issue: The documentation uses "GitHub Copilot," "Copilot CLI," "copilot" command, and references to @github/copilot-cli npm package without clearly explaining these are different tools with different purposes.

Why it's confusing:

• Quick Start mentions installing gh aw extension, then later says "use the copilot command"
• A beginner might think gh aw and copilot are the same thing
• The distinction between "GitHub Copilot as an AI engine" vs "GitHub Copilot CLI as a chat interface" is never explicitly stated

Recommendation:
Add a clear callout box early in Quick Start:

💡 **Tool Clarification**
- **`gh aw`** = GitHub CLI extension for compiling and managing agentic workflows
- **`copilot` CLI** = Interactive chat interface with GitHub Copilot (optional, for authoring)
- **GitHub Copilot** = The AI engine that runs your workflows (requires subscription)
```

### 2. **"Initialize your repository" Step Lacks Context**

**Location:** Quick Start, Step 2

**Issue:** The command `gh aw init --push` is presented without explaining:
- What it actually does to your repository
- Why you need it
- What files get created and committed
- Whether you can skip it if you're just testing

**Why it blocks getting started:**
- New users may be hesitant to run a command that pushes changes without understanding what's being committed
- The phrase "installs agents and tools for GitHub Copilot" doesn't explain the agent files are just markdown instructions

**Recommendation:**
Expand the explanation:
```
This command creates custom agent files in your repository:
- `.github/agents/agentic-workflows.agent.md` - Custom instructions for `/agent` command
- `.github/aw/github-agentic-workflows.instructions.md` - Copilot authoring hints

The `--push` flag commits and pushes these files. You can skip this step if you only want to test workflows without customization support.

3. "Daily" Schedule Is Mysterious

Location: Quick Start, workflow example

Issue: The configuration shows schedule: daily but doesn't explain:

• What time it runs (beginner worry: "Will it run at midnight? During work hours?")
• How to change it
• How to see when it last ran or when it will run next

Why it's blocking:

• Users can't predict when their first workflow will run
• No guidance on how to test immediately vs waiting for schedule

Current text:

schedule: daily — Runs once per day at a randomized time to distribute load

Better text:

schedule: daily — Runs once per day at a random time (usually during low-traffic hours). To test immediately, use gh aw run workflow-name. See Schedule Syntax for other options like hourly or weekly.

---

🟡 Confusing Areas (Slow down learning)

1. Jargon Without Definitions (First Encounter)

Location: Quick Start, early paragraphs

Terms that appear without explanation:

• "Frontmatter" - First mentioned in Step 3 with a link, but beginners may not click it immediately
• "Compilation" - Used throughout but only defined much later
• "Lock file" - The .lock.yml concept is critical but not explained upfront
• "Safe outputs" - Appears in frontmatter examples without prior introduction

Recommendation:
Add a "Key Concepts" callout box at the top of Quick Start with 2-3 sentence definitions of the most critical terms:

📖 **Key Concepts You'll Encounter**
- **Compilation**: Converting your human-friendly `.md` workflow file into a GitHub Actions `.lock.yml` file
- **Frontmatter**: YAML configuration at the top of your workflow between `---` markers  
- **Safe outputs**: Pre-approved operations (like creating issues) that don't require giving the AI full write access
```

### 2. **Prerequisites vs Quick Start Ordering**

**Location:** Quick Start structure

**Issue:** The Prerequisites section comes AFTER the installation steps, which is backwards. Users encounter commands before knowing if they're ready to run them.

**Current flow:**
1. Step 1: Install extension
2. Step 2: Initialize repo
3. (much later) Prerequisites section

**Better flow:**
1. Prerequisites (check if you're ready)
2. Installation steps
3. Configuration steps

**Recommendation:**
Move Prerequisites to the very top or add a "Before you begin" checklist.

### 3. **"Active GitHub Copilot Subscription" Requirement Is Buried**

**Location:** Quick Start, PAT creation tip box

**Issue:** The requirement for a Copilot subscription is hidden in a tip box within a tip box:
> Requires an active [GitHub Copilot subscription](...)

**Why it's confusing:**
- This is a MANDATORY requirement, not an optional tip
- Users might get through Steps 1-3 before realizing they can't proceed without a subscription
- The cost/subscription requirement should be mentioned in Prerequisites

**Recommendation:**
Add to Prerequisites section as the first item:
```
- ✅ **GitHub Copilot subscription** - Required for workflows to execute. [Sign up here](https://github.com/features/copilot) if you don't have one yet. (Agentic workflows use Copilot as the AI engine.)
```

### 4. **Copilot Requests Permission Is Non-Obvious**

**Location:** Quick Start, Step 4

**Issue:** Finding "Copilot Requests" in "Account permissions" (not Repository permissions) is counterintuitive. The documentation does explain it, but this trips up many users.

**Current help text is good but could be enhanced:**

Add a visual indicator:
```
⚠️ **Important**: "Copilot Requests" appears under "Account permissions" (near the bottom), NOT "Repository permissions" (near the top). Scroll down to find it.

5. What Happens After "gh aw run"?

Location: Quick Start, Step 5

Issue: After running gh aw run daily-team-status, the user is told to "check the status" but not:

• How long it typically takes
• What to do if it fails
• Where to find the generated issue
• What the issue will look like

Recommendation:
Add expected timing and troubleshooting:

gh aw run daily-team-status

# ⏱️ This typically takes 30-60 seconds for small repos, up to 3-5 minutes for large repos

# Check status (refresh until "conclusion" shows "success" or "failure")
gh aw status

# 📊 If successful, look for a new issue in your repository with label "daily-status"
# 🔍 If failed, view logs with: gh aw logs daily-team-status

6. Codespace Warning Comes Too Late

Location: Quick Start, Step 1 tip box

Issue: The Codespace installation workaround appears AFTER the standard installation command. Users in Codespaces will try the standard method, wait for it to fail, then see the workaround.

Recommendation:
Add detection hint before the standard installation:

# Are you in a GitHub Codespace? Check with:
echo $CODESPACES  # If this prints "true", use the standalone installer below

# Standard installation (for local machines):
gh extension install githubnext/gh-aw

# Codespace installation (if standard install fails):
curl -sL https://raw.githubusercontent.com/githubnext/gh-aw/main/install-gh-aw.sh | bash

---

🟢 What Worked Well

1. Progressive Disclosure of Complexity

The Quick Start doesn't try to explain everything at once. It gets you to a working workflow, then links to deeper docs. Good balance.

2. Concrete Examples

The daily team status workflow is a perfect "Hello World" - practical, understandable, and immediately useful.

3. Verification Steps

Including gh aw status output and explaining what to expect is excellent. More workflows should follow this pattern.

4. Visual Workflow Diagram

The simple ASCII diagram showing .md → compile → .lock.yml → GitHub Actions is clear and helpful.

5. Security Warnings

The warning at the top about AI agents making decisions is appropriate and well-placed.

6. Link Density

Appropriate use of links to glossary terms and deeper docs. Not overwhelming, but enough to dig deeper.

---

📝 Recommendations Summary

Quick Wins (High Impact, Low Effort)

1. Add "Tool Clarification" callout early in Quick Start
2. Move Prerequisites section to the top of Quick Start
3. Add "Key Concepts" callout with 3-4 critical definitions
4. Enhance timing expectations for workflow execution
5. Add Copilot subscription as first prerequisite item

Medium Priority

6. Expand gh aw init explanation with file list
7. Improve schedule documentation with testing guidance
8. Add visual hint for "Account permissions" location
9. Reorder Codespace warning before standard install
10. Add troubleshooting section for common first-run failures

Longer-Term Improvements

11. Create a "Common First-Run Issues" page linked from Quick Start
12. Add video walkthrough or animated GIFs for complex steps (especially PAT creation)
13. Build an interactive setup checker that validates prerequisites before starting
14. Add estimated time/cost for typical workflows in examples

---

🎯 Success Metrics

This documentation noob test identified:

• 3 critical blocking issues that prevent first-time users from succeeding
• 6 confusing areas that slow down the learning process significantly
• 6 things that work well and should be preserved/replicated

The documentation is structurally sound but needs clarity improvements for complete beginners, particularly around:

1. Tool relationships and prerequisites
2. Terminology definitions on first use
3. Expectation setting for timing and outcomes

Priority: Focus on the "Quick Wins" section first - these are simple text changes that will have immediate impact on new user success rates.

AI generated by Documentation Noob Tester

• expires on Jan 30, 2026, 7:44 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-01-30T07:44:56.042Z.