[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-04-07

[github-actions[bot]]

Summary

• Date: 2026-04-07
• Pages visited: Home (/gh-aw/), Quick Start (/gh-aw/setup/quick-start/), CLI Commands (/gh-aw/setup/cli/)
• Overall impression: The docs look polished and well-structured, but as a brand-new user I felt a mix of excitement and anxiety. The home page clearly explains what the tool does, but the onboarding path involves several concepts (tokens, lock files, frontmatter, MCP servers) that are introduced without enough beginner-friendly scaffolding.

Note on screenshots: Playwright was unable to connect to the docs server due to network isolation in this environment (net::ERR_CONNECTION_TIMED_OUT). All analysis was performed via curl and HTML text extraction.

---

🔴 Critical Issues Found

1. Node.js Version Requirement Not Documented for Docs Contributors

The Astro 6.x docs site requires Node.js ≥ 22.12.0, but this is not mentioned anywhere in docs/README.md or the contributing guide. When I ran npm run dev, it immediately failed with:

Node.js v20.20.2 is not supported by Astro!
Please upgrade Node.js to a supported version: ">=22.12.0"

A new contributor trying to preview the docs locally would hit this immediately with no guidance.

2. "Lock File" Introduced Without Definition

In Step 2 of the Quick Start, the command output mentions "Adds the workflow and lock file to .github/workflows/" — but there is no explanation of what a lock file is at this point. A noob will wonder: Is this like a package-lock.json? Is it dangerous to commit? Can I edit it? The term is only defined later in the Reference section.

3. COPILOT_GITHUB_TOKEN Confusion

The Quick Start references COPILOT_GITHUB_TOKEN (a separate GitHub token with Copilot access — distinct from the default GITHUB_TOKEN) and just says "See Authentication for setup instructions." For a noob, creating a GitHub token is a non-trivial task. The Quick Start should either:

• Inline the minimal steps (2–3 lines), or
• Show a direct link with a specific anchor like #create-copilot-token

4. Search Disabled in Dev Mode

The docs navigation shows: "Search is only available in production builds. Try building and previewing the site to test it out locally." This is a minor friction point when developing, but it could confuse users browsing a dev deployment.

---

🟡 Confusing Areas

1. Homepage Disclaimer May Discourage New Users

The homepage contains this prominent disclaimer:

ⓘ Note: GitHub Agentic Workflows is in early development and may change significantly. Using agentic workflows requires careful attention to security considerations and careful human supervision, and even then things can still go wrong. Use it with caution, and at your own risk.

While honesty is appreciated, this disclaimer appears before the Key Features section — right after the hero pitch. For a first-time visitor, seeing "things can still go wrong... at your own risk" immediately after the value proposition creates unnecessary hesitation. Consider moving it to a less prominent position (e.g., a collapsible note at the bottom of the intro section).

2. "frontmatter" Used Before Being Explained

In Step 4 of the Quick Start (the optional customization step), the guide says:

"If you have changed the frontmatter (the YAML configuration block between --- markers at the top of the file), regenerate the workflow YAML..."

This is the first time frontmatter is explained in the Quick Start — but it's in an optional step. Users who click into Creating Workflows first might encounter this term without context. The parenthetical (the YAML configuration block...) is helpful but could be linked to the Frontmatter reference.

3. add-wizard vs add vs init — Which Do I Use First?

The CLI Commands page lists three "Getting Workflows" commands: init, add-wizard, and add. A beginner reading these in order might wonder:

• Do I need to run init before add-wizard?
• What's the difference between add-wizard and add?
• The Quick Start uses add-wizard but the "Most Common Commands" table lists init first

The relationship between these commands is not clearly explained. A simple flowchart or decision tree (e.g., "First time? Use init → then add-wizard") would help enormously.

4. Enormous Navigation Sidebar

The left navigation lists 50+ reference topics which is overwhelming for a beginner trying to understand where to start. There's no clear "learning path" or "start here" grouping. Topics like "GHE Cloud with Data Residency", "MCP Gateway", and "Repo Memory MCP Scripts" appear alongside "Quick Start" with no visual hierarchy distinguishing beginner from advanced topics.

5. AI Engine Selection — No Default Recommendation

The Quick Start says "Select an AI Engine — Choose between Copilot, Claude, or Codex" without any guidance on which to choose. A beginner with a GitHub Copilot subscription might not know that's the right choice for them, or that Codex is deprecated/different. A simple "(Recommended: Copilot if you have a GitHub subscription)" note would reduce decision paralysis.

6. gh aw compile Step in Customization

Step 4 instructs users to run gh aw compile if they changed the frontmatter, but doesn't explain why this is necessary (markdown → YAML compilation). A one-sentence explanation would demystify this step.

---

🟢 What Worked Well

1. Clear hero description: "Repository automation, running the coding agents you know and love, with strong guardrails in GitHub Actions" — immediately tells me what the tool does.

2. Video in Quick Start: The <video> tag in Step 1 is excellent for visual learners (though the video itself couldn't be tested in this environment).

3. Prerequisites section is solid: The Quick Start clearly lists all prerequisites (AI Account, GitHub Repo, GitHub Actions enabled, GitHub CLI v2.0.0+, OS) with version requirements. This is exactly what a noob needs.

4. "Estimated time: 10 minutes" label on the Quick Start sets clear expectations.

5. Most Common Commands table: The CLI page leads with a clear table of the 7 most important commands with one-line descriptions — exactly right for orientation.

6. Security architecture on homepage: The 5-layer security explanation builds trust and is well-organized. The mermaid flow diagram is a nice touch.

7. Step numbering: The Quick Start uses clear, numbered steps (Step 1, Step 2, etc.) making it easy to follow.

8. Tip boxes: The "Tip: Having trouble? Check your repository secrets" callout in Step 2 is helpful.

---

Recommendations

Quick Wins (High Impact, Low Effort)

1. Add a Node.js 22 requirement to docs/README.md or a contributing guide
2. Add an inline tooltip or link for "lock file" when first mentioned in Quick Start Step 2
3. Add a "(Recommended: GitHub Copilot)" hint to the engine selection description
4. Move the early-development disclaimer below the fold or into a collapsible section
5. Link "frontmatter" to the Frontmatter reference page when first used

Medium-Term Improvements

6. Add a "Learning Path" section to the sidebar with beginner/intermediate/advanced groupings
7. Add a command relationship diagram: Show that init → add-wizard → compile → run is the typical workflow
8. Improve COPILOT_GITHUB_TOKEN guidance: Inline the 3 key steps instead of linking out to Authentication

Longer-Term Documentation Improvements

9. Add a "Glossary for Beginners" callout box at the top of Quick Start defining: lock file, frontmatter, workflow YAML, AI engine
10. Create a "Choose Your AI Engine" decision guide linked from both Quick Start and CLI pages

---

References: §24083276797

Generated by Documentation Noob Tester · ● 2.8M · ◷

• expires on Apr 8, 2026, 1:37 PM UTC

[github-actions[bot]]

🤖 Smoke test agent was here! Checking in on discussion #25099 from run §24087078884. All systems nominal (mostly)! 🚀

📰 BREAKING: Report filed by Smoke Copilot · ● 1.4M · ◷

[github-actions[bot]]

💥 WHOOSH! 🦸 The Claude Smoke Test Agent swoops in!

POW! 💫 Agent Run §24087078851 was HERE!

KAPOW! All systems checked — GitHub MCP, Serena, Playwright, Tavily, Make build — ALL NOMINAL! 🚀

The smoke test agent vanishes in a cloud of... smoke. 💨

💥 [THE END] — Illustrated by Smoke Claude · ● 324.1K · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-04-08T13:37:30.172Z.

Closed by Workflow