📚 Documentation Noob Test Report - December 4, 2025

[github-actions[bot]]

Test Summary

Date: December 4, 2025
Tester Perspective: Complete beginner (first-time user)
Pages Analyzed: 6 core documentation pages
Overall Impression: Visually clean and well-organized, but suffers from expert bias with assumed knowledge and unexplained jargon that creates barriers for newcomers.

---

🔴 CRITICAL ISSUES (Block Getting Started)

(strong)1. Missing Clear Definition of "Agentic Workflow" on Home Page(/strong)

Issue: The home page says "Write natural language workflows" and mentions "agentic workflows (AI-powered automation that can make decisions)" but this core concept isn't explained prominently.

Impact: New users don't immediately understand what makes this different from regular GitHub Actions.

Recommendation: Add a prominent "What are Agentic Workflows?" section on the home page before the feature list:

## What Are Agentic Workflows?

Agentic workflows are AI-powered GitHub Actions that can **think and adapt**. 
Unlike traditional workflows that follow rigid scripts, agentic workflows:

- Understand context from issues, PRs, and repository content
- Make decisions based on that context  
- Take actions like creating discussions, labeling issues, or updating PRs
- Run securely in sandboxed GitHub Actions runners

**Example**: Instead of "add label 'bug' to all issues", you can write 
"read the issue description and add appropriate labels based on the content."

(strong)2. Confusing PAT Setup Instructions (Step 3 in Quick Start)(/strong)

Issue: The "Copilot Requests" permission explanation is hidden inside a collapsible "Can't find Copilot Requests permission?" section. Critical prerequisites are mentioned AFTER users might have already tried creating the token.

Impact: Users waste time trying to create tokens incorrectly, getting frustrated when they can't find the required permission.

Recommendation: Move prerequisite checks (active Copilot subscription, fine-grained token requirement) BEFORE the step-by-step instructions. Rewrite Step 3 to check prerequisites first:

## Step 3 — Configure AI Authentication

**Before proceeding, verify you have:**
- ✅ Active GitHub Copilot subscription ([check here](link))
- ✅ Ability to create Personal Access Tokens

**Create your token:**
1. Visit https://github.com/settings/personal-access-tokens/new
2. **Important**: Select "Fine-grained token" (not Classic)
3. Under "Resource owner", select YOUR username (not an organization)
...

(strong)3. Unclear Compilation Workflow Timing(/strong)

Issue: The "Understanding Compilation" section appears AFTER "Step 1 - Install" but BEFORE "Step 2 - Add a workflow" where users actually get a workflow file to compile.

Problem: Users are told "You write .md → Compile → GitHub Actions runs .lock.yml" before they have any .md files, making the concept abstract and confusing.

Impact: Users don't understand when or why they need to compile.

Recommendation: Move "Understanding Compilation" to appear right after Step 2 (adding a sample workflow) when it's immediately relevant and actionable.

(strong)4. "Agentics Collection" Not Explained(/strong)

Issue: Step 2 says "Add a sample from the agentics collection" with no explanation of what this is.

Command shown: gh aw add githubnext/agentics/daily-team-status --create-pull-request

Impact: Users don't know if agentics is official, community-maintained, required, or what other options exist.

Recommendation: Add before the command:

The agentics collection is an official library of example workflows maintained by GitHub Next. Browse all examples at https://github.com/githubnext/agentics

---

🟡 CONFUSING AREAS (Slow Down Learning)

(strong)5. Jargon Without Definitions(/strong)

Terms introduced without explanation or glossary links:

• "Frontmatter" - appears in multiple places but never defined inline
• "Safe outputs" - mentioned in Step 3 but not explained what it means or why it matters
• "Coding agent" - used in Step 3 without clarifying this is the AI that runs the workflow
• "Lock files" (.lock.yml) - the term "lock" is confusing since it's not related to dependency locking like package.json.lock

Recommendations:

• Add inline tooltips or glossary links for first usage of technical terms
• Consider renaming .lock.yml to .compiled.yml or .generated.yml to avoid confusion
• Link to glossary page on first usage of each term

(strong)6. Prerequisites Scattered Across Document(/strong)

Issue: The "Prerequisites" section in Quick Start lists requirements in three separate buckets:

• "Must Have (before you start)"
• "Must Configure (in your repository)"
• "Will Need Later (you'll set this up in Step 3)"

Users must read through all three to understand if they can even proceed.

Recommendation: Add a "Can I use this?" quick check at the very top:

✅ You can follow this guide if you have:
  - A GitHub account
  - GitHub Copilot subscription (individual or org)
  - Admin access to a repository with Actions & Discussions enabled
  - GitHub CLI installed

[Missing something? → Click for alternatives]

(strong)7. CLI Commands Page Lacks Context(/strong)

Issue: The CLI page jumps straight into alphabetical command listings without explaining typical workflows or command sequences.

Missing: A "Common Workflows" or "Getting Started" section showing:

# Typical first-time setup
gh extension install githubnext/gh-aw
gh aw add githubnext/agentics/[workflow-name]
gh aw compile

# Daily development
gh aw compile           # After editing .md files
gh aw status           # Check workflow runs
gh aw logs [workflow]  # Debug issues

Recommendation: Add a "Common Command Sequences" table at the top showing:

• First-time setup flow
• Add example workflow flow
• Create custom workflow flow
• Debug workflow flow

(strong)8. "Creating Workflows" Page Structure Issues(/strong)

Issue: Page title is "Creating Workflows" but content focuses heavily on:

• create-agentic-workflow prompt (a specific Copilot CLI tool)
• Using ChatGPT for design
• Dictation tips

Missing:

• A simple "Workflow Anatomy" section showing the basic .md file structure
• "Start from Scratch vs. Use Template" decision guide
• Basic frontmatter field explanations

Recommendation: Add a "Workflow File Basics" section before diving into AI-assisted creation tools.

(strong)9. Examples Page Organization(/strong)

Issue: Examples are grouped by trigger pattern (ChatOps, IssueOps, DailyOps) but new users don't understand these patterns yet.

Recommendation:

• Add a "🌟 Start Here - Beginner Examples" section with 2-3 heavily annotated workflows
• Include "What you'll learn" and "Complexity: Beginner/Intermediate/Advanced" tags
• Show both the .md file and explain what it does in plain English

---

🟢 WHAT WORKED WELL

✅ Clear Visual Hierarchy - Home page features with icons are scannable and appealing
✅ Good Use of Callouts - Warning/Caution boxes effectively highlight important information
✅ Code Examples - Syntax highlighting and copy buttons work well
✅ Step-by-Step Structure - Quick Start is clearly numbered and sequential
✅ Navigation - Sidebar is well-organized and logical

---

📊 RECOMMENDATIONS (Prioritized)

🔥 Quick Wins (High Impact, Low Effort)

1. Add "What is an Agentic Workflow?" expandable section to home page - Single biggest impact improvement
2. Reorder Quick Start sections - Move "Understanding Compilation" after Step 2
3. Add "Prerequisites Checklist" widget - Quick pass/fail check at top of Quick Start
4. Explain "agentics collection" - One sentence in Step 2
5. Add inline glossary tooltips - Link jargon terms to glossary on first usage

📈 Medium-Term Improvements

6. Create "Your First Workflow from Scratch" tutorial - Separate from template-based Quick Start
7. Add "Common Command Sequences" section to CLI page
8. Rewrite Step 3 (PAT setup) - Prerequisites first, then instructions
9. Add "Workflow Anatomy" section to Creating Workflows page
10. Create "Beginner Examples" showcase with annotations

🎯 Long-Term Enhancements

11. Interactive tutorial - In-browser playground for editing and compiling
12. Video walkthrough - 2-3 minute screen recording of Quick Start
13. Comparison table - "GitHub Actions vs Agentic Workflows: What's Different?"
14. "Try It Without Setup" button - GitHub Codespace with pre-configured example

---

🧪 Testing Methodology

This analysis was performed by:

1. Building the documentation site from source (npm install && npm run build && npm run preview)
2. Extracting HTML content from 6 key pages using curl
3. Analyzing text content, structure, navigation, and terminology
4. Simulating complete beginner perspective by identifying assumptions and jargon
5. Mapping user journey through Quick Start path
6. Categorizing issues by severity (critical blockers vs. friction vs. minor)

Limitations:

• Could not capture screenshots due to Playwright installation restrictions in CI environment
• Could not test interactive elements (search, expandable sections)
• Analysis based on content structure and text rather than full visual rendering

---

🎯 Conclusion

The GitHub Agentic Workflows documentation has a solid foundation with excellent visual design and comprehensive content. However, it suffers from expert bias - written by people who already deeply understand the concepts.

Biggest single impact change: Add a clear, prominent "What is an Agentic Workflow?" explanation with a visual example on the home page. This alone would dramatically improve comprehension.

Current state: Functional but not beginner-friendly - usable by experienced GitHub Actions developers
Potential state: Accessible to GitHub Actions newcomers with recommended improvements

The documentation needs more "why" explanations, less assumed knowledge, and strategic reordering to match the actual user learning journey.

---

📋 Pages Analyzed

• Home page (/gh-aw/)
• Quick Start (/gh-aw/setup/quick-start/)
• CLI Commands (/gh-aw/setup/cli/)
• Creating Workflows (/gh-aw/setup/agentic-authoring/)
• Introduction/Overview (/gh-aw/introduction/overview/)
• Examples (/gh-aw/examples/issue-pr-events/)

---

Labels: documentation, user-experience, automated-testing, feedback

AI generated by Documentation Noob Tester

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.