📚 Documentation Noob Test Report - December 7, 2025

[github-actions[bot]]

Documentation Noob Testing - Fresh User Perspective

I navigated through the GitHub Agentic Workflows documentation as a complete beginner to identify confusing, broken, or unclear steps. Here's what I found.

🎯 Executive Summary

Overall Impression: The documentation has excellent visual design and solid structure, but assumes too much prior knowledge. As a complete beginner, I felt lost without understanding core concepts first.

Top 3 Blocking Issues:

1. No clear explanation of what "agentic" means - This is literally in the name, but never defined upfront!
2. Compilation concept introduced too late - Left wondering "why compile?" from the start
3. Prerequisites don't explain "why" - Lists requirements without context

---

🔴 CRITICAL ISSUES (Block Getting Started)

(strong)1. Missing Conceptual Foundation (Home Page)(/strong)

Issue: The site jumps straight into features without explaining the core concept.

What I see: "Write natural language workflows (AI-powered automation that can make decisions)"

What confuses me:

• What does "make decisions" mean in this context?
• How is this different from GitHub Actions?
• What's an example of a "decision" it can make?

Recommendation: Add a "What is GitHub Agentic Workflows?" section with:

• Simple 2-sentence explanation
• Concrete before/after example
• Comparison to familiar concepts (GitHub Actions, CI/CD)

(strong)2. Prerequisites Assume Too Much Knowledge (Quick Start)(/strong)

Issue: Prerequisites section doesn't explain why each item is needed.

What confuses me:

• Why do I need GitHub CLI? Can't I use the web interface?
• What's a "Personal Access Token with Copilot Requests permission"? I've never heard of this permission type
• Why do Discussions need to be enabled? The workflow example isn't explained yet

Recommendation:

• Add a sentence explaining WHY each prerequisite is needed
• Link to GitHub's PAT documentation
• Explain that "Copilot Requests" is a NEW permission type (many users won't have seen it before)

(strong)3. Installation Flow Breaks in Codespaces (Quick Start, Step 1)(/strong)

Issue: Warning about Codespaces installation failure comes AFTER the installation command.

What confuses me:

• This warning is AFTER the installation command - if I'm in Codespaces, I've already failed
• No explanation of WHY it fails
• Feels like an afterthought

Recommendation:

• Move Codespaces warning BEFORE the installation command
• Make it a prominent callout box
• Explain the permission/auth issue briefly

(strong)4. "Compilation" Concept Not Explained Early Enough (Quick Start)(/strong)

Issue: The guide mentions compilation in Step 1 but doesn't explain it until "Understanding Compilation" section later.

What confuses me:

• Why do I need to compile? Regular GitHub Actions don't need compilation
• What exactly gets compiled?
• Do I need to install a compiler?
• Is this like compiling C++ code?

Recommendation:

• Add a simple explanation at the TOP of Quick Start:

  "You'll write workflows in simple Markdown. GH Agentic Workflows compiles them into GitHub Actions YAML that GitHub can run."

• Keep the detailed "Understanding Compilation" section for reference

(strong)5. Step 3 Appears Incomplete (Quick Start)(/strong)

Issue: When extracting the page text, Step 3 content appears to cut off mid-sentence: "This permission allows your workflow to comm"

Problem: Can't complete the setup if instructions are incomplete!

Recommendation:

• Review the entire Step 3 to ensure all text is visible and accessible
• Add screenshots showing exactly where to find "Copilot Requests" permission in GitHub's PAT settings
• Include troubleshooting section if the permission doesn't appear

---

🟡 CONFUSING AREAS (Slow Down Learning)

(strong)1. Jargon Overload (Throughout)(/strong)

Terms used without definition:

• "Agentic" (used everywhere, never defined)
• "Safe outputs"
• "MCP server"
• "Engine" (Copilot vs Claude vs Codex - what's the difference?)
• "Frontmatter"
• "Lock file" (why the .lock.yml extension?)

Recommendation:

• Add a glossary link in the header navigation
• Implement hover tooltips for technical terms
• Define terms inline on first use with expandable explanations

(strong)2. Unclear Relationship Between Components (Overview, How It Works)(/strong)

Issue: Multiple concepts introduced without showing how they connect.

What confuses me:

• How does the Markdown file → Compilation → GitHub Actions flow actually work?
• What's the relationship between "engines" and "tools"?
• Where does Copilot CLI fit into the picture?
• What runs where?

Recommendation: Add a simple architecture diagram showing:

You write .md file
     ↓
gh aw compile creates .lock.yml  
     ↓
GitHub Actions runs .lock.yml
     ↓
Copilot CLI executes inside the action
     ↓
AI engine makes decisions
     ↓
Tools interact with GitHub

(strong)3. Examples Don't Show Expected Output (Examples)(/strong)

Issue: Examples show the workflow code but not what actually happens when it runs.

What I want to know:

• What does this workflow DO when it runs?
• What will I see in my GitHub repository?
• What does the output look like in Actions logs?
• What artifacts does it create (comments, issues, PRs)?

Recommendation: Add "Example Output" sections showing:

• GitHub Actions log snippets
• Issue comments that get created
• Discussion posts
• Whatever artifacts the workflow produces

(strong)4. CLI Commands Page is Reference Heavy (CLI Commands)(/strong)

Issue: Lists all commands without guiding me through common workflows.

What I need:

• Which commands do I use most often?
• In what order should I use them?
• What's a typical daily workflow look like?

Recommendation: Add "Common Workflows" section at the top:

• Making changes to a workflow: edit .md → compile → commit
• Testing locally: compile → inspect output
• Debugging a failed run: logs command → analyze
• Adding a new workflow: add command → customize

(strong)5. "Creating Workflows" Assumes I Know What to Create (Agentic Authoring)(/strong)

Issue: Explains HOW to write workflows without explaining WHAT to write.

What confuses me:

• What should my first custom workflow do?
• How do I come up with workflow ideas?
• What's realistic vs. too ambitious for a first workflow?

Recommendation:

• Add "Your First Custom Workflow" tutorial
• Start with a simple example: "Leave a comment when an issue is labeled 'needs-triage'"
• Walk through each section of the .md file step by step
• Build complexity gradually with follow-up tutorials

(strong)6. Security Concerns Not Addressed Upfront (Throughout)(/strong)

Issue: "AI making decisions in my repo" sounds scary to newcomers.

What worries me as a beginner:

• Can it accidentally delete things?
• What if it makes bad decisions?
• How do I control what it can access?
• Can it leak secrets or sensitive data?
• Who is responsible if something goes wrong?

Recommendation:

• Add "Safety & Security" section to Quick Start (before Step 1)
• Explain the sandboxed execution environment
• Show how to limit permissions with examples
• Emphasize that workflows run in GitHub Actions (familiar security model)
• Link to detailed security guide
• Include "AI Safety Best Practices" section

---

🟢 WHAT WORKED WELL

✅ Excellent Visual Design

• Feature cards on home page are easy to scan
• Icons help identify different workflow types quickly
• Badge system (command, automated, scheduled) is intuitive and helpful

✅ Effective Warning System

• The "Caution" boxes about research prototype and security stand out
• The "Never edit .lock.yml files" warning is prominent and clear
• Color-coded callouts draw attention appropriately

✅ Well-Organized Examples

• ChatOps, IssueOps, LabelOps categories make sense
• Easy to browse different use cases
• The card-based layout is inviting and scannable

✅ Good External Linking

• GitHub CLI installation guide link is helpful
• Links to source code repositories work
• Cross-references between related pages

✅ Helpful Page Navigation

• "On This Page" sidebar helps me know what's covered
• Jump links work smoothly
• Breadcrumb navigation aids orientation

---

📋 RECOMMENDATIONS (Prioritized)

🚀 Quick Wins (Help New Users Immediately)

1. Add "What is this?" section to home page

   • 3-4 sentences explaining the core concept
   • One concrete before/after example
   • Quick comparison to GitHub Actions
   • Effort: 30 minutes, Impact: High

2. Fix Step 3 in Quick Start

   • Complete any cutoff instructions
   • Add screenshot of PAT creation showing "Copilot Requests" permission
   • Effort: 1 hour, Impact: Critical

3. Move Codespaces warning before installation command

   • Prevents users from hitting an avoidable error
   • Effort: 5 minutes, Impact: Medium

4. Add inline definitions for key jargon

   • Tooltip or expandable definitions for "agentic", "MCP", "frontmatter", etc.
   • Effort: 2 hours, Impact: High

5. Show example outputs for 2-3 workflows

   • Screenshots of what users will actually see
   • Sample Actions logs
   • Effort: 1 hour per example, Impact: High

📈 Medium-Term Improvements

6. Create "Your First Custom Workflow" tutorial

   • Step-by-step guide building a workflow from scratch
   • Explain each piece of the markdown frontmatter
   • Show the entire process: write → compile → test → deploy
   • Effort: 4-6 hours, Impact: Very High

7. Add architecture diagram

   • Visual showing how components connect
   • Where does my code run?
   • Where does AI processing happen?
   • Flow from .md file to actual execution
   • Effort: 2-3 hours, Impact: High

8. Create troubleshooting flowchart

   • "Workflow didn't run" → check triggers
   • "Compilation failed" → check frontmatter syntax
   • "AI didn't do what I expected" → review prompt quality
   • Effort: 3-4 hours, Impact: Medium

9. Add video walkthrough

   • 5-minute "Hello World" screencast
   • Show the entire process from installation to first successful run
   • Effort: 4 hours (including editing), Impact: Very High

🎯 Longer-Term Documentation Improvements

10. Create progression learning paths

    • Beginner → Intermediate → Advanced tracks
    • "Start here if you've never used GitHub Actions"
    • "Start here if you're familiar with Actions but new to AI workflows"
    • Effort: 1-2 weeks, Impact: Very High

11. Add comprehensive FAQ section

    • "Why compilation?" "Why CLI?" "Why Copilot?"
    • Common pitfalls and solutions
    • Effort: 8 hours, Impact: Medium

12. Interactive tutorial/sandbox

    • In-browser code editor for writing workflows
    • Validate markdown syntax in real-time
    • Show compilation output immediately
    • Effort: Several weeks, Impact: Very High

13. Real-world case studies

    • How actual teams use agentic workflows
    • Problems solved, time saved
    • Lessons learned and best practices
    • Effort: Ongoing, Impact: High

---

📊 MISSING DOCUMENTATION

Topics I expected to find but didn't:

💰 Cost/Pricing Information

• Do I need a paid Copilot subscription?
• Are there API rate limits I should know about?
• What happens if I run out of tokens/quota?
• Any GitHub Actions minutes considerations?

⚖️ Comparison Guide

• vs. GitHub Actions (when to use which?)
• vs. GitHub Copilot for Pull Requests
• vs. Writing custom GitHub bots
• Decision matrix for choosing the right tool

📖 Best Practices

• How often should workflows run?
• How to write effective prompts for AI
• Common anti-patterns to avoid
• Workflow design principles

⚡ Performance

• How long do workflows typically take?
• Are there timeouts I should worry about?
• How to optimize slow workflows
• Caching strategies

🐛 Debugging Guide

• How to read and interpret logs
• Common error messages and their meanings
• Step-by-step debugging process
• Tools and techniques for troubleshooting

🔄 Migration Guide

• Converting existing GitHub Actions to agentic workflows
• What can vs. can't be converted
• Hybrid approaches (mixing Actions and agentic workflows)
• Migration checklist

---

🎬 CONCLUSION

Summary

The documentation has a solid foundation with excellent visual design and good structure, but it assumes too much prior knowledge. As a complete beginner, I feel lost without understanding the core concepts first.

Would I Be Able to Get Started?

Maybe, but with significant frustration and trial-and-error. The tool looks powerful and exciting, but the learning curve feels unnecessarily steep due to missing conceptual explanations.

Most Impactful Change

A beginner-focused rewrite of the Quick Start that:

• Explains concepts before diving into features
• Shows outputs and results, not just inputs
• Builds confidence step by step
• Assumes zero knowledge of AI workflows
• Addresses security concerns upfront

The Good News

The tool is clearly well-designed and the examples are compelling. With better onboarding documentation, this could be significantly more accessible to newcomers.

---

📝 Test Methodology

• Approach: Navigated documentation as a complete beginner with no prior knowledge of GitHub Agentic Workflows
• Tools Used: Built local documentation site with npm run build && npm run preview
• Pages Analyzed: Home, Quick Start, CLI Commands, Creating Workflows, Overview, How It Works
• Perspective: Assumed role of developer new to AI-powered automation, familiar with Git/GitHub basics but not GitHub Actions
• Focus: Identified blocking issues, confusing areas, and positive aspects

---

Labels: documentation, user-experience, onboarding, needs-improvement

AI generated by Documentation Noob Tester

[pelikhan]

/plan

[github-actions[bot]]

🎉 Yo ho ho! Plan Command found the treasure and completed successfully! ⚓💰