User Experience Analysis Report - 2026-01-26

[github-actions[bot]]

Executive Summary

Today's analysis focused on:

• 2 documentation files (quick-start.md, tools.md)
• 2 CLI commands (compile, new)
• 2 workflow message configurations (archie.md, brave.md)
• 1 validation file (sandbox_validation.go)

Overall Quality: ✅ Professional

Key Finding: The documentation demonstrates strong enterprise-quality patterns with excellent progressive disclosure and clear error messaging, but opportunities exist to improve information hierarchy in reference documentation and refine workflow messaging tone for enterprise contexts.

---

Quality Highlights ✅

Example 1: Quick Start Documentation - Excellent Progressive Disclosure

• File: docs/src/content/docs/setup/quick-start.md
• What works well:
  • Clear warning callouts for security considerations (lines 8-10)
  • Progressive disclosure with collapsible tips (lines 12-15, 29-37, 82-86)
  • Logical step-by-step flow with verification commands
  • Comprehensive troubleshooting guidance (lines 119-123)
  • Appropriate use of visual hierarchy with h2/h3 headers
• Quote/Reference: The prerequisite checklist (lines 143-150) provides clear, actionable requirements with verification commands

Example 2: Validation Error Messages - Clear and Actionable

• File: pkg/workflow/sandbox_validation.go
• What works well:
  • Specific error descriptions with context (lines 33-36, 62-68)
  • Actionable suggestions with code examples
  • Professional, non-blaming language
  • Appropriate technical detail for the audience
• Quote/Reference: Lines 62-68 provide both the issue ("mount mode must be 'ro' or 'rw'") and concrete solution with examples

---

Improvement Opportunities 💡

High Priority

Opportunity 1: Improve Information Hierarchy in Tools Reference - Single File Improvement

• File: docs/src/content/docs/reference/tools.md
• Current State:
  • Lines 99-129: GitHub tools "Modes and Restrictions" section contains critical security information (lockdown behavior, token precedence) presented in dense prose
  • Line 171: Token precedence information appears as a single line at the end of a complex section
  • Multiple important concepts compete for attention without clear visual hierarchy
• Issue:
  • Design Principle: Efficiency and Productivity - minimize cognitive load
  • Critical security information (automatic lockdown, token precedence) is buried in paragraph text
  • Enterprise users scanning for security implications may miss key details
  • No clear distinction between "what" (feature description) and "why it matters" (security implications)
• User Impact:
  • Developers configuring GitHub tools in production environments may miss critical security settings
  • Token configuration errors could lead to overprivileged workflows
  • Time wasted re-reading dense paragraphs to find specific configuration details
• Suggested Change:

Before (Lines 99-129):

**Remote Mode**: Use hosted MCP server for faster startup (no Docker). Requires `GH_AW_GITHUB_TOKEN`:

...dense paragraph...

**Read-Only**: Default behavior; restricts to read operations unless write operations configured.

**Lockdown**: Automatically determined based on repository visibility when using a custom token...

...more dense paragraphs...

See [Automatic GitHub Lockdown](/gh-aw/guides/security/#automatic-github-lockdown-on-public-repositories) for security implications.

After:

#### Configuration Modes

**Remote Mode** (Recommended for Speed)
- Uses hosted MCP server (no Docker required)
- Requires `GH_AW_GITHUB_TOKEN` secret
- Setup: `gh aw secrets set GH_AW_GITHUB_TOKEN --value "(your-pat)"`

```yaml wrap
tools:
  github:
    mode: remote  # Default: "local" (Docker)
```

**Read-Only Mode** (Default)
- Restricts operations to read-only
- Write operations require explicit configuration
- Recommended for audit and reporting workflows

#### Security Settings

**Automatic Lockdown for Public Repositories**

⚠️ **Important**: When using a custom token (`GH_AW_GITHUB_MCP_SERVER_TOKEN`), lockdown is automatically enabled for public repositories to prevent exposure of private repository content.

| Repository Type | Lockdown Behavior |
|----------------|-------------------|
| Public | ✅ Automatically enabled - filters content to push-access users only |
| Private/Internal | ⚠️ Automatically disabled - no filtering applied |

**Manual Override** (Use with caution):
```yaml wrap
tools:
  github:
    lockdown: true   # Force enable
    # or
    lockdown: false  # Explicitly disable (requires careful review)
```

**Security Impact**: See [Automatic GitHub Lockdown](/gh-aw/guides/security/#automatic-github-lockdown-on-public-repositories) for complete security implications.

#### Token Precedence (Most to Least Specific)

When multiple token sources are configured, the following precedence applies:

1. **GitHub App** - Highest security (short-lived, auto-revoked)
2. **`github-token`** - Workflow-specific override
3. **`GH_AW_GITHUB_MCP_SERVER_TOKEN`** - Organization-level custom token
4. **`GH_AW_GITHUB_TOKEN`** - User-level token (remote mode)
5. **`GITHUB_TOKEN`** - Default GitHub Actions token (most restrictive)

💡 **Recommendation**: Use GitHub App authentication for production workflows (see [GitHub App Tokens](/gh-aw/reference/tokens/#github-app-tokens-for-github-mcp-server)).

Why This Matters:

• User Impact: Security-critical information is immediately scannable with clear visual hierarchy
• Quality Factor: Reduces cognitive load by separating concerns (modes vs. security)
• Frequency: GitHub tools configuration is one of the most common tasks in workflow setup

Success Criteria:

• Changes made to docs/src/content/docs/reference/tools.md only
• Security information presented in scannable table format
• Token precedence displayed as numbered list with context
• Quality rating improves from ✅ Professional to ✅✅ Exemplary

Scope Constraint:

• Single file only: docs/src/content/docs/reference/tools.md
• No changes to other files required
• Can be completed independently

---

Medium Priority

Opportunity 2: Refine Workflow Message Tone for Enterprise Context - Single File Improvement

• File: .github/workflows/archie.md
• Current State:
  • Lines 27-32: Custom messages use personified language ("Archie here!", "sketching the architecture", "Blueprint complete!")
  • Mixing playful emoji use (📐, 🎨) with technical workflow context
  • Message tone appropriate for internal team tools but less suited for enterprise audit/reporting contexts
• Issue:
  • Design Principle: Professional Communication - business-appropriate tone
  • Playful personification ("Archie here!") may seem unprofessional in enterprise compliance contexts
  • Inconsistent with other workflows that use more neutral status reporting
  • Emojis used decoratively rather than to convey status clearly
• User Impact:
  • In audit trails or compliance reports, personified messages reduce professional credibility
  • Enterprise users expect neutral status updates, not personality-driven narratives
  • May create perception of "toy tool" rather than enterprise-grade automation
• Suggested Change:

Before (Lines 27-32):

messages:
  footer: "> 📊 *Diagram rendered by [{workflow_name}]({run_url})*"
  run-started: "📐 Archie here! [{workflow_name}]({run_url}) is sketching the architecture on this {event_type}..."
  run-success: "🎨 Blueprint complete! [{workflow_name}]({run_url}) has visualized the connections. The architecture speaks for itself! ✅"
  run-failure: "📐 Drafting interrupted! [{workflow_name}]({run_url}) {status}. The diagram remains incomplete..."

After:

messages:
  footer: "> 📊 *Diagram generated by [{workflow_name}]({run_url})*"
  run-started: "📐 Generating Mermaid diagrams for {event_type}... [{workflow_name}]({run_url})"
  run-success: "✅ Diagram generation complete - [{workflow_name}]({run_url}) analyzed relationships and created visualizations"
  run-failure: "❌ Diagram generation failed - [{workflow_name}]({run_url}) {status}. See run logs for details."

Why This Matters:

• User Impact: Professional status messages improve credibility in enterprise audit trails
• Quality Factor: Consistent, neutral tone across workflow messages reduces cognitive dissonance
• Frequency: Workflow messages appear in issue comments, run summaries, and audit logs

Success Criteria:

• Changes made to .github/workflows/archie.md only (lines 27-32)
• Messages use neutral, status-focused language
• Emojis convey status (✅ success, ❌ failure, 📐 in-progress) rather than decoration
• Quality rating improves from ⚠️ Needs Minor Work to ✅ Professional

Scope Constraint:

• Single file only: .github/workflows/archie.md
• No changes to other workflows required
• Can be completed independently

---

Files Reviewed

Documentation

• docs/src/content/docs/setup/quick-start.md - Rating: ✅ Professional
• docs/src/content/docs/reference/tools.md - Rating: ⚠️ Needs Minor Work

CLI Commands

• gh aw compile - Rating: ✅ Professional
• gh aw new - Rating: ✅ Professional

Workflow Messages

• archie.md - Rating: ⚠️ Needs Minor Work
• brave.md - Rating: ✅ Professional

Validation Code

• pkg/workflow/sandbox_validation.go - Rating: ✅ Professional

---

Metrics

• Files Analyzed: 7
• Quality Distribution:
  • ✅ Professional: 5
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

---

Design Principles Applied

This analysis evaluated user-facing aspects against enterprise software design principles:

1. Clarity and Precision ✅ - Error messages and documentation provide unambiguous guidance
2. Professional Communication ⚠️ - Some workflow messages use overly casual tone for enterprise context
3. Efficiency and Productivity ⚠️ - Dense prose in reference docs creates cognitive load
4. Trust and Reliability ✅ - Consistent error handling and validation messaging
5. Documentation Quality ✅ - Strong progressive disclosure patterns in tutorials

---

Analysis Methodology: Random sampling of 1-2 files per category (documentation, CLI, messages, validation) with focused evaluation against enterprise software design principles. Each improvement opportunity targets a single file to ensure changes are surgical and independently reviewable.

AI generated by Delight

• expires on Feb 2, 2026, 8:42 AM UTC