🎯 Repository Quality Improvement Report - Error Experience Engineering (Deep Dive)

[github-actions[bot]]

Analysis Date: 2026-01-28
Focus Area: Error Experience Engineering (Reused)
Strategy Type: Reuse (10% strategy)
Custom Area: Yes - Holistic developer experience when errors occur

Executive Summary

This is a deep dive follow-up to our January 7th error experience analysis. While we've seen positive progress (+57% console formatting adoption, +1,109% debug logging), the fundamental challenge remains: only 4.4% (69/1,581) of error output sites use proper console formatting. The codebase has grown significantly, and with it, error handling complexity.

Key Finding: The top 5 CLI files (add_interactive.go, init.go, add_command.go, audit_report_render.go, trial_command.go) contain 388 unformatted error outputs - representing 25% of all unformatted errors. Targeted improvements in these files would deliver outsized impact.

Critical Gap: Zero panic recovery implementations despite complex operations (compilation, GitHub API calls, file I/O). A single unhandled panic in production could crash an entire workflow run.

Full Analysis Report

Focus Area: Error Experience Engineering (Deep Dive)

This analysis revisits our January 7th focus area with deeper investigation into error formatting adoption, critical error paths, and user-facing error scenarios.

Current State Assessment

Metrics Collected:

Metric	Current Value	Jan 7 Value	Change	Status
Console-formatted errors	69 (4.4%)	44 (3%)	+57%	⚠️ Low adoption
Raw/unformatted errors	1,512 (95.6%)	178	+749%	❌ Massive growth
Properly wrapped errors (fmt.Errorf %w)	958 (90.9%)	809	+18%	✅ Good
Plain errors.New (context loss)	96 (9.1%)	80	+20%	⚠️ Increased
Debug logs for error paths	133	11	+1,109%	✅ Excellent
Panic recovery (defer recover)	0	0	0%	❌ Critical gap
Validation errors with guidance	77	N/A	N/A	⚠️ Only 29% of validations
Error documentation links	7	N/A	N/A	❌ Extremely low
Retry mechanisms	30	N/A	N/A	⚠️ Limited resilience
Error aggregation (errors.Join)	0	0	0%	❌ No multi-error support

Findings

Strengths

• Strong error wrapping culture (90.9%) preserves context across call stacks
• Debug logging explosion (+1,109%) shows improved observability mindset
• Structured validation errors exist with excellent patterns (e.g., GitHubToolsetValidationError)
• Error testing coverage is comprehensive (141 error assertions, 125 message content tests)
• 21 error type files provide reusable error infrastructure

Areas for Improvement

1. Console Formatting Adoption Crisis (Severity: 🔴 Critical)

• Only 4.4% of error sites use console.FormatErrorMessage()
• Top 5 CLI files contain 388 unformatted errors (25% of total)
• User-facing commands show inconsistent error presentation
• Impact: Users see raw Go error messages instead of helpful, formatted guidance

2. High-Value Target Files (Severity: 🟡 High)

File	Unformatted Errors	Impact
pkg/cli/add_interactive.go	107	Interactive wizard - critical UX
pkg/cli/init.go	100	First-time setup - critical onboarding
pkg/cli/add_command.go	72	Workflow creation - common operation
pkg/cli/audit_report_render.go	61	Audit reports - troubleshooting tool
pkg/cli/trial_command.go	48	Workflow testing - developer workflow

3. Zero Panic Recovery (Severity: 🔴 Critical)

• 0 panic recovery implementations across entire codebase
• Complex operations (GitHub API, file I/O, JSON parsing) lack safety nets
• A single unhandled panic crashes workflow runs
• Impact: Production incidents from unexpected panics, poor error messages

4. Error Aggregation Gap (Severity: 🟡 High)

• 0 uses of errors.Join() despite Go 1.20+ multi-error support
• Validation files return first error, hide subsequent issues
• Users must fix errors one-by-one instead of seeing full context
• Impact: Slow feedback loops, frustrated developers

5. Limited Error Documentation (Severity: 🟡 High)

• Only 7 error messages include documentation links
• 29% of validation errors include examples/suggestions
• No error code catalog or troubleshooting index
• Impact: Users lack guidance for common errors

Detailed Analysis

User-Facing Error Scenarios

High-Impact Categories (by frequency):

1. Permission errors: 4,727 references - access control, GitHub tokens, repository permissions
2. Network/API errors: 3,090 references - GitHub API, MCP servers, timeouts
3. Invalid configuration errors: 78 references - frontmatter, MCP config, workflow syntax
4. File not found errors: 56 references - workflow files, included files, templates
5. Compilation errors: 23 files - workflow validation, YAML generation

Error Recovery & Resilience Analysis

Current State:

• 30 retry mechanisms - limited to specific scenarios
• 401 fallback patterns - good defensive coding
• 99 graceful degradation patterns - reasonable resilience
• 66 context timeout handlers - good async operation management
• 0 error aggregation - critical gap for comprehensive feedback

Validation Error Quality Patterns

Gold Standard Examples:

• engine_validation.go - Clear error messages with examples: "invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot"
• GitHubToolsetValidationError - Structured, sorted, actionable multi-error reporting
• mcp_config_validation.go - Contextual error messages with configuration guidance

Improvement Opportunities:
5 validation files lack structured errors:

• bundler_runtime_validation.go
• repository_features_validation.go
• dispatch_workflow_validation.go
• template_validation.go
• strict_mode_validation.go

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for separate processing.

Improvement Tasks

Task 1: Add Panic Recovery to Critical Compilation Paths

Priority: High
Estimated Effort: Medium
Focus Area: Error Experience Engineering

Description:
Implement panic recovery (defer recover()) in critical compilation and GitHub API operation paths to prevent workflow crashes from unexpected panics. Focus on pkg/workflow/compiler*.go and pkg/cli/*_command.go files where panics could crash entire workflow runs.

Acceptance Criteria:

• Add panic recovery to main compiler entry points (Compile, CompileWorkflow, CompileAction)
• Add panic recovery to GitHub API client operations
• Add panic recovery to interactive CLI commands (init, add, trial)
• Recovered panics converted to proper errors with stack traces in debug logs
• Add test cases verifying panic recovery converts to errors

Code Region: pkg/workflow/compiler*.go, pkg/cli/*_command.go

Add panic recovery to critical compilation and GitHub API operations to prevent crashes.

Files to update:
- `pkg/workflow/compiler.go` - Main compilation entry points
- `pkg/workflow/compiler_yaml.go` - YAML generation operations
- `pkg/cli/trial_command.go` - Trial workflow execution
- `pkg/cli/add_interactive.go` - Interactive workflow creation
- `pkg/cli/init.go` - Repository initialization

Pattern to implement:
```go
func CriticalOperation() (err error) {
    defer func() {
        if r := recover(); r != nil {
            // Convert panic to error
            err = fmt.Errorf("panic during operation: %v\nstack trace:\n%s", r, string(debug.Stack()))
            
            // Log for debugging
            if log.Enabled() {
                log.Printf("Recovered from panic: %v", r)
            }
        }
    }()
    
    // Existing operation code...
    return nil
}
```

Requirements:
1. Add panic recovery to compilation entry points
2. Include stack traces in error messages (using `runtime/debug.Stack()`)
3. Log panics to debug logger for investigation
4. Return structured errors that work with console formatting
5. Add test cases for panic scenarios (nil pointer, index out of bounds, type assertion failures)

Example test:
```go
func TestCompiler_PanicRecovery(t *testing.T) {
    tests := []struct {
        name      string
        setupPanic func()
        wantErr   string
    }{
        {
            name: "nil pointer panic",
            setupPanic: func() { var x *int; _ = *x },
            wantErr: "panic during",
        },
    }
    
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            err := recoverableOperation(tt.setupPanic)
            assert.Error(t, err)
            assert.Contains(t, err.Error(), tt.wantErr)
        })
    }
}
```

Test with: `make test-unit` and verify panics are caught properly.

---

Task 2: Implement Error Aggregation for Validation

Priority: High
Estimated Effort: Large
Focus Area: Error Experience Engineering

Description:
Implement multi-error aggregation using errors.Join() (Go 1.20+) in validation files to show users all validation errors at once instead of failing on first error. This dramatically improves feedback loops and developer experience.

Acceptance Criteria:

• Update validation files to collect all errors before returning
• Use errors.Join() to aggregate multiple validation errors
• Maintain error ordering (most critical first)
• Update tests to verify multi-error behavior
• Ensure console formatting works with aggregated errors

Code Region: pkg/workflow/*validation*.go (focus on 5 files without structured errors)

Implement error aggregation in validation files to show all errors at once.

Priority files to update:
1. `pkg/workflow/bundler_runtime_validation.go`
2. `pkg/workflow/repository_features_validation.go`
3. `pkg/workflow/dispatch_workflow_validation.go`
4. `pkg/workflow/template_validation.go`
5. `pkg/workflow/strict_mode_validation.go`

Pattern to implement:
```go
func ValidateConfiguration(config *Config) error {
    var errs []error
    
    // Collect all validation errors
    if err := validateField1(config); err != nil {
        errs = append(errs, err)
    }
    
    if err := validateField2(config); err != nil {
        errs = append(errs, err)
    }
    
    if err := validateField3(config); err != nil {
        errs = append(errs, err)
    }
    
    // Return aggregated errors
    if len(errs) > 0 {
        return errors.Join(errs...)
    }
    
    return nil
}
```

Requirements:
1. Replace early-return pattern with error collection
2. Use `errors.Join()` for multi-error aggregation
3. Order errors by severity (critical → high → medium → low)
4. Ensure each error message is self-contained and actionable
5. Update tests to verify all errors are returned

Example test:
```go
func TestValidation_MultipleErrors(t *testing.T) {
    config := &Config{
        InvalidField1: "bad",
        InvalidField2: "worse",
    }
    
    err := ValidateConfiguration(config)
    
    require.Error(t, err)
    assert.Contains(t, err.Error(), "invalid field1")
    assert.Contains(t, err.Error(), "invalid field2")
}
```

Impact: Users see ALL validation issues in one run instead of fixing errors one-by-one.

Test with: `make test-unit` and verify multiple errors are reported together.

---

Task 3: Console Format Top 5 CLI Error-Heavy Files

Priority: High
Estimated Effort: Large
Focus Area: Error Experience Engineering

Description:
Apply console.FormatErrorMessage() formatting to the top 5 CLI files with highest unformatted error counts (388 total errors, representing 25% of all unformatted errors). This provides maximum user-facing impact.

Acceptance Criteria:

• Update all error outputs in add_interactive.go (107 errors)
• Update all error outputs in init.go (100 errors)
• Update all error outputs in add_command.go (72 errors)
• Update all error outputs in audit_report_render.go (61 errors)
• Update all error outputs in trial_command.go (48 errors)
• All fmt.Fprintln(os.Stderr, err) → fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))
• Verify no stdout error outputs remain
• Test commands manually to verify formatting

Code Region: pkg/cli/add_interactive.go, pkg/cli/init.go, pkg/cli/add_command.go, pkg/cli/audit_report_render.go, pkg/cli/trial_command.go

Apply console formatting to top 5 CLI files with most unformatted errors.

Files to update (in priority order):
1. `pkg/cli/add_interactive.go` - 107 unformatted errors
2. `pkg/cli/init.go` - 100 unformatted errors
3. `pkg/cli/add_command.go` - 72 unformatted errors
4. `pkg/cli/audit_report_render.go` - 61 unformatted errors
5. `pkg/cli/trial_command.go` - 48 unformatted errors

Pattern to replace:
```go
// ❌ BEFORE - Raw error output
if err != nil {
    fmt.Fprintln(os.Stderr, err)
    return err
}

// ✅ AFTER - Console formatted error
if err != nil {
    fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))
    return err
}
```

Also fix stdout anti-patterns:
```go
// ❌ BEFORE - Error to stdout
if err != nil {
    fmt.Println(err)
    return err
}

// ✅ AFTER - Error to stderr with formatting
if err != nil {
    fmt.Fprintln(os.Stderr, console.FormatErrorMessage(err.Error()))
    return err
}
```

Requirements:
1. Replace ALL `fmt.Fprintln(os.Stderr, err)` with `console.FormatErrorMessage(err.Error())`
2. Replace ALL `fmt.Fprintf(os.Stderr, "...%v", err)` with formatted errors
3. Fix any `fmt.Print*` error outputs (should use `os.Stderr`)
4. Import `"github.com/githubnext/gh-aw/pkg/console"` where needed
5. Test each command manually to verify formatting appears correctly

Testing:
```bash
# Test each command with intentional errors
./gh-aw init /nonexistent/path
./gh-aw add /invalid/workflow.md
./gh-aw trial nonexistent-workflow
./gh-aw audit 99999999

# Verify errors are colored/formatted in terminal
```

Impact: 388 errors (25% of all unformatted errors) will have proper console formatting.

Test with: `make agent-finish` to verify no test breakage.

---

Task 4: Add Documentation Links to Common Validation Errors

Priority: Medium
Estimated Effort: Medium
Focus Area: Error Experience Engineering

Description:
Enhance validation error messages with documentation links for common error scenarios. Currently only 7 error messages include documentation references. Target high-frequency validation errors in engine, MCP, permissions, and workflow configuration.

Acceptance Criteria:

• Identify top 20 most common validation errors from code analysis
• Add documentation links to each error message (format: See: https://github.com/githubnext/gh-aw/blob/main/docs/...)
• Ensure documentation pages exist or create missing pages
• Update validation error messages to include actionable examples
• Test error messages display correctly in terminal

Code Region: pkg/workflow/*validation*.go (focus on engine, MCP, permissions validation)

Add documentation links to common validation error messages.

Priority validation files:
1. `pkg/workflow/engine_validation.go` - Engine configuration errors
2. `pkg/workflow/mcp_config_validation.go` - MCP server configuration errors
3. `pkg/workflow/permissions_validation.go` - GitHub permissions errors
4. `pkg/workflow/github_tool_validation.go` - GitHub tools configuration errors
5. `pkg/workflow/safe_inputs_validation.go` - Safe inputs configuration errors

Pattern to implement:
```go
// ❌ BEFORE - Error without documentation
return fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom", engineID)

// ✅ AFTER - Error with documentation and example
return fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom.\n\nExample:\nengine: copilot\n\nSee: https://github.com/githubnext/gh-aw/blob/main/docs/reference/workflow-frontmatter.md#engine", engineID)
```

Requirements:
1. Add "See: [documentation URL]" to top 20 most common validation errors
2. Include inline examples in error messages where helpful
3. Verify documentation pages exist (create if missing)
4. Use consistent URL format: `https://github.com/githubnext/gh-aw/blob/main/docs/...`
5. Test error messages are readable and links are clickable in terminal

Documentation pages to reference:
- `docs/reference/workflow-frontmatter.md` - Workflow configuration
- `docs/reference/mcp-servers.md` - MCP server setup
- `docs/reference/permissions.md` - GitHub permissions
- `docs/guides/troubleshooting.md` - Common issues

Example enhanced error:
```go
return fmt.Errorf("MCP server '%s' missing required 'command' field.\n\nExample:\nmcp-servers:\n  %s:\n    command: npx\n    args:\n      - -y\n      - `@modelcontextprotocol/server-github`\n\nSee: https://github.com/githubnext/gh-aw/blob/main/docs/reference/mcp-servers.md#configuration", serverName, serverName)
```

Impact: Users get instant access to documentation when errors occur.

Test with: `make test-unit` and manually trigger validation errors.

---

Task 5: Implement Retry Logic for Transient Failures

Priority: Medium
Estimated Effort: Medium
Focus Area: Error Experience Engineering

Description:
Implement retry logic with exponential backoff for transient failures in GitHub API calls, MCP server connections, and network operations. Currently only 30 retry mechanisms exist. Add standardized retry infrastructure with configurable policies.

Acceptance Criteria:

• Create reusable retry utility package (pkg/retry/)
• Implement exponential backoff with jitter
• Add retry logic to GitHub API client operations
• Add retry logic to MCP server initialization
• Add retry logic to network-dependent operations
• Include retry attempts and delays in debug logs
• Make retry policies configurable (max attempts, initial delay, max delay)
• Test retry behavior with simulated transient failures

Code Region: Create pkg/retry/, update pkg/github/, pkg/workflow/mcp_*.go

Implement standardized retry logic for transient failures.

Create new retry infrastructure:
1. `pkg/retry/retry.go` - Core retry logic with exponential backoff
2. `pkg/retry/retry_test.go` - Comprehensive retry tests
3. `pkg/retry/policies.go` - Predefined retry policies

Apply retry logic to:
- `pkg/github/client.go` - GitHub API operations
- `pkg/workflow/mcp_gateway_config.go` - MCP Gateway initialization
- `pkg/workflow/mcp_local_config.go` - MCP local server startup
- Network-dependent validation operations

Pattern to implement:
```go
// pkg/retry/retry.go
package retry

import (
    "context"
    "fmt"
    "math"
    "math/rand"
    "time"
)

type Policy struct {
    MaxAttempts int
    InitialDelay time.Duration
    MaxDelay time.Duration
    Multiplier float64
}

var (
    DefaultPolicy = Policy{
        MaxAttempts: 3,
        InitialDelay: 100 * time.Millisecond,
        MaxDelay: 10 * time.Second,
        Multiplier: 2.0,
    }
    
    AggressivePolicy = Policy{
        MaxAttempts: 5,
        InitialDelay: 50 * time.Millisecond,
        MaxDelay: 5 * time.Second,
        Multiplier: 1.5,
    }
)

func Do(ctx context.Context, policy Policy, operation func() error) error {
    var err error
    
    for attempt := 0; attempt < policy.MaxAttempts; attempt++ {
        err = operation()
        if err == nil {
            return nil // Success
        }
        
        // Check if error is retryable
        if !IsRetryable(err) {
            return err
        }
        
        // Last attempt - don't sleep
        if attempt == policy.MaxAttempts-1 {
            break
        }
        
        // Calculate delay with exponential backoff and jitter
        delay := calculateDelay(attempt, policy)
        
        select {
        case <-ctx.Done():
            return fmt.Errorf("retry cancelled: %w", ctx.Err())
        case <-time.After(delay):
            // Continue to next attempt
        }
    }
    
    return fmt.Errorf("operation failed after %d attempts: %w", policy.MaxAttempts, err)
}

func calculateDelay(attempt int, policy Policy) time.Duration {
    delay := float64(policy.InitialDelay) * math.Pow(policy.Multiplier, float64(attempt))
    
    // Add jitter (±20%)
    jitter := delay * 0.2 * (rand.Float64()*2 - 1)
    delay += jitter
    
    // Cap at max delay
    if delay > float64(policy.MaxDelay) {
        delay = float64(policy.MaxDelay)
    }
    
    return time.Duration(delay)
}

func IsRetryable(err error) bool {
    // Check for transient errors
    // - Network timeouts
    // - HTTP 429 (rate limit)
    // - HTTP 500/502/503/504 (server errors)
    // - Connection refused
    // - EOF errors
    
    // Implementation details...
    return true // Placeholder
}
```

Usage example:
```go
// In GitHub API client
err := retry.Do(ctx, retry.DefaultPolicy, func() error {
    return gh.makeAPICall()
})

// With custom policy for MCP servers
err := retry.Do(ctx, retry.Policy{
    MaxAttempts: 5,
    InitialDelay: 200 * time.Millisecond,
    MaxDelay: 15 * time.Second,
    Multiplier: 2.0,
}, func() error {
    return connectToMCPServer()
})
```

Requirements:
1. Create reusable retry package with exponential backoff + jitter
2. Define common retry policies (Default, Aggressive, Conservative)
3. Implement retryable error detection
4. Add retry logic to GitHub API, MCP operations, network calls
5. Log retry attempts with debug logger
6. Respect context cancellation
7. Test with simulated transient failures

Testing:
```go
func TestRetry_TransientFailure(t *testing.T) {
    attempts := 0
    err := retry.Do(context.Background(), retry.DefaultPolicy, func() error {
        attempts++
        if attempts < 3 {
            return fmt.Errorf("transient error")
        }
        return nil // Success on 3rd attempt
    })
    
    assert.NoError(t, err)
    assert.Equal(t, 3, attempts)
}
```

Impact: Improved reliability for network operations and transient failures.

Test with: `make test-unit` and manual testing with flaky network.

---

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-01-26	example-workflow-portfolio-quality	Custom	Y	Identified need for beginner examples, catalog, learning path
2026-01-23	mcp-server-integration-quality	Custom	Y	Found consistency issues, documentation gaps
2026-01-22	validation-message-clarity-developer-guidance	Custom	Y	Discovered 29% guidance rate, zero console formatting
2026-01-21	dependencies	Standard	N	Found 278 deps, 134 on v0.x, zero vulnerability scanning
2026-01-20	command-interface-consistency-developer-ergonomics	Custom	Y	Only 12% implement RunX pattern, 0% have 3+ examples
2026-01-16	workflow-compilation-performance	Custom	Y	177 sequential workflows, no parallel compilation
2026-01-08	security	Standard	N	98.6% explicit permissions, but no CodeQL/gosec
2026-01-07	error-experience-engineering	Custom	Y	3% formatting, 0 panic recovery (↻ revisited today)
2026-01-06	testing	Standard	N	Exceptional 2.24:1 ratio, 320 skipped tests
2026-01-05	engine-configuration-best-practices	Custom	Y	32.6% missing engine config, no decision guide

Statistics:

• Total runs: 11 (including today)
• Custom rate: 72.7% (8/11) - exceeding 60% target ✅
• Reuse rate: 9.1% (1/11) - within 10% target ✅
• Unique areas explored: 10 distinct focus areas

---

🎯 Recommendations

Immediate Actions (This Week)

1. Add panic recovery to critical paths - Priority: High (Task 1)

   • Prevents production crashes from unexpected panics
   • Focus on compiler and GitHub API operations first

2. Console format top 5 CLI files - Priority: High (Task 3)

   • Immediate user experience improvement
   • 388 errors (25% of total) get proper formatting

Short-term Actions (This Month)

1. Implement error aggregation - Priority: High (Task 2)

   • Dramatically improves validation feedback loops
   • Users see all errors at once instead of one-by-one

2. Add documentation links to validation errors - Priority: Medium (Task 4)

   • Self-service support for common issues
   • Reduces support burden

Long-term Actions (This Quarter)

1. Implement retry logic - Priority: Medium (Task 5)

   • Improves reliability for transient failures
   • Better handling of GitHub API rate limits

2. Create error code catalog - Priority: Low

   • Standardized error reference system
   • Enables better error tracking and metrics

---

📈 Success Metrics

Track these metrics to measure improvement in Error Experience Engineering:

• Console formatting adoption: 4.4% → 25% (target after Task 3)
• Top 5 CLI files formatted: 0/5 → 5/5 (Task 3 completion)
• Panic recovery coverage: 0 → 15+ critical paths (Task 1 completion)
• Error aggregation adoption: 0% → 60% of validations (Task 2)
• Documentation links in errors: 7 → 27+ (Task 4)
• Retry-enabled operations: 30 → 60+ (Task 5)

---

Next Steps

1. Planner Agent: Split Tasks 1-5 into individual Copilot work items
2. Copilot Agent: Execute tasks in priority order (High → Medium → Low)
3. Track Progress: Monitor console formatting adoption rate weekly
4. Re-evaluate: Consider third error experience review in Q2 2026 to measure long-term impact

---

References:

• §21439978393

Generated by Repository Quality Improvement Agent
Next analysis: 2026-01-29 - Focus area will be selected based on diversity algorithm

AI generated by Repository Quality Improvement Agent

• expires on Feb 4, 2026, 1:33 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-04T13:33:57.837Z.