[prompt-analysis] Copilot PR Prompt Analysis - 2026-01-28

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (Dec 29, 2025 - Jan 28, 2026)
Total PRs: 1,000 | Merged: 648 (64.8%) | Closed: 342 (34.2%) | Open: 10 (1.0%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Refactor	10	9	90.0% ⭐
Test	19	15	78.9%
Other	72	52	72.2%
Feature	147	104	70.7%
Documentation	33	23	69.7%
Bug Fix	709	445	62.8%

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 156 words
• Most common keywords: update, add, file_reference, resolve, create
• Key insights:
  • 38% of merged PRs use file_reference (mention specific files)
  • 27% use code_reference (mention functions/classes)
  • 15% include fix verb (bug fixes)
  • 37% are short prompts (<50 words), 36% are long prompts (>100 words)

Example successful prompts:

View 5 Merged PR Examples

1. PR #12206: Fix TypeScript error in close_expired_discussions: add duplicateCount to return type → MERGED

   Prompt (395 words): Addresses CI failure investigating TypeScript typecheck failure. Prompt includes specific issue context, error details, and file references.

2. PR #12190: Replace redundant zero-capacity slice allocations with idiomatic zero-value declarations → MERGED

   Prompt (199 words): Specific refactoring task with clear objective to replace 10 instances of make([]T, 0) with zero-value slices. References discussion and code patterns.

3. PR #12189: Add package-level documentation to core packages → MERGED

   Prompt (382 words): Documentation task with clear scope - add package-level docs to major packages for better API discoverability.

4. PR #12176: Increase workflow health manager timeout to 30 minutes → MERGED

   Prompt (103 words): Short, focused bug fix addressing specific workflow failure with clear action.

5. PR #12174: Add automated cleanup policy for stale draft PRs → MERGED

   Prompt (364 words): Feature addition with problem statement, analysis, and proposed solution for draft PR cleanup automation.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 195 words (25% longer than merged)
• Most common keywords: resolve, long_prompt, add, fix, file_reference
• Issues identified:
  • Longer prompts don't help: Closed PRs average 39 words longer
  • Less code specificity: Only 14% use code_reference vs 15% in merged
  • More vague "resolve" tasks: 58% vs 37% in merged

Example unsuccessful prompts:

View 5 Closed PR Examples

1. PR #12197: [log] Add debug logging to utility and validation functions → CLOSED

   Prompt (131 words): Addressed formatting/linting validation but didn't provide clear value or proper implementation context.

2. PR #12191: Verify G116 Trojan Source detection is enabled (not G424) → CLOSED

   Prompt (207 words): Verification task with incorrect rule reference (G424 vs G116), suggesting confusion in requirements.

3. PR #12177: Fix: Require explicit assignee verification in auto-assign workflow → CLOSED

   Prompt (16 words): Extremely short prompt lacking context: "Debug this run and make sure the agent only assigns issues without an assignee"

4. PR #12157: Track campaign label permission errors in conclusion job failure reports → CLOSED

   Prompt (427 words): Very long prompt about permission warnings, possibly too complex or addressing symptoms rather than root cause.

5. PR #12146: Fix shell escaping for environment variable expansion in AWF → CLOSED

   Prompt (97 words): Technical fix but may have had implementation issues or been superseded by different approach.

Key Insights

Based on 1,000 PRs analyzed over 30 days, here are the most actionable findings:

1. 🎯 Refactoring > Bug Fixing: Refactoring tasks have 90% success rate vs 62.8% for bug fixes. The highest success comes from well-defined code improvements, not firefighting.

2. 📏 Length Sweet Spot: Both short (<50 words) and long (>100 words) prompts work - but extremely short prompts (<20 words) and extremely long prompts (>200 words) correlate with closures. The issue isn't length per se, but clarity.

3. 📁 File References Matter Less Than Expected: While 38% of merged PRs mention files, 37% of closed PRs also do. Simply mentioning files doesn't guarantee success - context matters more.

4. 🐛 Bug Fixes Are Hard: Bug fixes have the lowest success rate (62.8%) despite being the most common task type (709 PRs). This suggests bug fixes need better scoping or are inherently riskier.

5. 🔍 "Resolve" Is a Red Flag: The word "resolve" appears in 58% of closed PRs vs 37% of merged ones. Vague problem-solving tasks without clear solutions tend to fail.

Recommendations

Based on the data analysis, follow these best practices for higher PR success rates:

✅ DO:

1. Choose refactoring over reactive fixes (90% success)

   • Frame problems as code improvements, not just bug fixes
   • Example: "Replace X pattern with Y pattern" vs "Fix X"

2. Be specific about the desired outcome

   • Merged example: "Add duplicateCount to return type"
   • Not: "Fix TypeScript error"

3. Include concrete acceptance criteria

   • Testing tasks (78.9% success) work well because they have clear pass/fail
   • Define what "done" looks like upfront

4. Focus on one clear change

   • "Other" category (72.2%) succeeds when scope is crisp
   • Avoid multi-part tasks without clear relationship

❌ AVOID:

1. Vague "resolve" or "investigate" prompts

   • "Resolve the issue" → 58% appear in closed PRs
   • Be specific about what needs investigation and what success looks like

2. Extremely short prompts without context

   • PR Fix: Require explicit assignee verification in auto-assign workflow #12177 with 16 words closed
   • Always include: what, why, and expected outcome

3. Bug fixes without reproduction steps

   • Bug fixes have lowest success rate (62.8%)
   • Always include: current behavior, expected behavior, and how to test

4. Rambling context dumps

   • Closed PRs average 195 words vs 156 for merged
   • Front-load the ask, then provide supporting context

Historical Trends

Tracking success rates over the last 7 reports:

Date	PRs	Success Rate	Top Category	Best Success
2026-01-28	1,000	64.8%	Bug Fix (445)	Refactor (90.0%)
2026-01-26	1,000	65.5%	Bug Fix (430)	Refactor (70.3%)
2026-01-25	1,000	64.7%	Bug Fix (389)	Refactor (100.0%)
2026-01-24	1,000	64.3%	Bug Fix (328)	Refactor (83.3%)
2026-01-22	1,000	63.4%	Testing (492)	Refactor (100.0%)
2026-01-21	1,000	64.8%	Feature (480)	Testing (70.3%)
2026-01-20	1,000	65.4%	Testing (288)	Refactor (85.7%)

Trend: Success rates remain stable at 63-65% over the past week. Refactoring consistently shows highest success rates (80-100%), while bug fixes dominate by volume but have lower success rates (62-65%).

Methodology Notes

Prompt Extraction: Prompts extracted from PR bodies using the "Original prompt" section marker. For PRs without this marker, the first 500 characters of the body are used.

Categorization: Automated keyword-based categorization:

• bug_fix: Contains "fix", "resolve", "correct", "bug", "issue", "error"
• feature: Contains "add", "implement", "create", "new feature"
• refactor: Contains "refactor", "improve", "optimize", "restructure"
• documentation: Contains "document", "docs", "readme", "comment"
• test: Contains "test", "coverage", "spec"
• other: Does not match above patterns

Success Rate: Calculated as merged / (merged + closed) for completed PRs. Open PRs (10) excluded from success rate calculations.

---

Analysis generated from 1,000 Copilot PRs (Dec 29, 2025 - Jan 28, 2026)
Workflow Run: §21437296251

AI generated by Copilot PR Prompt Pattern Analysis

• expires on Feb 4, 2026, 12:04 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-04T12:04:17.982Z.