[prompt-analysis] Copilot PR Prompt Analysis - 2026-02-21

[github-actions[bot]]

Analysis of 1,000 Copilot-generated PRs from the last 30 days in github/gh-aw. Today's overall merge rate is 70.3% (703 merged, 297 closed), a slight improvement over recent days and the highest in the 13-day history tracked here.

Key Metrics

Metric	Value
Total PRs analyzed	1,000
Merged	703 (70.3%)
Closed (not merged)	297 (29.7%)
Open	0
Analysis date	2026-02-21
WIP PR merge rate	26.7%
Non-WIP PR merge rate	77.1%

---

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Bug Fix	895	642	71.7%
Feature Addition	88	56	63.6%
Refactoring	1	1	100% (too small)
Documentation	9	3	33.3% (too small)
Testing	1	0	0% (too small)
Update	4	1	25.0% (too small)

Note: The vast majority of PRs (89.5%) fall into the Bug Fix category due to broad keyword matching (fix, resolve, correct etc.). Feature Addition is the second largest distinct category.

---

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 603 words (vs 510 for closed)
• Code blocks in body: 71% of merged PRs (vs 55% of closed — +16pp)
• File references (`file.go`): 68% of merged PRs (vs 55% of closed — +13pp)
• No [WIP] tag: 77.1% merge rate vs 26.7% for WIP PRs

Example successful prompts:

1. PR fix(step-names): align Set up Go, Install UV, and Upload source inconsistencies #17386 – fix(step-names): align Set up Go, Install UV, and Upload source inconsistencies → Merged

   "Three step name inconsistencies across workflow lock files: one capitalization outlier against ~15 conforming workflows..."

2. PR fix: use 6-backtick fences in summaries to prevent escaping breakage #17384 – fix: use 6-backtick fences in summaries to prevent escaping breakage → Merged

   "Step summaries broke when safe-output body content contained triple-backtick code fences — the inner ``` prematurely closed..."

3. PR docs: add missing health command to CLI reference #14273 – docs: add missing health command to CLI reference → Merged

   "The health command was fully implemented and registered as an analysis command but absent from CLI documentation, making it undiscoverable to users."

Prompt body length success rates:

Body Length	Success Rate
Empty	0%
Medium (50–200 words)	67.8%
Long (200–500 words)	71.6%
Very Long (500+ words)	71.0%

---

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 510 words (shorter than merged)
• [WIP] tag present: 73.3% close rate (99 of 135 WIP PRs closed)
• Checklists (- [ ]) in body: 43% of closed vs 36% of merged (checklists may signal incomplete work)
• Vague/incomplete task descriptions correlated with closure

Example unsuccessful prompts:

1. PR Copilot/update entrypoint args lockdown #17382 – Copilot/update entrypoint args lockdown → Closed (empty body)

2. PR [WIP] Add initial documentation for LSP language #17343 – [WIP] Add initial documentation for LSP language → Closed

   "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan..."

3. PR fix: GitHub App token not overriding custom token in Copilot engine execution step #17260 – fix: GitHub App token not overriding custom token in Copilot engine → Closed (duplicate attempt — similar PR also closed)

---

View Detailed Structural Analysis

Body Feature Correlation with Merge Success

Feature	In Merged PRs	In Closed PRs	Difference
Has code block (```)	71%	55%	+16pp
File references (`file.ext`)	68%	55%	+13pp
References issue/PR (#123)	47%	48%	-1pp
Mentions "test"	61%	57%	+4pp
Has - [ ] checklist	36%	43%	-7pp
Mentions error/fail/bug	38%	37%	+1pp

WIP Analysis

Of 135 PRs with [WIP] in the title:

• 36 merged (26.7%)
• 99 closed (73.3%)

Of 865 non-WIP PRs:

• 667 merged (77.1%)
• 198 closed (22.9%)

WIP PRs are 2.9× more likely to be closed than merged.

Notable Patterns

• "failure" in title appeared in 22 closed PRs — these tend to be investigation/debug PRs that get superseded
• "codex" in title: evenly split (14 merged, 14 closed) — engine-specific PRs have average outcomes
• Very few truly empty bodies (1 PR) — Copilot almost always generates a description

View Recent 7-Day Trend

Date	PRs	Merged	Success Rate
2026-02-21	1,000	703	70.3% ↑
2026-02-20	1,000	693	69.6%
2026-02-19	1,000	689	69.2%
2026-02-18	1,000	697	69.7%
2026-02-17	1,000	698	69.9%
2026-02-16	1,000	687	68.7%
2026-02-14	1,000	670	67.0%

Trend: Steady improvement from 66-67% two weeks ago to 70.3% today. The last 7 days of activity show 293/394 PRs merged (74.4% — above the 30-day average).

---

Key Insights

• The single biggest predictor of closure is the [WIP] tag — WIP PRs close at 73.3% vs only 22.9% for non-WIP PRs. This is consistently observed across all 13 analysis days.
• Prompts with code blocks and file references merge more — Including specific code examples (code blocks: +16pp) and referencing specific files (file refs: +13pp) significantly correlates with successful merges, likely reflecting better task specificity.
• Prompt length matters but not dramatically — Long prompts (200–500 words) have the highest success rate (71.6%), but very long prompts (500+ words) are only marginally lower (71%). The key is quality/specificity, not just length.

---

Recommendations

1. DO: Always remove [WIP] from the title before submitting — or only create a PR once the implementation is complete. WIP PRs fail at 3× the rate of finished PRs.
2. DO: Include code blocks with concrete examples or expected behavior, and reference specific file paths in backticks — these structural elements appear in 71% and 68% of merged PRs respectively.
3. DO: Aim for 200–500 word prompt bodies that describe the problem context, the expected fix, and any relevant constraints — this sweet spot achieves 71.6% success.
4. AVOID: Checklists (- [ ]) that signal incomplete work — 43% of closed PRs use these vs only 36% of merged PRs. Use checklists only when all items will be completed.
5. AVOID: Vague or generic titles like "Update workflow" or "Improve performance" without specifics — concrete action verbs (fix, add, remove) with specific scope perform better.

---

References:

• Workflow Run §22250675243

AI generated by Copilot PR Prompt Pattern Analysis

• expires on Feb 28, 2026, 4:58 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-02-28T04:58:33.032Z.

Closed by Workflow