feat(changelog): Add AI-powered summaries for verbose changelog sections

README.md

@@ -44,6 +44,117 @@ craft publish 1.2.3
 - **Workspace Support** - Handle monorepos with NPM/Yarn workspaces
 - **CI Integration** - Wait for CI to pass, download artifacts, and publish
 - **GitHub Actions** - Built-in actions for release preparation and changelog previews
+- **AI Summaries** - Summarize verbose changelog sections using AI (GitHub Models or local fallback)
+
+## AI-Powered Changelog Summaries
+
+Craft can summarize changelog sections with many entries into concise descriptions. Uses [GitHub Models](https://github.com/marketplace/models) by default, with a local fallback when no token is available.
+
+### Example 1: Craft 2.16.0
+
+**Before (6 items):**
+```markdown
+### New Features
+- Strip commit patterns from changelog entries
+- Add support for custom changelog entries from PR descriptions
+- Support for multiple entries and nested items
+- Add changelog preview action and CLI command
+- Make release workflow reusable for external repos
+- Add version templating for layer names
+```
+
+**After (section summary with expandable details):**
+```markdown
+### New Features
+Changelog entries now support custom descriptions, multiple items, previews, reusable workflows, and version templating for layers.
+
+<details>
+<summary>Show 6 items</summary>
+
+- Strip commit patterns from changelog entries
+- Add support for custom changelog entries from PR descriptions
+- Support for multiple entries and nested items
+- Add changelog preview action and CLI command
+- Make release workflow reusable for external repos
+- Add version templating for layer names
+
+</details>
+```
+
+**Top-level summary** (with `topLevel: "always"`):
+> The software release includes several new features: the ability to strip commit patterns from changelog entries, support for custom changelog entries derived from pull request descriptions, and support for multiple entries and nested items. Additionally, a changelog preview action and CLI command have been added.
+
+### Example 2: Sentry 25.12.0 (Large Release)
+
+For large releases like [Sentry 25.12.0](https://github.com/getsentry/sentry/releases/tag/25.12.0) with 31+ items:
+
+| Section | Items | Words In → Out | Compression |
+|---------|-------|----------------|-------------|
+| ACI | 11 | 98 → 41 | **58%** |
+| Agents | 8 | 58 → 24 | **59%** |
+| Seer & Triage | 12 | 80 → 31 | **61%** |
+
+**Top-level summary** (106 words):
+> The latest software release includes several updates across three main areas: ACI, Agents, and Seer & Triage. In the ACI section, the metric monitor form now defaults to the number of errors, and alerts have been updated to include a disabled status and display test notification errors in the UI. The Agents section introduces markdown rendering, the ability to switch to raw values, and a new onboarding process for browser JavaScript. Additionally, the Seer & Triage updates involve changes to support repo type checks, column renaming for broader applicability, and the removal of unnecessary calls.
+
+### Configuration
+
+```yaml
+aiSummaries:
+  enabled: true
+  kickInThreshold: 5  # Only summarize sections with >5 items
+  model: "openai/gpt-4o-mini"  # optional, default
+  topLevel: "threshold"  # "always" | "never" | "threshold" | true | false
+```
+
+#### Top-Level Summary
+
+The `topLevel` option controls whether an executive summary is generated for the entire changelog:
+
+- `"always"` or `true`: Always generate a top-level summary paragraph
+- `"never"` or `false`: Never generate a top-level summary
+- `"threshold"` (default): Only generate if total items exceed `kickInThreshold`
+
+```yaml
+# Always include an executive summary
+aiSummaries:
+  topLevel: "always"
+
+# Never include an executive summary
+aiSummaries:
+  topLevel: false
+
+# Only summarize large releases (default behavior)
+aiSummaries:
+  topLevel: "threshold"
+  kickInThreshold: 10  # Only summarize releases with >10 total items
+```
+
+### Authentication
+
+The feature uses your GitHub token automatically:
+- From `GITHUB_TOKEN` environment variable, or
+- From `gh auth token` (GitHub CLI)
+
+If no token is available, Craft falls back to a local model ([Falconsai/text_summarization](https://huggingface.co/Falconsai/text_summarization)).
+
+### Available Models
+
+**GitHub Models** (requires token):
+
+```yaml
+aiSummaries:
+  model: "openai/gpt-4o-mini"   # Default, 71% compression
+  model: "openai/gpt-4o-mini"        # Fast and capable
+  model: "openai/gpt-4o"             # Most capable
+```
+
+**Local models** (no token needed):
+
+```yaml
+aiSummaries:
+  model: "local:Falconsai/text_summarization"  # 60MB, extractive
+```
 
 ## Configuration
 

eslint.config.mjs

@@ -4,7 +4,7 @@ import prettier from 'eslint-config-prettier';
 
 export default tseslint.config(
   {
-    ignores: ['docs/**', 'dist/**', 'node_modules/**', 'coverage/**', '*.mjs', '**/*.js'],
+    ignores: ['docs/**', 'dist/**', 'node_modules/**', 'coverage/**', '**/*.mjs', '**/*.js'],
   },
   eslint.configs.recommended,
   ...tseslint.configs.recommended,

package.json

@@ -12,6 +12,7 @@
   },
   "devDependencies": {
     "@aws-sdk/client-lambda": "^3.723.0",
+    "@eslint/js": "^9.17.0",
     "@google-cloud/storage": "^7.14.0",
     "@octokit/plugin-retry": "^7.1.2",
     "@octokit/request-error": "^7.0.0",
@@ -35,9 +36,6 @@
     "@types/tar": "^4.0.0",
     "@types/tmp": "^0.0.33",
     "@types/yargs": "^17",
-    "@eslint/js": "^9.17.0",
-    "typescript-eslint": "^8.18.2",
-    "zod": "^3.24.1",
     "async": "3.2.2",
     "aws4": "^1.11.0",
     "chalk": "4.1.1",
@@ -52,7 +50,6 @@
     "git-url-parse": "^16.1.0",
     "glob": "^11.0.0",
     "is-ci": "^2.0.0",
-    "vitest": "^3.0.2",
     "js-yaml": "4.1.1",
     "mkdirp": "^1.0.4",
     "mustache": "3.0.1",
@@ -71,7 +68,11 @@
     "tar": "6.2.1",
     "tmp": "0.2.4",
     "typescript": "^5.7.2",
-    "yargs": "^18"
+    "typescript-eslint": "^8.18.2",
+    "vitest": "^3.0.2",
+    "vitest-evals": "^0.5.0",
+    "yargs": "^18",
+    "zod": "^3.24.1"
   },
   "scripts": {
     "build:fat": "tsc -p tsconfig.build.json",
@@ -84,6 +85,7 @@
     "fix": "yarn lint --fix",
     "test": "vitest run",
     "test:watch": "vitest",
+    "test:evals": "vitest run --config vitest.evals.config.ts",
     "docs:dev": "cd docs && yarn dev",
     "docs:build": "cd docs && yarn build"
   },
@@ -92,6 +94,10 @@
     "yarn": "1.22.19"
   },
   "dependencies": {
-    "marked": "^17.0.1"
+    "@github/models": "^0.0.1-beta.2",
+    "@huggingface/transformers": "^3.8.1",
+    "ai": "^6.0.3",
+    "marked": "^17.0.1",
+    "node-llama-cpp": "^3.14.5"
   }
 }