Skip to content

Commit 130b290

Browse files
sjarmakclaude
sjarmak
and
claude
committed
Add 13 remaining tasks to reach 20-task target for ccb_test and ccb_document
ccb_test (6 new tasks — now 20 total): - test-unitgen-go-001: Generate unit tests for k8s storage/value package (Go) - test-unitgen-py-001: Generate unit tests for Django cache middleware (Python) - test-integration-001: Integration tests for Flipt evaluation API (Go) - test-integration-002: Integration tests for Navidrome media scanner (Go) - test-coverage-gap-001: Coverage gap analysis for Envoy HTTP conn manager (C++) - test-coverage-gap-002: Coverage gap analysis for Kafka group coordinator (Java) ccb_document (7 new tasks — now 20 total): - docgen-inline-001: Docstrings for Django cache middleware (Python) - docgen-inline-002: Javadoc for Kafka record batch serialization (Java) - docgen-changelog-001: Changelog generation for Terraform (Go) - docgen-changelog-002: Release notes for Flipt API changes (Go) - docgen-onboard-001: Developer onboarding guide for Istio control plane (Go) - docgen-runbook-001: Operational runbook for Prometheus TSDB compaction (Go) - docgen-runbook-002: Troubleshooting runbook for Envoy connection pools (C++) All tasks use pinned commits matching other tasks in the repo. Verifiers use pattern-matching checklists (no compilation required). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 17da13f commit 130b290

File tree

59 files changed

+3317
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

59 files changed

+3317
-0
lines changed

‎benchmarks/ccb_document/docgen-changelog-001/environment/Dockerfile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
FROM golang:1.22-bookworm
2+
3+
WORKDIR /workspace
4+
5+
RUN apt-get update && apt-get install -y \
6+
git \
7+
curl \
8+
python3 \
9+
npm \
10+
ripgrep \
11+
&& rm -rf /var/lib/apt/lists/*
12+
13+
RUN npm install -g @anthropic-ai/claude-code
14+
15+
RUN git clone --filter=blob:none --no-checkout https://github.com/hashicorp/terraform.git . && \
16+
git checkout a3dc571150a7651a1a4a8b302342d26089c97795 && \
17+
git config user.email "agent@example.com" && \
18+
git config user.name "Agent"
19+
20+
RUN mkdir -p /app

‎benchmarks/ccb_document/docgen-changelog-001/instruction.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Task: Generate Terraform Changelog
2+
3+
**Repository:** hashicorp/terraform
4+
**Output:** Write your changelog to `/workspace/CHANGELOG.md`
5+
6+
## Objective
7+
8+
Generate a structured changelog entry for Terraform covering significant changes in the repository. Analyze the commit history, existing CHANGELOG.md, and source changes to produce a well-categorized changelog.
9+
10+
## Scope
11+
12+
Analyze the repository to identify:
13+
- **New features**: New commands, configuration options, provider capabilities
14+
- **Bug fixes**: Resolved issues with state management, plan/apply correctness, or CLI behavior
15+
- **Breaking changes**: Any changes that require user action or break backward compatibility
16+
- **Deprecations**: Features or behaviors being phased out
17+
- **Performance improvements**: Changes that improve plan/apply speed or memory usage
18+
19+
Focus on changes visible in the `internal/`, `command/`, and `backend/` directories.
20+
21+
## Output Format
22+
23+
Write to `/workspace/CHANGELOG.md` using Terraform's established format:
24+
25+
```markdown
26+
## [Unreleased] / next release
27+
28+
### Breaking Changes
29+
30+
- **[component]**: Description of breaking change and migration path
31+
32+
### New Features
33+
34+
- **[component]**: Description of new feature with usage example if applicable
35+
36+
### Bug Fixes
37+
38+
- **[component]**: Description of the bug and what was fixed
39+
40+
### Performance Improvements
41+
42+
- **[component]**: Description
43+
44+
### Deprecations
45+
46+
- **[component]**: What is deprecated and what to use instead
47+
```
48+
49+
## Quality Bar
50+
51+
- Every entry must reference a specific component (e.g., `backend/s3`, `command/apply`, `internal/states`)
52+
- Breaking changes must include a migration path or workaround
53+
- At least 3 entries per category that has changes
54+
- Do not fabricate entries — base everything on actual code/commit evidence
55+
56+
## Anti-Requirements
57+
58+
- Do not copy the existing CHANGELOG.md verbatim
59+
- Do not include changes that are not evidenced in the codebase

‎benchmarks/ccb_document/docgen-changelog-001/task.toml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "docgen-changelog-001"
5+
description = "Generate a structured changelog for Terraform v1.8.0 covering features, bug fixes, and breaking changes"
6+
license = "BUSL-1.1"
7+
8+
[task]
9+
id = "docgen-changelog-001"
10+
repo = "hashicorp/terraform"
11+
category = "changelog_generation"
12+
language = "go"
13+
difficulty = "medium"
14+
time_limit_sec = 1200
15+
16+
[task.metadata]
17+
complexity_dimension = "documentation_reasoning"
18+
doc_type = "changelog"
19+
target_module = "internal/,command/,backend/"
20+
output_path = "/workspace/CHANGELOG.md"
21+
22+
[verification]
23+
type = "test"
24+
command = "bash /tests/test.sh"
25+
reward_type = "checklist"
26+
description = "Weighted checklist: change categorization accuracy (0.40) + completeness (0.30) + format compliance (0.30)"
27+
28+
[environment]
29+
build_timeout_sec = 900.0
30+
31+
[environment.setup_scripts]
32+
mcp_config = """#!/bin/bash
33+
if [ -n "$SOURCEGRAPH_ACCESS_TOKEN" ] && [ -n "$SOURCEGRAPH_URL" ]; then
34+
echo "Setting up Sourcegraph MCP configuration..."
35+
mkdir -p /root/.config/claude
36+
cat > /root/.config/claude/mcp.json << 'MCPEOF'
37+
{
38+
"mcpServers": {
39+
"sourcegraph": {
40+
"command": "npx",
41+
"args": ["-y", "@sourcegraph/mcp-server"],
42+
"env": {
43+
"SRC_ACCESS_TOKEN": "$SOURCEGRAPH_ACCESS_TOKEN",
44+
"SOURCEGRAPH_URL": "$SOURCEGRAPH_URL"
45+
}
46+
}
47+
}
48+
}
49+
MCPEOF
50+
echo "OK MCP configuration created"
51+
else
52+
echo "No Sourcegraph credentials provided, MCP disabled"
53+
fi
54+
exit 0
55+
"""

‎benchmarks/ccb_document/docgen-changelog-001/tests/ground_truth.json‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
{
2+
"task_id": "docgen-changelog-001",
3+
"description": "Terraform Changelog Generation",
4+
"weights": {
5+
"categorization": 0.4,
6+
"completeness": 0.3,
7+
"format": 0.3
8+
},
9+
"categorization": [
10+
{
11+
"id": "c1",
12+
"description": "Features section present",
13+
"patterns": [
14+
"## New Features|### New Features|## Features|### Features|## Enhancements"
15+
],
16+
"weight": 0.25
17+
},
18+
{
19+
"id": "c2",
20+
"description": "Bug fixes section present",
21+
"patterns": [
22+
"## Bug Fixes|### Bug Fixes|## Bugs|### Bugs"
23+
],
24+
"weight": 0.25
25+
},
26+
{
27+
"id": "c3",
28+
"description": "Breaking changes section present",
29+
"patterns": [
30+
"## Breaking|### Breaking|BREAKING"
31+
],
32+
"weight": 0.25
33+
},
34+
{
35+
"id": "c4",
36+
"description": "Component references in entries",
37+
"patterns": [
38+
"backend/|command/|internal/|provider"
39+
],
40+
"weight": 0.25
41+
}
42+
],
43+
"completeness": [
44+
{
45+
"id": "co1",
46+
"description": "At least 3 entries total",
47+
"patterns": [
48+
"^-\\s+|^\\*\\s+"
49+
],
50+
"weight": 0.5
51+
},
52+
{
53+
"id": "co2",
54+
"description": "References to actual Terraform subsystems",
55+
"patterns": [
56+
"state|plan|apply|provider|backend|workspace|module"
57+
],
58+
"weight": 0.5
59+
}
60+
],
61+
"format": [
62+
{
63+
"id": "f1",
64+
"description": "Markdown heading structure used",
65+
"patterns": [
66+
"^##|^###"
67+
],
68+
"weight": 0.5
69+
},
70+
{
71+
"id": "f2",
72+
"description": "Consistent entry format (dash or bullet)",
73+
"patterns": [
74+
"^- |^\\* "
75+
],
76+
"weight": 0.5
77+
}
78+
]
79+
}

‎benchmarks/ccb_document/docgen-changelog-001/tests/test.sh‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
#!/bin/bash
2+
# Reward: checklist (0.0-1.0) — documentation quality score
3+
set -uo pipefail
4+
5+
[ -f /tmp/.sg_only_mode ] && [ -f /tests/sgonly_verifier_wrapper.sh ] && source /tests/sgonly_verifier_wrapper.sh
6+
7+
mkdir -p /logs/verifier
8+
9+
OUTPUT_FILE="/workspace/CHANGELOG.md"
10+
EXTRA_FILES=""
11+
12+
# Fallback: search for any .md file the agent wrote
13+
if [ ! -f "$OUTPUT_FILE" ]; then
14+
ALT=$(find /workspace -maxdepth 2 -name "*.md" ! -name "README*" ! -name "CONTRIBUTING*" 2>/dev/null | head -1)
15+
OUTPUT_FILE="${ALT:-$OUTPUT_FILE}"
16+
fi
17+
18+
if [ ! -f "$OUTPUT_FILE" ]; then
19+
echo "No output file found at /workspace/CHANGELOG.md"
20+
echo "0.0" > /logs/verifier/reward.txt
21+
exit 0
22+
fi
23+
24+
# Write scorer script
25+
cat > /tmp/_scorer.py << 'SCOREEOF'
26+
import sys, re, json, os
27+
28+
output_path = sys.argv[1]
29+
extra_paths = sys.argv[2:-1]
30+
gt_path = sys.argv[-1]
31+
32+
try:
33+
content = open(output_path).read()
34+
gt = json.load(open(gt_path))
35+
except Exception as e:
36+
print(f"Error reading files: {e}", file=sys.stderr)
37+
print("0.0", end="")
38+
sys.exit(0)
39+
40+
# Also read any extra source files (for inline doc tasks)
41+
extra_content = ""
42+
for ep in extra_paths:
43+
if ep != gt_path and os.path.isfile(ep):
44+
try:
45+
extra_content += "\n" + open(ep).read()
46+
except Exception:
47+
pass
48+
49+
full_content = content + "\n" + extra_content
50+
51+
weights = gt.get("weights", {})
52+
53+
def check_section(section_items):
54+
if not section_items:
55+
return 0.0
56+
total_weight = sum(item.get("weight", 1.0 / len(section_items)) for item in section_items)
57+
section_score = 0.0
58+
for item in section_items:
59+
item_weight = item.get("weight", 1.0 / len(section_items))
60+
patterns = item.get("patterns", [])
61+
matched = False
62+
for p in patterns:
63+
try:
64+
if re.search(p, full_content, re.MULTILINE | re.IGNORECASE):
65+
matched = True
66+
break
67+
except re.error:
68+
if p.lower() in full_content.lower():
69+
matched = True
70+
break
71+
if matched:
72+
section_score += item_weight / total_weight if total_weight > 0 else 0.0
73+
print(f" PASS: {item['description']}", file=sys.stderr)
74+
else:
75+
print(f" FAIL: {item['description']}", file=sys.stderr)
76+
return section_score
77+
78+
total_score = 0.0
79+
for section_key, section_weight in weights.items():
80+
section_items = gt.get(section_key, [])
81+
section_score = check_section(section_items)
82+
contribution = section_score * float(section_weight)
83+
total_score += contribution
84+
print(f"Section '{section_key}': {section_score:.2f} * {section_weight} = {contribution:.3f}", file=sys.stderr)
85+
86+
total_score = min(1.0, max(0.0, total_score))
87+
print(f"Total score: {total_score:.2f}", file=sys.stderr)
88+
print(f"{total_score:.2f}", end="")
89+
90+
SCOREEOF
91+
92+
SCORE=$(python3 /tmp/_scorer.py "$OUTPUT_FILE" $EXTRA_FILES /tests/ground_truth.json 2>&1 | tail -1)
93+
# Re-run to get clean score (stderr goes to log, stdout is score)
94+
SCORE=$(python3 /tmp/_scorer.py "$OUTPUT_FILE" $EXTRA_FILES /tests/ground_truth.json 2>/dev/null)
95+
96+
echo "$SCORE" > /logs/verifier/reward.txt
97+
echo "Score: $SCORE"
98+
exit 0

‎benchmarks/ccb_document/docgen-changelog-002/environment/Dockerfile‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
FROM golang:1.22-bookworm
2+
3+
WORKDIR /workspace
4+
5+
RUN apt-get update && apt-get install -y \
6+
git \
7+
curl \
8+
python3 \
9+
npm \
10+
ripgrep \
11+
&& rm -rf /var/lib/apt/lists/*
12+
13+
RUN npm install -g @anthropic-ai/claude-code
14+
15+
RUN git clone --filter=blob:none --no-checkout https://github.com/flipt-io/flipt.git . && \
16+
git checkout 3d5a345f94c2adc8a0eaa102c189c08ad4c0f8e8 && \
17+
git config user.email "agent@example.com" && \
18+
git config user.name "Agent"
19+
20+
RUN mkdir -p /app

‎benchmarks/ccb_document/docgen-changelog-002/instruction.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Task: Generate Flipt Release Notes
2+
3+
**Repository:** flipt-io/flipt
4+
**Output:** Write your release notes to `/workspace/RELEASE_NOTES.md`
5+
6+
## Objective
7+
8+
Generate comprehensive release notes for Flipt summarizing API changes. Flipt is an open-source feature flag management system with REST and gRPC APIs. Analyze the codebase to identify what changed in terms of the public API surface.
9+
10+
## Scope
11+
12+
Analyze the following areas for API changes:
13+
- `rpc/flipt/` — protobuf API definitions (gRPC surface)
14+
- `internal/server/` — REST API handlers
15+
- `internal/storage/` — storage backend changes
16+
- `ui/` — frontend changes if any significant ones exist
17+
- `CHANGELOG.md` in the repository (use as reference, not as copy)
18+
19+
## Output Format
20+
21+
Write to `/workspace/RELEASE_NOTES.md`:
22+
23+
```markdown
24+
# Flipt Release Notes
25+
26+
## Breaking Changes
27+
28+
> These changes require action from users upgrading.
29+
30+
- **[API/Storage/Config]**: Description + migration path
31+
32+
## New Features
33+
34+
- **[component]**: Description with API example if applicable
35+
36+
## Deprecations
37+
38+
- **[component]**: What is deprecated, what to use instead, when it will be removed
39+
40+
## Bug Fixes
41+
42+
- **[component]**: Description
43+
44+
## Upgrade Guide
45+
46+
Step-by-step instructions for users upgrading from the previous version.
47+
```
48+
49+
## Quality Bar
50+
51+
- Every breaking change must include a migration path
52+
- Every deprecation must specify the replacement
53+
- New features must reference the specific API endpoint or config option
54+
- The upgrade guide must be actionable (numbered steps)
55+
56+
## Anti-Requirements
57+
58+
- Do not copy the existing CHANGELOG.md
59+
- Do not fabricate API endpoints — verify in rpc/ or internal/server/

0 commit comments

Comments
 (0)