Skip to content

Commit e531650

Browse files
sjarmakclaude
andcommitted
feat: add 12 mega-repo MCP-unique tasks (compiler-toolchain + mozilla-firefox)
Add 8 compiler-toolchain tasks (LLVM/GCC, IDs 101-108) and 4 mozilla-firefox tasks (IDs 109-112) targeting the largest public repos not yet in CCB. New repo sets: - compiler-toolchain: sg-evals/llvm-project--a8f3c97d, sg-evals/gcc--96dfb333 - mozilla-firefox: sg-evals/firefox--871325b8 Tasks span 6 MCP suites: domain, crossrepo_tracing, onboarding, platform, security, migration, incident, and the new ccb_mcp_crossrepo suite. Each task has full file set (task.toml, instruction.md, instruction_mcp.md, 3 Dockerfiles, eval.sh, test.sh, task_spec.json, oracle_answer.json, oracle_checks.py, sgonly_verifier_wrapper.sh). Also updates 5 K8s doc-gen tasks (sg-benchmarks -> sg-evals mirror names), mirror_creation_manifest.json (101 mirrors), and docs counts (222 total tasks). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 2208455 commit e531650

File tree

179 files changed

+14943
-104
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

179 files changed

+14943
-104
lines changed

AGENTS.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ This file is the operational quick-reference for benchmark maintenance.
44
`CLAUDE.md` mirrors this file.
55

66
## Benchmark Overview
7-
8 SDLC phase suites + 6 MCP-unique suites. SDLC tasks measure code quality
7+
8 SDLC phase suites + 10 MCP-unique suites. SDLC tasks measure code quality
88
across phases: build, debug, design, document, fix, secure, test, understand.
99
MCP-unique tasks measure org-scale cross-repo discovery and retrieval.
1010
See `README.md` for the full suite table and `docs/TASK_CATALOG.md` for

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ This file is the operational quick-reference for benchmark maintenance.
44
`AGENTS.md` mirrors this file.
55

66
## Benchmark Overview
7-
8 SDLC phase suites + 8 MCP-unique suites (6 active, 2 deferred). SDLC tasks
7+
8 SDLC phase suites + 10 MCP-unique suites. SDLC tasks
88
measure code quality across phases: build, debug, design, document, fix,
99
secure, test, understand. MCP-unique tasks measure org-scale cross-repo
1010
discovery and retrieval.

benchmarks/ccb_document/k8s-apiserver-doc-gen-001/environment/Dockerfile.sg_only

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
FROM golang:1.23-bookworm
66

7-
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--stripped
7+
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--8c9c67c0
88

99
ENV DEBIAN_FRONTEND=noninteractive
1010

benchmarks/ccb_document/k8s-apiserver-doc-gen-001/instruction_mcp.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
**Local source files are not present.** Your workspace does not contain source code. You **MUST** use Sourcegraph MCP tools to discover, read, and understand code before making any changes.
44

5-
**Target Repository:** `github.com/sg-evals/kubernetes--stripped`
6-
- Use `repo:^github.com/sg-evals/kubernetes--stripped$` filter in keyword_search
7-
- Use `github.com/sg-evals/kubernetes--stripped` as the `repo` parameter for go_to_definition/find_references/read_file
5+
**Target Repository:** `github.com/sg-evals/kubernetes--8c9c67c0`
6+
- Use `repo:^github.com/sg-evals/kubernetes--8c9c67c0$` filter in keyword_search
7+
- Use `github.com/sg-evals/kubernetes--8c9c67c0` as the `repo` parameter for go_to_definition/find_references/read_file
88

99

1010
## Required Workflow
@@ -67,7 +67,7 @@ If MCP search returns no results:
6767

6868
# Task: Kubernetes API Server Architecture and Extension Guide
6969

70-
**Repository:** github.com/sg-evals/kubernetes--stripped (mirror of kubernetes/kubernetes) (stripped snapshot)
70+
**Repository:** github.com/sg-evals/kubernetes--8c9c67c0 (mirror of kubernetes/kubernetes) (stripped snapshot)
7171
**Output:** Write your document to `/workspace/documentation.md`
7272

7373
## Objective

benchmarks/ccb_document/k8s-apiserver-doc-gen-001/task.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ license = "Apache-2.0"
77

88
[task]
99
id = "k8s-apiserver-doc-gen-001"
10-
repo = "sg-evals/kubernetes--stripped"
10+
repo = "sg-evals/kubernetes--8c9c67c0"
1111
category = "architecture_doc"
1212
language = "go"
1313
difficulty = "hard"

benchmarks/ccb_document/k8s-applyconfig-doc-gen-001/environment/Dockerfile.sg_only

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
FROM golang:1.23-bookworm
66

7-
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--stripped
7+
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--8c9c67c0
88

99
ENV DEBIAN_FRONTEND=noninteractive
1010

benchmarks/ccb_document/k8s-applyconfig-doc-gen-001/instruction_mcp.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
**Local source files are not present.** Your workspace does not contain source code. You **MUST** use Sourcegraph MCP tools to discover, read, and understand code before making any changes.
44

5-
**Target Repository:** `github.com/sg-evals/kubernetes--stripped`
6-
- Use `repo:^github.com/sg-evals/kubernetes--stripped$` filter in keyword_search
7-
- Use `github.com/sg-evals/kubernetes--stripped` as the `repo` parameter for go_to_definition/find_references/read_file
5+
**Target Repository:** `github.com/sg-evals/kubernetes--8c9c67c0`
6+
- Use `repo:^github.com/sg-evals/kubernetes--8c9c67c0$` filter in keyword_search
7+
- Use `github.com/sg-evals/kubernetes--8c9c67c0` as the `repo` parameter for go_to_definition/find_references/read_file
88

99

1010
## Required Workflow
@@ -67,7 +67,7 @@ If MCP search returns no results:
6767

6868
# Task: Kubernetes Server-Side Apply Configuration Guide
6969

70-
**Repository:** github.com/sg-evals/kubernetes--stripped (mirror of kubernetes/kubernetes) (stripped snapshot)
70+
**Repository:** github.com/sg-evals/kubernetes--8c9c67c0 (mirror of kubernetes/kubernetes) (stripped snapshot)
7171
**Output:** Write your document to `/workspace/documentation.md`
7272

7373
## Objective

benchmarks/ccb_document/k8s-applyconfig-doc-gen-001/task.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ license = "Apache-2.0"
77

88
[task]
99
id = "k8s-applyconfig-doc-gen-001"
10-
repo = "sg-evals/kubernetes--stripped"
10+
repo = "sg-evals/kubernetes--8c9c67c0"
1111
category = "architecture_doc"
1212
language = "go"
1313
difficulty = "hard"

benchmarks/ccb_document/k8s-clientgo-doc-gen-001/environment/Dockerfile.sg_only

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
FROM golang:1.23-bookworm
66

7-
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--stripped
7+
ENV SOURCEGRAPH_REPO_NAME=sg-evals/kubernetes--8c9c67c0
88

99
ENV DEBIAN_FRONTEND=noninteractive
1010

benchmarks/ccb_document/k8s-clientgo-doc-gen-001/instruction_mcp.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22

33
**Local source files are not present.** Your workspace does not contain source code. You **MUST** use Sourcegraph MCP tools to discover, read, and understand code before making any changes.
44

5-
**Target Repository:** `github.com/sg-evals/kubernetes--stripped`
6-
- Use `repo:^github.com/sg-evals/kubernetes--stripped$` filter in keyword_search
7-
- Use `github.com/sg-evals/kubernetes--stripped` as the `repo` parameter for go_to_definition/find_references/read_file
5+
**Target Repository:** `github.com/sg-evals/kubernetes--8c9c67c0`
6+
- Use `repo:^github.com/sg-evals/kubernetes--8c9c67c0$` filter in keyword_search
7+
- Use `github.com/sg-evals/kubernetes--8c9c67c0` as the `repo` parameter for go_to_definition/find_references/read_file
88

99

1010
## Required Workflow
@@ -67,7 +67,7 @@ If MCP search returns no results:
6767

6868
# Task: Kubernetes client-go Systems Guide
6969

70-
**Repository:** github.com/sg-evals/kubernetes--stripped (mirror of kubernetes/kubernetes) (stripped snapshot)
70+
**Repository:** github.com/sg-evals/kubernetes--8c9c67c0 (mirror of kubernetes/kubernetes) (stripped snapshot)
7171
**Output:** Write your document to `/workspace/documentation.md`
7272

7373
## Objective

0 commit comments

Comments
 (0)