Skip to content

Commit ae1e970

Browse files
sjarmakclaude
sjarmak
and
claude
committed
feat: add 21 mega-repo MCP-unique tasks across 8 large OSS repos
Add tasks for Firefox (121-124), GCC (125-127), OpenJDK (128-130), Rust (131-132), Chromium (133-135), AOSP (136-137), LibreOffice (138-139), and ArangoDB (140-141). Totals: 73 MCP-unique tasks, 243 total benchmark tasks across 19 repos. - 5 new repo set fixtures (java-platform, chromium-browser, android-platform, libreoffice-desktop, arangodb-database) - 5 new mirror entries in mirror_creation_manifest.json - 21 use cases added to use_case_registry.json (IDs 121-141) - All 21 oracle_answer.json curated (12 via Sourcegraph, 9 via GitHub) - Registration in selected_mcp_unique_tasks.json + selected_benchmark_tasks.json - Fixed instruction_mcp.md for tasks 131-132 (removed incorrect local repo text) - Updated customize_mcp_skeletons.py and register_new_mcp_tasks.py for new repo sets Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent e531650 commit ae1e970

File tree

273 files changed

+26253
-187
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

273 files changed

+26253
-187
lines changed

‎CLAUDE.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -64,9 +64,11 @@ Config names encode three dimensions: `{agent}-{source}-{verifier}`.
6464
**SDLC suites** (`ccb_build`, `ccb_debug`, etc.): use **baseline-local-direct**
6565
+ **mcp-remote-direct**. Agent produces code changes; verifier checks git diffs.
6666

67-
**MCP-unique suites** (`ccb_mcp_*`): use **baseline-local-artifact** +
67+
**MCP-unique suites** (`ccb_mcp_*`): default to **baseline-local-artifact** +
6868
**mcp-remote-artifact**. Agent produces `answer.json`; verifier scores against
69-
oracle. Never use `-direct` configs for MCP-unique suites.
69+
oracle. Tasks with `"verification_modes": ["artifact", "direct"]` in the use
70+
case registry also support `-direct` configs; the verifier dispatches
71+
automatically based on the `.artifact_only_mode` sentinel in test.sh.
7072

7173
MCP configs use `Dockerfile.sg_only` (direct) or `Dockerfile.artifact_only`
7274
(artifact) so the agent must discover code via MCP tools. The verifier clones

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/environment/Dockerfile‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
FROM ubuntu:22.04
2+
3+
ENV DEBIAN_FRONTEND=noninteractive
4+
5+
# Base tools
6+
RUN apt-get update && apt-get install -y --no-install-recommends \
7+
git \
8+
ca-certificates \
9+
curl \
10+
python3 \
11+
g++ make \
12+
&& rm -rf /var/lib/apt/lists/*
13+
14+
WORKDIR /workspace
15+
16+
# Clone local checkout repos (baseline config: agent has local access to these)
17+
# No local checkout repos specified for this fixture
18+
19+
# Initialize git identity for agent commits
20+
RUN git config --global user.email "agent@example.com" && \
21+
git config --global user.name "Agent" && \
22+
git config --global safe.directory '*'
23+
24+
# Create log directories
25+
RUN mkdir -p /logs/agent /logs/verifier
26+
27+
ENTRYPOINT []

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/environment/Dockerfile.artifact_only‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# ccx-compliance-124 — artifact_only variant
2+
# No local repo clone — agent uses Sourcegraph MCP exclusively for code access.
3+
# Agent produces answer.json artifact; verifier scores the artifact.
4+
5+
FROM ubuntu:22.04
6+
7+
ENV DEBIAN_FRONTEND=noninteractive
8+
ENV SOURCEGRAPH_REPOS="sg-evals/firefox--871325b8"
9+
10+
RUN apt-get update && apt-get install -y --no-install-recommends \
11+
git \
12+
ca-certificates \
13+
python3 \
14+
curl \
15+
&& rm -rf /var/lib/apt/lists/*
16+
17+
WORKDIR /workspace
18+
19+
# Empty workspace — agent discovers code via MCP tools only
20+
RUN git init && \
21+
git config user.email "agent@example.com" && \
22+
git config user.name "Agent" && \
23+
git config --global safe.directory '*'
24+
25+
# Create log directories
26+
RUN mkdir -p /logs/agent /logs/verifier
27+
28+
# Mark artifact-only mode — verifiers and eval scripts check this flag
29+
RUN touch /tmp/.artifact_only_mode
30+
31+
ENTRYPOINT []

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/environment/Dockerfile.sg_only‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# CCX-compliance-124 — sg_only variant
2+
# No local repo clone — agent uses Sourcegraph MCP exclusively for code access.
3+
# The verifier clones mirror repos at verification time (no /repo_full/ backup).
4+
5+
FROM ubuntu:22.04
6+
7+
ENV DEBIAN_FRONTEND=noninteractive
8+
ENV SOURCEGRAPH_REPOS="sg-evals/firefox--871325b8"
9+
10+
RUN apt-get update && apt-get install -y --no-install-recommends \
11+
git \
12+
ca-certificates \
13+
python3 \
14+
curl \
15+
&& rm -rf /var/lib/apt/lists/*
16+
17+
WORKDIR /workspace
18+
19+
# Empty workspace — agent discovers code via MCP tools only
20+
RUN git init && \
21+
git config user.email "agent@example.com" && \
22+
git config user.name "Agent" && \
23+
git config --global safe.directory '*'
24+
25+
# Create log directories
26+
RUN mkdir -p /logs/agent /logs/verifier
27+
28+
# Mark sg_only mode — verifiers and eval scripts check this flag
29+
RUN touch /tmp/.sg_only_mode
30+
31+
RUN echo '{"workdir":"/workspace","repos":[{"mirror":"sg-evals/firefox--871325b8","target_dir":"firefox"}]}' > /tmp/.sg_only_clone_manifest.json
32+
33+
ENTRYPOINT []

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/instruction.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Firefox Content Security Policy Enforcement Audit
2+
3+
## Your Task
4+
5+
Audit the Content Security Policy (CSP) enforcement infrastructure in Firefox. Find all C++ source files in `mozilla-firefox/firefox` under `dom/security/` that implement CSP parsing, evaluation, and violation reporting. Specifically: 1. The file that defines `nsCSPParser` — the CSP directive parser. 2. The file that implements `nsCSPContext` — the main CSP context that holds policies. 3. The file that implements CSP violation reporting (`nsCSPUtils` or similar). 4. The header file that declares the `nsIContentSecurityPolicy` XPCOM interface. 5. The file under `dom/security/` that performs script-src evaluation for inline scripts. Report each file path and its primary class or function.
6+
7+
## Context
8+
9+
You are working on a codebase task involving repos from the compliance domain.
10+
11+
## Available Resources
12+
13+
No local repositories are pre-checked out.
14+
15+
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
16+
- `sg-evals/firefox--871325b8` (mozilla-firefox/firefox)
17+
18+
## Output Format
19+
20+
Create a file at `/workspace/answer.json` with your findings in the following structure:
21+
22+
```json
23+
{
24+
"files": [
25+
{"repo": "org/repo-name", "path": "relative/path/to/file.cpp"}
26+
],
27+
"symbols": [
28+
{"repo": "org/repo-name", "path": "relative/path/to/file.cpp", "symbol": "SymbolName"}
29+
],
30+
"chain": [
31+
{"repo": "org/repo-name", "path": "relative/path/to/file.cpp", "symbol": "FunctionName"}
32+
],
33+
"text": "Narrative explanation of your findings, citing repos and file paths."
34+
}
35+
```
36+
37+
Include only the fields relevant to this task. Your answer is evaluated against a closed-world oracle — completeness matters.
38+
39+
## Evaluation
40+
41+
Your answer will be scored on:
42+
- **File recall and precision**: Did you find all relevant files?

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/instruction_mcp.md‎

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# IMPORTANT: Source Code Access
2+
3+
**Local source files are not present.** Your workspace does not contain source code. You **MUST** use Sourcegraph MCP tools to discover, read, and understand code before making any changes.
4+
5+
**Target Repositories (version-pinned mirrors):**
6+
7+
- `github.com/sg-evals/firefox--871325b8` — use `repo:^github.com/sg-evals/firefox--871325b8$` filter
8+
9+
Scope ALL keyword_search/nls_search queries to these repos.
10+
Use the repo name as the `repo` parameter for read_file/go_to_definition/find_references.
11+
12+
13+
## Required Workflow
14+
15+
1. **Search first** — Use MCP tools to find relevant files and understand existing patterns
16+
2. **Read remotely** — Use `sg_read_file` to read full file contents from Sourcegraph
17+
3. **Edit locally** — Use Edit, Write, and Bash to create or modify files in your working directory
18+
4. **Verify locally** — Run tests with Bash to check your changes
19+
20+
## Tool Selection
21+
22+
| Goal | Tool |
23+
|------|------|
24+
| Exact symbol/string | `sg_keyword_search` |
25+
| Concepts/semantic search | `sg_nls_search` |
26+
| Trace usage/callers | `sg_find_references` |
27+
| See implementation | `sg_go_to_definition` |
28+
| Read full file | `sg_read_file` |
29+
| Browse structure | `sg_list_files` |
30+
| Find repos | `sg_list_repos` |
31+
| Search commits | `sg_commit_search` |
32+
| Track changes | `sg_diff_search` |
33+
| Compare versions | `sg_compare_revisions` |
34+
35+
**Decision logic:**
36+
1. Know the exact symbol? -> `sg_keyword_search`
37+
2. Know the concept, not the name? -> `sg_nls_search`
38+
3. Need definition of a symbol? -> `sg_go_to_definition`
39+
4. Need all callers/references? -> `sg_find_references`
40+
5. Need full file content? -> `sg_read_file`
41+
42+
## Scoping (Always Do This)
43+
44+
```
45+
repo:^github.com/ORG/REPO$ # Exact repo (preferred)
46+
repo:github.com/ORG/ # All repos in org
47+
file:.*\.ts$ # TypeScript only
48+
file:src/api/ # Specific directory
49+
```
50+
51+
Start narrow. Expand only if results are empty.
52+
53+
## Efficiency Rules
54+
55+
- Chain searches logically: search -> read -> references -> definition
56+
- Don't re-search for the same pattern; use results from prior calls
57+
- Prefer `sg_keyword_search` over `sg_nls_search` when you have exact terms
58+
- Read 2-3 related files before synthesising, rather than one at a time
59+
- Don't read 20+ remote files without writing code — once you understand the pattern, start implementing
60+
61+
## If Stuck
62+
63+
If MCP search returns no results:
64+
1. Broaden the search query (synonyms, partial identifiers)
65+
2. Try `sg_nls_search` for semantic matching
66+
3. Use `sg_list_files` to browse the directory structure
67+
4. Use `sg_list_repos` to verify the repository name
68+
69+
---
70+
71+
**Sourcegraph Repositories:** `github.com/sg-evals/firefox--871325b8`
72+
73+
# Firefox Content Security Policy Enforcement Audit
74+
75+
## Your Task
76+
77+
Audit the Content Security Policy (CSP) enforcement infrastructure in Firefox. Find all C++ source files in `mozilla-firefox/firefox` under `dom/security/` that implement CSP parsing, evaluation, and violation reporting. Specifically: 1. The file that defines `nsCSPParser` — the CSP directive parser. 2. The file that implements `nsCSPContext` — the main CSP context that holds policies. 3. The file that implements CSP violation reporting (`nsCSPUtils` or similar). 4. The header file that declares the `nsIContentSecurityPolicy` XPCOM interface. 5. The file under `dom/security/` that performs script-src evaluation for inline scripts. Report each file path and its primary class or function.
78+
79+
## Context
80+
81+
You are working on a codebase task involving repos from the compliance domain.
82+
83+
## Available Resources
84+
85+
No local repositories are pre-checked out.
86+
87+
**Note:** Additional repositories are accessible via Sourcegraph MCP tools:
88+
- `sg-evals/firefox--871325b8` (mozilla-firefox/firefox)
89+
90+
## Output Format
91+
92+
Create a file at `/workspace/answer.json` with your findings in the following structure:
93+
94+
```json
95+
{
96+
"files": [
97+
{"repo": "org/repo-name", "path": "relative/path/to/file.go"}
98+
],
99+
"symbols": [
100+
{"repo": "org/repo-name", "path": "relative/path/to/file.go", "symbol": "SymbolName"}
101+
],
102+
"chain": [
103+
{"repo": "org/repo-name", "path": "relative/path/to/file.go", "symbol": "FunctionName"}
104+
],
105+
"text": "Narrative explanation of your findings, citing repos and file paths."
106+
}
107+
```
108+
109+
Include only the fields relevant to this task. Your answer is evaluated against a closed-world oracle — completeness matters.
110+
111+
## Evaluation
112+
113+
Your answer will be scored on:
114+
- **File recall and precision**: Did you find all relevant files?

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/task.toml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "CCX-compliance-124"
5+
description = "Firefox Content Security Policy Enforcement Audit"
6+
license = "Apache-2.0"
7+
8+
[task]
9+
id = "CCX-compliance-124"
10+
repo = "org/repo"
11+
category = "compliance-audit"
12+
language = "c++"
13+
difficulty = "hard"
14+
time_limit_sec = 900
15+
mcp_suite = "ccb_mcp_compliance"
16+
use_case_id = 124
17+
repo_set_id = "mozilla-firefox"
18+
mcp_unique = true
19+
20+
[verification]
21+
type = "test"
22+
command = "bash /tests/eval.sh"
23+
24+
reward_type = "score"
25+
description = "Firefox Content Security Policy Enforcement Audit"
26+
27+
[environment]
28+
build_timeout_sec = 600.0

‎benchmarks/ccb_mcp_compliance/ccx-compliance-124/tests/eval.sh‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
#!/bin/bash
2+
# eval.sh — MCP-unique benchmark evaluator for CCX-compliance-124
3+
# Exit-code-first (SWE-Factory pattern):
4+
# exit 0 — agent produced useful output (composite score > 0)
5+
# exit 1 — total failure (composite score == 0 or missing answer)
6+
#
7+
# Writes /logs/verifier/reward.txt with the composite score [0.0, 1.0]
8+
9+
set -euo pipefail
10+
11+
TASK_ID="CCX-compliance-124"
12+
ANSWER_PATH="/workspace/answer.json"
13+
TASK_SPEC_PATH="/tests/task_spec.json"
14+
ORACLE_CHECKS="/tests/oracle_checks.py"
15+
REWARD_PATH="/logs/verifier/reward.txt"
16+
17+
mkdir -p /logs/verifier
18+
19+
echo "=== CCX-compliance-124 evaluator ==="
20+
echo "Task spec: $TASK_SPEC_PATH"
21+
echo "Answer: $ANSWER_PATH"
22+
echo ""
23+
24+
# sg_only mode guard: restore full repo if verifier wrapper exists
25+
if [ -f /tmp/.sg_only_mode ] && [ -f /tests/sgonly_verifier_wrapper.sh ]; then
26+
echo "sg_only mode: sourcing verifier wrapper..."
27+
source /tests/sgonly_verifier_wrapper.sh
28+
fi
29+
30+
# Verify answer file exists
31+
if [ ! -f "$ANSWER_PATH" ]; then
32+
echo "ERROR: answer.json not found at $ANSWER_PATH"
33+
echo "0.0" > "$REWARD_PATH"
34+
exit 1
35+
fi
36+
37+
# Validate answer is valid JSON
38+
if ! python3 -c "import json; json.load(open('$ANSWER_PATH'))" 2>/dev/null; then
39+
echo "ERROR: answer.json is not valid JSON"
40+
echo "0.0" > "$REWARD_PATH"
41+
exit 1
42+
fi
43+
44+
echo "answer.json found and valid JSON"
45+
46+
# Run oracle checks
47+
if [ ! -f "$ORACLE_CHECKS" ]; then
48+
echo "ERROR: oracle_checks.py not found at $ORACLE_CHECKS"
49+
echo "0.0" > "$REWARD_PATH"
50+
exit 1
51+
fi
52+
53+
echo "Running oracle checks..."
54+
SCORE=$(python3 "$ORACLE_CHECKS" --answer "$ANSWER_PATH" --spec "$TASK_SPEC_PATH" --verbose 2>&1 | tee /dev/stderr | tail -1)
55+
56+
# Validate score is a number
57+
if ! echo "$SCORE" | python3 -c "import sys; float(sys.stdin.read().strip())" 2>/dev/null; then
58+
echo "ERROR: oracle_checks.py did not return a valid score: $SCORE"
59+
echo "0.0" > "$REWARD_PATH"
60+
exit 1
61+
fi
62+
63+
echo ""
64+
echo "Composite score: $SCORE"
65+
echo "$SCORE" > "$REWARD_PATH"
66+
67+
# Exit based on score (SWE-Factory exit-code-first pattern)
68+
python3 -c "import sys; sys.exit(0 if float('$SCORE') > 0 else 1)"

0 commit comments

Comments
 (0)