Skip to content

Commit 17da13f

Browse files
sjarmakclaude
sjarmak
and
claude
committed
Add Phase 2 SDLC benchmark suite tasks, configs, scripts, and skills
- benchmarks/ccb_build: 24 new tasks (impl, refac, deps-install patterns) - benchmarks/ccb_debug: 18 new tasks (prove, audit, fault patterns) - benchmarks/ccb_design: 20 new tasks (arch, sym, dep-chain patterns) - benchmarks/ccb_document: 13 new tasks (api/arch/migration doc-gen) - benchmarks/ccb_fix: 22 new tasks (bug fixes across OSS projects) - benchmarks/ccb_secure: new tasks (security review patterns) - benchmarks/ccb_test: 14 new tasks (code review, perf, search patterns) - benchmarks/ccb_understand: new tasks - configs: SDLC phase 2config runners (build/debug/design/document/fix/secure/test) - scripts: materialize_sdlc_suites.py, rename_tasks.py - skills: full skill runbook set (19 skills) - docs: backlog plans for ccb_test (6 remaining) and ccb_document (7 remaining) Note: ccb_test targets 20 tasks (14 scaffolded, 6 in backlog); ccb_document targets 20 tasks (13 scaffolded, 7 in backlog). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 1d6eb13 commit 17da13f

File tree

1,274 files changed

+156117
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,274 files changed

+156117
-0
lines changed

‎benchmarks/ccb_build/bustub-hyperloglog-impl-001/environment/Dockerfile‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# TAC Task Wrapper: sde-implement-hyperloglog
2+
FROM ghcr.io/theagentcompany/sde-implement-hyperloglog-image:1.0.0
3+
4+
# Create logs directory for Harbor compatibility
5+
RUN mkdir -p /logs /workspace
6+
7+
# Set working directory
8+
WORKDIR /workspace
9+
10+
# Environment variables
11+
ENV TAC_SERVER_HOSTNAME=localhost
12+
ENV DECRYPTION_KEY="theagentcompany is all you need"
13+
14+
CMD ["bash"]

‎benchmarks/ccb_build/bustub-hyperloglog-impl-001/environment/docker-compose.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
services:
2+
main:
3+
build:
4+
context: ${CONTEXT_DIR}
5+
image: ${MAIN_IMAGE_NAME}
6+
command: [ "sh", "-c", "sleep infinity" ]
7+
network_mode: host
8+
environment:
9+
- TEST_DIR=${TEST_DIR}
10+
- TAC_SERVER_HOSTNAME=localhost
11+
- DECRYPTION_KEY=theagentcompany is all you need
12+
volumes:
13+
- ${HOST_VERIFIER_LOGS_PATH}:${ENV_VERIFIER_LOGS_PATH}
14+
- ${HOST_AGENT_LOGS_PATH}:${ENV_AGENT_LOGS_PATH}

‎benchmarks/ccb_build/bustub-hyperloglog-impl-001/instruction.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Implement HyperLogLog Algorithm
2+
3+
**Repository:** bustub (TheAgentCompany GitLab)
4+
**Difficulty:** HARD
5+
**Category:** ccb_tac
6+
**Task Type:** Algorithm Implementation
7+
8+
## Description
9+
10+
Implement the HyperLogLog probabilistic data structure for cardinality estimation in the
11+
bustub database system. This requires understanding the existing codebase structure,
12+
following coding conventions, and implementing the algorithm across 4 files.
13+
14+
## Task
15+
16+
Clone http://the-agent-company.com:8929/root/bustub to /workspace folder and complete
17+
http://the-agent-company.com:8929/root/bustub/-/issues/759 locally.
18+
19+
Specifically, complete 4 files:
20+
- `bustub/src/include/primer/hyperloglog.h`
21+
- `bustub/src/include/primer/hyperloglog_presto.h`
22+
- `bustub/src/primer/hyperloglog.cpp`
23+
- `bustub/src/primer/hyperloglog_presto.cpp`
24+
25+
To ensure compatibility of testing across different operating systems, please change
26+
the line `hash = ((hash << 5) ^ (hash >> 27)) ^ bytes[i];` in your local file
27+
`src/include/common/util/hash_util.h` to:
28+
`hash = ((hash << 5) ^ (hash >> 27)) ^ static_cast<signed char>(bytes[i]);`
29+
30+
## Success Criteria
31+
32+
[x] Repository cloned successfully
33+
[x] All 4 implementation files completed
34+
[x] Unit tests pass (10 tests, graded by pass count)
35+
36+
## Notes
37+
38+
- This task uses TheAgentCompany's pre-built environment
39+
- Deterministic grading via unit tests
40+
- Score is based on number of passing tests (each test worth 0.5 points)

‎benchmarks/ccb_build/bustub-hyperloglog-impl-001/task.toml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
version = "1.0"
2+
3+
[metadata]
4+
name = "bustub-hyperloglog-impl-001"
5+
description = "Implement HyperLogLog algorithm in bustub database system"
6+
license = "MIT"
7+
author_name = "TheAgentCompany"
8+
author_email = "theagentcompany@cmu.edu"
9+
10+
[task]
11+
id = "bustub-hyperloglog-impl-001"
12+
category = "ccb_tac"
13+
language = "cpp"
14+
difficulty = "hard"
15+
time_limit_sec = 2400
16+
17+
[verification]
18+
type = "test"
19+
command = "bash /workspace/tests/test.sh"
20+
21+
reward_type = "checklist"
22+
description = "Task-specific evaluation checklist"
23+
[agent]
24+
timeout_sec = 2400.0
25+
26+
[environment]
27+
build_timeout_sec = 1800.0
28+
cpus = 4
29+
memory = "8G"
30+
storage = "20G"
31+
32+
[environment.setup_scripts]
33+
mcp_config = '''
34+
#!/bin/bash
35+
if [ -n "$SOURCEGRAPH_ACCESS_TOKEN" ] && [ -n "$SOURCEGRAPH_URL" ]; then
36+
mkdir -p /root/.config/claude
37+
cat > /root/.config/claude/mcp.json << EOF
38+
{
39+
"mcpServers": {
40+
"sourcegraph": {
41+
"command": "npx",
42+
"args": ["-y", "@sourcegraph/mcp-server"],
43+
"env": {
44+
"SRC_ACCESS_TOKEN": "$SOURCEGRAPH_ACCESS_TOKEN",
45+
"SOURCEGRAPH_URL": "$SOURCEGRAPH_URL"
46+
}
47+
}
48+
}
49+
}
50+
EOF
51+
echo "MCP configuration created"
52+
fi
53+
exit 0
54+
'''

‎benchmarks/ccb_build/bustub-hyperloglog-impl-001/tests/test.sh‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
#!/bin/bash
2+
# Reward: checklist (0.0-1.0) — task-specific evaluation criteria
3+
# TAC Task Verification: sde-implement-hyperloglog
4+
set -e
5+
6+
# sg_only_env: restore full repo before verification (no-op for regular runs)
7+
[ -f /tmp/.sg_only_mode ] && [ -f /tests/sgonly_verifier_wrapper.sh ] && source /tests/sgonly_verifier_wrapper.sh
8+
9+
10+
TRAJECTORY_PATH="${TRAJECTORY_PATH:-/logs/trajectory.jsonl}"
11+
OUTPUT_PATH="/logs/tac_result.json"
12+
13+
if [ ! -f "$TRAJECTORY_PATH" ]; then
14+
echo '[]' > "$TRAJECTORY_PATH"
15+
fi
16+
17+
echo "Running TAC evaluator for sde-implement-hyperloglog..."
18+
cd /utils
19+
20+
# Initialize TAC environment (sets up /etc/hosts and resets services)
21+
if [ -f "/utils/init.sh" ]; then
22+
SERVER_HOSTNAME="${TAC_SERVER_HOSTNAME:-localhost}" bash /utils/init.sh || true
23+
fi
24+
25+
DECRYPTION_KEY="${DECRYPTION_KEY:-theagentcompany is all you need}" \
26+
python_default /utils/eval.py \
27+
--trajectory_path "$TRAJECTORY_PATH" \
28+
--result_path "$OUTPUT_PATH" \
29+
2>&1 || {
30+
echo "TAC evaluator failed"
31+
echo '{"score": 0, "checkpoints": [], "error": "Evaluator failed"}' > "$OUTPUT_PATH"
32+
}
33+
34+
mkdir -p /logs/verifier
35+
36+
if [ -f "$OUTPUT_PATH" ]; then
37+
SCORE=$(python3 -c "
38+
import json
39+
d = json.load(open('$OUTPUT_PATH'))
40+
if 'final_score' in d:
41+
fs = d['final_score']
42+
total = fs.get('total', 0)
43+
print(round(fs['result'] / total, 4) if total > 0 else 0)
44+
elif 'score' in d:
45+
print(d['score'])
46+
else:
47+
print(0)
48+
" 2>/dev/null || echo "0")
49+
echo "TAC Score: $SCORE"
50+
echo "$SCORE" > /logs/verifier/reward.txt
51+
cp "$OUTPUT_PATH" /logs/verifier/reward.json 2>/dev/null || true
52+
exit 0
53+
else
54+
echo "0.0" > /logs/verifier/reward.txt
55+
exit 0
56+
fi

‎benchmarks/ccb_build/camel-fix-protocol-feat-001/CLAUDE.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# big-code-camel-feat-001: Implement camel-fix Component
2+
3+
This repository is large (~2M LOC). Use comprehensive search to understand existing patterns before implementing.
4+
5+
## Task Type: Feature Implementation
6+
7+
Your goal is to implement a new Camel component module. Focus on:
8+
9+
1. **Pattern discovery**: Study existing components (camel-kafka, camel-netty, camel-amqp) to understand conventions
10+
2. **File identification**: Identify ALL files that need creation (new module) and modification (parent POM)
11+
3. **Implementation**: Write code that follows existing Camel component patterns
12+
4. **Verification**: Ensure the module integrates with the parent Maven build
13+
14+
## Key Reference Files
15+
16+
- `components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaComponent.java` — component pattern
17+
- `components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaEndpoint.java` — endpoint pattern
18+
- `components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConsumer.java` — consumer pattern
19+
- `components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java` — producer pattern
20+
- `components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaConfiguration.java` — configuration pattern
21+
- `components/camel-netty/src/main/java/org/apache/camel/component/netty/NettyComponent.java` — network protocol component
22+
- `components/camel-netty/pom.xml` — POM pattern for protocol components
23+
24+
## Camel Component Conventions
25+
26+
- All components extend `DefaultComponent` and override `createEndpoint()`
27+
- Endpoints extend `DefaultEndpoint` and use `@UriEndpoint` annotation
28+
- Configuration POJOs use `@UriParams` class annotation and `@UriParam` field annotations
29+
- Consumers extend `DefaultConsumer` with `doStart()`/`doStop()` lifecycle
30+
- Producers extend `DefaultAsyncProducer` with `process(Exchange, AsyncCallback)`
31+
- Component discovery via `@Component("scheme")` annotation
32+
- Auto-generated files: configurers, URI factories, JSON schemas (via `camel-package-maven-plugin`)
33+
34+
## Output Format
35+
36+
Write your analysis to `/logs/agent/solution.md` with these required sections:
37+
38+
```markdown
39+
## Files Examined
40+
- path/to/file.ext — examined to understand [pattern/API/convention]
41+
42+
## Dependency Chain
43+
1. Define types/interfaces: path/to/types.ext
44+
2. Implement core logic: path/to/impl.ext
45+
3. Wire up integration: path/to/integration.ext
46+
4. Add tests: path/to/tests.ext
47+
48+
## Code Changes
49+
### path/to/file1.ext
50+
\`\`\`diff
51+
- old code
52+
+ new code
53+
\`\`\`
54+
55+
## Analysis
56+
[Implementation strategy, design decisions, integration approach]
57+
```
58+
59+
## Search Strategy
60+
61+
- Search for `DefaultComponent` in components/ to find component implementation patterns
62+
- Search for `@UriEndpoint` to understand endpoint annotation conventions
63+
- Search for `camel-kafka` or `camel-netty` to find reference implementations
64+
- Check `components/pom.xml` to understand module registration

‎benchmarks/ccb_build/camel-fix-protocol-feat-001/environment/Dockerfile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
FROM eclipse-temurin:17-jdk
2+
3+
WORKDIR /workspace
4+
5+
# Install dependencies
6+
RUN apt-get update && apt-get install -y \
7+
git \
8+
curl \
9+
python3 \
10+
python3-pip \
11+
npm
12+
13+
# Install Claude Code CLI
14+
RUN npm install -g @anthropic-ai/claude-code
15+
16+
# Clone Apache Camel at pinned commit (camel-4.18.0)
17+
RUN git clone --filter=blob:none --no-checkout https://github.com/apache/camel.git . && \
18+
git checkout 1006f047cdaba93c67df94219af608b993debc5a && \
19+
git config user.email "agent@example.com" && \
20+
git config user.name "Agent"
21+

‎benchmarks/ccb_build/camel-fix-protocol-feat-001/instruction.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# big-code-camel-feat-001: Implement camel-fix Component for FIX Protocol
2+
3+
## Task
4+
5+
Implement a new `camel-fix` component in Apache Camel that enables routing FIX (Financial Information eXchange) protocol messages through Camel routes. The FIX protocol is the standard electronic messaging protocol for securities trading, used by exchanges, brokers, and asset managers worldwide.
6+
7+
The component must follow Apache Camel's standard component architecture:
8+
9+
1. **FixComponent** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixComponent.java`):
10+
- Extends `DefaultComponent`
11+
- Annotated `@Component("fix")`
12+
- Creates `FixEndpoint` instances via `createEndpoint()`
13+
- Manages shared FIX engine lifecycle
14+
15+
2. **FixEndpoint** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixEndpoint.java`):
16+
- Extends `DefaultEndpoint`
17+
- Annotated `@UriEndpoint(scheme = "fix", syntax = "fix:sessionID", ...)`
18+
- Creates Consumer and Producer instances
19+
- URI format: `fix:sessionID?options`
20+
21+
3. **FixConsumer** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixConsumer.java`):
22+
- Extends `DefaultConsumer`
23+
- Receives inbound FIX messages and feeds them into Camel routes
24+
- Lifecycle management: starts/stops FIX acceptor sessions
25+
26+
4. **FixProducer** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixProducer.java`):
27+
- Extends `DefaultAsyncProducer`
28+
- Sends outbound FIX messages from Camel exchanges
29+
- Implements `process(Exchange, AsyncCallback)`
30+
31+
5. **FixConfiguration** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixConfiguration.java`):
32+
- POJO with `@UriParams` and `@UriParam` annotations
33+
- Fields: configFile, senderCompID, targetCompID, fixVersion, heartBeatInterval, socketConnectHost, socketConnectPort
34+
35+
6. **FixConstants** (`components/camel-fix/src/main/java/org/apache/camel/component/fix/FixConstants.java`):
36+
- Header constants for FIX message type, session ID, sender/target comp IDs
37+
38+
7. **Build and registration**:
39+
- `components/camel-fix/pom.xml` — Maven POM inheriting from `components` parent
40+
- `components/pom.xml` — Add `<module>camel-fix</module>` to modules list
41+
- Component descriptor files for Camel's service loader
42+
43+
Study existing components like `camel-kafka`, `camel-netty`, or `camel-amqp` for the complete pattern.
44+
45+
## Context
46+
47+
- **Repository**: apache/camel (Java, ~2M LOC)
48+
- **Category**: Feature Implementation
49+
- **Difficulty**: hard
50+
- **Subsystem Focus**: components/camel-fix/ (new module), components/pom.xml (registration)
51+
52+
## Requirements
53+
54+
1. Identify all files that need creation or modification
55+
2. Follow existing Camel component patterns (`DefaultComponent`, `DefaultEndpoint`, `@UriEndpoint`)
56+
3. Implement the component with actual code changes
57+
4. Ensure the module integrates correctly with the parent build
58+
59+
## Expected Output
60+
61+
Write your analysis to `/logs/agent/solution.md` with the following structure:
62+
63+
```
64+
## Files Examined
65+
- path/to/file1.ext — examined to understand [pattern/API/convention]
66+
- path/to/file2.ext — modified to add [feature component]
67+
...
68+
69+
## Dependency Chain
70+
1. Define types/interfaces: path/to/types.ext
71+
2. Implement core logic: path/to/impl.ext
72+
3. Wire up integration: path/to/integration.ext
73+
4. Add tests: path/to/tests.ext
74+
...
75+
76+
## Code Changes
77+
### path/to/file1.ext
78+
```diff
79+
- old code
80+
+ new code
81+
```
82+
83+
### path/to/file2.ext
84+
```diff
85+
- old code
86+
+ new code
87+
```
88+
89+
## Analysis
90+
[Explanation of implementation strategy, design decisions, and how the feature
91+
integrates with existing architecture]
92+
```
93+
94+
## Evaluation Criteria
95+
96+
- Compilation: Does the code compile after changes?
97+
- File coverage: Did you modify all necessary files?
98+
- Pattern adherence: Do changes follow existing codebase conventions?
99+
- Feature completeness: Is the feature fully implemented?

0 commit comments

Comments
 (0)