reasoning_seed_tasks.jsonl

{"id": "reasoning_001", "capability": "reasoning", "domain": "engineering", "difficulty": "medium", "prompt": "A cache-invalidation patch removes the per-key TTL check and replaces it with a global flush every 60 seconds. Constraints: latency P99 must stay under 50ms; no data may be served stale for more than 30s. Evaluate the proposal.", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "cache", "latency"]}
{"id": "reasoning_002", "capability": "reasoning", "domain": "engineering", "difficulty": "medium", "prompt": "A service has three downstream dependencies: A (critical, no fallback), B (degraded-mode fallback exists), C (optional enrichment). A circuit breaker is open on B. Under what conditions is it safe to serve a partial response vs. reject the request entirely?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "resilience"]}
{"id": "reasoning_003", "capability": "reasoning", "domain": "governance", "difficulty": "hard", "prompt": "Candidate v7 scores 0.741 overall vs incumbent 0.733. However v7's repair_success is 0.28, below the Gate v2 hard-reject floor of 0.30. Should v7 be promoted? Justify referencing the gate rules.", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "gate", "promotion"]}
{"id": "reasoning_004", "capability": "reasoning", "domain": "engineering", "difficulty": "medium", "prompt": "Engineer A's patch is 3 lines, fixes the root cause of a memory leak, but requires a 10-minute restart window. Engineer B's patch adds a background GC loop with zero downtime but masks the leak. Which approach is preferable and why?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "tradeoff"]}
{"id": "reasoning_005", "capability": "reasoning", "domain": "data", "difficulty": "hard", "prompt": "A migration script converts timestamps from local time to UTC. It has been tested on US/Eastern only. Production has records from 14 time zones, three of which observe DST on different dates than the US. What is the minimum set of facts required before running the migration safely?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "migration", "data-integrity"]}
{"id": "reasoning_006", "capability": "reasoning", "domain": "security", "difficulty": "hard", "prompt": "A feature reads user-uploaded files using the filename directly from the HTTP multipart form field as the path suffix. A reviewer says 'we validate the file extension so it is safe.' Is the reviewer correct? Name the specific attack vector if one exists.", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "security", "path-traversal"]}
{"id": "reasoning_007", "capability": "reasoning", "domain": "engineering", "difficulty": "medium", "prompt": "A load balancer routes 10% of traffic to a canary. Canary error rate: 0.8%. Baseline: 0.9%. The canary has processed 400 requests total. The team wants to promote to 100%. Is the sample size sufficient? What do you recommend?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "canary", "statistics"]}
{"id": "reasoning_008", "capability": "reasoning", "domain": "engineering", "difficulty": "medium", "prompt": "A CI pipeline runs all 4000 tests on every commit (22-minute median). A proposal adds a dependency-graph test-selection layer estimated to cut runtime to 4 minutes. What are the failure modes of this approach?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "ci", "test-selection"]}
{"id": "reasoning_009", "capability": "reasoning", "domain": "governance", "difficulty": "hard", "prompt": "A rollback restores the previous binary after a bad deploy, but a database migration that added a non-nullable column with a default has already run. What is the state of the system post-rollback and what risks remain?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "rollback", "schema"]}
{"id": "reasoning_010", "capability": "reasoning", "domain": "engineering", "difficulty": "hard", "prompt": "A feature flag system gates behaviour behind a per-user boolean in a shared cache with a 5-minute TTL. A/B testing requires a user sees the same variant for the full session duration (up to 2 hours). Is the design consistent with the A/B requirement? If not, what is the minimal fix?", "reference": {"rubric": {"identifies_constraint_conflict": true, "states_verdict": true}}, "scoring": "rubric", "tags": ["reasoning", "feature-flags", "consistency"]}