fix: navprove path mismatch + pytest config — root cause of 0% scoring

Two infrastructure bugs caused all 18 task-config pairs to fail:

1. Instruction/verifier path mismatch: instructions said
   `/workspace/regression_test` (no extension) but verifiers expected
   `/workspace/regression_test.py` (with extension). Agents interpreted
   the path as a directory name and created subdirectories. Fixed all 9
   instruction.md files to specify exact file paths with extensions.

2. Pytest config interference: qutebrowser's pytest.ini caused verifier
   crashes (deprecated --strict flag, unrecognized --timeout). Fixed 4
   qutebrowser test.sh files to use `-c /dev/null` to isolate from
   project config.

Also added directory fallback in find_and_prove_verifier.sh — if agent
creates a directory instead of a file, verifier searches for test files
inside it rather than failing immediately.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: LoCoBench Bot

benchmarks/ccb_navprove/CLAUDE.md

@@ -15,7 +15,7 @@ This suite tests your ability to navigate a large codebase, locate a reported bu
 
 ## Output Requirements
 
-Write your regression test to `/workspace/regression_test.{ext}` (use the appropriate extension for the project language — `.py`, `.go`, or `.test.ts`).
+Write your regression test as a **single file** at `/workspace/regression_test.{ext}` (use the appropriate extension for the project language — `.py`, `.go`, or `.test.ts`). Do NOT create a directory — write a single file directly at that path.
 
 Your regression test MUST:
 1. **Import or invoke** the buggy component directly

benchmarks/ccb_navprove/_shared/find_and_prove_verifier.sh

@@ -37,9 +37,29 @@ write_score() {
 }
 
 # --- Edge case: agent test does not exist or is empty ---
+# Fallback: if agent created a directory instead of a file, look for test files inside
 if [[ ! -f "$AGENT_TEST_PATH" ]]; then
-    echo "ERROR: Agent test not found at $AGENT_TEST_PATH" >> "$SUMMARY_LOG"
-    write_score "0.0"
+    # Strip extension to get potential directory name
+    DIR_PATH="${AGENT_TEST_PATH%.*}"
+    # Remove .test suffix if present (e.g., regression_test.test.ts -> regression_test)
+    DIR_PATH="${DIR_PATH%.test}"
+    FOUND_TEST=""
+    if [[ -d "$DIR_PATH" ]]; then
+        echo "NOTE: $AGENT_TEST_PATH not found as file, searching directory $DIR_PATH/" >> "$SUMMARY_LOG"
+        # Look for test files matching common patterns
+        for pattern in "regression_test.*" "test_*" "*_test.*" "*.test.*"; do
+            FOUND_TEST=$(find "$DIR_PATH" -maxdepth 1 -name "$pattern" -type f | head -1)
+            [[ -n "$FOUND_TEST" ]] && break
+        done
+        if [[ -n "$FOUND_TEST" ]]; then
+            echo "NOTE: Using fallback test file: $FOUND_TEST" >> "$SUMMARY_LOG"
+            AGENT_TEST_PATH="$FOUND_TEST"
+        fi
+    fi
+    if [[ ! -f "$AGENT_TEST_PATH" ]]; then
+        echo "ERROR: Agent test not found at $AGENT_TEST_PATH" >> "$SUMMARY_LOG"
+        write_score "0.0"
+    fi
 fi
 
 if [[ ! -s "$AGENT_TEST_PATH" ]]; then

benchmarks/ccb_navprove/navprove-ansible-vault-001/instruction.md

@@ -20,8 +20,8 @@ These issues affect `ansible-galaxy collection install` when processing collecti
 ## Your Task
 
 1. Investigate the codebase to find the root cause of the tar directory extraction fragility
-2. Write a regression test at `/workspace/regression_test` (Python)
-3. Your test must be self-contained and runnable with `python3 -m pytest --timeout=60`
+2. Write a regression test as a single file at `/workspace/regression_test.py`
+3. Your test must be self-contained and runnable with `python3 -m pytest --timeout=60 /workspace/regression_test.py`
 
 ## Constraints
 

benchmarks/ccb_navprove/navprove-flipt-cache-001/instruction.md

@@ -16,8 +16,8 @@ Specifically:
 ## Your Task
 
 1. Investigate the codebase to find the root cause of these authentication limitations
-2. Write a regression test at `/workspace/regression_test` (Go)
-3. Your test must be self-contained and runnable with `go test -run TestRegression -v -timeout 60s`
+2. Write a regression test as a single file at `/workspace/regression_test.go`
+3. Your test must be self-contained and runnable with `go test -run TestRegression -v -timeout 60s /workspace/regression_test.go`
 
 ## Constraints
 

benchmarks/ccb_navprove/navprove-qb-bookmark-001/instruction.md

@@ -19,8 +19,8 @@ The same bug affects HSVA notation (HSV with alpha channel). For example, `hsva(
 ## Your Task
 
 1. Investigate the codebase to find the root cause of the incorrect hue scaling
-2. Write a regression test at `/workspace/regression_test` (Python)
-3. Your test must be self-contained and runnable with `python3 -m pytest --timeout=60`
+2. Write a regression test as a single file at `/workspace/regression_test.py`
+3. Your test must be self-contained and runnable with `python3 -m pytest -c /dev/null --timeout=60 /workspace/regression_test.py`
 
 ## Constraints
 

benchmarks/ccb_navprove/navprove-qb-bookmark-001/tests/test.sh

@@ -3,7 +3,7 @@
 # Sources the shared find_and_prove_verifier to run 2-phase majority-of-3 verification.
 
 export AGENT_TEST_PATH="/workspace/regression_test.py"
-export TEST_COMMAND="python3 -m pytest --timeout=60"
+export TEST_COMMAND="python3 -m pytest -c /dev/null --timeout=60"
 export REFERENCE_PATCH="/tests/reference_fix.patch"
 export PATCH_APPLY_DIR="/workspace"
 

benchmarks/ccb_navprove/navprove-qb-download-001/instruction.md

@@ -16,8 +16,8 @@ This is particularly problematic because cache corruption can happen for reasons
 ## Your Task
 
 1. Investigate the codebase to find the root cause of the unhandled exception during cache loading
-2. Write a regression test at `/workspace/regression_test` (Python)
-3. Your test must be self-contained and runnable with `python3 -m pytest --timeout=60`
+2. Write a regression test as a single file at `/workspace/regression_test.py`
+3. Your test must be self-contained and runnable with `python3 -m pytest -c /dev/null --timeout=60 /workspace/regression_test.py`
 
 ## Constraints
 

benchmarks/ccb_navprove/navprove-qb-download-001/tests/test.sh

@@ -3,7 +3,7 @@
 # Sources the shared find_and_prove_verifier to run 2-phase majority-of-3 verification.
 
 export AGENT_TEST_PATH="/workspace/regression_test.py"
-export TEST_COMMAND="python3 -m pytest --timeout=60"
+export TEST_COMMAND="python3 -m pytest -c /dev/null --timeout=60"
 export REFERENCE_PATCH="/tests/reference_fix.patch"
 export PATCH_APPLY_DIR="/workspace"
 

benchmarks/ccb_navprove/navprove-qb-tab-001/instruction.md

@@ -14,8 +14,8 @@ This only affects the text/foreground threshold setting. Other dark mode setting
 ## Your Task
 
 1. Investigate the codebase to find the root cause of the ignored threshold setting
-2. Write a regression test at `/workspace/regression_test` (Python)
-3. Your test must be self-contained and runnable with `python3 -m pytest --timeout=60`
+2. Write a regression test as a single file at `/workspace/regression_test.py`
+3. Your test must be self-contained and runnable with `python3 -m pytest -c /dev/null --timeout=60 /workspace/regression_test.py`
 
 ## Constraints
 

benchmarks/ccb_navprove/navprove-qb-tab-001/tests/test.sh

@@ -3,7 +3,7 @@
 # Sources the shared find_and_prove_verifier to run 2-phase majority-of-3 verification.
 
 export AGENT_TEST_PATH="/workspace/regression_test.py"
-export TEST_COMMAND="python3 -m pytest --timeout=60"
+export TEST_COMMAND="python3 -m pytest -c /dev/null --timeout=60"
 export REFERENCE_PATCH="/tests/reference_fix.patch"
 export PATCH_APPLY_DIR="/workspace"