Real-world SSH tests bypass host-key verification (AutoAddPolicy at 37 sites) #148

Summary

clustrix/ssh_security.py says, in its module docstring:

Every call site in clustrix must call configure_host_key_policy on its SSHClient instead of calling set_missing_host_key_policy directly.

CLAUDE.md repeats it as a hard rule. clustrix/ itself complies — the tests do not. set_missing_host_key_policy(paramiko.AutoAddPolicy()) appears at 37 call sites across 29 files under tests/.

Why this matters beyond style

These are the real-world tests — the ones that exist specifically to prove clustrix behaves correctly against real hardware. Because they build their own SSHClient and opt into AutoAddPolicy, the host-key verification path that every real user gets is never exercised by a real SSH connection. The unit tests in tests/unit/test_host_key_policy.py cover the policy in isolation; nothing covers it end to end.

So the security fix landed in production code is, in practice, tested only against itself.

The files

tests/real_world/api_validation/validate_slurm_job_submission.py
tests/real_world/api_validation/validate_ssh_cluster_access.py
tests/real_world/api_validation/validate_ssh_venv.py
tests/real_world/cluster_validation/check_error_pickle.py
tests/real_world/cluster_validation/check_recent_job.py
tests/real_world/cluster_validation/check_slurm_logs.py
tests/real_world/cluster_validation/debug_slurm_cluster_environment.py
tests/real_world/cluster_validation/debug_slurm_jobs.py
tests/real_world/cluster_validation/debug_slurm_output_location.py
tests/real_world/cluster_validation/test_remote_filesystem_comprehensive.py
tests/real_world/cluster_validation/test_slurm_basic.py
tests/real_world/cluster_validation/test_slurm_packaging_jobs.py
tests/real_world/cluster_validation/test_slurm_shared_filesystem.py
tests/real_world/cluster_validation/test_ssh_key_automation_real_clusters.py
tests/real_world/cluster_validation/test_ssh_packaging.py
tests/real_world/test_check_latest_job.py
tests/real_world/test_examine_failing_slurm.py
tests/real_world/test_find_actual_slurm_jobs.py
tests/real_world/test_slurm_cluster_debug_correct_path.py
tests/real_world/test_slurm_cluster_full_manual.py
tests/real_world/test_slurm_cluster_job_debug.py
tests/real_world/test_slurm_cluster_job_env.py
tests/real_world/test_slurm_cluster_job_script.py
tests/real_world/test_slurm_cluster_logs_detailed.py
tests/real_world/test_slurm_cluster_logs.py
tests/real_world/test_slurm_cluster_script_debug.py
tests/real_world/test_slurm_cluster_venv_check.py
tests/real_world/test_slurm_cluster_workflow_simple.py
tests/real_world/test_ssh_real.py

(tests/unit/test_host_key_policy.py also names AutoAddPolicy, but legitimately — it is the test for the policy. Excluded from the list and the count.)

Why this is not fixed in the backend-removal PR that found it

The mechanical change is one line per site:

-    ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    configure_host_key_policy(ssh_client)

but it is not behaviour-preserving, and that is the whole point. configure_host_key_policy with no config defaults to "reject", so any of these scripts pointed at a cluster absent from the runner's known_hosts will start failing with HostKeyVerificationError instead of silently trusting the key. That is the correct behaviour and it is what a user would hit — but it means each file needs to be run against its real cluster to confirm the host is trusted and the test still passes.

None of these can be run without cluster credentials and VPN access, so a blind sweep would convert a silent security gap into a silently broken test suite. Doing it properly needs a session with real cluster access.

Definition of done

• Every site outside tests/unit/test_host_key_policy.py uses configure_host_key_policy
• Each affected test is run against its real cluster and passes
• Cluster host keys are documented as a prerequisite for real-world tests (add the ssh-keyscan command to the real-world testing guide), or the runner adds them explicitly as a setup step
• At least one real-world test asserts that an unknown host is rejected — the guarantee currently has no end-to-end coverage at all
• A lint check or test fails if AutoAddPolicy reappears outside the policy's own test