Repository navigation
Debugging Pattern: Remote Execution Environment Inspection #92
Description
Summary
Document valuable debugging patterns for remote cluster execution found during repository cleanup.
Debug Pattern 1: Basic Cluster Execution Verification
From debug_cluster_execution.py - Simple test to verify if code executes remotely:
@cluster(cluster_type="ssh", cluster_host="host.edu", username="user", remote_work_dir="/tmp/test")
def simple_test():
import socket, os
return {
"hostname": socket.gethostname(),
"current_dir": os.getcwd(),
"python_version": os.sys.version,
"message": "Hello from cluster"
}
# Verify execution location
result = simple_test()
if "expected_domain.edu" in result['hostname']:
print("❌ EXECUTED LOCALLY")
else:
print("✅ EXECUTED REMOTELY")Debug Pattern 2: Progressive SSH Testing
From debug_cluster_test.py - Systematic approach to isolate cluster issues:
- Test basic SSH connection before cluster execution
- Test simple commands via direct SSH
- Test minimal cluster function with basic settings
- Disable complex features (auto_parallel, GPU detection) during debugging
def test_basic_connection():
# Direct paramiko SSH test
ssh_client = paramiko.SSHClient()
ssh_client.connect(hostname, username, password, timeout=30)
# Test command execution
stdin, stdout, stderr = ssh_client.exec_command('echo "Hello"')
output = stdout.read().decode().strip()
return output == "Hello"Debug Pattern 3: Remote Environment Inspection
From debug_venv2_execution.py - Deep inspection of remote execution environment:
@cluster(cleanup_on_success=False) # Keep files for debugging
def debug_environment():
import sys, os, torch
return {
"python_executable": sys.executable,
"python_version": sys.version,
"working_directory": os.getcwd(),
"environment_vars": dict(os.environ),
"torch_available": torch.cuda.is_available(),
"gpu_count": torch.cuda.device_count()
}Plus manual SSH inspection of remote directories and log files.
Debugging Value
These patterns provide systematic approaches for:
- Verifying remote vs local execution
- Testing SSH connectivity independently
- Inspecting remote execution environments
- Debugging environment setup issues
- Keeping debugging artifacts with
cleanup_on_success=False
Source: Repository cleanup Issue #72
Activity
Issue #72: Delete obsolete development scripts
jeremymanning commentedon Aug 17, 2026
Closing as archival, not actionable.
This issue was created as a content archive when #72 (repository cleanup) removed ad-hoc debugging scripts from the repo root. It records a debugging pattern but has no acceptance criteria, no deliverable, and no owner.
The patterns it describes now exist as executable code under
tests/real_world/(e.g.test_ndoli_slurm_logs.py,test_ndoli_slurm_logs_detailed.py,test_ndoli_debug.py,test_core_functionality_*.py), which is the right home for them.Closing to keep the backlog to items with real deliverables. The underlying need — that these real-world diagnostics actually run somewhere, rather than sitting unexecuted — is now tracked as a first-class task in the new production-readiness plan, since the scheduled real-world workflow has failed on every one of its last 60 runs.