Fix the cloud execution path: KeyError, provider interface mismatch, placeholder hostnames #119

Part of #108 · Phase 3

Problem

The cloud execution path has never run successfully. Four independent defects, any one of which is fatal.

1. Producer/consumer key mismatch — guaranteed KeyError

clustrix/utils.py:151-160 (serialize_function) emits:

{"function": <bytes>, "args": <pickled bytes>, ...}

clustrix/executor_cloud.py:390-392 reads:

func_data['func']            # KeyError
func_data.get('args', ())

This is unconditional — not a regression, not an edge case. It fails on the first line of the first real invocation. Its survival in the tree is proof the path was never executed, and is the clearest single piece of evidence for why this plan prioritizes real execution over more tests.

2. Provider interface mismatch — 4 of 5 providers dead on arrival

executor_cloud.py:222 requires hasattr(cloud_provider, "create_instance"). The providers define something else:

All four therefore fall through to raise NotImplementedError at executor_cloud.py:234.

3. Fabricated hostnames escape into the SSH path

get_cluster_config() error branches return invented hosts, which executor_cloud.py:260 then tries to SSH into:

• cloud_providers/gcp.py:544 -> placeholder.gcp.com
• cloud_providers/azure.py:669 -> placeholder.azure.com
• cloud_providers/lambda_cloud.py:327,338 -> placeholder.lambdalabs.com

An error path that returns a plausible-looking value instead of raising is worse than a crash: the failure surfaces later, somewhere else, as a confusing connection error.

4. Unsafe result handling in the cloud/K8s paths

executor_kubernetes.py:339 returns results by parsing stdout with ast.literal_eval on print(f'CLUSTRIX_RESULT:{result}'). Any non-literal return value silently degrades to a string. And executor_kubernetes.py:304-308 returns "completed" on any API exception for tracked jobs.

Acceptance criteria

• A single serialization contract, defined in one place, with a test asserting producer and consumer agree on every key
• One provider interface (ABC), implemented by all providers; hasattr duck-typing dispatch removed
• Every provider either satisfies the interface or is removed — no shipped NotImplementedError on an advertised feature
• Error paths raise; no function returns a fabricated hostname. grep -rn "placeholder\." clustrix/ returns nothing
• K8s results travel as a structured, typed payload — not literal_eval of stdout
• K8s status never reports "completed" on an exception
• At least one cloud backend runs a real function end to end in CI (see the HF Jobs issue)
• README.md claims are reduced to what actually works (see the docs issue)

Verification

grep -rn "NotImplementedError" clustrix/executor_cloud.py    # empty
grep -rn "placeholder\." clustrix/                            # empty