Repository navigation
Worker Self-Reported Sandbox & Hardware Compatibility - #2381
shrutiyam-glitch wants to merge 8 commits into
Conversation
…boxCompat Unify worker hardware compatibility and sandbox runtime version identity into SandboxRuntime and VersionedSandboxCompat on WorkerStatus and RegisterWorkerRequest in ateapipb and ateletpb. Instead of a standalone unversioned HardwareIdentity map, workers report a default_runtime and optional restorable_runtimes, each carrying a sandbox_class, optional informational name, and a VersionedSandboxCompat (schema version plus unique key-value AttributeEntry pairs covering both hardware features and runtime asset hashes). This allows the control plane to match snapshot compatibility as an opaque versioned attribute set and lets workers advertise multiple restorable runtime versions alongside their default cold-boot runtime.
…edSandboxCompat Update hardware.ProbeHost to return a v1 VersionedSandboxCompat carrying the host architecture as an AttributeEntry instead of the removed HardwareIdentity map. Update hardware.Matches and add hardware.MatchesCompat to evaluate compatibility on SandboxRuntime by requiring matching sandbox_class, non-empty compatibility schema version, and an exact 1-to-1 match on all key-value attributes while ignoring the informational runtime name and attribute slice ordering.
…gistration Update ateom.Report to require a SandboxClass in ReportConfig and construct the worker's DefaultRuntime combining its sandbox class with the host's v1 VersionedSandboxCompat from hardware.ProbeHost. Pass SandboxClassGvisor in cmd/ateom-gvisor and SandboxClassMicroVM in cmd/ateom-microvm when starting the worker registration loop.
Update atelet's AteomSupport.RegisterWorker handler to convert and forward DefaultRuntime and RestorableRuntimes from ateletpb to ateapipb when registering a worker with the control plane. In ate-api-server's WorkerService.RegisterWorker, replace the previous HardwareIdentity status update with DefaultRuntime and RestorableRuntimes, validate that all reported runtimes match the Worker's configured sandbox_class, and skip the store write when capacity and runtimes are unchanged. Update fake-workersync and validation tests to use DefaultRuntime with CompatVersion and SchemaVersion.
…gistration Return ateletpb.VersionedSandboxCompat from hardware.ProbeHost so ateom does not convert between ateapipb and ateletpb, and guarantee that attributes are emitted sorted by key so repeated registrations compare equal under proto.Equal. Return FailedPrecondition instead of InvalidArgument when a reported runtime's sandbox_class contradicts the stored Worker's class, document AttributeEntry fields for apitool validation, and drop the error-substring assertion in TestReportFailsFastWithoutSandboxClass.
d12dafa to
dad9dfe
Compare
dad9dfe to
b6d3345
Compare
| // Key-value compatibility attributes defined by this schema version, covering | ||
| // both runtime assets (e.g. "gvisor_asset_hash") and effective guest hardware | ||
| // features after masking (e.g. "architecture", "cpu_features"). | ||
| // Matched by exact equality of all keys and values within the same schema |
There was a problem hiding this comment.
// Matched by exact equality of all keys and values within the same schema
// version.
I think it should describe "matched against snapshots, and only the keys exist in snaphost is checked"
| // TODO: Add support for additional compatibility versions. | ||
| // | ||
| // +k8s:required | ||
| VersionedSandboxCompat compat_version = 3; |
There was a problem hiding this comment.
nit: compat_version -> version?
| // +k8s:eachVal=+k8s:maxLength=256 | ||
| map<string, string> attributes = 1; | ||
| // +k8s:maxLength=253 | ||
| string name = 2; |
There was a problem hiding this comment.
Let's drop this field since its not used yet, we can add it later if needed.
| return out | ||
| } | ||
|
|
||
| func toSandboxRuntimes(in []*ateletpb.SandboxRuntime) []*ateapipb.SandboxRuntime { |
There was a problem hiding this comment.
nit: toAteAPISandboxRuntimes?
| } | ||
| wAttrs := worker.GetAttributes() | ||
| sAttrs := snap.GetAttributes() | ||
| if len(wAttrs) != len(sAttrs) { |
There was a problem hiding this comment.
🤖 should-fix 🟡 – This length check makes matching an exact 1:1 comparison. Before this PR, Matches only checked the keys present on the snapshot. The two semantics fail differently. With exact matching, adding cpu_features (#2345) without bumping schema_version makes every existing snapshot unrestorable, and they silently fall back to cold boot. With snapshot-keys-only matching, adding a key without a bump lets old snapshots restore on any CPU. Please pick one semantics, and document in the proto that adding an attribute requires a new schema_version.
|
|
||
| // MatchesCompat reports whether two VersionedSandboxCompat values have the same | ||
| // non-empty schema version and an exact 1-to-1 match on all attributes. | ||
| func MatchesCompat(worker, snap *ateapipb.VersionedSandboxCompat) bool { |
There was a problem hiding this comment.
🤖 should-fix 🟡 – MatchesCompat is exported but has no direct table test. TestMatches does not cover these branches: both schema_version values empty, snapshot compat_version nil while the worker's is set, and both nil.
| if proto.Equal(worker.GetStatus().GetCapacity(), reported) && proto.Equal(worker.GetStatus().GetHardware(), reportedHardware) { | ||
| status := worker.GetStatus() | ||
| if proto.Equal(status.GetCapacity(), reported) && | ||
| proto.Equal(status.GetDefaultRuntime(), defaultRuntime) && |
There was a problem hiding this comment.
🤖 should-fix 🟡 – Nothing tests a change to default_runtime on its own. TestRegisterWorker_RecordsRuntimes changes capacity (1 to 4094) in the same step, so removing this proto.Equal would still pass. The documented "reordering restorable runtimes causes a rewrite" behavior of sameRuntimes is also untested. Consider folding RecordsRuntimes, RejectsRuntimeOfAnotherClass and UnclassedWorkerAcceptsAnyClass into one table with {workerClass, default, restorable, wantCode, wantWrite}. That table should also cover an unclassed Worker reporting restorable runtimes of mixed classes.
| // both runtime assets (e.g. "gvisor_asset_hash") and effective guest hardware | ||
| // features after masking (e.g. "architecture", "cpu_features"). | ||
| // Matched by exact equality of all keys and values within the same schema | ||
| // version. Producers must emit attributes sorted by key so repeated |
There was a problem hiding this comment.
🤖 should-fix 🟡 – The server neither enforces nor normalizes this ordering contract. A producer that does not sort writes to the store on every registration, which breaks the idempotency the RegisterWorker doc promises. Sorting attributes in RegisterWorker before comparing and storing would remove the contract from every producer.
| // registrations of the same runtime compare equal without spurious writes. | ||
| // | ||
| // +k8s:required | ||
| // +k8s:maxItems=32 |
There was a problem hiding this comment.
🤖 should-fix 🟡 – Neither ateapi nor atelet validation tests cover maxItems=32, for attributes or for restorable_runtimes. The atelet tests also lack the "invalid restorable_runtimes entry" case that the ateapi tests have. Validation of WorkerStatus.default_runtime and restorable_runtimes through ValidateWorkerUpdate is untested as well.
2 participants
Fixes #2348
Restoring a sandboxed actor from a checkpointed snapshot requires strict compatibility between the worker that created the snapshot and the worker restoring it — across the sandbox class (
gvisor,microvm), the host hardware (architecture,cpu_features), and the sandbox runtime assets (gvisor_asset_hash, microVM kernel/rootfs digests).SandboxRuntime&VersionedSandboxCompat):Combines
sandbox_class, an informational versionname, and acompat_version(schema_version+repeated AttributeEntry attributes) into a single self-contained message. When an actor is checkpointed, the control plane can copy the worker'sdefault_runtimedirectly onto the snapshot, and during warm restore the scheduler can match the snapshot'sSandboxRuntimeagainst candidate workers without inspecting runtime-specific fields.default_runtimevs.restorable_runtimes):Workers report both a
default_runtime(used for cold starts and new checkpoints) andrestorable_runtimes(other enabled runtime versions on the worker that can still warm-restore existing snapshots while a new default version rolls out).schema_version+AttributeEntry):compat_version.schema_version("v1") versions the compatibility attribute schema itself, whileattributesuses a list-map ofAttributeEntry(+k8s:listType=map,+k8s:listMapKey=key) so declarative validation enforces required keys, values, and length bounds on every entry.Scope of This PR vs. Follow-ups
In this PR: Establishes the
SandboxRuntime/VersionedSandboxCompatproto schema and declarative validation, updatesinternal/hardware(ProbeHostandMatches), and wires end-to-end registration ofdefault_runtime(withsandbox_classand host architecture) andrestorable_runtimesthroughateom,atelet,ate-api-server, andfake-workersync.Follow-ups:
default_runtime(name,gvisor_asset_hash) andrestorable_runtimesfromSandboxConfigonceSandboxConfigmoves toWorkerPool.cpu_features) inhardware.ProbeHost()(hardware: report per-runtime CPU attributes at worker registration #2345 - WIP)worker.status.default_runtimeonto snapshots at pause/checkpoint time and filter candidate workers withhardware.Matchesduring warm-restore scheduling.Tests pass
Appropriate changes to documentation are included in the PR