Skip to content

Add per-CPU memory cap support for SLURM clusters - #354

Open
mpvanderschelling wants to merge 1 commit into
developfrom
feat/cluster-mem-per-cpu-cap
Open

Add per-CPU memory cap support for SLURM clusters#354
mpvanderschelling wants to merge 1 commit into
developfrom
feat/cluster-mem-per-cpu-cap

Conversation

@mpvanderschelling

Copy link
Copy Markdown
Collaborator

Summary

Follow-up to #352 / #353 (per-CPU memory accounting). Adds a cluster-level
per-CPU memory cap so an entire pipeline is submittable on sites whose
job_submit filter rejects --mem and caps per-CPU memory (e.g. TU Delft's
DelftBlue, 3968 MB/core).

SlurmCluster.mem_per_cpu becomes str | None carrying the cap (e.g.
"3968M"). When set, every generated script (orchestrator + steps) emits
--mem-per-cpu derived from each step's declared per-node mem:

  • per-CPU value = ceil(mem / cpus_per_task)
  • cpus_per_task is bumped up to ceil(mem / cap) when the node's core:memory
    ratio can't supply the declared memory on the requested cores (the only way to
    buy more memory on such a cluster is more cores). It is only ever increased,
    never reduced.

An explicit per-resource SlurmResources.mem_per_cpu (from #352) still renders
verbatim and wins. Clusters that accept per-node --mem leave the cap None
behaviour unchanged (e.g. Brown's Oscar).

Why

With only the per-resource field from #352, every experiment/step would need a
manual DelftBlue-specific override, and the library-default orchestrator
resources (--mem=1G) were themselves unsubmittable there. This keeps
total-memory declarations cluster-agnostic: each cluster maps them to its own
core:memory ratio.

Tests

Added 8 cases to TestPerCpuMemory covering the cap fit / bump / divide paths,
explicit-override precedence, orchestrator rendering, and the default-None
(Oscar) path. Full test_slurm.py + test_resources.py pass (57).

🤖 Generated with Claude Code

Clusters whose job_submit filter rejects --mem and caps per-CPU memory (e.g. DelftBlue) can now set SlurmCluster.mem_per_cpu to that cap (e.g. "3968M"). Generated scripts (orchestrator + steps) then emit --mem-per-cpu derived from each step's declared per-node mem as ceil(mem / cpus_per_task), bumping cpus_per_task up to ceil(mem / cap) when the node's core:memory ratio can't supply the declared memory on the requested cores. An explicit per-resource mem_per_cpu still renders verbatim, and clusters that accept per-node --mem leave the cap None (behaviour unchanged).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant