Add per-CPU memory cap support for SLURM clusters - #354
Open
mpvanderschelling wants to merge 1 commit into
Open
Conversation
Clusters whose job_submit filter rejects --mem and caps per-CPU memory (e.g. DelftBlue) can now set SlurmCluster.mem_per_cpu to that cap (e.g. "3968M"). Generated scripts (orchestrator + steps) then emit --mem-per-cpu derived from each step's declared per-node mem as ceil(mem / cpus_per_task), bumping cpus_per_task up to ceil(mem / cap) when the node's core:memory ratio can't supply the declared memory on the requested cores. An explicit per-resource mem_per_cpu still renders verbatim, and clusters that accept per-node --mem leave the cap None (behaviour unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #352 / #353 (per-CPU memory accounting). Adds a cluster-level
per-CPU memory cap so an entire pipeline is submittable on sites whose
job_submitfilter rejects--memand caps per-CPU memory (e.g. TU Delft'sDelftBlue, 3968 MB/core).
SlurmCluster.mem_per_cpubecomesstr | Nonecarrying the cap (e.g."3968M"). When set, every generated script (orchestrator + steps) emits--mem-per-cpuderived from each step's declared per-nodemem:ceil(mem / cpus_per_task)cpus_per_taskis bumped up toceil(mem / cap)when the node's core:memoryratio can't supply the declared memory on the requested cores (the only way to
buy more memory on such a cluster is more cores). It is only ever increased,
never reduced.
An explicit per-resource
SlurmResources.mem_per_cpu(from #352) still rendersverbatim and wins. Clusters that accept per-node
--memleave the capNone—behaviour unchanged (e.g. Brown's Oscar).
Why
With only the per-resource field from #352, every experiment/step would need a
manual DelftBlue-specific override, and the library-default orchestrator
resources (
--mem=1G) were themselves unsubmittable there. This keepstotal-memory declarations cluster-agnostic: each cluster maps them to its own
core:memory ratio.
Tests
Added 8 cases to
TestPerCpuMemorycovering the cap fit / bump / divide paths,explicit-override precedence, orchestrator rendering, and the default-
None(Oscar) path. Full
test_slurm.py+test_resources.pypass (57).🤖 Generated with Claude Code