fix(serve,pdf): container-aware resource tuning — CPU quota budget, arena-off default, memory-ceiling admission control - #266
Merged
artiz merged 1 commit intoAug 14, 2026
Conversation
…rena-off default, memory-ceiling admission control Two production reports from constrained deployments, one PR. cgroup-aware budget (docling_core::env::cpu_budget): host parallelism clamped by the container quota, read straight from cpu.max (v2) and cfs_quota_us/period_us (v1) as a belt-and-braces clamp for runtimes where available_parallelism still reports the host cores — the issue's 8 threads under a 4-CPU limit. The shared TableFormer session, which used to take the raw host width on top of the already-sized worker pools (the oversubscription behind 128 throttled periods and ~66% higher peak memory in the report), now takes the shared budget, and DOCLING_RS_TF_INTRA narrows it further for memory-first deployments (the reporter's 2-thread config: 2.2 GB -> 1.3 GB peak). The ASR session sizes from the same budget. Quota parsers are unit-tested; arena on/off output is bit-identical on the PDF corpus, and the unconstrained-host default budget is unchanged, so snapshots hold. with redp5110_sampled.pdf: a warm server retained ~2.0 GB RSS after one conversion and kept GROWING per request — ONNX Runtime's CPU arena grows a slab for every new tensor shape (every PDF page differs) and never returns a byte, and glibc keeps freed bitmap heap mapped on top. Two levers, both measured: - DOCLING_RS_NO_ARENA registers the CPU EP with DisableCpuMemArena (+ initializers off-arena) at the end of ep::apply, so explicit GPU EPs keep priority. The serve binaries default it ON (explicit =0 restores): retained RSS 2031 MB and climbing -> ~700 MB and flat across large-PDF requests, at no measurable warm latency cost (33.5s vs 33.3s). Library and CLI batch use keep the arena. - malloc_trim(0) after every conversion (sync, streaming, async; glibc-Linux only) returns the freed page-bitmap heap. Admission control: --max-memory-mb / DOCLING_RS_MAX_MEMORY_MB, else the container's own cgroup memory limit (v2 memory.max, v1 limit_in_bytes; 0 disables). Once process RSS (VmRSS) crosses the watermark — 85%, DOCLING_RS_MEMORY_WATERMARK_PCT overrides — sync and async submissions answer 503 + Retry-After with a message naming the numbers, instead of being accepted and OOM-killing the whole server (exit 137 takes every in-flight request with it). In-flight work finishes; /v1/config reports max_memory_mb and live rss_mb, which is also how the retention measurements above were taken. Router tests pin the 503 path, the Retry-After header, async shedding, and 0-disables. Closes docling-project#262 Closes docling-project#263 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
artiz
force-pushed
the
claude/resource-tuning-262-263
branch
from
August 14, 2026 20:02
921d8a4 to
09271f6
Compare
This was referenced Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two production reports from constrained deployments, one PR.
#262 — thread pools vs cgroup CPU limits. Every pool now sizes from a cgroup-aware budget (docling_core::env::cpu_budget): host parallelism clamped by the container quota, read straight from cpu.max (v2) and cfs_quota_us/period_us (v1) as a belt-and-braces clamp for runtimes where available_parallelism still reports the host cores — the issue's 8 threads under a 4-CPU limit. The shared TableFormer session, which used to take the raw host width on top of the already-sized worker pools (the oversubscription behind 128 throttled periods and ~66% higher peak memory in the report), now takes the shared budget, and DOCLING_RS_TF_INTRA narrows it further for memory-first deployments (the reporter's 2-thread config: 2.2 GB -> 1.3 GB peak). The ASR session sizes from the same budget. Quota parsers are unit-tested; arena on/off output is bit-identical on the PDF corpus, and the unconstrained-host default budget is unchanged, so snapshots hold.
#263 — memory ceiling + the arena ratchet. Measured on this 8-core box with redp5110_sampled.pdf: a warm server retained ~2.0 GB RSS after one conversion and kept GROWING per request — ONNX Runtime's CPU arena grows a slab for every new tensor shape (every PDF page differs) and never returns a byte, and glibc keeps freed bitmap heap mapped on top. Two levers, both measured:
Admission control: --max-memory-mb / DOCLING_RS_MAX_MEMORY_MB, else the container's own cgroup memory limit (v2 memory.max, v1 limit_in_bytes; 0 disables). Once process RSS (VmRSS) crosses the watermark — 85%, DOCLING_RS_MEMORY_WATERMARK_PCT overrides — sync and async submissions answer 503 + Retry-After with a message naming the numbers, instead of being accepted and OOM-killing the whole server (exit 137 takes every in-flight request with it). In-flight work finishes; /v1/config reports max_memory_mb and live rss_mb, which is also how the retention measurements above were taken. Router tests pin the 503 path, the Retry-After header, async shedding, and 0-disables.
Closes #262
Closes #263
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT