Skip to content

fix(serve,pdf): container-aware resource tuning — CPU quota budget, arena-off default, memory-ceiling admission control - #266

Merged
artiz merged 1 commit into
docling-project:masterfrom
artiz:claude/resource-tuning-262-263
Aug 14, 2026
Merged

fix(serve,pdf): container-aware resource tuning — CPU quota budget, arena-off default, memory-ceiling admission control#266
artiz merged 1 commit into
docling-project:masterfrom
artiz:claude/resource-tuning-262-263

Conversation

@artiz

@artiz artiz commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Two production reports from constrained deployments, one PR.

#262 — thread pools vs cgroup CPU limits. Every pool now sizes from a cgroup-aware budget (docling_core::env::cpu_budget): host parallelism clamped by the container quota, read straight from cpu.max (v2) and cfs_quota_us/period_us (v1) as a belt-and-braces clamp for runtimes where available_parallelism still reports the host cores — the issue's 8 threads under a 4-CPU limit. The shared TableFormer session, which used to take the raw host width on top of the already-sized worker pools (the oversubscription behind 128 throttled periods and ~66% higher peak memory in the report), now takes the shared budget, and DOCLING_RS_TF_INTRA narrows it further for memory-first deployments (the reporter's 2-thread config: 2.2 GB -> 1.3 GB peak). The ASR session sizes from the same budget. Quota parsers are unit-tested; arena on/off output is bit-identical on the PDF corpus, and the unconstrained-host default budget is unchanged, so snapshots hold.

#263 — memory ceiling + the arena ratchet. Measured on this 8-core box with redp5110_sampled.pdf: a warm server retained ~2.0 GB RSS after one conversion and kept GROWING per request — ONNX Runtime's CPU arena grows a slab for every new tensor shape (every PDF page differs) and never returns a byte, and glibc keeps freed bitmap heap mapped on top. Two levers, both measured:

  • DOCLING_RS_NO_ARENA registers the CPU EP with DisableCpuMemArena (+ initializers off-arena) at the end of ep::apply, so explicit GPU EPs keep priority. The serve binaries default it ON (explicit =0 restores): retained RSS 2031 MB and climbing -> ~700 MB and flat across large-PDF requests, at no measurable warm latency cost (33.5s vs 33.3s). Library and CLI batch use keep the arena.
  • malloc_trim(0) after every conversion (sync, streaming, async; glibc-Linux only) returns the freed page-bitmap heap.

Admission control: --max-memory-mb / DOCLING_RS_MAX_MEMORY_MB, else the container's own cgroup memory limit (v2 memory.max, v1 limit_in_bytes; 0 disables). Once process RSS (VmRSS) crosses the watermark — 85%, DOCLING_RS_MEMORY_WATERMARK_PCT overrides — sync and async submissions answer 503 + Retry-After with a message naming the numbers, instead of being accepted and OOM-killing the whole server (exit 137 takes every in-flight request with it). In-flight work finishes; /v1/config reports max_memory_mb and live rss_mb, which is also how the retention measurements above were taken. Router tests pin the 503 path, the Retry-After header, async shedding, and 0-disables.

Closes #262
Closes #263

Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

…rena-off default, memory-ceiling admission control

Two production reports from constrained deployments, one PR.

cgroup-aware budget (docling_core::env::cpu_budget): host parallelism
clamped by the container quota, read straight from cpu.max (v2) and
cfs_quota_us/period_us (v1) as a belt-and-braces clamp for runtimes
where available_parallelism still reports the host cores — the issue's
8 threads under a 4-CPU limit. The shared TableFormer session, which
used to take the raw host width on top of the already-sized worker
pools (the oversubscription behind 128 throttled periods and ~66%
higher peak memory in the report), now takes the shared budget, and
DOCLING_RS_TF_INTRA narrows it further for memory-first deployments
(the reporter's 2-thread config: 2.2 GB -> 1.3 GB peak). The ASR
session sizes from the same budget. Quota parsers are unit-tested;
arena on/off output is bit-identical on the PDF corpus, and the
unconstrained-host default budget is unchanged, so snapshots hold.

with redp5110_sampled.pdf: a warm server retained ~2.0 GB RSS after one
conversion and kept GROWING per request — ONNX Runtime's CPU arena
grows a slab for every new tensor shape (every PDF page differs) and
never returns a byte, and glibc keeps freed bitmap heap mapped on top.
Two levers, both measured:
- DOCLING_RS_NO_ARENA registers the CPU EP with DisableCpuMemArena
  (+ initializers off-arena) at the end of ep::apply, so explicit GPU
  EPs keep priority. The serve binaries default it ON (explicit =0
  restores): retained RSS 2031 MB and climbing -> ~700 MB and flat
  across large-PDF requests, at no measurable warm latency cost
  (33.5s vs 33.3s). Library and CLI batch use keep the arena.
- malloc_trim(0) after every conversion (sync, streaming, async;
  glibc-Linux only) returns the freed page-bitmap heap.

Admission control: --max-memory-mb / DOCLING_RS_MAX_MEMORY_MB, else
the container's own cgroup memory limit (v2 memory.max, v1
limit_in_bytes; 0 disables). Once process RSS (VmRSS) crosses the
watermark — 85%, DOCLING_RS_MEMORY_WATERMARK_PCT overrides — sync and
async submissions answer 503 + Retry-After with a message naming the
numbers, instead of being accepted and OOM-killing the whole server
(exit 137 takes every in-flight request with it). In-flight work
finishes; /v1/config reports max_memory_mb and live rss_mb, which is
also how the retention measurements above were taken. Router tests pin
the 503 path, the Retry-After header, async shedding, and 0-disables.

Closes docling-project#262
Closes docling-project#263

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
@artiz
artiz force-pushed the claude/resource-tuning-262-263 branch from 921d8a4 to 09271f6 Compare August 14, 2026 20:02
@artiz
artiz merged commit b5943ae into docling-project:master Aug 14, 2026
10 checks passed
@artiz
artiz deleted the claude/resource-tuning-262-263 branch August 14, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant