feat: memory debug mode (RSS vs tracked heap + top allocations)#100
Merged
Conversation
Diagnostic to find what actually holds the resident memory under backup load (the OOM), instead of inferring. Enabled only when S3PROXY_TRACEMALLOC is set (zero overhead otherwise): starts tracemalloc at startup and logs the top live Python allocations (size + call site) every S3PROXY_TRACEMALLOC_INTERVAL secs and on SIGUSR1. Chart gains an extraConfig passthrough so one replica can set the flag via values; revert after capture.
Turn the tracemalloc diagnostic into a proper memory debug mode. The OOM's defining trait is the gap between real RSS (what the kernel kills on) and the Python-tracked heap: prod hit ~957MB RSS with only ~87MB tracked, i.e. the memory lived in C-level buffers (uvicorn/httptools sockets, allocator), which no top-allocations list can explain. So every interval the mode now logs RSS, tracked, untracked (rss-tracked) and the governor's active bytes side by side, then the top live Python allocations. One dump tells us which world we're in: - large untracked gap -> C-level (transport buffers), not a Python call site - small gap -> Python, and the top list names the exact line Gated by S3PROXY_MEMORY_DEBUG (alias S3PROXY_TRACEMALLOC), zero overhead when unset; dumps every S3PROXY_MEMORY_DEBUG_INTERVAL secs and on SIGUSR1.
ServerSideHannes
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A gated memory debug mode to find what actually holds resident memory under backup load — the s3proxy OOM that concurrency caps haven't resolved and that local repros can't reproduce (a pod dies at effective-6/10 while 8×52MB uploads cost only ~149MiB locally).
The defining trait of this OOM is the gap between real RSS and the Python-tracked heap: prod hit ~957MB RSS with only ~87MB tracked — the memory lives in C-level buffers (uvicorn/httptools sockets, allocator retention), which no top-allocations list can explain. So the mode logs, every interval:
One dump tells us which world we're in:
Details
s3proxy/app.py—_rss_mb()(from /proc),_dump_tracemalloc(RSS+tracked+untracked+governed),_periodic_tracemalloc,_maybe_start_tracemalloc.S3PROXY_MEMORY_DEBUG(aliasS3PROXY_TRACEMALLOC), zero overhead when unset. Dumps everyS3PROXY_MEMORY_DEBUG_INTERVALsecs (default 15) and on SIGUSR1.chart—extraConfigpassthrough;tests/unit/test_tracemalloc_profiling.py.Usage
extraConfig: { S3PROXY_MEMORY_DEBUG: "1" }+ temporarily raise pod memory to ~2Gi (survive long enough to dump).MEMORY_DEBUG/MEMORY_DEBUG_TOPfrom logs under real backup load.No behavior change unless the flag is set. Lint + unit tests pass (3.14).