fix(npy): avoid unbounded allocation when reading the .npy header (CWE-770) - #3962
Open
li-jin-quan wants to merge 1 commit into
Open
li-jin-quan wants to merge 1 commit into
li-jin-quan wants to merge 1 commit into
Conversation
li-jin-quan
force-pushed
the
fix/npy-header-unbounded-alloc-cwe770
branch
from
September 7, 2026 08:34
a30e1f2 to
529ea05
Compare
…E-770) `read_header` allocated `vec![0u8; header_len]` where `header_len` is a length prefix read from the (untrusted) .npy file (2 bytes for v1.0, 4 bytes for v2.0), before a single byte of the header was read. A 14-byte v2.0 file can therefore make the parser request up to 4 GiB of memory up front. Measured with a process-global counting allocator: a 14-byte file declaring a 1 GiB header produced a 1,073,741,824-byte zeroed allocation from 2 bytes of actual content (~76.7M x amplification). On memory-constrained hosts this aborts the process via `handle_alloc_error`; on larger hosts it silently commits gigabytes for data that is not present (CWE-770, denial of service). Read incrementally with `take(header_len).read_to_end(..)` so the allocation is bounded by the bytes actually present, and report truncation as a normal error. `take` caps consumption at `header_len`, so the remainder of the stream is untouched and well-formed files behave exactly as before. Add unit tests for the valid-header roundtrip and the truncated error path.
li-jin-quan
force-pushed
the
fix/npy-header-unbounded-alloc-cwe770
branch
from
September 7, 2026 09:30
529ea05 to
28b000c
Compare
Author
|
Hi — gentle ping on this one too, about a week old. The .npy header reader trusts header_size straight from the file. A crafted value makes it allocate before validating anything. This bounds the read by what's actually in the file. Same question as my other PR — worth taking, or would you rather I close it? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I found a small DoS in
read_header.It reads a length from the
.npyfile, then allocates that much before checking the bytes are actually there:The v2.0 length is 4 bytes, so a 14-byte file can ask for up to 4 GiB.
I measured it with a counting allocator: 1 GiB allocated from 14 bytes. On a small machine this aborts the process.
Fix: read with
take(header_len)instead of pre-allocating, then check the length.takecaps how much is consumed, so the rest of the stream is untouched — well-formed files behave exactly as before.Added two tests: header roundtrip, and the truncated case.
Does this look right to you? Happy to adjust anything.
One note:
Tensor::from_readerhas the same shape (vec![0f32; elem_count]). I left it out on purpose to keep this PR small — happy to do it as a follow-up if you want.