Skip to content

fix(npy): avoid unbounded allocation when reading the .npy header (CWE-770) - #3962

Open
li-jin-quan wants to merge 1 commit into
huggingface:mainfrom
li-jin-quan:fix/npy-header-unbounded-alloc-cwe770
Open

li-jin-quan wants to merge 1 commit into
huggingface:mainfrom
li-jin-quan:fix/npy-header-unbounded-alloc-cwe770

Conversation

@li-jin-quan

@li-jin-quan li-jin-quan commented Sep 7, 2026 •

Copy link
Copy Markdown

I found a small DoS in read_header.

It reads a length from the .npy file, then allocates that much before checking the bytes are actually there:

let mut header_len = vec![0u8; header_len_len];  // 2 bytes (v1.0) or 4 bytes (v2.0)
reader.read_exact(&mut header_len)?;
let mut header = vec![0u8; header_len];          // <-- size comes from the file
reader.read_exact(&mut header)?;

The v2.0 length is 4 bytes, so a 14-byte file can ask for up to 4 GiB.

I measured it with a counting allocator: 1 GiB allocated from 14 bytes. On a small machine this aborts the process.

Fix: read with take(header_len) instead of pre-allocating, then check the length.

file before after
14 bytes, 1 GiB header 1,073,741,824 bytes 92 bytes + a clean "truncated" error
well-formed file unchanged unchanged

take caps how much is consumed, so the rest of the stream is untouched — well-formed files behave exactly as before.

Added two tests: header roundtrip, and the truncated case.

Does this look right to you? Happy to adjust anything.

One note: Tensor::from_reader has the same shape (vec![0f32; elem_count]). I left it out on purpose to keep this PR small — happy to do it as a follow-up if you want.

@li-jin-quan
li-jin-quan force-pushed the fix/npy-header-unbounded-alloc-cwe770 branch from a30e1f2 to 529ea05 Compare September 7, 2026 08:34
…E-770)

`read_header` allocated `vec![0u8; header_len]` where `header_len` is a length
prefix read from the (untrusted) .npy file (2 bytes for v1.0, 4 bytes for
v2.0), before a single byte of the header was read. A 14-byte v2.0 file can
therefore make the parser request up to 4 GiB of memory up front.

Measured with a process-global counting allocator: a 14-byte file declaring a
1 GiB header produced a 1,073,741,824-byte zeroed allocation from 2 bytes of
actual content (~76.7M x amplification). On memory-constrained hosts this
aborts the process via `handle_alloc_error`; on larger hosts it silently
commits gigabytes for data that is not present (CWE-770, denial of service).

Read incrementally with `take(header_len).read_to_end(..)` so the allocation
is bounded by the bytes actually present, and report truncation as a normal
error. `take` caps consumption at `header_len`, so the remainder of the stream
is untouched and well-formed files behave exactly as before.

Add unit tests for the valid-header roundtrip and the truncated error path.
@li-jin-quan
li-jin-quan force-pushed the fix/npy-header-unbounded-alloc-cwe770 branch from 529ea05 to 28b000c Compare September 7, 2026 09:30
@li-jin-quan

Copy link
Copy Markdown
Author

Hi — gentle ping on this one too, about a week old.

The .npy header reader trusts header_size straight from the file. A crafted value makes it allocate before validating anything. This bounds the read by what's actually in the file.

Same question as my other PR — worth taking, or would you rather I close it?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant