Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FileDataLoader fails to read the file when size > INT32_MAX (#4435)
Summary: On macOS, the `read` function will fail with an `EINVAL` error if the size parameter exceeds `INT32_MAX`. This update addresses the issue by adding a check to ensure that the read size does not surpass `INT32_MAX`. On Linux, the maximum permissible read size is 2,147,479,552 bytes ( < `INT32_MAX`), so attempting to read beyond this limit is inconsequential. Pull Request resolved: #4435 Test Plan: Exporting llama3 with `python -m examples.models.llama2.export_llama --checkpoint examples/models/llama-2-7B/consolidated.00.pth --params examples/models/llama-2-7B/params.json --coreml --disable_dynamic_shape -kv ` Without fix Fails with `invalid argument` error. With fix Succeeds. Reviewed By: kirklandsign Differential Revision: D60321719 Pulled By: shoumikhin fbshipit-source-id: fca265c6c1edc628b38a5044693ec7bbe0c0b43a
- Loading branch information