Enormous CPU and memory consumption on crafted input

[Shnatsel]

Given a crafted input lewton allocates over 10 gigabytes of physical memory and spends over a minute decoding a tiny 20kb file. This issue can be used to cause denial of service.

Steps to reproduce:
RUSTFLAGS='--cfg=fuzzing' cargo run --release --example perf /path/to/malformed/file.ogg

Testcase: where-did-my-memory-go.ogg.gz

Found using AFL.rs, which categorized this issue as a hang.

This issue is distinct from #34 which is about allocating terabytes of virtual memory that is not actually filled.

[Shnatsel]

For reference, there is a long history of this kind of properties in binary format decoders. See how libpng dealt with this for an example.

@est31 since lewton's value proposition is memory safety, which makes it a good fit for handling untrusted data, but this issue undermines this use case. There has been no progress on this for 5 months. Are you okay with me including this issue in "call for participation" section in This Week In Rust?

[est31]

I haven't thought that this would be a great problem in e.g. web browsers as you as website author can bring them to a grind already if you want to do so. Just do a while true loop in js for example.

But sure, feel free to propose it in the help wanted section.

[est31]

So I've tried to unsuccessfully reproduce this bug, both on a 4 GB system and a 32 GB system. Seems that now we are returning a HeaderBadFormat in header.rs line 720 (as of current master). I guess the bug was fixed by commit b312ce4 . Maybe there's later on still a hang like issue but it isn't triggered any more. I'll check how complicated creation of a regression test would be and then close the issue.

[est31]

Here is the repacked file with the corrected CRC checksums. The original file also had an ogg-level error, so using the repacking code without any modifications to it wasn't possible, but when I made it only repack the first 4 packets and set the end of stream bit for the fourth packet, I was able to create a reproducing test case.

[Shnatsel]

Thanks! I'll run the fuzzer again to see if it can produce any more inputs that trigger similar conditions.

[est31]

@Shnatsel wonderful, thank you for your contributions!

[Shnatsel]

Here's a sample triggering the problem on latest master, obtained after 5 minutes of running AFL: issue_35_hang_new.ogg.gz

This sample is less severe than before and does eventually encounter "Error: Vorbis bitstream header decode problem", but it still spends 7 seconds and 1500Mb of RAM on decoding a 18kb file before hitting the error.

[Shnatsel]

Here are some more samples from a previous run that still exhibit the issue on latest master (2c64469). The most egregious one consumes 5 seconds and 800Mb RAM for a 9Kb file.

hangs_from_older_run.zip

[est31]

@Shnatsel could you file a separate issue?