huge memory allocation vulnerability in mp42aac

[mondaylord]

Hi, developers of Bento4:
In the test of the binary mp42aac instrumented with ASAN. There is a huge memory allocation vulnerability in mp42aac, Bento4/Source/C++/Core/Ap4RtpAtom.cpp:50. Here is the ASAN mode output:

=================================================================
==28243==ERROR: AddressSanitizer: requested allocation size 0xffffffffe7000003 (0xffffffffe7001008 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T0)
#0 0x4f2778 in operator new[](unsigned long) /root/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:102
#1 0x6065e7 in AP4_RtpAtom::AP4_RtpAtom(unsigned int, AP4_ByteStream&) (/root/Bento4/build/mp42aac+0x6065e7)

==28243==HINT: if you don't care about these errors you may set allocator_may_return_null=1
SUMMARY: AddressSanitizer: allocation-size-too-big /root/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:102 in operator new[](unsigned long)
==28243==ABORTING

Crash input

https://github.com/17ssDP/fuzzer_crashes/blob/main/Bento4/mp42aac-alloc

Validation steps

git clone https://github.com/axiomatic-systems/Bento4
cd Bento4/
mkdir check_build && cd check_build
cmake ../ -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_FLAGS="-fsanitize=address" -DCMAKE_CXX_FLAGS="-fsanitize=address" -DCMAKE_BUILD_TYPE=Release
make -j
./mp42aac mp42aac-alloc /dev/null

Environment

Ubuntu 16.04
Clang 10.0.1
gcc 5.5