Leak detection doesn't work

[mikea]

Leak detection is disabled because it always gives a false positive.

To reproduce:

docker run -ti -e ASAN_OPTIONS=detect_leaks=1 ossfuzz/expat run expat_parse_fuzzer -runs=100

This will generate lots of leaks like:

Indirect leak of 1 byte(s) in 1 object(s) allocated from:
    #0 0x50d000 in operator new(unsigned long) /src/llvm/projects/compiler-rt/lib/asan/asan_new_delete.cc:82
    #1 0x53fd89 in std::__1::__allocate(unsigned long) /usr/local/bin/../include/c++/v1/new:171:10
    #2 0x53fd89 in std::__1::allocator<unsigned char>::allocate(unsigned long, void const*) /usr/local/bin/../include/c++/v1/memory:1771
    #3 0x53fd89 in std::__1::allocator_traits<std::__1::allocator<unsigned char> >::allocate(std::__1::allocator<unsigned char>&, unsigned long) /usr/local/bin/../include/c++/v1/memory:1526
    #4 0x53fd89 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::allocate(unsigned long) /usr/local/bin/../include/c++/v1/vector:923
    #5 0x53f2a4 in _ZNSt3__16vectorIhNS_9allocatorIhEEE6assignIPhEENS_9enable_ifIXaasr21__is_forward_iteratorIT_EE5valuesr16is_constructibleIhNS_15iterator_traitsIS7_E9referenceEEE5valueEvE4typeES7_S7_ /usr/local/bin/../include/c++/v1/vector:1403:9
    #6 0x52d096 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::operator=(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&) /usr/local/bin/../include/c++/v1/vector:1348:9
    #7 0x52d096 in fuzzer::InputCorpus::AddToCorpus(std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > const&, unsigned long, bool) /src/libfuzzer/FuzzerCorpus.h:71
    #8 0x583213 in fuzzer::Fuzzer::ShuffleAndMinimize(std::__1::vector<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >, std::__1::allocator<std::__1::vector<unsigned char, std::__1::allocator<unsigned char> > > >*) /src/libfuzzer/FuzzerLoop.cpp:427:14
    #9 0x52059f in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:519:5
    #10 0x5aac38 in main /src/libfuzzer/FuzzerMain.cpp:20:10
    #11 0x7f9b61f2982f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)

[kcc]

Solution: add --cap-add SYS_PTRACE to docker run command line.
lsan needs ptrace and by default docker denies it.

[mikea]

I have verified that ClusterFuzz has detect_leaks=1 (@oliverchang). We'll keep detect_leaks=0 meanwhile in docker images.

lsan issue also filed: google/sanitizers#728

[kcc]

Could you please double-check again?
We haven't reported a single leak yet. This is very, very strange.

[inferno-chromium]

It is explicitly disabled in all job types. Oliver might know why. We can turn it on easily, we should just make sure it does not cause startup crashes, might cause some noise with fixed testcases. But CF should blacklist them automatically over time.

[oliverchang]

I just enabled this after doing some quick tests. Let's see if we get any reports tomorrow morning. Did something in LSan change recently? Last I tried (>1 month ago) this didn't work on our bots.

[kcc]

I don't remember any changes in LSan.
What didn't work?

[inferno-chromium]

No, changes in LSan. Just that we had this disabled in ClusterFuz via a job type environment variable. I think this might be done due to overcaution since both asan and lsan are in same job type. On chromium, we had these in different job type. But now that we have well tested automatic blacklisting and deblacklisting, this will work smoothly. And we see new leaks, see https://clusterfuzz-external.appspot.com/v2/testcases?q=leak&showall=1

[kcc]

lovely leaks!
Two of them have short stack traces, most likely because some of
the deps are built w/o -fno-omit-frame-pointers.
In the long run we'll solve this by rebuilding the world (needed for msan too).
In short term, for those two, we'll just rerun manually with fast_unwind_on_malloc=0

[mikea]

Can CF use fast_unwind_on_malloc=0 for reproducer only?

[oliverchang]

Unfortunately that might break CF, since we expect the crash state to be the same/similar during fuzzing and running a testcase for reproducibility purposes.

[kcc]

can we make lsan work in the docker? (by enabling --cap-add SYS_PTRACE)
It's pretty confusing when leaks are not reported inside the docker but then are reported by CF (e.g. #168)