Memory leak during automatic topic compaction

[racorn]

Describe the bug
When the automatic log compaction is triggered for a topic within a namespace with active compaction-threshold, there is severe leak of direct memory, eventually causing broker shutdown:

[bookkeeper-io-12-3] ERROR org.apache.pulsar.PulsarBrokerStarter - -- Shutting down - Received OOM exception: failed to allocate 16777216 byte(s) of direct memory (used: 6442450944, max: 6442450944)
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 6442450944, max: 6442450944)
        at ...

Digging into this, it seems the phase two of compaction leaks most.

To Reproduce
Steps to reproduce the behavior:

1. start pulsar-broker with leak detection 'Paranoid'
   -Dpulsar.allocator.leak_detection=Paranoid

2. Create namespace and set compaction-threshold

bin/pulsar-admin namespaces create -c pulsar-cluster default/compacted
bin/pulsar-admin namespaces set-compaction-threshold -t 4M default/compacted

3. Start producing messages on a topic in the compacted namespace, wait for compaction to be triggered. Batching must be used for the produces, and no compression must be used (to leak direct memory)
4. grep for 'LEAK' in the pulsar broker log file
5. May take some time to wait for OOM error, but leaks should be logged at first compaction run

Expected behavior
Topic compaction should not leak memory

Screenshots

Showing typical direct memory usage before OOM

Additional context
Linux 4.15.x

[racorn]

I will create a pull request to fix this.

[jiazhai]

Thanks @racorn for your great help.