WaitForFlushMemTables may hang up if mempurge is used

[ywave620]

Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://groups.google.com/forum/#!forum/rocksdb or https://www.facebook.com/groups/rocksdb.dev

Expected behavior

DB::Impl::Flush() returns Status::OK()

Actual behavior

DB::Impl::Flush() hangs up

Steps to reproduce the behavior

Let a flush job starts and make sure it use mempurge (by doing writes against a small set of keys), before it finishes, call DB::Impl::Flush().

See the db/db_flush_test.cc MempurgeInterleavedWithNormalFlush in #11251 for more details.

Explanation

We have two flush jobs, the first one uses mempurge while the second one use normal flush.

Recall that mempurge adds its output, a MemTable with the same id of one of its input to the immutable memtable list. The existence of the output let DBImpl::WaitForFlushMemTables hang because this wait breaking condition
cfds[i]->imm()->GetEarliestMemTableID() > *flush_memtable_ids[i] will never be met until a third flush job is started.