Huge compaction after upgrading from 8.3.2 to 9.7.3

[ahamlat]

Hi team,
When upgrading our rocksdbjni lib from 8.3.2 to 9.7.3, we noticed a huge IO writes and reads for a pretty long time, depending on the size of the database. We've seen a range of 20 minutes to two hours period of high IO activity. Those IO are caused by a huge compaction that resulted in a different organization in the levels for each Column Family. Knowing that we're using the default configuration related to Leveled compaction, you can see below that when running 8.3.2 version, there is the expected distribution across different levels, but with 9.7.3, the levels 1,2 and 3 are empty

RocksDB 8.3.2

RocksDB 9.7.3

Disk activity after upgrade

Thread causing IO activity especially writes is a compaction thread (rockdb:low)

The LOG file : LOG.zip

Wall Clock Profiling

Expected behavior

Upgrading RocksDB from 8.3.2 to 9.7.3 version shouldn't trigger a huge compaction, and move data from lower levels to last levels.

Actual behavior

Upgrading RocksDB from 8.3.2 to 9.7.3 triggers a huge compaction, that can take up to two hours, moving data from lower levels to last levels.

Steps to reproduce the behavior

Upgrade rocksdb version (in our case rocksdbjni) from 8.3.2 to 9.7.3.

[zaidoon1]

it's related to the change in 8.4.0. if you look at the changelog:

• Change the default value for option level_compaction_dynamic_level_bytes to true. This affects users who use leveled compaction and do not set this option explicitly. These users may see additional background compactions following DB open. These compactions help to shape the LSM according to level_compaction_dynamic_level_bytes such that the size of each level Ln is approximately size of Ln-1 * max_bytes_for_level_multiplier. Turning on this option has other benefits too: see more detail in wiki: https://github.com/facebook/rocksdb/wiki/Leveled-Compaction#option-level_compaction_dynamic_level_bytes-and-levels-target-size and in option comment in advanced_options.h (Make option level_compaction_dynamic_level_bytes true by default #11525).

[ahamlat]

Thanks @zaidoon1, that explains the behaviour I see.