Add simple heuristics for experimental mempurge. #8583
Conversation
|
@bjlemaire has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
bjlemaire
force-pushed
the
enhanced_heuristics_experimental_mempurge
branch
from
5b0eb03 to
e0040fe
Compare
|
@bjlemaire has updated the pull request. You must reimport the pull request before landing. |
|
@bjlemaire has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
Specifically, likely hits to CPU usage, read efficiency, and maximum burst write throughput, though not always so. |
bjlemaire
changed the title
|
@bjlemaire has updated the pull request. You must reimport the pull request before landing. |
|
@bjlemaire has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
…O print about mempurge/memflush execution times.
bjlemaire
force-pushed
the
enhanced_heuristics_experimental_mempurge
branch
from
fccf5eb to
a40adb1
Compare
|
@bjlemaire has updated the pull request. You must reimport the pull request before landing. |
|
@bjlemaire has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@bjlemaire merged this pull request in 4361d6d. |
Summary: Add `experimental_mempurge_policy` option flag and introduce two new `MemPurge` (Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE. `ALWAYS`: every flush will first go through a `MemPurge` process. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on. `ALWAYS` is designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput). `ALTERNATE`: a flush is transformed into a `MemPurge` except if one of the memtables being flushed is the product of a previous `MemPurge`. `ALTERNATE` is a good tradeoff between reduction in number of L0 SST files created and performance. `ALTERNATE` perform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios. This PR also includes support for `experimental_mempurge_policy` in `db_bench`. Testing was done locally by replacing all the `MemPurge` policies of the unit tests with `ALTERNATE`, as well as local testing with `db_crashtest.py` `whitebox` and `blackbox`. Overall, if an `ALWAYS` mempurge policy passes the tests, there is no reasons why an `ALTERNATE` policy would fail, and therefore the mempurge policy was set to `ALWAYS` for all mempurge unit tests. Pull Request resolved: facebook#8583 Reviewed By: pdillinger Differential Revision: D29888050 Pulled By: bjlemaire fbshipit-source-id: e2cf26646d66679f6f5fb29842624615610759c1
Add
experimental_mempurge_policyoption flag and introduce two newMemPurge(Memtable Garbage Collection) policies: 'ALWAYS' and 'ALTERNATE'. Default value: ALTERNATE.ALWAYS: every flush will first go through aMemPurgeprocess. If the output is too big to fit into a single memtable, then the mempurge is aborted and a regular flush process carries on.ALWAYSis designed for user that need to reduce the number of L0 SST file created to a strict minimum, and can afford a small dent in performance (possibly hits to CPU usage, read efficiency, and maximum burst write throughput).ALTERNATE: a flush is transformed into aMemPurgeexcept if one of the memtables being flushed is the product of a previousMemPurge.ALTERNATEis a good tradeoff between reduction in number of L0 SST files created and performance.ALTERNATEperform particularly well for completely random garbage ratios, or garbage ratios anywhere in (0%,50%], and even higher when there is a wild variability in garbage ratios.This PR also includes support for
experimental_mempurge_policyindb_bench.Testing was done locally by replacing all the
MemPurgepolicies of the unit tests withALTERNATE, as well as local testing withdb_crashtest.pywhiteboxandblackbox. Overall, if anALWAYSmempurge policy passes the tests, there is no reasons why anALTERNATEpolicy would fail, and therefore the mempurge policy was set toALWAYSfor all mempurge unit tests.