Skip to content

Latest commit

 

History

History
49 lines (28 loc) · 1.93 KB

pip-318.md

File metadata and controls

49 lines (28 loc) · 1.93 KB

PIP-318: Don't retain null-key messages during topic compaction

Background knowledge

Apache Pulsar is supported Topic Compaction which is a key-based data retention mechanism.

Motivation

Currently, we retain all null-key messages during topic compaction, which I don't think is necessary because when you use topic compaction, it means that you want to retain the value according to the key, so retaining null-key messages is meaningless.

Additionally, retaining all null-key messages will double the storage cost, and we'll never be able to clean them up since the compacted topic has not supported the retention policy yet.

In summary, I don't think we should retain null-key messages during topic compaction.

Goals

High Level Design

In order to avoid introducing break changes to release version, we need to add a configuration to control whether to retain null-key messages during topic compaction. If the configuration is true, we will retain null-key messages during topic compaction, otherwise, we will not retain null-key messages during topic compaction.

Public-facing Changes

Configuration

Add config to broker.conf/standalone.conf

topicCompactionRetainNullKey=false

Backward & Forward Compatibility

  • Make topicCompactionRetainNullKey=false default in the 3.2.0.
  • Cherry-pick it to a branch less than 3.2.0 make topicCompactionRetainNullKey=true default.
  • Delete the configuration topicCompactionRetainNullKey in 3.3.0 and don't supply an option to retain null-keys.

Revert

Make topicCompactionRetainNullKey=true in broker.conf/standalone.conf.

Upgrade

Make topicCompactionRetainNullKey=false in broker.conf/standalone.conf.

Links