Skip to content

Introduce continuous offset for pulsar #9038

@aloyszhang

Description

@aloyszhang

Is your feature request related to a problem? Please describe.
Currently, pulsar provides <LedgerId, EntryId> to locate an Entry, and provides BatchIndex for the inner message of Entry.
But there is not a continuous offset for each Message in Entry and we could not use the offset to search an Entry.

For protocol handler like KOP, the map between kakfa offset and pulsar position is implenment by split 64 bits to three parts which represents ledgerId, entryId and batchIndex seperatelly. This way makes the max range of ledgerId much lower than Long.MAX_VALUE and ledgerId space will used up quickly, see streamnative/kop#175.

Describe the solution you'd like
We should introduce continuous offset for pulsar, which can be achieved by broker entry metadata introduced in PIP-70.

Additional context
We can introduce offset for each message first, and with this feature, we can more things futher

  1. implement a more stable offset manager for KOP
  2. provide more preceise metrics on message level like backlog
  3. ...

Metadata

Metadata

Assignees

Labels

type/featureThe PR added a new feature or issue requested a new feature

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions