-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Description
Is your feature request related to a problem? Please describe.
Currently, pulsar provides <LedgerId, EntryId> to locate an Entry, and provides BatchIndex
for the inner message of Entry.
But there is not a continuous offset for each Message in Entry and we could not use the offset to search an Entry.
For protocol handler like KOP, the map between kakfa offset and pulsar position is implenment by split 64 bits to three parts which represents ledgerId, entryId and batchIndex seperatelly. This way makes the max range of ledgerId much lower than Long.MAX_VALUE and ledgerId space will used up quickly, see streamnative/kop#175.
Describe the solution you'd like
We should introduce continuous offset for pulsar, which can be achieved by broker entry metadata introduced in PIP-70.
Additional context
We can introduce offset for each message first, and with this feature, we can more things futher
- implement a more stable offset manager for KOP
- provide more preceise metrics on message level like backlog
- ...