PIP-17 : Tiered storage (Master Issue)

[ivankelly]

Storing backlogs on bookies for a long time can get expensive. If
there are other cheaper forms of storage available (S3/HDFS), capex
can be reduced by moving older data to this storage.

The proposed design proposes an interface to do this, with an initial
implementation targeted at S3. However the interface is general enough
that HDFS could also be used.

Design: https://github.com/apache/incubator-pulsar/wiki/PIP-17:-Tiered-storage-for-Pulsar-topics
Mailing List discussion: https://lists.apache.org/thread.html/3f1c98658395c92e3c858ff966aa829dba0e165a28ef350e0ec3a33f@%3Cdev.pulsar.apache.org%3E
Prototype: https://github.com/ivankelly/incubator-pulsar/tree/s3-offload-proto

[haphut]

I'm loving Pulsar's design. I came here to suggest offloading data into object storage so that the client would not even notice and you are already working on it. Thank you!

Food for thought: It is the Pulsar client's responsibility to compress and decompress messages. However, I wonder if for some use cases it might make sense to store only compressed data in the object storage while within the cluster the data would stay in decompressed form. I would find it simpler if in most cases the client would not need to worry about storage yet the cluster admins could use storage space efficiently.

Do you think it makes sense to include compression in the scope of tiered storage? Or is this an object storage implementation issue that should be handled by Minio, for example?

For reference, xz compresses our most voluminous JSON events into 5 % of the original size.

[ivankelly]

@haphut Are you suggesting that the offloading implementation compresses the data as it's sent to the long term storage, regardless of whether the topic is compressed or not? This seems like something that should be pretty easy to do with how we're building tiered storage. It likely won't make it into the first cut, but as I said, should be pretty easy to add to second iteration.

[haphut]

Yes, that's what I mean. Good to hear that you do not see major obstacles for adding it later. Keep up the great work!