Configure Kafka consumption from a specific timestamp

[mayankshriv]

Currently, we only seem to support smallest/largest values for consumer.prop.auto.offset.reset in case of OSS Kafka consumer plugin. It would be great to also support consuming from a specific timestamp. With the KafkaConsumer#offsetsForTime() and KafkaConsumer#seek() api's we should be able to do so (or perhaps there's a simpler way?).

[mayankshriv]

cc: @npawar

[mcvsubbu]

Native pinot code supports any string there, not just smallest/largest. The underlying plugin can interpret the string however it wants.

We only want that the underlying plugin interpret the basic strings we need them to (smallest/largest).

in LinkedIn, we use "2d", for example. The stream then attempts to provide an approximate offset that is about 2d old.

[mayankshriv]

Yes, that's what was the intent of the issue, updated for clarity.

[npawar]

This would be a useful feature. Tagging this as a beginner issue.