Skip to content

[BackPort for 0.11.0 Release] Add Protocol Buffer Stream Decoder#9164

Closed
KKcorps wants to merge 1 commit intoapache:release-0.11.0from
KKcorps:release-0.11.0-with-protobuf-decoder
Closed

[BackPort for 0.11.0 Release] Add Protocol Buffer Stream Decoder#9164
KKcorps wants to merge 1 commit intoapache:release-0.11.0from
KKcorps:release-0.11.0-with-protobuf-decoder

Conversation

@KKcorps
Copy link
Contributor

@KKcorps KKcorps commented Aug 4, 2022

(cherry picked from commit 8806dc3)
Original PR - #8972

Supported configs

  • descriptorFile - Path of the descriptor file. You can generate this file use protoc -o file.desc --include_imports file.proto command. The path can be a local path in which case it needs to be available on all servers OR it can be a DFS path such as s3, gcs etc. If providing a DFS path, you need to ensure pinot is configured to use that filesystem.

  • protoClassName - If the descriptor file contains multiple proto object, you can mention the name of the class to use for parsing.

Example

    "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.consumer.type": "lowLevel",
        "stream.kafka.topic.name": "metrics_pb",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder",
        "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc",
        "stream.kafka.decoder.prop.protoClassName": "Metrics",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.broker.list": "localhost:19092",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest",
        "realtime.segment.flush.threshold.time": "12h",
        "realtime.segment.flush.threshold.size": "100M"
      }

@KKcorps KKcorps changed the title [BackPort for 0.11.0 Release] Add Protocol Buffer Stream Decoder (#8972) [BackPort for 0.11.0 Release] Add Protocol Buffer Stream Decoder Aug 4, 2022
@KKcorps KKcorps added the release-notes Referenced by PRs that need attention when compiling the next release notes label Aug 4, 2022
Copy link
Contributor

@walterddr walterddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm for the cherry-pick.

@walterddr
Copy link
Contributor

let's wait for #9176

@xiangfu0 xiangfu0 deleted the branch apache:release-0.11.0 August 31, 2022 21:08
@xiangfu0 xiangfu0 closed this Aug 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-notes Referenced by PRs that need attention when compiling the next release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants