Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configuring outbox connector; #7

Merged
merged 1 commit into from
Jun 25, 2021
Merged

Conversation

gunnarmorling
Copy link
Collaborator

The "Expand JSON SMT" (https://github.com/RedHatInsights/expandjsonsmt/)
is used for un-stringifying the outbox event payload. As this SMT only works
for specific fields and not the entire message (see RedHatInsights/expandjsonsmt#10),
the payload is first hoisted into a temporary field, then un-stringified and finally
promoted to the top level of the Kafka message value again.

Hey @kbastani, this gives a nice event structure finally:

{"createdAt":1624638360969,"lastModified":1624638371355,"status":"ORDER_ASSIGNED","accountId":62941725,"restaurant":{"id":4,"storeId":7946,"name":"California & Battery","city":"San Francisco","country":"US","latitude":37.793060302734375,"longitude":-122.3997573852539},"links":[],"orderId":976}

The topic is debezium.Order now. I think you can consume that data and sink it into Pinot now? Or is anything else needed for doing so?

The "Expand JSON SMT" (https://github.com/RedHatInsights/expandjsonsmt/)
is used for un-stringifying the outbox event payload. As this SMT only works
for specific fields and not the entire message (see RedHatInsights/expandjsonsmt#10),
the payload is first hoisted into a temporary field, then un-stringified and finally
promoted to the top level of the Kafka message value again.
@kbastani
Copy link
Owner

Awesome! I'll merge in and get to work on updating the schema configuration for the Pinot tables.

@kbastani kbastani merged commit 05adb55 into kbastani:master Jun 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants