Add Input Parameters for numeric type and decimal scale for the destination-s3-glue connector #48486
quazi-h
started this conversation in
Connector Ideas and Features
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We were running into an issue where our data included fractional numeric fields that we needed to handle differently but there were no such options under the configuration settings for the destination-s3-glue connector.
To solve this issue, we would like to implement a "Numeric Type" setting that will allow us to map fractional numeric values to a Hive compatible type.
See <a href=\"https://cwiki.apache.org/confluence/display/hive/languagemanual+types#LanguageManualTypes-NumericTypes\\\">here for more details."
Revisiting this older PR with the relevant changes that was closed and never merged upstream: #26517
This PR includes changes that add a Stringify option to the destination-s3-glue connector as well as adding input parameters for fractional numeric type and decimal scale. We had originally bundled the decimal_scale and numeric_type changes along with the Stringify feature changes for the s3-glue connector. This was because we had created the MetastoreFormatConfig and MetastoreJsonlFormatConfig in our older PR since it was cleaner than having two separate PRs with a lot of overlapping code.
I rebased my dev branches against airbytehq/airbyte as well as Stephane's changes to get local builds working against the latest CDK. It seems a handful of changes from our old PRs did make it in, including the MetastoreFormatConfigs so I have been cherry picking the changes from the old PR to add the decimal_scale config option as well as the numeric_type argument.
Tagging @stephane-airbyte to get input on this issue.
Should this be handled in its own standalone PR or is it possible to include these changes along with a PR for the stringify feature add for the s3-glue connector?
I am almost done working on my isolated branch with the upstream changes along with the missing pieces for the decimal / numeric_type input. I am just working through some errors when I try a local build for the connector and when I have a successful build, I'll be able to make the separate PR if needed. I created this issue/discussion in order to document the problem in case we need that for the separate PR. I believe there are already issues for the Stringify work for the s3 and s3-glue connector.
Please let me know if this is not the right place for this issue/discussion and feel free to point me towards the proper channels to get this seen.
Beta Was this translation helpful? Give feedback.
All reactions