-
Notifications
You must be signed in to change notification settings - Fork 522
fix(binder): enhance RecordBinder and TypeAdapter to support STRUCT type conversion #3005
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This pull request refactors the Avro-to-Iceberg record binding and type conversion logic to better support nested struct types within lists and maps. The key changes include:
- Introduced a new
StructConverterfunctional interface to enable recursive struct conversion - Extended
TypeAdapterinterface with an additional method accepting aStructConverterparameter - Refactored
RecordBinderto pre-compute binders for nested structs in lists and maps - Extracted
FieldMappinginto a separate public class for better code organization - Simplified field mapping initialization logic in
RecordBinder - Added test coverage for lists of records and maps with record values
- Removed the
writeRecordWithConvertedMapEntrieshelper method as the functionality is now handled by the refactored type adapters
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| TypeAdapter.java | Added new convert method overload accepting StructConverter for recursive struct conversion |
| StructConverter.java | New functional interface for converting struct-typed values during type adaptation |
| AbstractTypeAdapter.java | Updated convert method to delegate STRUCT type conversion to StructConverter callback |
| AvroValueAdapter.java | Implemented convertStruct method and bridged old/new convert method signatures |
| RecordBinder.java | Refactored field mapping initialization and extended binder pre-computation to support structs in lists/maps |
| FieldMapping.java | Extracted as a standalone class to represent Avro-to-Iceberg field mappings |
| AvroRecordBinderTest.java | Added tests for list-of-records and map-with-record-values conversion; removed obsolete helper method |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
core/src/main/java/kafka/automq/table/binder/AvroValueAdapter.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
core/src/main/java/kafka/automq/table/binder/AvroValueAdapter.java:134
- The recursive call to
convertinconvertListshould use the 4-parameter overload with theStructConverterto maintain consistency with the new callback pattern. Currently it calls the 3-parameter method which will usethis::convertStructfrom line 191, but this breaks encapsulation and could cause issues if the caller intended to use a differentStructConverter.
Object convert = convert(element, elementSchema, targetType.elementType());
core/src/main/java/kafka/automq/table/binder/AvroValueAdapter.java:165
- The recursive calls to
convertinconvertMapshould use the 4-parameter overload with theStructConvertercallback parameter to maintain consistency with the new callback pattern and ensure proper handling of nested structs.
Object key = convert(record.get(keyField.pos()), keySchema, keyType);
Object value = convert(record.get(valueField.pos()), valueSchema, valueType);
core/src/main/java/kafka/automq/table/binder/AvroValueAdapter.java:183
- The recursive calls to
convertshould use the 4-parameter overload with theStructConvertercallback parameter to maintain consistency with the new callback pattern and ensure proper handling of nested structs in map values.
Object key = convert(rawKey, STRING_SCHEMA_INSTANCE, keyType);
Object value = convert(entry.getValue(), valueSchema, valueType);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This pull request refactors and extends the Avro-to-Iceberg record binding and conversion logic, improving support for struct, list, and map types while simplifying field mapping and adapter interfaces.
fix #3009
fixed protobuf timestamp type