BigQuery trim schema with selected fields #32514
Conversation
Trim BQ schema directly instead of converting to avro schema and back
| if (selectedFields != null | ||
| && selectedFields.isAccessible() | ||
| && selectedFields.get() != null) { | ||
| return selectedFields.get().contains(field.getName()); |
There was a problem hiding this comment.
selected nested field were always filtered out the beam schema
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
RustedBones
force-pushed
the
trim-table-schema
branch
from
87d9b71
to
d251f07
Compare
|
Assigning reviewers. If you would like to opt out of this review, comment R: @Abacn for label java. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java
Outdated
Show resolved
Hide resolved
| if (selectedFieldsProvider != null && selectedFieldsProvider.isAccessible()) { | ||
| tableSchema = BigQueryUtils.trimSchema(tableSchema, selectedFieldsProvider.get()); |
There was a problem hiding this comment.
Can you explain a little (maybe include in the PR description too) why it's better to trim based on selectedFields instead of the read session's schema? We pass selected fields as a component when building the read session, so I would think it returns a trimmed schema already?
There was a problem hiding this comment.
Yes, the operation here is correct. It is just for consistency/simplicity:
- We also trim the schema in the
BigQueryIOwhere we do not have access to a trimmed avro schema. - when we direct read with
ARROW, to get access to the avro schema we doarrow -> beam -> avroschema conversions just for the sake of filtering known fields.
There was a problem hiding this comment.
I see, thanks for explaining. Just wanted to make sure I wasn't missing something bigger.
...d-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageQueryTest.java
Outdated
Show resolved
Hide resolved
Extracted from #32360
Trim BQ schema using selected fields instead of avro schema.