Skip to content

[Java][BigQueryIO] Avoid reflective lookup in unknown-column collectors - #40491

Open
khaledh wants to merge 1 commit into
apache:masterfrom
khaledh:perf/unknown-column-collector-map-access
Open

khaledh wants to merge 1 commit into
apache:masterfrom
khaledh:perf/unknown-column-collector-map-access

Conversation

@khaledh

@khaledh khaledh commented Oct 9, 2026 •

Copy link
Copy Markdown

Use backing maps for ordinary unknown-column collector keys to avoid repeated reflective lookup and string interning during Storage Write API conversion.

Why

Unknown-column capture, nested collector attachment, and empty-STRUCT pruning currently use TableRow map operations. Those operations consult GenericData model metadata and intern field names, even for ordinary keys. With automatic schema updates, collector bookkeeping also runs for known nested fields, so deeply repeated records pay this cost even without unknown columns.

This patch uses the existing backing map for ordinary keys while retaining TableRow collectors and their allocation behavior. Annotated model fields still use the original setter and map operations. The emptiness shortcut treats every non-null f value as an entry, including an empty cell list, and falls back to the original behavior if future model fields appear.

Context

Closes #40492.

Related to #40203 (descriptor lookup) and #40204 (required-field validation), but independent of both. This patch changes collector access only; it does not change descriptor lookup, message validation, timestamp conversion, or public converter signatures.

What changed

  • Use guarded backing-map access for unknown-column capture, nested attachment, lookup, and removal.
  • Keep model-field behavior, repeated positions, and empty-container pruning unchanged.
  • Add regression tests for protobuf bytes, encoded unknown JSON, preexisting collectors, model-field errors, and cell-format conversion.

Validation

  • Spotless and Checkstyle pass.
  • All 42 converter tests pass before and after the patch, including 17 new collector tests, in an isolated Java 11-target build against published Beam 2.77.0 dependencies.
  • The normal Beam Gradle test task remains blocked locally: the configured protoc-gen-grpc-java executable is x86-only on macOS ARM. The isolated build does not replace a full master build or upstream CI.
  • A downstream Beam 2.68.0 backport showed 27.1% more committed offsets/s and 20.3% lower estimated CPU per committed offset in one matched staging pair. This is supporting evidence for the collector change, not a benchmark of this master port or a production guarantee. Full pipeline payload parity and long-run validation remain open.

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Avoid reflective lookup and interning in BigQuery Storage Write API unknown-column collectors

1 participant