Skip to content

Handle hex tensor decoding across compatible value types - #36385

Merged
bratseth merged 2 commits into
vespa-engine:masterfrom
SahilKumar75:issue-35611-hex-tensor-field-type-sensitivity
Apr 13, 2026
Merged

Handle hex tensor decoding across compatible value types#36385
bratseth merged 2 commits into
vespa-engine:masterfrom
SahilKumar75:issue-35611-hex-tensor-field-type-sensitivity

Conversation

@SahilKumar75

Copy link
Copy Markdown
Contributor

Why

Hex-encoded tensor input currently depends on field cell type in a brittle way. Feeding float-encoded hex to a tensor<bfloat16>(...) field can fail with an out-of-bounds error in JSON tensor parsing.

What

  • Added expected-size-aware hex decoding in tensor JSON parsing.
  • When hex length matches the expected number of values but a different supported cell width, decode using inferred source type and let normal numeric assignment convert to target field type.
  • Applied this in both:
    • vespajlib JSON tensor decoding (JsonFormat)
    • document feed JSON reader (TensorReader)
  • Kept empty-string behavior intact with existing error messages.

Tests

  • mvn -pl vespajlib -Dtest=JsonFormatTestCase test
  • mvn -pl document -Dtest=JsonReaderTestCase test
  • mvn -pl document -am -Dtest=JsonReaderTestCase -Dsurefire.failIfNoSpecifiedTests=false test

Regression coverage added

  • float-hex -> bfloat16 decoding for dense values
  • float-hex -> bfloat16 decoding for mixed block values
  • document JSON reader coverage for both direct dense and mixed block hex forms

Fixes #35611

@bratseth bratseth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@bratseth
bratseth merged commit 071515f into vespa-engine:master Apr 13, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hex encoded tensors are field type sensitive

2 participants