fix: read ARRAY<DOUBLE> through Arrow's Float8Vector - #2
Closed
LuciferYang wants to merge 1 commit into
Closed
Conversation
LuciferYang
marked this pull request as draft
August 21, 2026 16:45
LuciferYang
force-pushed
the
fix/converter-array-double-cast
branch
from
August 21, 2026 16:51
3b7adfe to
9ee3bfd
Compare
readArrayData cast the child vector to the private inner Double8Vector wrapper instead of org.apache.arrow.vector.Float8Vector. The wrapper is never instantiated anywhere, so every read of an array-of-double column (List or FixedSizeList, both map to ARRAY<DOUBLE>) threw ClassCastException. The write path and the Float read branch already used the Arrow vector directly; this was a leftover from an unfinished refactor. RowDataConverterTest covers both list representations for double and float reads, plus empty/null arrays and a write batch that forces the child vector to reallocate (the production sink path); each guard was verified to redden under a targeted regression mutation. The tests allocate off-heap memory, so the root pom also sets the --add-opens=java.base/java.nio argLine that JDK 17 CI legs require for Arrow's MemoryUtil to initialize.
LuciferYang
force-pushed
the
fix/converter-array-double-cast
branch
from
August 21, 2026 16:53
9ee3bfd to
c7e7596
Compare
Owner
Author
|
Superseded by upstream PR lance-format#69 (issue tracked at lance-format#68). Closing this fork-local PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1
Problem
The
DoubleTypebranch ofRowDataConverter.readArrayDatacast the child vector to the private inner classDouble8Vector— a wrapper that is never instantiated anywhere (its javadoc says "alias for Float8Vector"), a leftover from an unfinished refactor. Every read of anARRAY<DOUBLE>column, via either Arrow representation that maps to it (List<Float64>andFixedSizeList<Float64>, i.e. Lance f64 vector columns), threwClassCastException. The type layer accepts the schema (LanceTypeConverterdocumentsFixedSizeList<Float64> <-> ARRAY<DOUBLE>), so DDL and planning succeed and the failure only surfaces at scan time. All read entry points (LanceSource,LanceInputFormat,LanceAggregateSource,LanceVectorSearch) funnel into this one private method and were all affected.Fix
org.apache.arrow.vector.Float8Vectordirectly, matching the siblingFloatTypebranch and the write path.Double8Vectorwrapper.Tests (
RowDataConverterTest, 4 guards)testWriteThenReadArrayOfDoubleRoundTriptestReadFixedSizeListOfDoubletestWriteBeyondInitialListCapacitysetSafereallocation — the path the production sink hits on every 1024-row batch (defaultwrite.batch-size)testReadFixedSizeListOfFloatFloatTypebranch (f32 vector columns, the primary embedding path): values and null elementVerification: 4/4 green; full module unit suite 234/0/0; six targeted mutations (reverting the cast fix, FSL start-index off-by-one, dropping the element null guard, dropping the parent-null short-circuit,
setSafe->set, wrong cast in the Float branch) each reddened exactly the corresponding test while the others stayed green; with the production fix reverted, the suite-level red set was exactly the tests above.Build note
The new tests allocate off-heap Arrow memory, which requires
--add-opens=java.base/java.nio=ALL-UNNAMEDforMemoryUtilto initialize on JDK 17 (a CI matrix leg). The root pom now sets thatargLine; jacoco appends its agent to the same property, so the two coexist.