Move buffer position in ByteBufferIndexWriter#writeFloats#607
Move buffer position in ByteBufferIndexWriter#writeFloats#607michaeljmarshall wants to merge 2 commits intodatastax:mainfrom
Conversation
|
In working with this code, I noticed that it'd be very helpful to have custom implementations to write a |
There was a problem hiding this comment.
Pull request overview
This PR fixes a critical bug in ByteBufferIndexWriter#writeFloats where the buffer position was not being advanced after writing float arrays, causing record size mismatches during graph serialization.
Changes:
- Fixed buffer position tracking in
ByteBufferIndexWriter#writeFloatsby explicitly advancing the position after writing - Added
writeTo(IndexWriter)method toVectorFloatinterface and implementations to simplify vector serialization - Added comprehensive test coverage for the
writeFloatsfix and the newwriteTomethods
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| jvector-base/src/main/java/io/github/jbellis/jvector/disk/ByteBufferIndexWriter.java | Fixed bug by advancing buffer position after writing floats |
| jvector-base/src/main/java/io/github/jbellis/jvector/vector/types/VectorFloat.java | Added writeTo method to interface |
| jvector-base/src/main/java/io/github/jbellis/jvector/vector/ArrayVectorFloat.java | Implemented writeTo using writeFloats |
| jvector-native/src/main/java/io/github/jbellis/jvector/vector/MemorySegmentVectorFloat.java | Implemented writeTo with element-by-element writing |
| jvector-tests/src/test/java/io/github/jbellis/jvector/disk/TestByteBufferIndexWriter.java | Added comprehensive test suite for buffer position tracking |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| for (int i = 0; i < length(); i++) { | ||
| writer.writeFloat(get(i)); | ||
| } |
There was a problem hiding this comment.
The writeTo implementation writes floats one at a time in a loop, which is less efficient than bulk writing. Consider converting to a float array first and using writeFloats for better performance, similar to the ArrayVectorFloat implementation.
| for (int i = 0; i < length(); i++) { | |
| writer.writeFloat(get(i)); | |
| } | |
| int len = length(); | |
| float[] data = (float[]) segment.heapBase().get(); | |
| writer.writeFloats(data, 0, len); |
Fixes #606
This PR fixes a bug where we put the vector into the buffer without moving the position forward.
When attempting to upgrade to
4.0.0-rc7, I hit the following exception:The issue seems to be in the
ByteBufferIndexWriter#writeFloats. By callingasFloatBuffer(), we're creating a new object with its own position tracking, and as a result, putting the float[] moves theFloatBuffer's position, but not the sourceBufferposition.This PR also adds a convenience method to simplify writing a vector to the
IndexWriterto reduce the need for branching in the calling code.