fix: fix memory safety issue in native c2r #3367
Open
+66
−2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… references NativeRowIterator reuses a single UnsafeRow object that points to native memory. The buffer is cleared on each convert() call, so any held row references become stale. Return unsafeRow.copy() to copy each row to Java heap memory. Closes apache#3308 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3367 +/- ##
============================================
+ Coverage 56.12% 59.99% +3.86%
- Complexity 976 1464 +488
============================================
Files 119 175 +56
Lines 11743 16165 +4422
Branches 2251 2681 +430
============================================
+ Hits 6591 9698 +3107
- Misses 4012 5114 +1102
- Partials 1140 1353 +213 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
The regression test was directly using Arrow classes (RootAllocator, IntVector, VarCharVector, VectorSchemaRoot) which get shaded during the Maven package phase. This caused NoSuchMethodError at runtime because the test called NativeUtil.rootAsBatch() with unshaded Arrow types, but the compiled method expected shaded types. Refactored to use CometArrowConverters.rowToArrowBatchIter() which handles Arrow shading internally. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
comphead
reviewed
Feb 3, 2026
| currentIdx += 1 | ||
|
|
||
| unsafeRow | ||
| unsafeRow.copy() |
Contributor
There was a problem hiding this comment.
Makes sense... as unsafeRow can be reused. it would inflate mem usage a bit, but it is better than corrupted memory
comphead
approved these changes
Feb 3, 2026
Contributor
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @andygrove especially for the local test, if we manage to find a solution without extra copy in future the test will help to guard against corruption issues
Member
Author
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #3308
Rationale for this change
NativeRowIteratorreuses a singleUnsafeRowobject that points to native memory. The buffer is cleared on each convert() call, so any held row references become stale. ReturnunsafeRow.copy()to copy each row to Java heap memory.What changes are included in this PR?
How are these changes tested?