-
Notifications
You must be signed in to change notification settings - Fork 3.9k
ARROW-6199: [Java] Avro adapter avoid potential resource leak. #5059
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5059 +/- ##
==========================================
+ Coverage 87.63% 89.75% +2.11%
==========================================
Files 1014 674 -340
Lines 144947 100382 -44565
Branches 1437 0 -1437
==========================================
- Hits 127030 90093 -36937
+ Misses 17555 10289 -7266
+ Partials 362 0 -362Continue to review full report at Codecov.
|
|
@emkornfield Hi Micah, Please help take a look on this. About avro adapter there are Array, Map, Fixed type left, I would like to start follow-ups(support all types and iterator) as soon as possible to merge it before 0.15 branch cut together with Jdbc iterator. Many thanks :) |
|
@emkornfield Please help take a look at this one :) |
java/adapter/avro/src/main/java/org/apache/arrow/consumers/AvroBytesConsumer.java
Outdated
Show resolved
Hide resolved
|
@emkornfield build passed, and I am curious about which time zone you are in? |
|
+1 thank you. US/Pacific. |
| */ | ||
| public void consume(Decoder decoder, VectorSchemaRoot root) throws IOException { | ||
| int valueCount = 0; | ||
| while (true) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lets fix in one of the follow-ups. but I think looping should be done outside of the consumer interfaces.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you Micah, will fix in next PR. I plan to start two PRs one by one(support all type, iterator API), then I think i could be called a initial version and hope to catch up 0.15 branch cut.
Related to [ARROW-6199](https://issues.apache.org/jira/browse/ARROW-6199). Currently, avro consumer interface has no close API, which may cause resource leak like AvroBytesConsumer#cacheBuffer. To resolve this, make consumer extends AutoCloseable and create CompositeAvroConsumer to encompasses consume and close logic. Closes apache#5059 from tianchen92/ARROW-6199 and squashes the following commits: d60d94c <tianchen> fix 42f22da <tianchen> clear vectors in close 5b91da7 <tianchen> fix comments 3ffc076 <tianchen> ARROW-6199: Avro adapter avoid potential resource leak. Authored-by: tianchen <niki.lj@alibaba-inc.com> Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
Related to ARROW-6199.
Currently, avro consumer interface has no close API, which may cause resource leak like AvroBytesConsumer#cacheBuffer.
To resolve this, make consumer extends AutoCloseable and create CompositeAvroConsumer to encompasses consume and close logic.