Skip to content

Conversation

@tianchen92
Copy link
Contributor

Related to ARROW-6199.

Currently, avro consumer interface has no close API, which may cause resource leak like AvroBytesConsumer#cacheBuffer.
To resolve this, make consumer extends AutoCloseable and create CompositeAvroConsumer to encompasses consume and close logic.

@codecov-io
Copy link

codecov-io commented Aug 11, 2019

Codecov Report

Merging #5059 into master will increase coverage by 2.11%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #5059      +/-   ##
==========================================
+ Coverage   87.63%   89.75%   +2.11%     
==========================================
  Files        1014      674     -340     
  Lines      144947   100382   -44565     
  Branches     1437        0    -1437     
==========================================
- Hits       127030    90093   -36937     
+ Misses      17555    10289    -7266     
+ Partials      362        0     -362
Impacted Files Coverage Δ
r/src/recordbatch.cpp
r/R/Table.R
js/src/util/fn.ts
go/arrow/array/bufferbuilder.go
r/src/symbols.cpp
rust/datafusion/src/execution/projection.rs
rust/datafusion/src/execution/filter.rs
rust/arrow/src/csv/writer.rs
rust/datafusion/src/bin/main.rs
go/arrow/ipc/file_reader.go
... and 332 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cc12ab...d60d94c. Read the comment docs.

@tianchen92
Copy link
Contributor Author

tianchen92 commented Aug 13, 2019

@emkornfield Hi Micah, Please help take a look on this. About avro adapter there are Array, Map, Fixed type left, I would like to start follow-ups(support all types and iterator) as soon as possible to merge it before 0.15 branch cut together with Jdbc iterator. Many thanks :)

@tianchen92
Copy link
Contributor Author

@emkornfield Please help take a look at this one :)

@tianchen92
Copy link
Contributor Author

@emkornfield build passed, and I am curious about which time zone you are in?

@emkornfield
Copy link
Contributor

+1 thank you. US/Pacific.

*/
public void consume(Decoder decoder, VectorSchemaRoot root) throws IOException {
int valueCount = 0;
while (true) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets fix in one of the follow-ups. but I think looping should be done outside of the consumer interfaces.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you Micah, will fix in next PR. I plan to start two PRs one by one(support all type, iterator API), then I think i could be called a initial version and hope to catch up 0.15 branch cut.

pribor pushed a commit to GlobalWebIndex/arrow that referenced this pull request Oct 24, 2025
Related to [ARROW-6199](https://issues.apache.org/jira/browse/ARROW-6199).

Currently, avro consumer interface has no close API, which may cause resource leak like AvroBytesConsumer#cacheBuffer.
To resolve this, make consumer extends AutoCloseable and create CompositeAvroConsumer to encompasses consume and close logic.

Closes apache#5059 from tianchen92/ARROW-6199 and squashes the following commits:

d60d94c <tianchen> fix
42f22da <tianchen> clear vectors in close
5b91da7 <tianchen> fix comments
3ffc076 <tianchen> ARROW-6199:  Avro adapter avoid potential resource leak.

Authored-by: tianchen <niki.lj@alibaba-inc.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants