Add support for repeated columns in the filter2 API

[asfimport]

They currently are not supported. They would need their own set of operators, like contains() and size() etc.

Reporter: Alex Levenson / @isnotinvain

PRs and other links:

• GitHub Pull Request #1328
• GitHub Pull Request #1370
• GitHub Pull Request #1328

_{Note: This issue was originally created as PARQUET-34. Please see the migration documentation for further details.}

[asfimport]

Viktor Szathmáry / @phraktle:
Any plans on supporting this?

[asfimport]

Alex Levenson / @isnotinvain:
We'd like to, contains() at least shouldn't be too hard, it may not even require any changes except removing the check that prevents it.

That said we don't have any concrete plans to do it right away – pull requests always welcome and we're happy to help!

[asfimport]

Flavio Pompermaier:
Repeated columns (protobuf) and lists (thrift) support would be VERY useful..

[asfimport]

Stefano Bortoli:
I agree with Flavio. +1 for supporting multi-valued columns.

[asfimport]

Nicola:
+1

[asfimport]

Julien Le Dem / @julienledem:
@alexlevenson did you have any plan for this?

[asfimport]

Flavio Pompermaier:
If someone could give me some advice I could try to implement it by myself (I never looked into Parquet code...)

[asfimport]

Julien Le Dem / @julienledem:
You can start by looking at the filter2 api contribution:
commit:
ad32bf0
PR:
#4
older PR with feedback:
https://github.com/Parquet/parquet-mr/pull/412

[asfimport]

Flavio Pompermaier:
I think that introducing contains() is definitely not as simple as mentioned at the beginning of this post.
It would require a huge rework of parquet-column in order to handle Collections (PARQUET-35) and filtering on row groups (see PARQUET-43).

I'll be glad to contribute but I can't do it alone without a direct support from your side (at least to agree on the changes to apport at the core code).
Do you think you can spend some time to actively focus with me on this task in the next days...?

[asfimport]

Flavio Pompermaier:
Anyone willing to push this feature..? :(

[asfimport]

Ryan Blue / @rdblue:
[~f.pompermaier], I don't think anyone has extra cycles to spend implementing this right now, but if you are interested in building it, we'll work with you to get it reviewed and included.

I think the next step is to write up what you think needs to be done so we can look at it and help you in the right direction. It may be that the disconnect between Alex's comment about this being easy and your assessment is a different level of support.

[asfimport]

Flavio Pompermaier:
Am I still the only one asking for this feature..?

[asfimport]

Ryan Blue / @rdblue:
I think it could be useful, but it doesn't seem high priority for anyone. We're happy to review it if you put something together.

[asfimport]

Sean Scott:
I was surprised when my search failed when attempting to filter on a repeated column.  

 

I know its been a few years since there has been activity here.  What is everyone doing for pushdown predicates to avoid huge data transfers/ row scans?

[asfimport]

Claire McGinty / @clairemcginty:
Current status of this ticket:

 

• PARQUET-34: Add #contains FilterPredicate for Array columns #1328 (support for Contains(eq)) has been merged if anyone would like to test it

• PARQUET-34: Extend Contains support to all ColumnFilterPredicate types #1370 (support for Contains(lt), Contains(gt), Contains(in), etc) is open

• Todo: Support not(contains(...))

• Todo: Support size()

   

  I can finish up PR PARQUET-34: Extend Contains support to all ColumnFilterPredicate types #1370 but probably won't have bandwidth to work on the remaining Todos until July.