Catalog abstraction

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10902

This issue is intended to be a design discussion around the introduction of the notion of data catalog into DataFusion.

 

A first draft containing some observations can be found here: [https://docs.google.com/document/d/1dy2eiJDtmhEHKit1ekR-7_LguMO8z-72ORIPCCIHWOo/edit#|https://docs.google.com/document/d/1dy2eiJDtmhEHKit1ekR-7_LguMO8z-72ORIPCCIHWOo/edit?usp=sharing]

 

To add a bit of context, a higher level description of DataFusion's interfaces can be found here: [https://docs.google.com/document/d/1S5bR48Apzvp1cXOVrlk-YJnc-Aml-IapRrEs3cLvz6c/edit?usp=sharing]

[alamb]

Comment from Remi Dettai(rdettai) @ 2020-12-14T15:51:33.144+0000:

If anyone wants write access, feel free to ask!

Comment from Jorge Leitão(jorgecarleitao) @ 2020-12-14T16:00:50.035+0000:

cc [~andygrove] [~alamb]

Comment from Remi Dettai(rdettai) @ 2020-12-14T16:35:44.060+0000:

Catalogs typically associate to each partition/file an expression bounding the content (min/max statistics). To assert which partitions/files will need to be processed from the catalog, some sort of constant folding as that mentioned in ARROW-9770 can be used.

[houqp]

I think this can be closed now?

[alamb]

I defer to @rdettai -- we have a catalog abstraction in DataFusion but it is only part of what is described in the document linked in the description

[rdettai]

There are different definitions of what a Catalog really is that are conflicting. I don't think that this issue format is the right way to track the topic I was interested in initially. Also the attached document is kind of outdated, this one is more relevant. I would say we can close this. Thanks @houqp 😃

[alamb]

Good call -- we can also always open a new issue for follow on work.