Draft proposal to move DataFusion to a new top level apache project

[alamb]

Is your feature request related to a problem or challenge?

In general, the idea of moving DataFusion to a new top level project within the Apache Software Foundation seems to have consensus (see #6475)

Describe the solution you'd like

I would like a clear and concise proposal with a writeup of what it would take to "promote" DataFusion to a new top level project.

Describe alternatives you've considered

Tasks

• Create a draft proposal
• Submit a name search to ensure we can use the name https://issues.apache.org/jira/browse/PODLINGNAMESEARCH
• Submit the proposal to the board for consideration. For reference, this is the proposal sent for Arrow
• (Hopefully) get approval for the board (for example, see the minutes of the meeting that created Arrow)

[alamb]

Update: I spent a while researching the approproate procedure and haven't found one yet. Thus, I sent a note to the incubator email list asking for help: https://lists.apache.org/list.html?general@incubator.apache.org

Update: https://lists.apache.org/thread/r4n73pmms1lv0jbohyx1o1z13d615t99

Except from response

I believe the process is for the Board to create a new PMC whose IP and PMC members are an exact copy of the parent PMC. (Like the Unix “fork” function.) And then both PMCs delete the stuff, and PMC members, they no longer need.

Arrow itself was created via this process (from Drill). You could search the archives for the board resolution that created Arrow. The Incubator is not involved in the process. (Except in some minor ways, such as a name search.)

[alamb]

Update: I think I now understand the process required for the propsal and I researched what is necessary (and I updated this ticket with the relevant items).

The first thing will be to ensure the naming is fine. I think it is but we should go through the official naming approval process

@andygrove in order to do a name search / validation on DataFusion, one of the fields is "derivation". Do you have any background / text I can refer to related to how you picked the name DataFusion?

For reference, here is the relevant entry for the Arrow project:
https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-92?jql=project%20%3D%20PODLINGNAMESEARCH%20AND%20description%20~%20Arrow

[alamb]

I started a discussion on initial PMC membership on the mailing list: https://lists.apache.org/thread/pymrzcdw4qdptvby85f69rg3pcckl15b

[liurenjie1024]

Just curious, will arrow-ballista be included in the new top project or still remain under arrow?

[alamb]

Just curious, will arrow-ballista be included in the new top project or still remain under arrow?

@liurenjie1024 that is an excellent question. I think ballista would naturally belong in the DataFusion project, but I don't know for sure.

I have stared a (new) discussion about that on the mailing list: https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q

[liurenjie1024]

Just curious, will arrow-ballista be included in the new top project or still remain under arrow?

@liurenjie1024 that is an excellent question. I think ballista would naturally belong in the DataFusion project, but I don't know for sure.

I have stared a (new) discussion about that on the mailing list: https://lists.apache.org/thread/ob3n0d9ky0bgrryl3xn39w9k566bq00q

+1, it would be more nature to move ballista along with datafusion.

[alamb]

I have filed a name search request / research item on https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-219

The next step will be to pull the entire proposal together into one coherent document for discussion

[alamb]

Here is a draft proposal https://docs.google.com/document/d/11WTNYS8KWScOt3ySTX39WVS6krPhUvHsuJRY9PZQx4g/edit

I also sent a note to the arrow dev list:
https://lists.apache.org/thread/c150t1s1x0kcb3r03cjyx31kqs5oc341

[alamb]

The proposal is drafted and under review. ETA on a board vote is April 2024. Details #6475 (comment)

I am claiming the proposal is written so closing this ticket