[EPIC] Tasks for a new Top Level Apache Project

[alamb]

Is your feature request related to a problem or challenge?

Update: the new project was approved by the board and we are now in the process of setting it all up

We are on track to have a new top level apache project for DataFusion #6475 (see Proposal Document for more details)

Once this is approved by the ASF board, we will have some logistical things to change. This ticket tracks the work items

Describe the solution you'd like

Rename github repositories

• https://issues.apache.org/jira/browse/INFRA-25726

Note it is critical in my opinion that all existing links continue to work (as in the url https://github.com/apache/arrow-datafusion/issues/9691 will be redirected to the new repo). I believe this is the default behavior in Github when a repository is renamed but we should double check.

• https://github.com/apache/arrow-datafusion/ → apache/datafusion
• https://github.com/apache/arrow-datafusion-python → apache/datafusion-python
• https://github.com/apache/arrow-ballista → apache/datafusion-ballista
• https://github.com/apache/arrow-ballista-python → apache/datafusion-ballista-python
• https://github.com/apache/arrow-datafusion-comet → apache/datafusion-comet

Update docs

• Update documentation to replace Apache Arrow DataFusion with Apache DataFusion #10130
• Change hosting url from https://arrow.apache.org/datafusion/ to https://datafusion.apache.org (with a redirect from https://arrow.apache.org/datafusion/)
• Change link under subprojects from https://arrow.apache.org/ to point at new docsite (or maybe we should just remove the link all together)
• Replace NOTICES.txt with version that is relevant to DataFusion #10131

Infrastructure

• Create Project Website (datafusion.apache.org) #10151
• Redirect existing docs from https://arrow.apache.org/ to https://datafusion.apache.org/ #10194
• Create a DOAP (description of a project file) #10133
• Create mailing lists (dev@datafusion.apache.org, github@datafusion.apache.org, commits@datafusion.apache.org, private@datafusion.apache.org)
• (email trouble) https://issues.apache.org/jira/browse/INFRA-25727
• (DNS trouble): https://issues.apache.org/jira/browse/INFRA-25731
• Update ASF config https://github.com/apache/datafusion/blob/main/.asf.yaml
• Update committers list: Update committer list for new top level DataFusion project #10154

Process

• Updates to the release scripts and associated documentation to be datafusion rather than arrow-datafusion #10134
• Update communications section https://arrow.apache.org/datafusion/contributor-guide/communication.html
• Create a publicly accessable list of committers / affiliations (ideally from the data in https://people.apache.org/phonebook.html?unix=datafusion, similarly to https://arrow.apache.org/committers/)
• Consider adding a "Downloads" page #10236
• May 2024 ASF Board Report #10281

Announcements

• Blog Post about graduating to a new top level project #10135

[andygrove]

We will also need to make some updates to the release scripts and associated documentation (link to reporter URL, mailing list name, and so on)

[phillipleblanc]

Note it is critical in my opinion that all existing links continue to work (as in the url https://github.com/apache/arrow-datafusion/issues/9691 will be redirected to the new repo). I believe this is the default behavior in Github when a repository is renamed but we should double check.

Yes, I just verified this behavior is the default. I created a repo at https://github.com/phillipleblanc/datafusion-rename-test - created an issue: https://github.com/phillipleblanc/datafusion-rename-test/issues/1 and renamed the repo to datafusion-rename-success - and now the above links redirect to the new repo name.

[alamb]

Thank you for double checking @phillipleblanc

[alamb]

Its official -- DataFusion is now its own Top Level Project!

https://projects.apache.org/project.html?datafusion

I plan to file individual tickets for the tasks listed above later this morning

[alamb]

My plan / hope is that we leave as much of the operations of DataFusion the same day to day as we transition to our own top level project. Once we have gotten ourselves running independently then we can discuss any potential changes

I think mailing lists are the first important thing to setup, so following https://github.com/apache/arrow/blob/main/.asf.yaml, I submitted requests via https://selfserve.apache.org for the following emails

• dev@datafusion.apache.org
• private@datafusion.apache.org
• commits@datafusion.apache.org
• github@datafusion.apache.org

In continuing with the current norms of the DataFusion community, I did not create a users@datafusion.apache.org mailing list, because according to the docs "The vast majority of communication occurs in the open on our github repository in the form of tickets, issues, discussions, and Pull Requests."

Thus I think having another potential list that people might accidentally send a request would be confusing.

[metesynnada]

🎉🎉

[andygrove]

I created #10130 to make some documentation updates to change the name from Apache Arrow DataFusion to Apache DataFusion

[viirya]

Regarding mailing lists, I think this may not be possible or hard to do, but just curious about if there are similar cases before. Is it possible to move previous threads related to DataFusion to new mailing lists?

[andygrove]

I filed an INFRA ticket to rename the repos.

https://issues.apache.org/jira/browse/INFRA-25726

[alamb]

Regarding mailing lists, I think this may not be possible or hard to do, but just curious about if there are similar cases before. Is it possible to move previous threads related to DataFusion to new mailing lists?

I am not sure -- maybe we could look in the INFRA JIRA tickets or file a request

[alamb]

I created #10133 and #10134 for logistics

I also filed #10135 to make a blog post -- I would be interested in anyone has opinions on the venue (should we post to the arrow blog or make the first post on a new datafusion specific blog)?