create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases #1253

houqp · 2021-11-06T05:27:21Z

Rationale for this change

Prepare for the 6.0.0 release.

We have way more changes than I initially expected :D

Also cc @rdettai

What changes are included in this PR?

New changelogs
Improve release automations

Are there any user-facing changes?

no

houqp · 2021-11-06T05:28:42Z

I am quite comfortable with the release automation tools and documentation now, I think next time, any committer should be able to run the release so I won't be the only one who is running the releases.

houqp · 2021-11-06T05:29:35Z

.github_changelog_generator

 #

-# point to the old changelog in apache/arrow
-front-matter=For older versions, see [apache/arrow/CHANGELOG.md](https://github.com/apache/arrow/blob/master/CHANGELOG.md)\n


not needed anymore because it's already included in the 5.0.0 changelog and we are appending 6.0.0 changelog on top of it.

houqp · 2021-11-06T05:31:34Z

dev/release/README.md

-Create local release rc tags:
-
-```
-git tag -f 5.1.0-rc-local
-# if there is ballista release
-git tag -f ballista-0.5.0-rc-local
-# if there is python binding release
-git tag -f python-0.3.0-rc-local
-```
-
-Manully edit the previous release version tag in


turns out these local tags are not being used at all in changelog automation.

houqp · 2021-11-06T05:32:48Z

dev/release/update_change_log.sh

    --project arrow-datafusion \
-    --since-tag "${FROM_VER}" \
    --include-labels "${PROJECT}" \
+    --base "${OUTPUT_PATH}" \


this argument appends new version changelog to the existing one file instead of overwriting the full history.

jimexist · 2021-11-06T06:24:12Z

thanks for the update @houqp - i guess i can help with releasing next time (or a minor one, like 6.1)

xudong963 · 2021-11-06T07:09:24Z

Nice work 👍🏻@houqp, looking forward to being a committer and running the release, LOL 😁

alamb

This is great -- thank you very much @houqp ❤️

Do you plan a release blog? I would love to help make one. There are all sorts of cool things I think we could write about in this release (AVRO support, explain analyze, constant folding, hive partitioning support, and the new roadmap

alamb · 2021-11-06T10:01:50Z

datafusion/CHANGELOG.md

+
+**Breaking changes:**
+
+- Removed deprecated with\_concurrency [\#1200](https://github.com/apache/arrow-datafusion/pull/1200) ([rdettai](https://github.com/rdettai))


There is a lot of good stuff in DF 6.0.0 😮

houqp · 2021-11-06T16:39:01Z

Do you plan a release blog? I would love to help make one.

Please go ahead @alamb ! I won't have time to do this until the release has been cut and voted. We can all collaborate to the blog like the last one @andygrove wrote.

jimexist · 2021-11-07T13:56:25Z

oh by the way after this is cut and release i will help with the homebrew formula update

houqp · 2021-11-08T04:30:22Z

Going to wait for @xudong963 's set changes for 6.0.0 release since those PRs look pretty complete and mostly ready to merge :)

alamb · 2021-11-08T22:16:25Z

Please go ahead @alamb ! I won't have time to do this until the release has been cut and voted. We can all collaborate to the blog like the last one @andygrove wrote.

Here is an initial shot: apache/arrow-site#160

I may have missed something

xudong963 · 2021-11-13T06:52:17Z

@houqp Maybe README also need update: https://github.com/apache/arrow-datafusion/blob/master/README.md#using-datafusion-as-a-library

[dependencies]
datafusion = "5.0.0"

to

[dependencies]
datafusion = "6.0.0"

houqp · 2021-11-14T01:04:12Z

Thanks @xudong963 for catching that, I have updated the version bump automation to cover README.md going forward.

alamb · 2021-11-14T17:12:11Z

🎉

…ache#1253) * create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases

… in window aggregates (apache#1253)

…rrow errors (apache#1253) * Refactor schema, config, dataframe, and expression classes to use RwLock and Mutex for interior mutability * Add error handling to CaseBuilder methods to preserve builder state * Refactor to use parking_lot for interior mutability in schema, config, dataframe, and conditional expression modules * Add concurrency tests for SqlSchema, Config, and DataFrame * Add tests for CaseBuilder to ensure builder state is preserved on success * Add test for independent handles in CaseBuilder to verify behavior * Fix CaseBuilder to preserve state correctly in when() method * Refactor to use named constant for boolean literals in test_expr.py * fix ruff errors * Refactor to introduce type aliases for cached batches in dataframe.rs * Cherry pick from apache#1252 * Add most expr - cherry pick from apache#1252 * Add source root - cherry pick apache#1252 * Fix license comment formatting in config.rs * Refactor caching logic to use a local variable for IPython environment check * Add test for ensuring exposed pyclasses default to frozen * Add PyO3 class mutability guidelines reference to contributor guide * Mark boolean expression classes as frozen for immutability * Refactor PyCaseBuilder methods to eliminate redundant take/store logic * Refactor PyConfig methods to improve readability by encapsulating configuration reads * Resolve patch apply conflicts for CaseBuilder concurrency improvements - Added CaseBuilderHandle guard that keeps the underlying CaseBuilder alive while holding the mutex and restores it on drop - Updated when, otherwise, and end methods to operate through the guard and consume the builder explicitly - This prevents transient None states during concurrent access and improves thread safety * Resolve Config optimization conflicts for improved read/write concurrency - Released Config read guard before converting values to Python objects in get and get_all - Ensures locks are held only while collecting scalar entries, not during expensive Python object conversion - Added regression test that runs Config.get_all and Config.set concurrently to guard against read/write contention regressions - Improves overall performance by reducing lock contention in multi-threaded scenarios * Refactor PyConfig get methods for improved readability and performance * Refactor test_expr.py to replace positional boolean literals with named constants for improved linting compliance * fix ruff errors * Add license header to test_pyclass_frozen.py for compliance * Alternate approach to case expression * Replace case builter with keeping the expressions and then applying as required * Update unit tests * Refactor case and when functions to utilize PyCaseBuilder for improved clarity and functionality * Update src/expr/conditional_expr.rs --------- Co-authored-by: ntjohnson1 <24689722+ntjohnson1@users.noreply.github.com> Co-authored-by: Tim Saucer <timsaucer@gmail.com>

houqp added this to the 6.0.0 milestone Nov 6, 2021

houqp requested review from Dandandan, alamb, andygrove, jimexist, jorgecarleitao and kszucs November 6, 2021 05:27

github-actions bot added ballista development-process Related to development process of DataFusion labels Nov 6, 2021

houqp commented Nov 6, 2021

View reviewed changes

jimexist approved these changes Nov 6, 2021

View reviewed changes

alamb mentioned this pull request Nov 6, 2021

Release Datafusion 6.0.0 #890

Closed

alamb approved these changes Nov 6, 2021

View reviewed changes

houqp force-pushed the qp_release branch from e7763ae to 87c8eaa Compare November 6, 2021 23:57

andygrove mentioned this pull request Nov 8, 2021

Ballista integration tests fail to run (feature edition2021 is required) #1272

Closed

alamb mentioned this pull request Nov 8, 2021

Add DataFusion 6.0.0 blog apache/arrow-site#160

Merged

houqp force-pushed the qp_release branch from 87c8eaa to cfe3c85 Compare November 12, 2021 05:59

houqp mentioned this pull request Nov 12, 2021

python: update docs to use new APIs #1287

Merged

houqp added 3 commits November 13, 2021 13:19

create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases

4c441d5

update datafusion version in readme

84c3486

regenerate changelog

97f65fb

houqp force-pushed the qp_release branch from cfe3c85 to 97f65fb Compare November 13, 2021 22:50

github-actions bot added the documentation Improvements or additions to documentation label Nov 13, 2021

houqp merged commit 7824a8d into apache:master Nov 14, 2021

houqp deleted the qp_release branch November 14, 2021 01:04

matthewmturner pushed a commit to matthewmturner/arrow-datafusion that referenced this pull request Nov 16, 2021

create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases (ap…

5257304

…ache#1253) * create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases

unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025

fix: Fall back to Spark for unsupported partition or sort expressions…

be48839

… in window aggregates (apache#1253)


		Breaking changes:

		- Removed deprecated with\_concurrency [\#1200](https://github.com/apache/arrow-datafusion/pull/1200) ([rdettai](https://github.com/rdettai))

create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases #1253

create datafusion 6.0.0, ballista 0.6.0 and python 0.4.0 releases #1253

Uh oh!

Conversation

houqp commented Nov 6, 2021

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

houqp commented Nov 6, 2021

Uh oh!

houqp Nov 6, 2021

Choose a reason for hiding this comment

Uh oh!

houqp Nov 6, 2021

Choose a reason for hiding this comment

Uh oh!

houqp Nov 6, 2021

Choose a reason for hiding this comment

Uh oh!

jimexist commented Nov 6, 2021

Uh oh!

xudong963 commented Nov 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Nov 6, 2021

Choose a reason for hiding this comment

Uh oh!

houqp commented Nov 6, 2021

Uh oh!

jimexist commented Nov 7, 2021

Uh oh!

houqp commented Nov 8, 2021

Uh oh!

alamb commented Nov 8, 2021

Uh oh!

xudong963 commented Nov 13, 2021

Uh oh!

houqp commented Nov 14, 2021

Uh oh!

alamb commented Nov 14, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xudong963 commented Nov 6, 2021 •

edited

Loading