Skip to content

Update Upgrade Guide for 48.0.1 #16699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 7, 2025
Merged

Update Upgrade Guide for 48.0.1 #16699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/index.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ To get started, see
:caption: Library User Guide

library-user-guide/index
library-user-guide/upgrading
Copy link
Contributor Author

@alamb alamb Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved the upgrade link up higher in the index so it was easier to find

library-user-guide/extensions
library-user-guide/using-the-sql-api
library-user-guide/working-with-exprs
Expand All @@ -138,7 +139,6 @@ To get started, see
library-user-guide/extending-operators
library-user-guide/profiling
library-user-guide/query-optimizer
library-user-guide/upgrading

.. .. _toc.contributor-guide:

Expand Down
38 changes: 20 additions & 18 deletions docs/source/library-user-guide/upgrading.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,24 +62,7 @@ DataFusionError::SchemaError(

[#16652]: https://github.com/apache/datafusion/issues/16652

### `datafusion.execution.collect_statistics` now defaults to `true`

The default value of the `datafusion.execution.collect_statistics` configuration
setting is now true. This change impacts users that use that value directly and relied
on its default value being `false`.

This change also restores the default behavior of `ListingTable` to its previous. If you use it directly
you can maintain the current behavior by overriding the default value in your code.

```rust
# /* comment to avoid running
ListingOptions::new(Arc::new(ParquetFormat::default()))
.with_collect_stat(false)
// other options
# */
```

### Metadata is now represented by `FieldMetadata`
### Metadata on Arrow Types is now represented by `FieldMetadata`

Metadata from the Arrow `Field` is now stored using the `FieldMetadata`
structure. In prior versions it was stored as both a `HashMap<String, String>`
Expand Down Expand Up @@ -137,6 +120,25 @@ SET datafusion.execution.spill_compression = 'zstd';

For more details about this configuration option, including performance trade-offs between different compression codecs, see the [Configuration Settings](../user-guide/configs.md) documentation.

## DataFusion `48.0.1`

### `datafusion.execution.collect_statistics` now defaults to `true`

The default value of the `datafusion.execution.collect_statistics` configuration
setting is now true. This change impacts users that use that value directly and relied
on its default value being `false`.

This change also restores the default behavior of `ListingTable` to its previous. If you use it directly
you can maintain the current behavior by overriding the default value in your code.

```rust
# /* comment to avoid running
ListingOptions::new(Arc::new(ParquetFormat::default()))
.with_collect_stat(false)
// other options
# */
Comment on lines +135 to +139
Copy link
Contributor

@comphead comphead Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# /* comment to avoid running
ListingOptions::new(Arc::new(ParquetFormat::default()))
.with_collect_stat(false)
// other options
# */
/* comment to avoid running
ListingOptions::new(Arc::new(ParquetFormat::default()))
.with_collect_stat(false)
// other options
*/

Not sure why those hashes were needed

Copy link
Contributor Author

@alamb alamb Jul 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The # hides the comments from the docs (so /* comment to avoid running doesn't show up in the docs)

Screenshot 2025-07-07 at 11 47 39 AM

```

## DataFusion `48.0.0`

### `Expr::Literal` has optional metadata
Expand Down