37.0.0.md

37.0.0 (2024-03-28)

Full Changelog

Breaking changes:

• refactor: Change SchemaProvider::table to return Result<Option<..> rather than Option<..> #9307 (crepererum)
• feat: issue_9285: port builtin reg function into datafusion-function-* (1/3 regexpmatch) #9329 (Lordworms)
• Cache common plan properties to eliminate recursive calls in physical plan #9346 (mustafasrepo)
• Consolidate TreeNode transform and rewrite APIs #8891 (peter-toth)
• Extend argument types for udf return_type_from_exprs #9522 (jayzhan211)
• Systematic Configuration in 'Create External Table' and 'Copy To' Options #9382 (metesynnada)
• Move trim functions (btrim, ltrim, rtrim) to datafusion_functions, make expr_fn API consistent #9730 (Omega359)

Performance related:

• perf: improve to_field performance #9722 (haohuaijin)

Implemented enhancements:

• feat: support for defining ARRAY columns in CREATE TABLE #9381 (jonahgao)
• feat: support unnest in FROM clause #9355 (jonahgao)
• feat: support nvl2 function #9364 (guojidan)
• feat: issue #9224 substitute tlide in table path #9259 (Lordworms)
• feat: replace std Instant with wasm-compatible wrapper #9189 (waynexia)
• feat: support unnest with additional columns #9400 (jonahgao)
• feat: Support EscapedStringLiteral, update sqlparser to 0.44.0 #9268 (JasonLi-cn)
• feat: add support for fixed list wildcard in type signature #9312 (universalmind303)
• feat: Add projection to HashJoinExec. #9236 (my-vegetable-has-exploded)
• feat: function name hints for UDFs #9407 (SteveLauC)
• feat: Introduce convert Expr to SQL string API and basic feature #9517 (backkem)
• feat: implement more expr_to_sql functionality #9578 (devinjdangelo)
• feat: implement aggregation and subquery plans to SQL #9606 (devinjdangelo)
• feat: track memory usage for recursive CTE, enable recursive CTEs by default #9619 (jonahgao)
• feat: Between expr to sql string #9803 (sebastian2296)
• feat: Expose array_empty and list_empty functions as alias of empty function #9807 (erenavsarogullari)
• feat: Not expr to string #9802 (sebastian2296)
• feat: pass SessionState not SessionConfig to FunctionFactory::create #9837 (tshauck)

Fixed bugs:

• fix: use JoinSet to make spawned tasks cancel-safe #9318 (DDtKey)
• fix: nvl function's return type #9357 (guojidan)
• fix: panic in isnan() when no args are given #9377 (SteveLauC)
• fix: using test data sample for catalog example #9372 (korowa)
• fix: sort_batch function unsupported mixed types with list #9410 (JasonLi-cn)
• fix: casting to ARRAY types failed #9441 (jonahgao)
• fix: reading from partitioned json & arrow tables #9431 (korowa)
• fix: coalesce function should return correct data type #9459 (viirya)
• fix: generate_series and range panic on edge cases #9503 (jonahgao)
• fix: substr_index not handling negative occurrence correctly #9475 (jonahgao)
• fix: support two argument TRIM #9521 (tshauck)
• fix: incorrect null handling in range and generate_series #9574 (jonahgao)
• fix: recursive cte hangs on joins #9687 (jonahgao)
• fix: parallel parquet can underflow when max_record_batch_rows < execution.batch_size #9737 (devinjdangelo)
• fix: change placeholder errors from Internal to Plan #9745 (erratic-pattern)
• fix: ensure mutual compatibility of the two input schemas from recursive CTEs #9795 (jonahgao)

Documentation updates:

• docs: put flatten in top fn list #9376 (SteveLauC)
• Update documentation so list_to_string alias to point to array_to_string #9374 (monkwire)
• Uplift keys/dependencies to use more workspace inheritance #9293 (Jefffrey)
• docs: update contributor guide (migration to sqllogictest is done) #9408 (SteveLauC)
• Move the to_timestamp* functions to datafusion-functions #9388 (Omega359)
• NEW Logo #9385 (pinarbayata)
• Minor: docs: rm duplicate words. #9449 (my-vegetable-has-exploded)
• Update contributor guide with updated scalar function howto #9438 (Omega359)
• docs: fix extraneous char in array functions table of contents #9560 (tshauck)
• doc: Add missing doc link #9631 (Weijun-H)
• chore: remove repetitive word the the --> the in docs / comments #9673 (InventiveCoder)
• Update example-usage.md to remove reference to simd and rust nightly. #9677 (Omega359)
• Minor: Improve documentation for LogicalPlan::expressions #9698 (alamb)
• Add Minimum Supported Rust Version policy to docs #9681 (alamb)
• doc: Updated known users list and usage dependency description #9718 (comphead)

Merged pull requests:

• refactor: Change SchemaProvider::table to return Result<Option<..> rather than Option<..> #9307 (crepererum)
• fix write_partitioned_parquet_results test case bug #9360 (guojidan)
• fix: use JoinSet to make spawned tasks cancel-safe #9318 (DDtKey)
• Update nix requirement from 0.27.1 to 0.28.0 #9344 (dependabot[bot])
• Replace usages of internal_err with exec_err where appropriate #9241 (Omega359)
• feat : Support for deregistering user defined functions #9239 (mobley-trent)
• fix: nvl function's return type #9357 (guojidan)
• refactor: move acos() to function crate #9297 (SteveLauC)
• docs: put flatten in top fn list #9376 (SteveLauC)
• Update documentation so list_to_string alias to point to array_to_string #9374 (monkwire)
• feat: issue_9285: port builtin reg function into datafusion-function-* (1/3 regexpmatch) #9329 (Lordworms)
• Add test to verify issue #9161 #9265 (jonahgao)
• refactor: fix error macros hygiene (always import DataFusionError) #9366 (crepererum)
• feat: support for defining ARRAY columns in CREATE TABLE #9381 (jonahgao)
• fix: panic in isnan() when no args are given #9377 (SteveLauC)
• feat: support unnest in FROM clause #9355 (jonahgao)
• feat: support nvl2 function #9364 (guojidan)
• refactor: move asin() to function crate #9379 (SteveLauC)
• fix: using test data sample for catalog example #9372 (korowa)
• delete tail space, fix error: unused import: DataFusionError #9386 (Tangruilin)
• Run cargo-fmt on datafusion-functions/core #9367 (alamb)
• Cache common plan properties to eliminate recursive calls in physical plan #9346 (mustafasrepo)
• Run cargo-fmt on all of datafusion-functions #9390 (alamb)
• feat: issue #9224 substitute tlide in table path #9259 (Lordworms)
• port range function and change gen_series logic #9352 (Lordworms)
• [MINOR]: Generate physical plan, instead of logical plan in the bench test #9383 (mustafasrepo)
• Add to_date function #9019 (Tangruilin)
• Minor: clarify performance in docs for ScalarUDF, ScalarUDAF and ScalarUDWF #9384 (alamb)
• feat: replace std Instant with wasm-compatible wrapper #9189 (waynexia)
• Uplift keys/dependencies to use more workspace inheritance #9293 (Jefffrey)
• Improve documentation for ExecutionPlanProperties, use consistent field name #9389 (alamb)
• Doc: Workaround for Running cargo test locally without signficant memory #9402 (devinjdangelo)
• feat: support unnest with additional columns #9400 (jonahgao)
• Minor: improve the display name of unnest expressions #9412 (jonahgao)
• Minor: Move function signature check to planning stage #9401 (2010YOUY01)
• chore(deps): update substrait requirement from 0.24.0 to 0.25.1 #9406 (dependabot[bot])
• docs: update contributor guide (migration to sqllogictest is done) #9408 (SteveLauC)
• Move the to_timestamp* functions to datafusion-functions #9388 (Omega359)
• Minor: Support LargeList List Range indexing and fix large list handling in ConstEvaluator #9393 (jayzhan211)
• NEW Logo #9385 (pinarbayata)
• Handle serde for ScalarUDF #9395 (yyy1000)
• Minior: Add tests with sqrt with negative argument #9426 (caicancai)
• Move SpawnedTask from datafusion_physical_plan to new datafusion_common_runtime crate #9414 (mustafasrepo)
• Re-export datafusion-functions-array #9433 (andygrove)
• Minor: Support LargeList for ListIndex #9424 (PsiACE)
• move ArrayDims, ArrayNdims and Cardinality to datafusion-function-crate #9425 (Weijun-H)
• refactor: make instr() an alias of strpos() #9396 (SteveLauC)
• Add test case for invalid tz in timestamp literal #9429 (MohamedAbdeen21)
• Minor: simplify call #9434 (alamb)
• Support IGNORE NULLS for LEAD window function #9419 (comphead)
• fix sqllogicaltest result #9444 (jackwener)
• Minor: docs: rm duplicate words. #9449 (my-vegetable-has-exploded)
• minor: fix cargo clippy some warning #9442 (jackwener)
• port regexp_like function and port related tests #9397 (Lordworms)
• fix: sort_batch function unsupported mixed types with list #9410 (JasonLi-cn)
• refactor: add join_unwind to SpawnedTask #9422 (DDtKey)
• Ignore null LEAD support for small batch sizes. #9445 (mustafasrepo)
• fix: casting to ARRAY types failed #9441 (jonahgao)
• fix: reading from partitioned json & arrow tables #9431 (korowa)
• feat: Support EscapedStringLiteral, update sqlparser to 0.44.0 #9268 (JasonLi-cn)
• Minor: fix LEAD test description #9451 (comphead)
• Consolidate TreeNode transform and rewrite APIs #8891 (peter-toth)
• Support Date32 arguments for generate_series #9420 (Lordworms)
• Minor: change doc for range #9455 (Lordworms)
• doc: add missing function index in scalar_expression.md #9462 (Weijun-H)
• build: Update bigdecimal version in Cargo.toml #9471 (comphead)
• chore(deps): update base64 requirement from 0.21 to 0.22 #9446 (dependabot[bot])
• Port regexp_replace functions and related tests #9454 (Lordworms)
• Update contributor guide with updated scalar function howto #9438 (Omega359)
• feat: add support for fixed list wildcard in type signature #9312 (universalmind303)
• Add a ScalarUDFImpl::simplfy() API, move SimplifyInfo et al to datafusion_expr #9304 (jayzhan211)
• Implement IGNORE NULLS for FIRST_VALUE #9411 (huaxingao)
• Add plugable handler for CREATE FUNCTION #9333 (milenkovicm)
• Enable configurable display of partition sizes in the explain statement #9474 (jayzhan211)
• Reduce casts for LEAD/LAG #9468 (comphead)
• [CI build] fix chrono suggestions #9486 (comphead)
• Make regex dependency optional in datafusion-functions, add CI checks for function packages #9473 (alamb)
• fix: coalesce function should return correct data type #9459 (viirya)
• LEAD/LAG calculate default value once #9485 (comphead)
• chore: simplify the return type of validate_data_types() #9491 (waynexia)
• minor: use arrow-rs casting from Float to Timestamp #9500 (comphead)
• chore(deps): update substrait requirement from 0.25.1 to 0.27.0 #9502 (dependabot[bot])
• fix: generate_series and range panic on edge cases #9503 (jonahgao)
• Fix undeterministic behaviour of schema nullability of lag window query #9508 (mustafasrepo)
• Add to_unixtime function #9077 (Tangruilin)
• Minor: fixed transformed state in UDF Simplify #9484 (alamb)
• test: port strpos test in physical_expr/src/functions to sqllogictest #9439 (SteveLauC)
• Port ArrayHas family to functions-array #9496 (jayzhan211)
• port array_empty and array_length to datafusion-function-array crate #9510 (Weijun-H)
• fix: substr_index not handling negative occurrence correctly #9475 (jonahgao)
• [minor] extract collect file statistics method and add doc #9490 (Ted-Jiang)
• test: sqllogictests for multiple tables join #9480 (korowa)
• Add support for ignore nulls for LEAD, LAG in WindowAggExec #9498 (Lordworms)
• Minior: Improve log expr description #9516 (caicancai)
• port flatten to datafusion-function-array #9523 (Weijun-H)
• feat: Add projection to HashJoinExec. #9236 (my-vegetable-has-exploded)
• Add example for FunctionFactory #9482 (milenkovicm)
• Move date_part, date_trunc, date_bin functions to datafusion-functions #9435 (Omega359)
• fix: support two argument TRIM #9521 (tshauck)
• Remove physical expr of ListIndex and ListRange, convert to array_element and array_slice functions #9492 (jayzhan211)
• feat: function name hints for UDFs #9407 (SteveLauC)
• Minor: Improve documentation for registering AnalyzerRule #9520 (alamb)
• Extend argument types for udf return_type_from_exprs #9522 (jayzhan211)
• move make_array array_append array_prepend array_concat function to datafusion-functions-array crate #9504 (guojidan)
• Port StringToArray to function-arrays subcrate #9543 (erenavsarogullari)
• Minor: remove .. pattern matching in sql planner #9531 (alamb)
• Minor: Fix document Interval syntax #9542 (yyy1000)
• Port struct to datafusion-functions #9546 (yyy1000)
• UDAF and UDWF support aliases #9489 (lewiszlw)
• docs: fix extraneous char in array functions table of contents #9560 (tshauck)
• [MINOR]: Fix undeterministic test #9559 (mustafasrepo)
• Port arrow_typeof to datafusion-function #9524 (yyy1000)
• feat: Introduce convert Expr to SQL string API and basic feature #9517 (backkem)
• Port ArraySort to function-arrays subcrate #9551 (erenavsarogullari)
• refactor: unify some plan optimization in CommonSubexprEliminate #9556 (jackwener)
• Port ArrayDistinct to functions-array subcrate #9549 (erenavsarogullari)
• Minor: add a sql_planner benchmarks to reflecte select many field on a huge table #9536 (haohuaijin)
• Support IGNORE NULLS for FIRST/LAST window function #9470 (huaxingao)
• Systematic Configuration in 'Create External Table' and 'Copy To' Options #9382 (metesynnada)
• fix: incorrect null handling in range and generate_series #9574 (jonahgao)
• Update README.md #9572 (Abdullahsab3)
• Port tan, tanh to datafusion-functions #9535 (ongchi)
• feat(9493): provide access to FileMetaData for files written with ParquetSink #9548 (wiedld)
• Export datafusion-functions UDFs publically #9585 (alamb)
• Update the comment and Add a check #9571 (colommar)
• Port ArrayRepeat to functions-array subcrate #9568 (erenavsarogullari)
• Fix ApproxPercentileAccumulator on zero values #9582 (Dandandan)
• Add FunctionRewrite API, Move Array specific rewrites to datafusion_functions_array #9583 (alamb)
• Move from_unixtime, now, current_date, current_time functions to datafusion-functions #9537 (Omega359)
• minor: update Debug trait impl for WindowsFrame #9587 (comphead)
• Initial support LogicalPlan to SQL String #9596 (backkem)
• refactor: use a common macro to define math UDFs #9598 (jonahgao)
• Move all crypto related functions to datafusion-functions #9590 (Lordworms)
• Remove physical expr of NamedStructField, convert to get_field function call #9563 (yyy1000)
• Add /benchmark github command to comparison benchmark between base and pr commit #9461 (gruuya)
• support unnest as subexpression #9592 (YjyJeff)
• feat: implement more expr_to_sql functionality #9578 (devinjdangelo)
• Port ArrayResize to functions-array subcrate #9570 (erenavsarogullari)
• Move make_date, to_char to datafusion-functions #9601 (Omega359)
• Fix to_timestamp benchmark #9608 (Omega359)
• feat: implement aggregation and subquery plans to SQL #9606 (devinjdangelo)
• Port ArrayElem/Slice/PopFront/Back into functions-array #9615 (jayzhan211)
• Minor: Remove datafusion-functions-array dependency from datafusion-optimizer #9621 (alamb)
• Enable TTY during bench data generation #9626 (gruuya)
• Remove constant expressions from SortExprs in the SortExec #9618 (mustafasrepo)
• Try fixing missing results name in the benchmark step #9632 (gruuya)
• feat: track memory usage for recursive CTE, enable recursive CTEs by default #9619 (jonahgao)
• doc: Add missing doc link #9631 (Weijun-H)
• Add explicit move of PR bench results if they were placed in HEAD dir #9636 (gruuya)
• Add array_reverse function to datafusion-function-* crate #9630 (Weijun-H)
• Move parts of InListSimplifier simplify rules to Simplifier #9628 (jayzhan211)
• Port Array Union and Intersect to functions-array #9629 (jayzhan211)
• Port ArrayPosition and ArrayPositions to functions-array subcrate #9617 (erenavsarogullari)
• Optimize make_date (#9089) #9600 (vojtechtoman)
• Support AT TIME ZONE clause #9647 (tinfoil-knight)
• Window Linear Mode use smaller buffers #9597 (mustafasrepo)
• Port ArrayExcept to functions-array subcrate #9634 (erenavsarogullari)
• chore: improve array expression doc and clean up array_expression.rs #9650 (Weijun-H)
• Minor: remove clone in exprlist_to_fields #9657 (jayzhan211)
• Port ArrayRemove, ArrayRemoveN, ArrayRemoveAll to functions-array subcrate #9656 (erenavsarogullari)
• Minor: Remove redundant dependencies from datafusion-functions/Cargo.toml #9622 (alamb)
• Support IGNORE NULLS for NTH_VALUE window function #9625 (huaxingao)
• Improve Robustness of Unparser Testing and Implementation #9623 (devinjdangelo)
• Adding Constant Check for FilterExec #9649 (Lordworms)
• chore(deps-dev): bump follow-redirects from 1.15.4 to 1.15.6 in /datafusion/wasmtest/datafusion-wasm-app #9609 (dependabot[bot])
• move array_replace family functions to datafusion-function-array crate #9651 (Weijun-H)
• chore: remove repetitive word the the --> the in docs / comments #9673 (InventiveCoder)
• Update example-usage.md to remove reference to simd and rust nightly. #9677 (Omega359)
• [MINOR]: Remove some .unwraps from nth_value.rs file #9674 (mustafasrepo)
• minor: Remove deprecated methods #9627 (comphead)
• Migrate arrow_cast to a UDF #9610 (alamb)
• parquet: Add rowgroups_matched{statistics,bloom_filter} statistics #9640 (progval)
• Make COPY TO align with CREATE EXTERNAL TABLE #9604 (metesynnada)
• Support "A column is known to be entirely NULL" in PruningPredicate #9223 (appletreeisyellow)
• Suppress self update for windows CI runner #9661 (jayzhan211)
• add schema to SQL ast builder #9624 (sardination)
• core/tests/parquet/row_group_pruning.rs: Add tests for strings #9642 (progval)
• Fix incorrect results with multiple COUNT(DISTINCT..) aggregates on dictionaries #9679 (alamb)
• parquet: Add support for Bloom filters on binary columns #9644 (progval)
• Update Arrow/Parquet to 51.0.0, tonic to 0.11 #9613 (tustvold)
• Move inlist rule to expr_simplifier #9692 (jayzhan211)
• Support Serde for ScalarUDF in Physical Expressions #9436 (yyy1000)
• Support Union types in ScalarValue #9683 (avantgardnerio)
• parquet: Add support for row group pruning on FixedSizeBinary #9646 (progval)
• Minor: Improve documentation for LogicalPlan::expressions #9698 (alamb)
• Make builtin window function output datatype to be derived from schema #9686 (comphead)
• refactor: Extract array_to_string and string_to_array from functions-array subcrate' s kernels and udf containers #9704 (erenavsarogullari)
• Add Minimum Supported Rust Version policy to docs #9681 (alamb)
• doc: Add DataFusion profiling documentation for MacOS #9711 (comphead)
• Minor: add ticket reference to commented out test #9715 (alamb)
• Minor: Rename path from common_runtime to common-runtime #9717 (alamb)
• Use object_store:BufWriter to replace put_multipart #9648 (yyy1000)
• Fix COPY TO failing on passing format options through CLI #9709 (tinfoil-knight)
• fix: recursive cte hangs on joins #9687 (jonahgao)
• Move starts_with, to_hex, trim, upper to datafusion-functions (and add string_expressions) #9541 (Tangruilin)
• Support for extract(x from time) / date_part from time types #8693 (Jefffrey)
• doc: Updated known users list and usage dependency description #9718 (comphead)
• Minor: improve documentation for CommonSubexprEliminate #9700 (alamb)
• build: modify code to comply with latest clippy requirement #9725 (comphead)
• Minor: return internal error rather than panic on unexpected error in COUNT DISTINCT #9712 (alamb)
• fix(9678): short circuiting prevented population of visited stack, for common subexpr elimination optimization #9685 (wiedld)
• perf: improve to_field performance #9722 (haohuaijin)
• Minor: Run ScalarValue size test on aarch again #9728 (alamb)
• Move trim functions (btrim, ltrim, rtrim) to datafusion_functions, make expr_fn API consistent #9730 (Omega359)
• make format prefix optional for format options in COPY #9723 (tinfoil-knight)
• refactor: Extract range and gen_series functions from functions-array subcrate' s kernels and udf containers #9720 (erenavsarogullari)
• Move ascii function to datafusion_functions #9740 (PsiACE)
• adding expr to string for IsNotNull IsTrue IsFalse and IsUnkown #9739 (Lordworms)
• fix: parallel parquet can underflow when max_record_batch_rows < execution.batch_size #9737 (devinjdangelo)
• support format in options of COPY command #9744 (tinfoil-knight)
• Move lower, octet_length to datafusion-functions #9747 (Omega359)
• Fixed missing trim() in rust api #9749 (Omega359)
• refactor: Extract array_length, array_reverse and array_sort functions from functions-array subcrate' s kernels and udf containers #9751 (erenavsarogullari)
• refactor: Extract array_empty and array_repeat functions from functions-array subcrate' s kernels and udf containers #9762 (erenavsarogullari)
• Minor: remove an outdated TODO in TypeCoercion #9752 (jonahgao)
• refactor: Extract array_resize and cardinality functions from functions-array subcrate' s kernels and udf containers #9766 (erenavsarogullari)
• fix: change placeholder errors from Internal to Plan #9745 (erratic-pattern)
• Move levenshtein, uuid, overlay to datafusion-functions #9760 (Omega359)
• improve null handling for to_char #9689 (tinfoil-knight)
• Add Expr->String for ScalarFunction and InList #9759 (yyy1000)
• Move repeat, replace, split_part to datafusion_functions #9784 (Omega359)
• refactor: Extract array_dims, array_ndims and flatten functions from functions-array subcrate' s kernels and udf containers #9786 (erenavsarogullari)
• Minor: Improve documentation about ColumnarValues::values_to_array #9774 (alamb)
• Fix panic in struct function with mixed scalar/array arguments #9775 (alamb)
• refactor: Apply minor refactorings to functions-array crate #9788 (erenavsarogullari)
• Move bit_length and chr functions to datafusion_functions #9782 (PsiACE)
• Support tencent cloud COS storage in datafusion-cli #9734 (harveyyue)
• Make it easier to register configuration extension ... #9781 (milenkovicm)
• Expr to Sql : Case #9798 (yyy1000)
• feat: Between expr to sql string #9803 (sebastian2296)
• feat: Expose array_empty and list_empty functions as alias of empty function #9807 (erenavsarogullari)
• Support Expr Like to sql #9805 (Weijun-H)
• feat: Not expr to string #9802 (sebastian2296)
• [Minor]: Move some repetitive codes to functions(proto) #9811 (mustafasrepo)
• Implement IGNORE NULLS for LAST_VALUE #9801 (huaxingao)
• [MINOR]: Move some repetitive codes to functions #9810 (mustafasrepo)
• fix: ensure mutual compatibility of the two input schemas from recursive CTEs #9795 (jonahgao)
• Add support for constant expression evaluation in limit #9790 (mustafasrepo)
• Projection Pushdown through user defined LogicalPlan nodes. #9690 (mustafasrepo)
• chore(deps): update substrait requirement from 0.27.0 to 0.28.0 #9809 (dependabot[bot])
• Run TPC-H SF10 during PR benchmarks #9822 (gruuya)
• Expose parser on DFParser to enable user controlled parsing #9729 (tshauck)
• Disable parallel reading for gziped ndjson file #9799 (Lordworms)
• Optimize to_timestamp (with format) (#9090) #9833 (vojtechtoman)
• Create unicode module in datafusion/functions/src/unicode and unicode_expressions feature flag, move char_length function #9825 (Omega359)
• [Minor] Update TCPDS tests, remove some #[ignore]d tests #9829 (Dandandan)
• doc: Adding baseline benchmark example #9827 (comphead)
• Add name method to execution plan #9793 (matthewmturner)
• chore(deps-dev): bump express from 4.18.2 to 4.19.2 in /datafusion/wasmtest/datafusion-wasm-app #9826 (dependabot[bot])
• feat: pass SessionState not SessionConfig to FunctionFactory::create #9837 (tshauck)