Python Polars 1.10.0

🚀 Performance improvements

• Add/fix unordered row decode, change unordered format (#19284)
• Fast decision for Parquet dictionary encoding (#19256)
• Make date_range / datetime_range ~10x faster for constant durations (#19216)
• Batch utf8-validation in csv 18% / 25% on 1.9.0 (#19124)
• Use two-pass algorithm for csv to ensure correctness and SIMDize more ~17% (#19088)

✨ Enhancements

• Add SQL support for bit_count and bitwise &, |, and xor operators (#19114)
• Add credential provider utility classes for AWS, GCP (#19297)
• Support decoding Float16 in Parquet (#19278)
• Experimental credential_provider argument for scan_parquet (#19271)
• Allow DeltaTable input to scan_delta and read_delta (#19229)
• New quantile interpolation method & QUANTILE_DISC function in SQL (#19139)
• Conserve Parquet SortingColumns for ints (#19251)
• Low level flight interface (#19239)
• Improved list arithmetic support (#19162)
• Add Expr.struct.unnest() as alias for Expr.struct.field("*") (#19212)
• Add 'drop_empty_rows' parameter for read_ods (#19202)
• Add 'drop_empty_rows' parameter for read_excel (#18253)
• Expose LTS CPU in show_versions() (#19193)
• Check Python version when deserializing UDFs (#19175)
• Raise an error when users try to use Polars API in a fork()-without-execve() child (#19149)
• Quantile function in SQL (#18047)
• Improve scalar strict message (#19117)
• Add Series::{first, last, approx_n_unique} (#19093)
• Allow for rolling_*_by to use index count as window (#19071)
• Delay deserialization of python function until physical plan (#19069)
• Add cum(_min/_max) for pl.Boolean (#19061)

🐞 Bug fixes

• Don't produce duplicate column names in Series.to_dummies (#19326)
• Use of HAVING outside of GROUP BY should raise a suitable SQLSyntaxError (#19320)
• More accurate from_dicts typing/signature (#19322)
• Fix empty array gather (#19316)
• Merge categorical rev-map in unpivot (#19313)
• DataFrame descending sorting by single list element (#19233)
• Fix cse union schema (#19305)
• Correctly load Parquet statistics for f16 (#19296)
• Error on invalid query (#19303)
• Fix enum scalar output (#19301)
• Fix list gather invalid fast path (#19299)
• Fix quoting style of decimal csv output (#19298)
• Don't vertically parallelize literal select (#19295)
• Fix struct reshape fast path (#19294)
• Also split on forward slashes during hive path inference on Windows (#19282)
• Don't cse as_struct (#19280)
• Only apply string parsing to String dtype (#19222)
• Make the SQLAlchemy connection check more robust (#19270)
• Ensure that read_database takes advantage of Arrow return from a duckdb_engine connection when using a SQLAlchemy Selectable (#19255)
• Compilation error missing use JsonLineReader (#19244)
• Don't remember Parquet statistics if filtered (#19248)
• Do not check dtypes of non-projected columns for parquet (#19254)
• Parquet predicate pushdown for lit(_) != (#19246)
• Use all chunks in Series from arrow struct (#19218)
• Don't trigger row limit in array construction (#19215)
• Fix struct literals (#19214)
• Plotting was not interacting well with Altair schema wrappers (#19213)
• Fixing infer_schema for DataType::Null (#19201)
• Migrate to PyO3 0.22 and released verion of rust-numpy crate (#19199)
• Add 'drop_empty_rows' parameter for read_excel (#18253)
• Don't unwrap() expansion (#19196)
• Properly handle non-nullable nested Parquet (#19192)
• Fix invalid list collection in expression engine (#19191)
• Fix use of "hidden_columns" parameter in write_excel (#19029)
• Implement to_arrow functionality properly for Arrays (#19077)
• Remove incorrect warning when using an IO[bytes] instance (#19154)
• Don't fail test if e.g. jax has been used first, since jax installs a fork handler that warns (#19178)
• Fix incorrect (eq|ne)_missing on List/Array types (#19155)
• Properly broadcast Struct when then validity (#19148)
• Allow partial name overlap in join_where resolution (#19128)
• Fix floordiv / modulo with scalar 0 on LHS (#19143)
• Ensure aligned chunks in OOC sort (#19118)
• Recursively align when converting to ArrowArray (#19097)
• Raise on invalid shape of shape 1, empty combination (#19113)
• Use two-pass algorithm for csv to ensure correctness and SIMDize more ~17% (#19088)
• Allow converting DatetimeOwned to ChunkedArray (#19094)
• Throw proper error for empty char params in scan_csv (#19100)
• Ensure parquet schema arg is propagated to IR (#19084)
• Only rewrite numeric ineq joins (#19083)
• Check validity of columns of keys/aggs in dsl->ir (#19082)
• Bitwise aggregations should ignore null values (#19067)
• Remove failing datetime subclass test (#19068)
• Don't ignore multiple columns in LazyFrame.unnest (#19035)

📖 Documentation

• Remove ecosystem viz section since there is one in misc already (#18408)
• Fix typo in custom expressions docs (#19292)
• Add SQL docs for new QUANTILE_CONT and QUANTILE_DISC functions (#19272)
• Add marimo to ecosystem.md (#19250)
• Improve DataFrame.write_database docstring (#19189)
• Link to main website from banner (#19177)
• Fix example of as_struct (#19116)
• Clarify difference between bitwise/logical ops (#19180)
• Add non-equi joins to, and revise, joins docs page (#19127)
• Add Series.first,last,approx_n_unique to docs (#19146)
• Annotate Config kwarg options (#18988)
• Revise and improve 'Concepts' section (#19087)

🛠️ Other improvements

• Add/fix unordered row decode, change unordered format (#19284)
• Move from parquet-format-safe to polars-parquet-format (#19275)
• Skip flaky test (#19242)
• Add more tests for list arithmetic (#19225)
• Remove unused IPC async (#19223)
• Make get_list_builder infallible (#19217)
• Migrate to PyO3 0.22 and released verion of rust-numpy crate (#19199)
• Make expression output type known (#19195)
• Revert "feat(python): Raise an error when users try to use Polars API in a fork()-without-execve() child (#19149) (#19188)
• Zero-Field Structs and DataFrame with Height Property (#19123)
• Make pl.repeat part of the IR (#19152)
• Expose IEJoin IR node to python (#19104)
• Clean remove_prefix since python3.9 is now the minimum Python (#19070)
• Add new streaming engine to CI (#19051)

Thank you to all our contributors for making this release possible!
@Bidek56, @MarcoGorelli, @Rashik-raj, @adamreeve, @alexander-beedie, @alonme, @balbok0, @coastalwhite, @deanm0000, @dependabot, @dependabot[bot], @eitsupi, @etrotta, @itamarst, @jbutterwick, @joelostblom, @kenkoooo, @khalidmammadov, @laurentS, @mcrumiller, @mscolnick, @nameexhaustion, @orlp, @pomo-mondreganto, @ritchie46, @rodrigogiraoserrao, @siddharth-vi, @stinodego, @sunadase and @wence-

Python Polars 1.9.0

🚀 Performance improvements

• Use List's TotalEqKernel (#18984)

✨ Enhancements

• Bitwise operations / aggregations (#18994)
• Allow insert_column to take expressions (#19024)
• Improved error message DSL -> IR resolving (#19032)
• Add strict param to eager/lazy frame "rename" (#19017)
• Support schema arg in read/scan_parquet() (#19013)
• Add include_file_paths parameter to read_parquet (#19008)
• Add allow_missing_columns option to read/scan_parquet (#18922)
• Drop python 3.8 support (#18965)
• Use FFI to extract Series from different Polars binaries (#18964)
• Allow for zero-width fixed size lists (#18940)

🐞 Bug fixes

• Remove failing temporal lit tests (#19056)
• Divide-by-zero in OOC sort (#19048)
• Ensure must_flush flag is not reset (#19046)
• Error node should be on top (#19045)
• Force nested struct missing equality (#19031)
• Fix invalid alias udf (#19021)
• Raise invalid predicate join_where (#19020)
• Fix nested flag of functions with multiple arguments (#19016)
• Fix projection pushdown bug in IEJOINS (#19015)
• Separate temporal tests (#19012)
• Return the truth values of ne_missing and eq_missing operations for struct instead of null (#18930)
• Fix list to numpy conversion (#19009)
• Fix struct broadcasting comparisons (#19003)
• Wrong result on when().then().otherwise() on struct when both result are broadcast (#19000)
• Improve literals for temporal subclasses (#18998)
• Ensure same fmt in Series/AnyValue to string cast (#18982)
• Return correct value for when().then().else() on structs when using first()\last() (#18969)
• IPC don't write variadic_buffer_counts in blocks, but only dictionaries (#18980)
• Respect allow_threading in TernaryExpr (#18977)
• Make join test order-agnostic (#18975)
• Fix lit().shrink_dtype() broadcasting (#18958)
• Parallel evaluation of cumulative_eval (#18959)
• Properly implement AnyValue::Binary into_py (#18960)
• Fix Expr.over with order_by did not take effect if group keys were sorted (#18947)
• Properly fetch type of full None List Series (#18916)
• Incorrect mode for sorted input (#18945)
• Properly choose inner physical type for Array (#18942)
• Disable very old date in timezone test for CI (#18935)
• Infer reshape dims when determining schema (#18923)
• Incorrect broadcasting on list-of-string set ops (#18918)
• Adding with_row_index() to previously collected lazy scan does not take effect (#18913)

📖 Documentation

• Fix example of lazy schema verification (#19059)
• Rewrite 'Getting started' page (#19028)
• Fix is_not_nan description (#18985)
• Recommend targetDir for rust-analyzer (#18973)
• Fix LazyFrame fetch method references (#18033)

📦 Build system

• Bump Rust toolchain to nightly-2024-09-29 (#19006)
• Bump simd-json to 0.14 (#18999)

🛠️ Other improvements

• Remove built info (#19057)
• Mark schema arg in read/scan_parquet as unstable (#19018)
• Fix new-streaming test_lazy_parquet::test_row_index (#19019)
• Preserve scalar in more places (#18898)
• Mention allow_missing_columns in error message when column not found (parquet) (#18972)
• Disable CSE-specific test on new streaming engine (#18971)
• Add FixedSizeList equality broadcasting (#18967)
• Divide ChunkCompare into Eq and Ineq variants (#18963)
• Another set of new-stream test skip/fixes (#18952)
• Fix/skip variety of new-streaming tests, cont (#18928)
• Fix/skip variety of new-streaming tests (#18924)

Thank you to all our contributors for making this release possible!
@LukasFolwarczny, @Plutone11011, @aleexharris, @alexander-beedie, @barak1412, @coastalwhite, @dependabot, @dependabot[bot], @edwinvehmaanpera, @kgv, @mcrumiller, @nameexhaustion, @orlp, @ritchie46, @rodrigogiraoserrao, @stinodego and @xhiroga

Python Polars 1.8.2

🚀 Performance improvements

• Improve rename performace for Lazy API (#18890)
• Collapse cross-joins to faster joins (#18633)

✨ Enhancements

• Improve scalar strict message (#18904)

🐞 Bug fixes

• Properly zip struct validities (#18886)
• Out-of-bounds gather in categorical->int cast (#18897)
• AnyValue Series from Categorical/Enum (#18893)
• Properly cast AnyValue string (#18888)
• Fix SO in json inference (#18887)
• Use proper thread pool in cumulative_eval (#18885)

📖 Documentation

• Fix broken user-guide API links (#18872)

Thank you to all our contributors for making this release possible!
@coastalwhite, @npielawski, @orlp, @ritchie46 and @siddharth-vi

Python Polars 1.8.1

🚀 Performance improvements

• Cache register plugin function (#18860)

🐞 Bug fixes

• Properly calculate duration units (#18869)
• Check values in strict cast Int to Time (#18854)
• Fix typo in DuplicateError error message (#18855)
• Properly merge live- and dead columns in prefiltered (#18862)

📖 Documentation

• Fix minor rogue apostrophes (#18865)

🛠️ Other improvements

• Make with_column_unchecked take Column (#18863)
• Keep scalar in more places (#18775)

Thank you to all our contributors for making this release possible!
@MarcoGorelli, @coastalwhite, @mcrumiller, @ritchie46 and @rodrigogiraoserrao

Python Polars 1.8.0

✨ Enhancements

• Support arithmetic between Series with dtype list (#17823)
• Relaxed schema alignment for parquet file list read (#18803)
• Always preserve sorted flag for .dt.date (#18692)
• Enable additional ruff lint rule sets (#18721)
• Implement single inequality joins for join_where (#18727)

🐞 Bug fixes

• DataFrame plot was raising when some extra keywords were passed to encodings (e.g. x=alt.X(a, axis=alt.Axis(labelAngle=30))) (#18836)
• Respect strictness in list constructor (#18853)
• Properly broadcast array arithmetic (#18851)
• Throw error for comparison of unequal length series (#18816)
• Raise when parquet file has extra columns and no select() was done (#18843)
• Pass missing user params in write_csv (#18845)
• Improve join argument checks (#18847)
• Struct filter by index (#18778)
• Proper dtype casting for struct embedded categoricals in chunked categoricals (#18815)
• Fixed some error/assertion types (#18811)
• Remove panic in arr.to_struct (#18804)
• Allow empty sort by columns (#18774)
• Broadcast zip_with for structs (#18770)
• Dropped/shifted rows in parquet scan with streaming=True (#18766)
• Fix cum_max using exception text of cum_min for invalid dtype (#18780)
• Fix accidental raise on shape 1 (#18748)

📖 Documentation

• Fix link to issue tracker and code snippet format in GPU docs (#18850)
• Clarify documentation for schema in read_csv function (#18759)
• Fix literal type mapping example in lit docstrings (#18756)
• Refactor docs directory hierarchy (#18773)
• Minor improvements to contributing guide (#18777)
• Improve over docs, add example with order_by (#18796)
• Add documentation for beta gpu support (#18762)

🛠️ Other improvements

• Re-export PyO3 in polars-python crate (#18835)
• Make NodeTraverser struct public (#18822)
• Add panic to unchecked DataFrame constructors in debug mode (#18807)
• Fix parquet file metadata is dropped after first DSL->IR conversion (#18789)
• Remove extra hashmap construction in new-streaming parquet (#18792)
• Remove TODO comment regarding NumPy pinning (#18776)
• Remove unused methods (#18744)
• Make DataFrame a Vec of Column instead of Series (#18664)
• Run benchmark on PR labeled 'needs-bench' (#18737)
• Enable additional ruff lint rule sets (#18721)

Thank you to all our contributors for making this release possible!
@3ok, @Manishearth, @MarcoGorelli, @adamreeve, @alexander-beedie, @barak1412, @beckernick, @bradfordlynch, @coastalwhite, @deanm0000, @eitsupi, @i64, @itamarst, @mcrumiller, @nameexhaustion, @orlp, @r-brink, @ritchie46, @rodrigogiraoserrao, @squnit, @stinodego and @t-ded

Rust Polars 0.43.1

🐞 Bug fixes

• Revert automatically turning on Parquet prefiltered (#18720)
• Parquet prefiltered with projection pushdown (#18714)
• Correctly display multilevel nested Arrays (#18687)
• Fix scalar literals (#18707)
• Missing activation for serde for PlSmallStr from some crates (#18702)
• Add missing PhantomDatas to BackingStorage (#18699)
• Fix use of undeclared crate or module error (#18701)
• Refactor decompression checks and add support for decompressing JSON (#18536)
• Qcut all nulls panics (#18667)

🛠️ Other improvements

• Remove IR info from DSL (#18712)
• Refactor code into functions in new parquet source (#18711)
• Remove unused feature flags from polars-mem-engine (#18679)
• Remove hive_parts from DSL source (#18694)

Thank you to all our contributors for making this release possible!
@alexander-beedie, @ankane, @attila-lin, @coastalwhite, @eitsupi, @nameexhaustion, @ohanf, @orlp, @ritchie46 and @yarimiz

Python Polars 1.7.1

🐞 Bug fixes

• Revert automatically turning on Parquet prefiltered (#18720)
• Parquet prefiltered with projection pushdown (#18714)
• Fix scalar literals (#18707)

🛠️ Other improvements

• Remove IR info from DSL (#18712)
• Remove unused feature flags from polars-mem-engine (#18679)
• Remove hive_parts from DSL source (#18694)

Thank you to all our contributors for making this release possible!
@ankane, @attila-lin, @coastalwhite, @eitsupi, @nameexhaustion, @orlp and @ritchie46

Rust Polars 0.43.0

🏆 Highlights

• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)

🚀 Performance improvements

• Back arrow arrays with SharedStorage which can have non-refcounted static slices (#18666)
• Don't traverse file list twice for extension validation (#18620)
• Remove cloning of ColumnChunkMetadata (#18615)
• Add upfront partitioning in ColumnChunkMetadata (#18584)
• Enable Parquet parallel=prefiltered for auto (#18514)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Added optimizer rules for is_null().all() and similar expressions to use null_count() (#18359)
• Parquet do not copy uncompressed pages (#18441)
• Several large parquet optimizations (#18437)
• Batch Plain Parquet UTF-8 verification (#18397)
• Partition metadata for parquet statistic loading (#18343)
• Fix accidental quadratic parquet metadata (#18327)
• Lazy decompress Parquet pages (#18326)
• Don't rechunk aligned chunks in owned_binary_chunk_align (#18314)
• Batch DELTA_LENGTH_BYTE_ARRAY decoding (#18299)
• Slice pushdown for SimpleProjection (#18296)
• Use direct path for time/timedelta literals (#18223)
• Speedup ndjson reader ~40% (#18197)

✨ Enhancements

• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)
• Make expressions containing Python UDFs serializable (#18135)
• Support Serde for IRPlan (#18433)
• Respect input time zone if input is pandas Timestamp (#18346)
• Add POLARS_BACKTRACE_IN_ERR for debugging (#18333)
• IR serde (#18298)
• Improve decimal_comma error message (#18269)
• Support pre-signed URLs for cloud scan (#18274)
• Support empty structs (#18249)
• Allow float in interpolate_by by column (#18015)

🐞 Bug fixes

• Scalar checks (#18627)
• Scanning hive partitioned files where hive columns are partially included in the file (#18626)
• Enable "polars-json/timezones" feature from "polars-io" (#18635)
• Use Buffer<T> in ObjectSeries, fixes variety of offset bugs (#18637)
• Properly slice validity mask on pl.Object series (#18631)
• Indicative error in list.gather when wrong indices type is supplied (#18611)
• Fix group first value after group-by slice (#18603)
• Functions for streaming require streaming feature (#18602)
• Allow for date/datetime subclasses (e.g. pd.Timestamp, FreezeGun) in pl.lit (#18497)
• Fix UnitVec inline clone and with_capacity (#18586)
• Ensure result name of pow matches schema in grouped context (#18533)
• Decimal mean agg dtype was incorrect in IR (#18577)
• Fix output type for list.eval in certain cases (#18570)
• Fix map_elements for List return dtypes (#18567)
• Do not remove double-sort if maintain_order=True (#18561)
• Empty any_horizontal should be false, not true (#18545)
• Fix type inference error in map_elements for List types (#18542)
• Added proper handling of file.write for large remote csv files (#18424)
• Handle Parquet projection pushdown with only row index (#18520)
• Properly raise on invalid selector expressions (#18511)
• Wrong output column name in or and xor operations (#18512)
• Various schema corrections (#18474)
• Don't drop objects on empty buffers (#18469)
• Add missing chunk align in pipe sink (#18457)
• Expr.sign should preserve dtype (#18446)
• Enable CSE in eager if struct are expanded (#18426)
• Treat explode as gather (#18431)
• Fencepost error in debug assertion in splitfields (#18423)
• Unsoundness in CSV SplitFields (#18413)
• Parquet nested values that span several pages (#18407)
• Support reading empty parquet files (#18392)
• Recurse on map field during type conversion (#15075)
• Allow search_sorted on boolean series (#18387)
• Mark Expr.(lower|upper)_bound as returning scalar (#18383)
• Fix broken feature gate for ParquetReader (#18376)
• Fix compressed ndjson row count (#18371)
• Use correct column names when there are no value columns in unpivot (#18340)
• Parquet several smaller issues (#18325)
• Fix group-by slice on all keys (#18324)
• Compute joint null mask before calling rolling corr/cov stats (#18246)
• Several scan_parquet(parallel='prefiltered') problems (#18278)
• Json feature flag missing imports (#18305)
• Check groups in group-by filter (#18300)
• Make json readers ignore BOM character (#18240)
• Parquet delta encoding for 0-bitwidth miniblocks (#18289)
• Arguments for upsample only have to be sorted within groups (#18264)
• Use appropriate bins in hist when bin_count specified (#16942)
• Raise suitable error on unsupported SQL set op syntax (#18205)
• Fix invalid state due to cached IR (#18262)
• Fix failed AWS credential load from '~/.aws/credentials' due to formatting (#18259)
• Fix panic streaming parquet scan from cloud with slice (#18202)
• Consistently round half-way points down in dt.round (#18245)
• Fix duplicate column output and panic for include_file_paths (#18255)
• Fix unit null rank (#18252)
• Use physical for row-encoding (#18251)

📖 Documentation

• Fix multiprocessing docs regarding fork method check (#18563)
• Pre-compute plugin_path before defining plugin (#18503)
• Fix BinViewChunkedBuilder arguments (#17277) (#18439)
• Add date_range and datetime_ranges examples without eager=True (#18379)
• Document POLARS_BACKTRACE_IN_ERR env var (#18354)
• Document DataFrame.__getitem__ and Series.__getitem__ (#18309)
• Improve decimal_comma error message (#18269)
• Clarify coalesce behaviour in join_asof (#18273)
• Add note to Expr.shuffle differentiating from df method (#18266)

📦 Build system

• Remove extension-module from polars-python (#18554)
• Bump Rust toolchain to nightly-2024-08-26 (#18370)

🛠️ Other improvements

• Push down max row group height calc to file metadata (#18674)
• Re-use already decoded metadata for first path (new-parquet-source) (#18656)
• Remove duplicate byte range calc from new parquet source (#18655)
• Fix a bunch of tests for new-streaming (#18659)
• Rename MemSlice::from_slice -> MemSlice::from_static (#18657)
• Don't raise on multiple same names in ie_join (#18658)
• Split parquet_source.rs in new-streaming (#18649)
• Check predicates in join_where (#18648)
• Feature gate iejoin (#18646)
• Scan from BytesIO in new-streaming parquet source (#18643)
• Rename MetaData -> Metadata (#18644)
• Change join_where semantics (#18640)
• Fix unimplemented panics to give todo!s for AUTO_NEW_STREAMING (#18628)
• Remove extra schema traits (#18616)
• One simplify expression module and keep utility local (#18621)
• Check number of binary comparisons in join_where predicates (#18608)
• Raise on suffixed predicate in join_where (#18607)
• Fix Python docs build (#18605)
• Fix nan-ignoring max/min in new-streaming (#18593)
• Correctly support more types in new-streaming sum (#18580)
• Bump NodeTraverser major version (#18576)
• Fix mean reduction in new-streaming (#18572)
• Rename data_type -> dtype (#18566)
• Refactor ArrowSchema to use polars_schema::Schema<D> (#18564)
• Remove NotifyReceiver from new-streaming parquet source (#18540)
• Refactor Schema to use generic struct from new polars-schema crate (#18539)
• Temporarily pin NumPy in CI to address dependency resolving issue (#18544)
• Fix and extend AnyValue comparison (#18534)
• Remove top-level metadata from ArrowSchema (#18527)
• Add FromIterator impls for PlSmallStr (#18509)
• Update PlSmallStr comment (#18518)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Make expressions containing Python UDFs serializable (#18135)
• Allow polars to pass cargo check on windows (#18498)
• Remove From<&&str> for PlSmallStr (#18507)
• Change naming to new benchmark setup (#18473)
• More refactor for PlSmallStr (#18456)
• Split Reduction into it plus ReductionState (#18460)
• Remove a string allocation in Parquet (#18466)
• Unify internal string type (#18425)
• Remove network call in hf docs (#18454)
• Remove old streaming flag if we're going into new streaming (#18438)
• Address spurious hypothesis test failure (#18434)
• Add pl.length() reduction and small new-streaming fixes (#18429)
• Fencepost error in debug assertion in splitfields (#18423)
• Group arguments in conversion in a Context (#18418)
• Turn all Binary/Utf8 into BinaryView/Utf8View in Parquet (#18331)
• Recursively evaluate is_elementwise for function expressions (#18385)
• Various small fixes for the new streaming engine (#18384)
• Temporarily add ability to disable parquet source node (#18378)
• Improve dot formatting of new-streaming parquet source (#18367)
• Fix the required version of rust in README.md (#18357)
• Only instantiate used portion of graph (#18337)
• Fix new_streaming parameter (#18342)
• Add parquet source node to new streaming engine (#18152)
• Disable common sub-expr elim for new streaming engine (#18330)
• Remove unused Parquet indexes (#18329)
• Lower arbitrary expressions in the new streaming engine (#18315)
• Expose many more function expressions to python IR (#18317)
• Add Graphviz physical plan visualization for new streaming engine (#18307)
• Add DataFrame::new_with_broadcast and simplify column uniqueness checks (#18285)
• Add output_schema to all PhysNodes (#18272)
• Change fn schema to fn collect_schema (#18261)
• Add multiplexer node to new streaming engine (#18241)
• Add feature gates for polars-python crate (#18232)
• Split py-polars crate (#18204)
• Update the required version of rust in README.md (#18203)
• Add itertools in utils (#18213)
• Use or_else for raising (#18206)
• Remove unused Parquet source files (#18193)

Thank you to all our contributors for making this release possible...

Python Polars 1.7.0

🏆 Highlights

• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)

🚀 Performance improvements

• Back arrow arrays with SharedStorage which can have non-refcounted static slices (#18666)
• Don't traverse file list twice for extension validation (#18620)
• Remove cloning of ColumnChunkMetadata (#18615)
• Add upfront partitioning in ColumnChunkMetadata (#18584)
• Enable Parquet parallel=prefiltered for auto (#18514)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Added optimizer rules for is_null().all() and similar expressions to use null_count() (#18359)

✨ Enhancements

• Update BytecodeParser for upcoming Python 3.13 (#18677)
• Add tooltip by default to charts (#18625)
• Add support for IO[bytes] and bytes in scan_{...} functions (#18532)
• Support shortcut eval of common boolean filters in SQL interface "WHERE" clause (#18571)
• Add IEJoin algorithm for non-equi joins and support Full non-equi joins (#18365)
• Make expressions containing Python UDFs serializable (#18135)

🐞 Bug fixes

• Use IO[bytes] instead of BytesIO in DataFrame.write_parquet() (#18652)
• Scalar checks (#18627)
• Scanning hive partitioned files where hive columns are partially included in the file (#18626)
• Enable "polars-json/timezones" feature from "polars-io" (#18635)
• Use Buffer<T> in ObjectSeries, fixes variety of offset bugs (#18637)
• Properly slice validity mask on pl.Object series (#18631)
• Raise if single argument form in replace/replace_strict is not a mapping (#18492)
• Fix group first value after group-by slice (#18603)
• Allow for date/datetime subclasses (e.g. pd.Timestamp, FreezeGun) in pl.lit (#18497)
• Fix output type for list.eval in certain cases (#18570)
• Fix map_elements for List return dtypes (#18567)
• Check for duplicate column names in read_database cursor result, raising DuplicateError if found (#18548)
• Do not remove double-sort if maintain_order=True (#18561)
• Empty any_horizontal should be false, not true (#18545)
• Fix type inference error in map_elements for List types (#18542)
• Address incorrect align_frames result when the alignment column contains NULL values (#18521)
• Fix advertised version in source builds (#18523)
• Handle Parquet projection pushdown with only row index (#18520)
• DataFrame write_database not passing down "engine_options" when using ADBC (#18451)
• Properly raise on invalid selector expressions (#18511)
• Wrong output column name in or and xor operations (#18512)
• Normalize by default in Series.entropy like Expr.entropy does (#18493)
• Various schema corrections (#18474)
• Don't drop objects on empty buffers (#18469)
• Expr.sign should preserve dtype (#18446)
• Ensure assert_frame_not_equal and assert_series_not_equal raise on mismatched input types (#18402)
• Fixed Worksheet definition in write_excel type annotations (#18452)

📖 Documentation

• Update join_where docs to clarify behaviour (#18670)
• Fix multiprocessing docs regarding fork method check (#18563)
• Various docstring improvements to testing.assert_* functions (#18494)
• Fix formula in ewm_mean_by (#18506)
• Pre-compute plugin_path before defining plugin (#18503)
• Add Expr.null_count to aggregations (#18459)

🛠️ Other improvements

• Fix a bunch of tests for new-streaming (#18659)
• Don't raise on multiple same names in ie_join (#18658)
• Check predicates in join_where (#18648)
• Change join_where semantics (#18640)
• Add benchmark tests for join_where with inequalities (#18614)
• Check number of binary comparisons in join_where predicates (#18608)
• Raise on suffixed predicate in join_where (#18607)
• Fix Python docs build (#18605)
• Use streaming argument in test_parquet_slice_pushdown_non_zero_offset (#18529)
• Fix delta test merge (#18601)
• Alter/skip some tests for new streaming (#18574)
• Add lower-bound pin for numba (#18555)
• Temporarily pin NumPy in CI to address dependency resolving issue (#18544)
• Change PlSmallStr impl from Arc<str> to compact_str (#18508)
• Make expressions containing Python UDFs serializable (#18135)
• Change naming to new benchmark setup (#18473)
• Ensure physical arguments to np ufuncs are rechunked (#18471)
• Remove a string allocation in Parquet (#18466)
• Remove network call in hf docs (#18454)

Thank you to all our contributors for making this release possible!
@0xbe7a, @MarcoGorelli, @WbaN314, @adamreeve, @alexander-beedie, @alonme, @barak1412, @coastalwhite, @dependabot, @dependabot[bot], @eitsupi, @henryharbeck, @ion-elgreco, @krasnobaev, @megaserg, @nameexhaustion, @ohanf, @orlp, @philss, @r-brink, @ritchie46, @skellys, @squnit, @stinodego, @wence- and @yarimiz

Python Polars 1.6.0

💥 Unstable Breaking changes

These API's were marked unstable and are allowed to change.

• Use Altair in DataFrame.plot (#17995)

🚀 Performance improvements

• Parquet do not copy uncompressed pages (#18441)
• Several large parquet optimizations (#18437)
• Batch Plain Parquet UTF-8 verification (#18397)
• Partition metadata for parquet statistic loading (#18343)
• Fix accidental quadratic parquet metadata (#18327)
• Lazy decompress Parquet pages (#18326)
• Don't rechunk aligned chunks in owned_binary_chunk_align (#18314)
• Batch DELTA_LENGTH_BYTE_ARRAY decoding (#18299)
• Slice pushdown for SimpleProjection (#18296)
• Use direct path for time/timedelta literals (#18223)
• Speedup ndjson reader ~40% (#18197)
• Skip parquet page when unneeded (#18192)

✨ Enhancements

• Use Altair in DataFrame.plot (#17995)
• Allow mapping as syntactic sugar in str.replace_many (#18214)
• Respect input time zone if input is pandas Timestamp (#18346)
• Improve Schema and DataType interop with Python types (#18308)
• Add POLARS_BACKTRACE_IN_ERR for debugging (#18333)
• IR serde (#18298)
• Improve decimal_comma error message (#18269)
• Support pre-signed URLs for cloud scan (#18274)
• Support the most recent version of "duckdb_engine" connections via read_database (#18277)
• Support empty structs (#18249)
• Allow float in interpolate_by by column (#18015)
• Make show_versions more responsive (#18208)

🐞 Bug fixes

• Enable CSE in eager if struct are expanded (#18426)
• Treat explode as gather (#18431)
• Parquet nested values that span several pages (#18407)
• Support reading empty parquet files (#18392)
• Recurse on map field during type conversion (#15075)
• Allow search_sorted on boolean series (#18387)
• Mark Expr.(lower|upper)_bound as returning scalar (#18383)
• Fix compressed ndjson row count (#18371)
• Use correct column names when there are no value columns in unpivot (#18340)
• Parquet several smaller issues (#18325)
• Fix group-by slice on all keys (#18324)
• Compute joint null mask before calling rolling corr/cov stats (#18246)
• Several scan_parquet(parallel='prefiltered') problems (#18278)
• Json feature flag missing imports (#18305)
• Check groups in group-by filter (#18300)
• Parquet delta encoding for 0-bitwidth miniblocks (#18289)
• Arguments for upsample only have to be sorted within groups (#18264)
• Use appropriate bins in hist when bin_count specified (#16942)
• Raise suitable error on unsupported SQL set op syntax (#18205)
• Fix invalid state due to cached IR (#18262)
• Fix failed AWS credential load from '~/.aws/credentials' due to formatting (#18259)
• Fix panic streaming parquet scan from cloud with slice (#18202)
• Consistently round half-way points down in dt.round (#18245)
• Fix duplicate column output and panic for include_file_paths (#18255)
• Fix unit null rank (#18252)
• Use physical for row-encoding (#18251)
• Convert date and datetime in literal construction (#16018)
• Fix gather str as lit (#18207)

📖 Documentation

• Add date_range and datetime_ranges examples without eager=True (#18379)
• Fix incorrect comments in group_by_dynamic (#18415)
• Alphabetise methods in Python API reference (#18380)
• Document POLARS_BACKTRACE_IN_ERR env var (#18354)
• Add missing aggregation entries (#18334) (#18341)
• Add missing Series methods to API reference (#18312)
• Document DataFrame.__getitem__ and Series.__getitem__ (#18309)
• Fix typos and add see also links to struct name expressions (#18282)
• Improve decimal_comma error message (#18269)
• Clarify coalesce behaviour in join_asof (#18273)
• Add note to Expr.shuffle differentiating from df method (#18266)
• Improve formatting and consistency of various docstrings (#18237)
• Add missing "Parameters" section to bin.size expr docstring (#18222)
• Fix column name output in example of DataFrame.map_rows (#18227)

📦 Build system

• Bump Rust toolchain to nightly-2024-08-26 (#18370)

🛠️ Other improvements

• Address spurious hypothesis test failure (#18434)
• Turn all Binary/Utf8 into BinaryView/Utf8View in Parquet (#18331)
• Fix the required version of rust in README.md (#18357)
• Remove unused Parquet indexes (#18329)
• Deprecate serialize json for LazyFrame (#18283)
• Don't add sink node to cloud query (#18280)
• Split py-polars crate (#18204)
• Fix test for new deltalake release (#18211)
• Update the required version of rust in README.md (#18203)
• Fix version bifurcation for test_read_database_cx_credentials (#18220)
• Use or_else for raising (#18206)
• Remove unused Parquet source files (#18193)

Thank you to all our contributors for making this release possible!
@BartSchuurmans, @ChayimFriedman2, @MarcoGorelli, @StepfenShawn, @agossard, @alexander-beedie, @cgbur, @coastalwhite, @corwinjoy, @deanm0000, @henryharbeck, @ion-elgreco, @jqnatividad, @krasnobaev, @liufeimath, @markxwang, @mcrumiller, @nameexhaustion, @orlp, @ritchie46, @stinodego, @sunadase, @thomascamminady and @wence-