feat(python): Optimise read_excel when using "calamine" engine with the latest fastexcel

[lukapeschke]

Since worksheets are directly converted to polars, fastexcel's eager loading functions can be used. This makes fastexcel use borrowed types under the hood, which allows it to run faster and have a slightly lower memory footprint. See ToucanToco/fastexcel#147 (comment)

[lukapeschke]

I'm not really sure how to tackle the coverage issue here... Is there an easy way to run the CI with different versions of a dependency ?

[lukapeschke]

depends on ToucanToco/fastexcel#260

[alexander-beedie]

I'm not really sure how to tackle the coverage issue here... Is there an easy way to run the CI with different versions of a dependency ?

Unfortunately not... looks like this change can't be merged here until ToucanToco/fastexcel#260 is merged (and released) on your side though - unless you want to update this PR's fastexcel version check to additionally detect that the currently-failing parameter is being used, and fall-back if so? Otherwise we can park this in draft until it's good to go.

[lukapeschke]

@alexander-beedie I've updated the PR to check for fastexcel >= 0.11.2. However, I'm not sure on how to proceed to have coverage on both branches. Do you have a suggestion/preference ?

[alexander-beedie]

@alexander-beedie I've updated the PR to check for fastexcel >= 0.11.2. However, I'm not sure on how to proceed to have coverage on both branches. Do you have a suggestion/preference ?

Only real option is that you locally test the earlier version 😅

Also, looks like we have to wait for the 0.11.4 release for tests to be able to complete:

Run uv pip install --compile-bytecode -r requirements-dev.txt -r requirements-ci.txt
  × No solution found when resolving dependencies:
  ╰─▶ Because only the following versions of fastexcel are available:
          fastexcel<=0.11.2
          fastexcel==0.11.3
      and fastexcel==0.11.2 has no wheels are available with a matching
      platform, we can conclude that fastexcel>=0.11.2,<0.11.3 cannot be used.
      And because fastexcel==0.11.3 was yanked (reason: doesn't work on
      macos 14) and you require fastexcel>=0.11.2, we can conclude that the
      requirements are unsatisfiable.

[lukapeschke]

Yup, I've marked the PR as draft until ToucanToco/fastexcel#270 is merged. These macos runner updates are a real pain 😕

All right, I'll run the tests locally with both versions then. I'll let you know when we're all good, thanks!

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 80.49%. Comparing base (9eec688) to head (5503c17).
Report is 1 commits behind head on main.

Files	Patch %	Lines
py-polars/polars/io/spreadsheet/functions.py	50.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #17735   +/-   ##
=======================================
  Coverage   80.49%   80.49%           
=======================================
  Files        1503     1503           
  Lines      197027   197031    +4     
  Branches     2795     2796    +1     
=======================================
+ Hits       158589   158598    +9     
+ Misses      37918    37912    -6     
- Partials      520      521    +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lukapeschke]

@alexander-beedie macos seem to be finally working! I've ran the tests in tests/unit/io/test_spreadsheet.py with fastexcel 0.11.5 and fastexcel 0.10.4, and they passed with both versions 🙂 I couldn't find any other mention of calamine in the tests, but there are other test files I should run, please let me know!

Regarding the perf and memory gain, I've ran the following script on this branch with both fastexcel versions:

#!/usr/bin/env python3

import polars as pl
from sys import argv

pl.read_excel(argv[1], engine="calamine")

I got the following results with fastexcel 0.10.4:

And the following ones with 0.11.5:

[alexander-beedie]

Good stuff! 👍