Add support for Python 3.12

[abey79]

What

• Bumps arrow to 14.0.2
• Fixes Support Python 3.12 #3699
• Relates to CI: systematic testing of SDK and examples on all supported (major) Python version #2445

I added a bunch of fixes to scripts/run_all.py, which has a marked tendency to bit rot:

• Prevent scripts/run_all.py from bit-rotting #4710

Some workaround introduced to deal with PyTorch not 3.12 ready, see:

• Clean up the torch and torch vision python 3.12 gating when PyTorch is 3.12-compatible #4704

TODO:

•  nox -s tests-3.12 passing
• nox -s run_all-3.12 --save passing
• nox -s roundtrips-3.12 passing
• Solve this:

  ❯ pixi add pyarrow==14.0.0 
  × could not determine any available versions for pyarrow on linux-64. Either the package could not be found or version constraints on other dependencies result in a conflict.
  ╰─▶ Cannot solve the request because of: No candidates were found for pyarrow ==14.0.0.
  

• Test won't pass until torch is updated for 3.12: Pytorch for Python 3.12 not available pytorch/pytorch#110436

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested demo.rerun.io (if applicable)
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

• PR Build Summary
• Docs preview
• Examples preview
• Recent benchmark results
• Wasm size tracking

[genemerewether]

Hi, any plans to merge this or are you waiting for the pytorch test dep? thanks!

[abey79]

PyTorch is still not released for python 3.12 sadly, but apparently the nightly works. Maybe we could have test work by updating our requirements_*.txt with:

torch ; python_version < "3.12"
--pre --index-url https://download.pytorch.org/whl/nightly torch ; python_version >= "3.12"

EDIT: ☝🏻 that doesn't work (--pre and --index-url apply to the entire requirement.txt file). I found another way to get tests to pass.

[genemerewether]

PyTorch is still not released for python 3.12 sadly, but apparently the nightly works. Maybe we could have test work by updating our requirements_*.txt with:

torch ; python_version < "3.12"
--pre --index-url https://download.pytorch.org/whl/nightly torch ; python_version >= "3.12"

That would be great, if you're open to it. My env requires pyarrow >= 14 and I've been eagerly waiting to integrate rerun (having tested in a separate env)

[abey79]

I could get nox -s tests-3.12 to pass by gating torch in requirements_*.txt and explicitly installing its nightly in noxfile.py.