Add Python 3.14 support #757

forsyth2 · 2025-11-18T22:18:35Z

Summary

Important: this should be merged to main after the production release of zppy v3.1.0.

Objectives:

Add support for Python 3.14

Issue resolution:

Equivalent of Add Python 3.14 support zppy-interfaces#44, Add Python 3.14 support zstash#402

Select one: This pull request is...

a bug fix: increment the patch version
a small improvement: increment the minor version
a new feature: increment the minor version
an incompatible (non-backwards compatible) API change: increment the major version

Small Change

To merge, I will use "Squash and merge". That is, this change should be a single commit.
Logic: I have visually inspected the entire pull request myself.
Pre-commit checks: All the pre-commits checks have passed.

forsyth2 · 2025-11-18T22:19:24Z

Xylar had noted:

we would want python >=3.11,<3.15 by next spring and python >=3.12,<3.15 by next fall. That is, we add a python version each spring and drop one each fall.

So, for the Spring release (tentatively planned as: testing period beginning April 1, release on May 1), we'd want to support four versions of Python: 3.11, 3.12, 3.13, 3.14.

forsyth2 · 2025-11-18T22:21:18Z

Tom also noted:

numba/llvmlite don't [yet support Python 3.14]. Both are dependencies of packages xesmf, which is a dependency of xcdat. Related issue with xcdat: xCDAT/xcdat#813 (comment)

forsyth2 · 2025-11-18T22:34:20Z

Re xcdat: it looks like zppy itself doesn't use it.

> git grep -n xcdat
zppy/logger.py:1:"""Logger module for setting up a logger. Based on xcdat/xcdat/_logger.py"""

forsyth2 · 2025-11-18T22:34:45Z

All the GitHub Actions are passing, including the 3.14 test.

forsyth2 · 2025-12-02T02:36:08Z

Unlike zppy-interfaces & zstash, this PR isn't running into any problems with GitHub Actions when using Python 3.14.

Before merging:

Run integration test suite, to confirm no other issues arise.
Ideally also merge Add Python 3.14 support zppy-interfaces#44 (but I suppose that's not strictly necessary)

forsyth2 · 2025-12-04T01:06:29Z

Testing steps

cd ~/ez/zppy
git status
# On branch fix-mpas-analysis-cache
# nothing to commit, working tree clean
git checkout add-python3.14
git log
# Add Python 3.14 support
# Multithread image tests (#752)
# Fixes for rc4 (#755)
# Bump to 3.1.0rc3 (#754)

# Let's do some rebasing
git fetch upstream main
git rebase upstream/main
git log
# Add Python 3.14 support
# Add NCO version parameter (#761)
# Bump to 3.1.0 (#759)

# Edit tests/integration/utils.py

# This should actually be a permanent change:
# get_chyrsalis_expansions:
#        "path_pcmdi_diags_obs_ts": "/lcrc/group/e3sm/diagnostics/observations/Atm/time-series",

# TEST_SPECIFICS:
# TEST_SPECIFICS: Dict[str, Any] = {
#     # These are custom environment_commands for specific tasks.
#     # Never set these to "", because they will print the line
#     # `environment_commands = ""` for the corresponding task,
#     # thus overriding the value set higher up in the cfg.
#     # That is, there will be no environment set.
#     # (`environment_commands = ""` only redirects to Unified
#     # if specified under the [default] task)
#     "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
#     "global_time_series_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
#     "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
#     # This is the environment setup for other tasks.
#     # Leave as "" to use the latest Unified environment.
#     "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
#     "cfgs_to_run": [
#         "weekly_bundles",
#         "weekly_comprehensive_v2",
#         "weekly_comprehensive_v3",
#         "weekly_legacy_3.0.0_bundles",
#         "weekly_legacy_3.0.0_comprehensive_v2",
#         "weekly_legacy_3.0.0_comprehensive_v3",
#     ],
#     "tasks_to_run": [
#         "e3sm_diags",
#         "mpas_analysis",
#         "global_time_series",
#         "ilamb",
#         "pcmdi_diags",
#     ],
#     "unique_id": "test_zppy_py3.14",
# }

git diff
# Diff looks good

# Set up conda
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n test-py3.14
conda activate test-py3.14
pre-commit run --all-files
python -m pip install .
conda list | grep python
# python 3.14.0 h32b2ec7_102_cp314 conda-forge
# Good, python is at 3.14

pytest tests/test_*.py
# 35 passed in 0.44s
python tests/integration/utils.py
git status
# Good, only generated files changed
sq
# Good, no currently queued jobs.
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
# Count the number of jobs
sq | wc -l # Tue 12/02 13:16 => 144 - header row = 143 jobs
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq | wc -l # Tue 12/02 13:54 => 30 - header row = 29 jobs
sq | wc -l # Tue 12/02 14:55 => 1 - header row = 0 jobs

# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
git status
# On branch add-python3.14
# Good, correct branch

# Current conda env: test-py3.14
# Good, correct environment

zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Tue 12/02 14:56 => 5 - header row = 4 jobs

Hit disk space limit while testing E3SM-Project/zstash#407:

sq
# USER     ACCOUNT NODE PARTITION JOBID   ST     REASON       TIME TIME_LIMIT NAME
# ac.forsy    e3sm    1   compute 1029464 CG NonZeroExi       4:33    7:00:00 bundle3
# ac.forsy    e3sm    1   compute 1029461 CG NonZeroExi       5:27    7:00:00 ilamb_1985-1986
# ac.forsy    e3sm    1   compute 1029462 CG NonZeroExi       5:02    7:00:00 bundle3
# ac.forsy    e3sm    1   compute 1029463 CG NonZeroExi       5:24    7:00:00 ilamb_1985-1986
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# bundle3.status:ERROR
# e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.status:ERROR (11)
# ilamb_1985-1986.status:ERROR (3)
emacs bundle3.o1029462
# OSError: [Errno 28] No space left on device

lcrc-quota
# ----------------------------------------------------------------------------------------
# Home                          Current Usage   Space Avail    Quota Limit    Grace Time
# ----------------------------------------------------------------------------------------
# ac.forsyth2                        61 GB          38 GB         100 GB               
# ----------------------------------------------------------------------------------------
# Project                       Current Usage   Space Avail    Quota Limit    Grace Time
# ----------------------------------------------------------------------------------------
/usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2
# /usr/lpp/mmfs/bin/mmlsquota -u ac.forsyth2 --block-size T fs2
#                          Block Limits                                               |     File Limits
# Filesystem Fileset    type             TB      quota      limit   in_doubt    grace |    files   quota    limit in_doubt    grace  Remarks
# fs2        root       USR              84        300        300          1     none |  5986737       0        0      148     none lcrcstg.lcrc.anl.gov

# Neither of these indicate any problems with disk space...

Attempting to pick up testing without starting over:

cd ~/ez/zppy
git status
# On branch fix-mpas-analysis-cache

# Handle uncommitted changes
git add -A
git commit -m "Testing " --no-verify
git checkout add-python3.14
# On branch add-python3.14
# Good, correct branch

conda activate test-py3.14
# Current conda env: test-py3.14
# Good, correct environment

python -m pip install .

zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Wed 12/03 15:30 => 5 - header row = 4 jobs
sq | wc -l # Wed 12/03 16:33 => 1 - header row = 0 jobs

# Review finished runs
### v2  ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/test_zppy_py3.14/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/test_zppy_py3.14/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors

cd ~/ez/zppy
ls tests/integration/test_*.py
pytest tests/integration/test_bash_generation.py
# 1 failed in 2.12s -- that's because of changes in https://github.com/E3SM-Project/zppy/pull/761
pytest tests/integration/test_campaign.py
# 6 failed in 1.93s -- same reason
pytest tests/integration/test_defaults.py
# 1 failed in 0.37s -- same reason
pytest tests/integration/test_last_year.py
# 1 passed in 0.29s
pytest tests/integration/test_bundles.py
# 1 failed, 1 passed in 1.39s
# Looks like we might be missing some ts_rof_monthly code? Possibly because of the restart?
salloc --nodes=1 --partition=debug --time=00:30:00 --account=e3sm
lcrc_conda
conda activate test-py3.14
pytest tests/integration/test_images.py
# 1 passed in 711.75s (0:11:51)
cat test_images_summary.md

Test name	Total images	Correct images
bundles_e3sm_diags	1692	1692
bundles_global_time_series	3	3
bundles_ilamb	388	388
comprehensive_v2_e3sm_diags	3691	3691
comprehensive_v2_mpas_analysis	856	856
comprehensive_v2_global_time_series	12	12
comprehensive_v2_ilamb	776	776
comprehensive_v3_e3sm_diags	5210	5210
comprehensive_v3_mpas_analysis	856	856
comprehensive_v3_global_time_series	1404	1404
comprehensive_v3_ilamb	837	837
comprehensive_v3_pcmdi_diags	617	617
legacy_3.0.0_bundles_e3sm_diags	1692	1692
legacy_3.0.0_bundles_global_time_series	3	3
legacy_3.0.0_bundles_ilamb	388	388
legacy_3.0.0_comprehensive_v2_e3sm_diags	3691	3691
legacy_3.0.0_comprehensive_v2_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v2_global_time_series	12	12
legacy_3.0.0_comprehensive_v2_ilamb	776	776
legacy_3.0.0_comprehensive_v3_e3sm_diags	5210	5210
legacy_3.0.0_comprehensive_v3_mpas_analysis	856	856
legacy_3.0.0_comprehensive_v3_global_time_series	90	90
legacy_3.0.0_comprehensive_v3_ilamb	837	837

Bundles debugging

Full test output:

platform linux -- Python 3.14.0, pytest-9.0.1, pluggy-1.6.0
rootdir: /gpfs/fs1/home/ac.forsyth2/ez/zppy
configfile: pyproject.toml
plugins: cov-7.0.0
collected 2 items                                                                                                                           

tests/integration/test_bundles.py .F                                                                                                  [100%]

================================================================= FAILURES ==================================================================
______________________________________________________ test_bundles_bash_file_content _______________________________________________________

    def test_bundles_bash_file_content():
        expansions = get_expansions()
        user_output = expansions["user_output"]
        expected_dir = expansions["expected_dir"]
        unique_id = expansions["unique_id"]
        actual_directory = f"{user_output}zppy_weekly_bundles_output/{unique_id}/{V3_CASE_NAME}/post/scripts"
        expected_directory = f"{expected_dir}expected_bundles/bundle_files"
        # Check that bundle files are correct
        assert (
            os.system(
                f"diff -bu -I 'zppy_weekly_bundles_output/' {actual_directory}/bundle1.bash {expected_directory}/bundle1.bash"
            )
            == 0
        )
        assert (
            os.system(
                f"diff -bu -I 'zppy_weekly_bundles_output/' {actual_directory}/bundle2.bash {expected_directory}/bundle2.bash"
            )
            == 0
        )
>       assert (
            os.system(
                f"diff -bu -I 'zppy_weekly_bundles_output/' {actual_directory}/bundle3.bash {expected_directory}/bundle3.bash"
            )
            == 0
        )
E       assert 256 == 0
E        +  where 256 = <built-in function system>("diff -bu -I 'zppy_weekly_bundles_output/' /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.../post/scripts/bundle3.bash /lcrc/group/e3sm/public_html/zppy_test_resources/expected_bundles/bundle_files/bundle3.bash")
E        +    where <built-in function system> = os.system

tests/integration/test_bundles.py:84: AssertionError
----------------------------------------------------------- Captured stdout call ------------------------------------------------------------
--- /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts/bundle3.bash	2025-12-03 17:29:39.401933024 -0600
+++ /lcrc/group/e3sm/public_html/zppy_test_resources/expected_bundles/bundle_files/bundle3.bash	2025-11-13 18:01:58.484043149 -0600
@@ -32,6 +32,26 @@
 # Loop over all tasks
 error=false
 
+echo === ts_rof_monthly_1985-1986-0002.bash  ===
+task=ts_rof_monthly_1985-1986-0002.bash
+taskStatusFile="${task%.*}.status"
+if [ ! -f ${taskStatusFile} ] || [ "`cat ${taskStatusFile}`" != "OK" ] ; then
+  ./ts_rof_monthly_1985-1986-0002.bash
+  if [ $? -ne 0 ]; then
+    error=true
+  fi
+fi
+
+echo === ts_rof_monthly_1987-1988-0002.bash  ===
+task=ts_rof_monthly_1987-1988-0002.bash
+taskStatusFile="${task%.*}.status"
+if [ ! -f ${taskStatusFile} ] || [ "`cat ${taskStatusFile}`" != "OK" ] ; then
+  ./ts_rof_monthly_1987-1988-0002.bash
+  if [ $? -ne 0 ]; then
+    error=true
+  fi
+fi
+
 echo === e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash  ===
 task=e3sm_diags_atm_monthly_180x360_aave_mvm_model_vs_model_1987-1988_vs_1985-1986.bash
 taskStatusFile="${task%.*}.status"
========================================================== short test summary info ==========================================================
FAILED tests/integration/test_bundles.py::test_bundles_bash_file_content - assert 256 == 0
======================================================== 1 failed, 1 passed in 1.39s ========================================================

# Let's check the ts_rof_monthly error:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/test_zppy_py3.14/v3.LR.historical_0051/post/scripts
grep "ts_rof" *.o*
# bundle3.o1029462:=== ts_rof_monthly_1985-1986-0002.bash ===
# bundle3.o1029462:=== ts_rof_monthly_1987-1988-0002.bash ===

grep "ts_rof" *.bash # The test is actually checking the bash file
# Many matches

grep "./ts_rof_monthly_1985-1986-0002.bash" *.bash
# No results
grep "./ts_rof_monthly_1987-1988-0002.bash" *.bash
# No results

ls bundle3.o*
# bundle3.o1029462  bundle3.o1033271

# So, yes, ts_rof ran on the initial run before we ran into the disk space limit.
# So, it was already marked as completed and the bash file it was in got overwritten during the rerun.

Ok, I think this PR does actually pass the full test suite. Two issues came up, neither of which are actually because of this PR:

The merging of Add NCO version parameter #761 to main changes some of the expected results, but they haven't been updated yet.
The jobs had to be restarted because of the space limitations hit while testing Potential fix to token timeout zstash#407. That evidently broke the bundles test because it's checking a bash file which gets overwritten on reruns. (This doesn't end up being a problem for users because they can still get results, but it does mean the bash file didn't have everything the test was expecting.) I can rerun from scratch, but I don't think it's necessary. Additionally, once this is merged to main, it will be tested as part of the weekly testing of main anyway.

forsyth2 · 2025-12-04T01:09:03Z

@andrewdnolan @tomvothecoder I've run the full test suite on this change. Is there any reason to hold off on merging?

I shouldn't have to wait until zppy-interfaces and zstash are ready too, right? That would be a longer wait -- the former needs dependencies to support Python 3.14 and the latter needs further debugging/code changes.

andrewdnolan

@forsyth2 This looks great to me! My personal opinion would be that if your testing and CI passes then I'd merge this.

Having this in will ensure all development between now and the next unified release in March can and will be tested with 3.14, which should save alot of headache.

forsyth2 · 2025-12-04T18:34:01Z

Awesome, thanks @andrewdnolan! I'll go ahead and merge then.

forsyth2 self-assigned this Nov 18, 2025

forsyth2 added the DevOps CI/CD, configuration, etc. label Nov 18, 2025

Add Python 3.14 support

ed56120

forsyth2 force-pushed the add-python3.14 branch from 7ac2efe to ed56120 Compare November 18, 2025 22:25

This was referenced Nov 18, 2025

Add Python 3.14 support E3SM-Project/zppy-interfaces#44

Draft

Add Python 3.14 support E3SM-Project/zstash#402

Draft

forsyth2 mentioned this pull request Dec 2, 2025

Potential fix to token timeout E3SM-Project/zstash#407

Closed

7 tasks

forsyth2 marked this pull request as ready for review December 4, 2025 01:06

andrewdnolan approved these changes Dec 4, 2025

View reviewed changes

forsyth2 merged commit 406751e into main Dec 4, 2025
7 checks passed

forsyth2 deleted the add-python3.14 branch December 4, 2025 18:34

forsyth2 mentioned this pull request Dec 5, 2025

Fix regressions due to xarray 2025.11.0 attributes change E3SM-Project/e3sm_diags#1021

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Python 3.14 support #757

Add Python 3.14 support #757

Uh oh!

forsyth2 commented Nov 18, 2025 •

edited

Loading

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Dec 2, 2025 •

edited

Loading

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

andrewdnolan left a comment

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add Python 3.14 support #757

Add Python 3.14 support #757

Uh oh!

Conversation

forsyth2 commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Small Change

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Nov 18, 2025

Uh oh!

forsyth2 commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

andrewdnolan left a comment

Choose a reason for hiding this comment

Uh oh!

forsyth2 commented Dec 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

forsyth2 commented Nov 18, 2025 •

edited

Loading

forsyth2 commented Dec 2, 2025 •

edited

Loading