Updating catalog builder package version

[Ciheim]

Describe your changes

Issue ticket number and link (if applicable)

Checklist before requesting a review

• I ran my code
• I tried to make my code readable
• I tried to comment my code
• I wrote a new test, if applicable
• I wrote new instructions/documentation, if applicable
• I ran pytest and inspected it's output
• I ran pylint and attempted to implement some of it's feedback
• No print statements; all user-facing info uses logging module

[ceblanton]

Looks good to me. 2025.01.01 is the latest catalogbuilder version on our channel (https://anaconda.org/channels/NOAA-GFDL/packages/catalogbuilder/files)

[ceblanton]

It's not passing the tests but the failures don't clearly look catalogbuilder related

[singhd789]

It's not passing the tests but the failures don't clearly look catalogbuilder related

I don't think it's catalog builder related either - I saw the same failure in my #679 PR but haven't taken a closer look yet

[ilaflott]

we need green checks!

this doesn't look associated with catalogbuilder directly, the error message is associated with a lack of the IPython, which is required by the intake_esm package, which catalogbuilder seems to require a specific version of that's not working here.

it's not impossible something about intake_esm changed from underneath catalogbuilder.

pay close attention to the trace in the logs!

https://github.com/NOAA-GFDL/fre-cli/actions/runs/20758852348/job/59608111480#step:8:169

[ilaflott]

actually, i'm realizing this is something to do with analysis-scripts / fre.analysis / #542 , and catalogbuilder changes just revealed the issue. consider the following:

the error/stack trace from pytest starts here. I clipped the output to the important bits, see link for all gory details:

fre/analysis/tests/test_subtools.py:52: AssertionError
----------------------------- Captured stdout call -----------------------------
Traceback (most recent call last):
  File "/tmp/tmpwu2tkcqs/python_script", line 3, in <module>
    paths = run_plugin('freanalysis_clouds', '/tmp/tmpvcmwustk/fake-catalog', '.', config=config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/analysis_scripts/plugins.py", line 140, in run_plugin
    return _plugin_object(name).run_analysis(catalog, png_dir, config, reference_catalog)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/freanalysis_clouds/__init__.py", line 97, in run_analysis
    catalog = intake.open_esm_datastore(catalog)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/__init__.py", line 82, in __getattr__
    driver = registry[attr[5:]]  # "open_..."
             ~~~~~~~~^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/source/__init__.py", line 30, in __getitem__
    return it.load()
           ^^^^^^^^^
...
...
...

  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/__init__.py", line 10, in <module>
    from intake_esm.core import esm_datastore
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/core.py", line 24, in <module>
    from fastprogress.fastprogress import progress_bar

...
...
...

ModuleNotFoundError: No module named 'IPython'

So, IPython is required by intake_esm, but isn't there. intake_esm is required by intake which is required by freanalysis_clouds... and this is where it gets odd! freanalysis_clouds should be getting pip install'd into it's own venv, and NOT the same environment as fre-cli / analysis-scripts.

NOAA-GFDL/analysis-scripts/user_scripts/freanalysis_clouds does require intake_esm, but again, it should be trying to use an intake_esm pip installed into a separate venv with which it is compatible

[ceblanton]

actually, i'm realizing this is something to do with analysis-scripts / fre.analysis / #542 , and catalogbuilder changes just revealed the issue. consider the following:

the error/stack trace from pytest starts here. I clipped the output to the important bits, see link for all gory details:

fre/analysis/tests/test_subtools.py:52: AssertionError
----------------------------- Captured stdout call -----------------------------
Traceback (most recent call last):
  File "/tmp/tmpwu2tkcqs/python_script", line 3, in <module>
    paths = run_plugin('freanalysis_clouds', '/tmp/tmpvcmwustk/fake-catalog', '.', config=config)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/analysis_scripts/plugins.py", line 140, in run_plugin
    return _plugin_object(name).run_analysis(catalog, png_dir, config, reference_catalog)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/freanalysis_clouds/__init__.py", line 97, in run_analysis
    catalog = intake.open_esm_datastore(catalog)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/__init__.py", line 82, in __getattr__
    driver = registry[attr[5:]]  # "open_..."
             ~~~~~~~~^^^^^^^^^^
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake/source/__init__.py", line 30, in __getitem__
    return it.load()
           ^^^^^^^^^
...
...
...

  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/__init__.py", line 10, in <module>
    from intake_esm.core import esm_datastore
  File "/tmp/tmpvcmwustk/env/lib/python3.11/site-packages/intake_esm/core.py", line 24, in <module>
    from fastprogress.fastprogress import progress_bar

...
...
...

ModuleNotFoundError: No module named 'IPython'

So, IPython is required by intake_esm, but isn't there. intake_esm is required by intake which is required by freanalysis_clouds... and this is where it gets odd! freanalysis_clouds should be getting pip install'd into it's own venv, and NOT the same environment as fre-cli / analysis-scripts.

NOAA-GFDL/analysis-scripts/user_scripts/freanalysis_clouds does require intake_esm, but again, it should be trying to use an intake_esm pip installed into a separate venv with which it is compatible

@ilaflott thank you for getting to the bottom of this, and I'm sorry the trail leads to fre-analysis.

The existing fre-analysis stuff is being refactored, and I strongly suggest we turn off the fre-analysis tests for now.

[ilaflott]

I agree- @skip the fre.analysis tests... A bit sad, though

[ilaflott]

@Ciheim @ceblanton @singhd789

bizarre- this might actually be an intake-esm bug, catalogbuilder is having the same problem in it's pipelines that it ran successfully for recent updates. perplexing.

see https://github.com/NOAA-GFDL/CatalogBuilder/actions/runs/20970566497/job/60272504792

[singhd789]

@Ciheim @ceblanton @singhd789

bizarre- this might actually be an intake-esm bug, catalogbuilder is having the same problem in it's pipelines that it ran successfully for recent updates. perplexing.

see https://github.com/NOAA-GFDL/CatalogBuilder/actions/runs/20970566497/job/60272504792

Also happened in my fre make PR: #679

[Ciheim]

@ilaflott Good catch. This is why we test!

[ilaflott]

the mystery thickens...

AnswerDotAI/fastprogress#117

[ilaflott]

actually, i think intake_esm's fix for this isn't up to snuff: intake/intake-esm#770

i think i officially blame intake_esm, BUT, analysis-script's failure to use an isolated environment is troubling, independently.

[ilaflott]

In light of exploring this, I've opened intake/intake-esm#773

@ceblanton @Ciheim @singhd789 @aradhakrishnanGFDL

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.30%. Comparing base (982868f) to head (21d4523).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #693   +/-   ##
=======================================
  Coverage   85.30%   85.30%           
=======================================
  Files          68       68           
  Lines        4498     4498           
=======================================
  Hits         3837     3837           
  Misses        661      661           

Flag	Coverage Δ
unittests	85.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.