-
Notifications
You must be signed in to change notification settings - Fork 15
Enable land only ilamb and global time-series (for J case spin up) #765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
@forsyth2 If time allows, could you make a code review to see if there are any issues i should pay attention to? This PR does 4 things: |
forsyth2
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chengzhuzhang I think it would be a good idea to run the integration tests to make sure nothing breaks there, but my visual inspection appears fine:
✅ Enable land only ilamb when no atm data present.
✅ Enable global time-series when no atm data present.
✅ Fix a bug in climo task for enabling generating mosart climatology using ncclimo.
✅ For global time-series tasks: copy ocean results to case_dir and clean up results dir
I added some explanatory comments on the PR just for my understanding. The code may benefit from increased commenting, but this is non-essential.
| # Create output directory | ||
| # Create local links to input cmip time-series files | ||
| lnd_ts_for_ilamb={{ output }}/post/lnd/{{ ts_land_grid }}/cmip_ts/monthly/ | ||
| {% if not land_only %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Part 1 of fix for "Enable land only ilamb when no atm data present."
| echo 'ERROR (1)' > {{ prefix }}.status | ||
| exit 1 | ||
| fi | ||
| {% if not land_only %} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Part 2 of fix for "Enable land only ilamb when no atm data present."
| # Only require ATM if we have non-ocean plots | ||
| if has_atm_plots: | ||
| c["use_atm"] = True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Fix for "Enable global time-series when no atm data present."
| prc_typ = tmp | ||
| elif tmp in ("mosart",): | ||
| component = "rof" | ||
| prc_typ = tmp |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Fix for "Fix a bug in climo task for enabling generating mosart climatology using ncclimo."
|
|
||
| # Copy ocean results to case directory | ||
| echo | ||
| echo ===== COPY OCEAN RESULTS TO CASE DIRECTORY ===== |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Part 1 of fix for "For global time-series tasks: copy ocean results to case_dir and clean up results dir"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As described in the "Path explanation, part 1:" comment, both of these paths are actually in case_dir so a better title of this section would be something like: "Copy files from scripts/{{ prefix }}_results/ocn/ to ocn/."
|
|
||
| # Clean up temporary results directory | ||
| echo | ||
| echo ===== CLEANUP TEMPORARY RESULTS DIRECTORY ===== |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅ Part 2 of fix for "For global time-series tasks: copy ocean results to case_dir and clean up results dir"
| ################################################################################ | ||
| case={{ case }} | ||
| www={{ www }} | ||
| case_dir={{ output }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this makes sense because we had --case_dir {{ output }}
|
|
||
| if [ -d "${results_dir_absolute_path}/ocn" ]; then | ||
| mkdir -p ${case_dir}/post/ocn | ||
| rsync -av ${results_dir_absolute_path}/ocn/ ${case_dir}/post/ocn/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Path explanation, part 1:
results_dir={{ prefix }}_results
results_dir_absolute_path={{ scriptDir }}/${results_dir}
So, results_dir_absolute_path is
${case_dir}/post/scripts/{{ prefix }}_results
So, we're syncing
${case_dir}/post/scripts/{{ prefix }}_results/ocn/ -> ${case_dir}/post/ocn/
| # Copy files | ||
| rsync -a --delete ${results_dir_absolute_path} ${top_level} | ||
| # Copy files (excluding ocn directory) | ||
| rsync -a --delete --exclude='ocn' ${results_dir_absolute_path} ${top_level} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Path explanation, part 2:
top_level=${www}/${case}/global_time_series/
So, we're syncing
# Except .../ocn:
${case_dir}/post/scripts/{{ prefix }}_results -> ${www}/${case}/global_time_series/
Furthermore, the --delete will delete files in ${www}/${case}/global_time_series/ that are not in ${case_dir}/post/scripts/{{ prefix }}_results.
| echo | ||
| echo ===== CLEANUP TEMPORARY RESULTS DIRECTORY ===== | ||
| echo | ||
| rm -rf ${results_dir_absolute_path} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Path explanation, part 3:
We can now remove
${case_dir}/post/scripts/{{ prefix }}_results
Putting all that together we get:
# Step 1: Copy files from `scripts/{{ prefix }}_results/ocn/` to `ocn/`.
rsync -av ${case_dir}/post/scripts/{{ prefix }}_results/ocn/ ${case_dir}/post/ocn/
# Step 2: Copy non-ocean files from the output directory `case_dir` to the web path `www` for viewing on the web server.
rsync -a --delete --exclude='ocn' ${case_dir}/post/scripts/{{ prefix }}_results ${www}/${case}/global_time_series/
# Step 3: At this point, all the files have been copied to one directory or another; delete them at source.
rm -rf ${case_dir}/post/scripts/{{ prefix }}_results
|
@forsyth2 , Thanks for a review! |
|
@forsyth2 There was a new mpas-a related PR merged to main, would you prefer to also merge this for the integration test, or rebase this branch then test? |
|
@chengzhuzhang I'd prefer to do a preliminary test before merging. I'm currently setting that up for this PR & E3SM-Project/zppy-interfaces#46. The mpas-a related PR (#760) at least had an initial integration test done, whereas this one hasn't had any done yet. I am being careful with integration testing to base branches off the same exact commit history as was used to generate expected results, so we can be sure diffs are from the PR itself. |
|
Actually, I have done zppy test using this PR with the recent v3.HR spin-up run which can be thought of as an integration test. So if the code change is solid, it is also fine to merge and then test. Though I don't have a preference when to merge. |
|
My test is almost wrapped up; we might as well get those results before merging. |
Testing steps# Set up zppy-interfaces env
cd ~/ez/zppy-interfaces
git status
# On branch main
# nothing to commit, working tree clean
git fetch upstream fix-missing-year-key-original-plots
git checkout -b test-fix-missing-year-key-original-plots upstream/fix-missing-year-key-original-plots
git log
# Good, commits match https://github.com/E3SM-Project/zppy-interfaces/pull/46/commits
# Good, based on last commit from expected results: Bump to 0.2.0 (#45)
lcrc_conda # Activate conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zi-pr46-env
conda activate zi-pr46-env
python -m pip install .
pytest tests/unit/global_time_series/test_*.py
# 10 passed in 21.72s
pytest tests/unit/pcmdi_diags/test_*.py
# 7 passed in 8.27s
cd ~/ez/zppy
git status
# On branch zppy-weekly-test-20251216
# Handle uncommitted changes:
git add -A
git commit -m "Testing" --no-verify
git fetch upstream enable_land_only_ilamb
git checkout -b test_enable_land_only_ilamb upstream/enable_land_only_ilamb
git log
# Good, commits match https://github.com/E3SM-Project/zppy/pull/765/commits
# Bad, based on: Add Python 3.14 support (#757)
# But expected results are based on: Add legacy cfgs for zppy v3.1.0 (#762)
# https://github.com/E3SM-Project/zppy/commits/main was updated this morning by the merging of
# https://github.com/E3SM-Project/zppy/pull/760, so we can't simply rebase off `main`.
# We need to get the commit directly.
git fetch upstream main
git rebase ac7dca27232d63e72c1824f6662d020e9d251698 # Commit for: Add legacy cfgs for zppy v3.1.0 (#762)
git log
# Good, now 5 commits on top of: Add legacy cfgs for zppy v3.1.0 (#762)
# That means any diffs will truly be from this PR.
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr765-env
conda activate zppy-pr765-env
python -m pip install .
pytest tests/test_*.py
# 35 passed in 0.47s
# Get testing commit: https://github.com/E3SM-Project/zppy/pull/767
# 806dfa2e1ce947930ecf7483eeb3f35e18c41bc6
git fetch upstream issue-766-test-links
git cherry-pick 806dfa2e1ce947930ecf7483eeb3f35e18c41bc6
git log
# Good, commit is added
# Edit tests/integration/utils.py
# There haven't been any changes to the test cfgs, so let's keep using the 3.0.0 legacy cfgs for now.
# TEST_SPECIFICS:
# TEST_SPECIFICS: Dict[str, Any] = {
# # These are custom environment_commands for specific tasks.
# # Never set these to "", because they will print the line
# # `environment_commands = ""` for the corresponding task,
# # thus overriding the value set higher up in the cfg.
# # That is, there will be no environment set.
# # (`environment_commands = ""` only redirects to Unified
# # if specified under the [default] task)
# "diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# "global_time_series_environment_commands": "source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pr46-env",
# "pcmdi_diags_environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# # This is the environment setup for other tasks.
# # Leave as "" to use the latest Unified environment.
# "environment_commands": "source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
# # For a complete test, run the set of latest cfgs and at least one set of legacy cfgs
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# # "weekly_legacy_3.1.0_bundles",
# # "weekly_legacy_3.1.0_comprehensive_v2",
# # "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_bundles",
# "weekly_legacy_3.0.0_comprehensive_v2",
# "weekly_legacy_3.0.0_comprehensive_v3",
# ],
# "tasks_to_run": [
# # "e3sm_diags",
# # "mpas_analysis",
# "global_time_series",
# "ilamb",
# # "pcmdi_diags",
# ],
# "unique_id": "zppy_pr765_zi_pr46",
# }
git diff
# Diff looks good
python tests/integration/utils.py
# CFG FILES HAVE BEEN GENERATED FROM TEMPLATES WITH THESE SETTINGS:
# UNIQUE_ID=zppy_pr765_zi_pr46
# diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# global_time_series_environment_commands=source /gpfs/fs1/home/ac.forsyth2/miniforge3/etc/profile.d/conda.sh; conda activate zi-pr46-env
# pcmdi_diags_environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# environment_commands=source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
# Reminder: `environment_commands=''` => the latest E3SM Unified environment will be used
# For reference:
alias sq
# alias sq='sqa -u ac.forsyth2'
alias sqa
# alias sqa='squeue -o "%8u %.7a %.4D %.9P %7i %.2t %.10r %.10M %.10l %j" --sort=P,-t,-p'
sq
# No jobs currently queued
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_comprehensive_v3_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v2_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_comprehensive_v3_chrysalis.cfg
sq | wc -l # Wed 12/17 11:52 => 90 - header row = 89 jobs
sq | wc -l # Wed 12/17 12:34 => 1 - header row = 0 jobs
# Check on bundles status
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
# Now, run bundles part 2
cd ~/ez/zppy
git status
# On branch test_enable_land_only_ilamb
# Good, correct branch
# Current conda env: zppy-pr765-env
# Good, correct environment
zppy -c tests/integration/generated/test_weekly_bundles_chrysalis.cfg
zppy -c tests/integration/generated/test_weekly_legacy_3.0.0_bundles_chrysalis.cfg
sq | wc -l # Wed 12/17 12:37 => 1 - header row = 0 jobs
# No new jobs were launched
# Let's review the command line output from the `zppy` launches:Debug issue with test cfg setupFor bundles: Similarly, for legacy_3.0.0_bundles: ls /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts/ts_land_monthly_1985-1986-0002.status
# No such file or directory
ls /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts/ts_land_monthly_1987-1988-0002.status
# No such file or directory
emacs /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts/ilamb_1985-1986.settingsThat includes: emacs /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts/provenance.20251217_195100_579267.cfg
# [[ land_monthly ]]
# active = False
emacs tests/integration/template_weekly_bundles.cfg
# [[ land_monthly ]]
# active = #expand active_e3sm_diags#
emacs tests/integration/template_weekly_comprehensive_v2.cfg
# [[ land_monthly ]]
emacs tests/integration/template_weekly_comprehensive_v3.cfg
# [[ land_monthly ]]
# So, the bundles test cfgs have a bug: `land_monthly` is only launched if e3sm_diags is requested
# The comprehensive tests do not have that bug.Testing steps, continued# Review finished runs
### v2 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v2_output/zppy_pr765_zi_pr46/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v2_output/zppy_pr765_zi_pr46/v2.LR.historical_0201/post/scripts
grep -v "OK" *status
# Good, no errors
### v3 ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_comprehensive_v3_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_comprehensive_v3_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
### bundles ###
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
# Good, no errors
cd /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_legacy_3.0.0_bundles_output/zppy_pr765_zi_pr46/v3.LR.historical_0051/post/scripts
grep -v "OK" *status
cd ~/ez/zppy
git status
# On branch test_enable_land_only_ilamb
# Good, correct branch
# Env: zppy-pr765-env
# Good, correct env
ls tests/integration/test_*.py
# # Can't run because we didn't run all possible jobs:
# pytest tests/integration/test_bash_generation.py
# pytest tests/integration/test_bundles.py
pytest tests/integration/test_campaign.py
# 3 failed, 3 passed in 3.90s; caused by prc_typ change in this PR
pytest tests/integration/test_defaults.py
# 1 passed in 2.18s
pytest tests/integration/test_last_year.py
# 1 passed in 1.63s
salloc --nodes=1 --partition=debug --time=00:30:00 --account=e3sm
# We need to reactivate conda now that we're on a compute node
lcrc_conda # Activate conda
conda activate zppy-pr765-env
pytest tests/integration/test_images.py
# 1 failed in 129.88s (0:02:09)
cat test_images_summary.mdOutput:
Let's remove the bundles lines since we know the relevant test cfgs were set up incorrectly.
✅ Passes relevant subset of integration tests |
|
@chengzhuzhang Thanks, my preliminary testing passed. I have updated the PR description checklists accordingly and will now merge both PRs (this one & E3SM-Project/zppy-interfaces#46) |
|
Great! Thank you, @forsyth2 . |
|
I made the merge commit's description the following (from the earlier comment): |
Summary
Objectives:
Based on a thread with @xuezhengllnl , for the alternating B/J cases spin-up runs from v3.HR runs,
zppy cfg file cannot run ilamb for the J-case segment of the v3HR spinup run because J-case configuration has no eam hist outputs. We will need to check the time evolution of the land condition for both J-case segments and B-case segments. Starting 0106-0165 for 60years J case (no atm), 0166-0175 10 years Bcase (fully coupled with atm). And recycle.
Note: J-case means interactive ocean, seaice, land model with atmospheric forcing (no EAM model). B-case means fully coupled model configuration. The v3HR spin-up simulation includes both configurations.
Select one: This pull request is...
Small Change