Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tests/test_zppy_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ def test_set_component_and_prc_typ():
c = {"input_component": "mosart"}
set_component_and_prc_typ(c)
assert c["component"] == "rof"
assert c["prc_typ"] == "sgs"
assert c["prc_typ"] == "mosart"

# Test with input_files
c = {"input_component": "", "input_files": "cam.extension"}
Expand Down Expand Up @@ -181,7 +181,7 @@ def test_set_component_and_prc_typ():
c = {"input_component": "", "input_files": "mosart.extension"}
set_component_and_prc_typ(c)
assert c["component"] == "rof"
assert c["prc_typ"] == "sgs"
assert c["prc_typ"] == "mosart"

# Test error case
c = {"input_component": "", "input_files": ""}
Expand Down
24 changes: 17 additions & 7 deletions zppy/global_time_series.py
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,24 @@ def determine_components(c: Dict[str, Any]) -> None:
c["use_lnd"] = False
c["use_ocn"] = False
if c["plots_original"]:
c["use_atm"] = True
has_original_ocn_plots = (
("change_ohc" in c["plots_original"])
or ("max_moc" in c["plots_original"])
or ("change_sea_level" in c["plots_original"])
)
if has_original_ocn_plots:
# Define ocean-specific plots that don't require ATM data
ocean_plots = {"change_ohc", "max_moc", "change_sea_level"}

# Parse the plots_original into individual plot names
original_plots = set(plot.strip() for plot in c["plots_original"].split(","))

# Check for ocean plots
has_ocean_plots = bool(original_plots & ocean_plots)
if has_ocean_plots:
c["use_ocn"] = True

# Check for non-ocean plots (ATM plots)
non_ocean_plots = original_plots - ocean_plots
has_atm_plots = bool(non_ocean_plots)

# Only require ATM if we have non-ocean plots
if has_atm_plots:
c["use_atm"] = True
Comment on lines +115 to +117
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fix for "Enable global time-series when no atm data present."

else:
# For better string processing in global_time_series.bash
c["plots_original"] = "None"
Expand Down
34 changes: 32 additions & 2 deletions zppy/templates/global_time_series.bash
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,25 @@ results_dir_absolute_path={{ scriptDir }}/${results_dir}
################################################################################
case={{ case }}
www={{ www }}
case_dir={{ output }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this makes sense because we had --case_dir {{ output }}


# Copy ocean results to case directory
echo
echo ===== COPY OCEAN RESULTS TO CASE DIRECTORY =====
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Part 1 of fix for "For global time-series tasks: copy ocean results to case_dir and clean up results dir"

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As described in the "Path explanation, part 1:" comment, both of these paths are actually in case_dir so a better title of this section would be something like: "Copy files from scripts/{{ prefix }}_results/ocn/ to ocn/."

echo

if [ -d "${results_dir_absolute_path}/ocn" ]; then
mkdir -p ${case_dir}/post/ocn
rsync -av ${results_dir_absolute_path}/ocn/ ${case_dir}/post/ocn/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path explanation, part 1:

results_dir={{ prefix }}_results
results_dir_absolute_path={{ scriptDir }}/${results_dir}

So, results_dir_absolute_path is

${case_dir}/post/scripts/{{ prefix }}_results

So, we're syncing

${case_dir}/post/scripts/{{ prefix }}_results/ocn/ -> ${case_dir}/post/ocn/ 

if [ $? != 0 ]; then
cd {{ scriptDir }}
echo 'ERROR (6)' > {{ prefix }}.status
exit 6
fi
echo "Ocean results copied to ${case_dir}/post/ocn/"
else
echo "No ocean results directory found at ${results_dir_absolute_path}/ocn"
fi

# Copy output to web server
echo
Expand Down Expand Up @@ -49,8 +68,8 @@ do
done
{% endif %}

# Copy files
rsync -a --delete ${results_dir_absolute_path} ${top_level}
# Copy files (excluding ocn directory)
rsync -a --delete --exclude='ocn' ${results_dir_absolute_path} ${top_level}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path explanation, part 2:

top_level=${www}/${case}/global_time_series/

So, we're syncing

# Except .../ocn:
${case_dir}/post/scripts/{{ prefix }}_results -> ${www}/${case}/global_time_series/

Furthermore, the --delete will delete files in ${www}/${case}/global_time_series/ that are not in ${case_dir}/post/scripts/{{ prefix }}_results.

if [ $? != 0 ]; then
cd {{ scriptDir }}
echo 'ERROR (8)' > {{ prefix }}.status
Expand All @@ -72,6 +91,17 @@ chmod -R go+rX,go-w ${results_dir}
popd
{% endif %}

# Clean up temporary results directory
echo
echo ===== CLEANUP TEMPORARY RESULTS DIRECTORY =====
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Part 2 of fix for "For global time-series tasks: copy ocean results to case_dir and clean up results dir"

echo
rm -rf ${results_dir_absolute_path}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Path explanation, part 3:

We can now remove

${case_dir}/post/scripts/{{ prefix }}_results

Putting all that together we get:

# Step 1: Copy files from `scripts/{{ prefix }}_results/ocn/` to `ocn/`.
rsync -av ${case_dir}/post/scripts/{{ prefix }}_results/ocn/ ${case_dir}/post/ocn/ 

# Step 2: Copy non-ocean files from the output directory `case_dir` to the web path `www` for viewing on the web server.
rsync -a --delete --exclude='ocn' ${case_dir}/post/scripts/{{ prefix }}_results ${www}/${case}/global_time_series/

# Step 3: At this point, all the files have been copied to one directory or another; delete them at source.
rm -rf ${case_dir}/post/scripts/{{ prefix }}_results

if [ $? = 0 ]; then
echo "Successfully cleaned up ${results_dir_absolute_path}"
else
echo "Warning: Failed to clean up ${results_dir_absolute_path}"
fi

################################################################################
# Update status file and exit
{% raw %}
Expand Down
4 changes: 4 additions & 0 deletions zppy/templates/ilamb.bash
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ echo
# Create output directory
# Create local links to input cmip time-series files
lnd_ts_for_ilamb={{ output }}/post/lnd/{{ ts_land_grid }}/cmip_ts/monthly/
{% if not land_only %}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Part 1 of fix for "Enable land only ilamb when no atm data present."

atm_ts_for_ilamb={{ output }}/post/atm/{{ ts_atm_grid }}/cmip_ts/monthly/
{% endif %}
# Go through the time series files for between year1 and year2,
# using a step size equal to the number of years per time series file
start_year=$(echo $Y1 | sed 's/^0*//')
Expand All @@ -56,12 +58,14 @@ do
echo 'ERROR (1)' > {{ prefix }}.status
exit 1
fi
{% if not land_only %}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Part 2 of fix for "Enable land only ilamb when no atm data present."

cp -s ${atm_ts_for_ilamb}/*_*_*_*_*_*_${start_year}??-${end_year}??.nc .
if [ $? != 0 ]; then
cd {{ scriptDir }}
echo 'ERROR (2)' > {{ prefix }}.status
exit 2
fi
{% endif %}
done
cd ../..

Expand Down
1 change: 1 addition & 0 deletions zppy/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,7 @@ def set_component_and_prc_typ(c: Dict[str, Any]) -> None:
prc_typ = tmp
elif tmp in ("mosart",):
component = "rof"
prc_typ = tmp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Fix for "Fix a bug in climo task for enabling generating mosart climatology using ncclimo."

else:
raise ValueError(
f"Cannot extract output component name from {c['input_component']} or {c['input_files']}."
Expand Down
Loading