Skip to content

Add additional files to the mpas-analysis cache for time series analysis #760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chengzhuzhang merged 3 commits into from
Dec 17, 2025
Merged

Add additional files to the mpas-analysis cache for time series analysis #760

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2084313
Add additional files to the mpas-analysis cache for time series analysis
cbegeman Nov 25, 2025
0860a1f
Fix filepath for new cache files
cbegeman Dec 5, 2025
c8b131a
Add to cache after subdirs are created
cbegeman Dec 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions zppy/templates/mpas_analysis.bash
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,11 @@ do
mkdir -p cache/${subdir} ${identifier}/${subdir}
rsync -av cache/${subdir}/ ${identifier}/${subdir}/
done
files=( "mpasIndexOcean.nc" "mpasTimeSeriesOcean.nc" "seaIceAreaVolNH.nc" "seaIceAreaVolSH.nc")
for file in "${files[@]}"
do
cp cache/timeseries/${file} ${identifier}/timeseries/${file}
done
{% endif %}

# Run MPAS-Analysis
Expand Down Expand Up @@ -297,10 +302,16 @@ fi
echo
echo ===== CACHE OUTPUT FILES =====
echo
for file in "${files[@]}"
do
cp ${identifier}/timeseries/${file} cache/timeseries/${file}
done
Comment on lines +305 to +308
Copy link
Contributor Author

@cbegeman cbegeman Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonbob This is the set of lines that will need to be run once manually to cache the files that weren't cached with the released version of MPAS-Analysis

Note: defined above files=( "mpasIndexOcean.nc" "mpasTimeSeriesOcean.nc" "seaIceAreaVolNH.nc" "seaIceAreaVolSH.nc")

Copy link
Collaborator

@chengzhuzhang chengzhuzhang Dec 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cbegeman I think this line might be the trouble that caused cp: cannot stat 'cache/mpasIndexOcean.nc': No such file or directory issues in my second run test. The Cache output files are saved in cache/timeseries, however when the second run is reading files from cache, it tried to find files from cache

for subdir in "${cached[@]}"
do
rsync -av ${identifier}/${subdir}/ cache/${subdir}/
done
# Remove one particularly large file which does not need to be cached
rm ${identifier}/timeseries/mpasTimeSeriesSeaIce.nc
Comment on lines +313 to +314
Copy link
Contributor

@xylar xylar Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to just be deleting a time-series file form the analysis directory, and not to be related to caching at all. Is this perhaps a relic of an earlier approach to copying cache files and something no longer needed? I would think it wouldn't be a great idea to delete output from the mpas-analysis output directory even if it is large unless we decide on a systematic way of cleaning things up that doesn't lose important data.

Copy link
Contributor Author

@cbegeman cbegeman Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing large files from MPAS-Analysis that are not needed was requested by HR (specifically @wlin7). Do you think that MPAS-Analysis would be a better place to do this clean-up? I'm not particularly tied to including this in this PR but maybe some more context would be helpful:

This file seems to be somewhat of an outlier in MPAS-Analysis output files in that it has dimensions of Time (equal the length of the requested time series) and nCells (equal to the size of the whole mesh, which seems particularly egregious given that only a portion of those cells would contain sea ice). Correct me if there are other files like this. It's not clear to me that this would be "losing important data" because it's just copying information from history files that would (ideally) be archived as opposed to derived analysis variables.

ncinfo timeseries/mpasTimeSeriesSeaIce.nc
...
    dimensions(sizes): Time(72), StrLen(64), nCells(465044)
    variables(dimensions): |S1 xtime_startMonthly(Time, StrLen), |S1 xtime_endMonthly(Time, StrLen), float32 timeMonthly_avg_iceAreaCell(Time, nCells), float32 timeMonthly_avg_iceVolumeCell(Time, nCells), float32 timeMonthly_avg_snowVolumeCell(Time, nCells)

Copy link
Contributor

@xylar xylar Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I don't think zppy should be deleting MPAS-Analysis files. This needs to be fixed in MPAS-Analysis instead. Sorry for extra work for, though!

Copy link
Contributor

@xylar xylar Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is a corresponding file for the the ocean as far as I know.

Copy link
Contributor

@xylar xylar Dec 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's chat about this on Slack.

Copy link
Contributor

@xylar xylar Dec 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After having a conversation on Slack, I've come around to this approach.

I think MPAS-Analysis would ideally delete this file itself. I think MPAS-Analysis would also ideally extract only the variables that it plots directly from mpasTimeSeriesOcean.nc into other, smaller time series files that could be cached instead. But this was not how it was designed and making these changes in a robust way will likely take more time than we have before HR needs a working solution.

So I am fine with this approach as a less ideal but working alternative.

{% endif %}

# Copy output to web server
Expand Down
Loading