Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce memory footprint #397

Merged
merged 1 commit into from
Nov 23, 2023
Merged

Reduce memory footprint #397

merged 1 commit into from
Nov 23, 2023

Conversation

sriharshakandala
Copy link
Member

@sriharshakandala sriharshakandala commented Nov 22, 2023 •
edited
Loading

Purpose

Analyze memory footprint

  • I have read and checked the items on the review checklist.

@sriharshakandala
Copy link
Member Author

sriharshakandala commented Nov 22, 2023 •
edited
Loading

RRTMGP's memory footprint seems to be primarily dominated by (2) arrays, random_sw and random_lw

test % julia --project all_sky_tuning.jl
FT = Float64, ncols = 131658; size per field = 0.04119899868965149 GB
"---------------------" = "---------------------"
CPU_slv_mem = 25.158858485519886 GB
"---------------------" = "---------------------"
fldname = context; fldsize = 0.0 GB
fldname = as; fldsize = 23.609969824552536 GB
fldname = op; fldsize = 0.12359713017940521 GB
fldname = src_lw; fldsize = 0.41591424494981766 GB
fldname = src_sw; fldsize = 0.3737342208623886 GB
fldname = bcs_lw; fldsize = 0.01569490134716034 GB
fldname = bcs_sw; fldsize = 0.029428035020828247 GB
fldname = fluxb_lw; fldsize = 0.1265399158000946 GB
fldname = fluxb_sw; fldsize = 0.16871988773345947 GB
fldname = flux_lw; fldsize = 0.1265399158000946 GB
fldname = flux_sw; fldsize = 0.16871988773345947 GB
"---------------------" = "---------------------"
fldname = lon; fldsize = 0.0 GB
fldname = lat; fldsize = 0.0 GB
fldname = p_lay; fldsize = 0.041199035942554474 GB
fldname = p_lev; fldsize = 0.04217996448278427 GB
fldname = t_lay; fldsize = 0.041199035942554474 GB
fldname = t_lev; fldsize = 0.04217996448278427 GB
fldname = t_sfc; fldsize = 0.000980965793132782 GB
fldname = col_dry; fldsize = 0.041199035942554474 GB
fldname = vmr; fldsize = 0.7827810272574425 GB
fldname = cld_r_eff_liq; fldsize = 0.041199035942554474 GB
fldname = cld_r_eff_ice; fldsize = 0.041199035942554474 GB
fldname = cld_path_liq; fldsize = 0.041199035942554474 GB
fldname = cld_path_ice; fldsize = 0.041199035942554474 GB
fldname = cld_frac; fldsize = 0.041199035942554474 GB
fldname = random_lw; fldsize = 10.546943709254265 GB
fldname = random_sw; fldsize = 9.228575751185417 GB
fldname = cld_mask_lw; fldsize = 1.3183680027723312 GB
fldname = cld_mask_sw; fldsize = 1.3183680027723312 GB
fldname = cld_mask_type; fldsize = 0.0 GB
fldname = ice_rgh; fldsize = 7.450580596923828e-9 GB
fldname = nlay; fldsize = 7.450580596923828e-9 GB
fldname = ncol; fldsize = 7.450580596923828e-9 GB
fldname = ngas; fldsize = 7.450580596923828e-9 GB
"---------------------" = "---------------------"

@sriharshakandala sriharshakandala mentioned this pull request Nov 22, 2023
Closed
@sriharshakandala
Copy link
Member Author

sriharshakandala commented Nov 23, 2023 •
edited
Loading

After fixing, we see

test % julia --project all_sky_tuning.jl
FT = Float64, ncols = 131658; size per field = 0.04119899868965149 GB
"---------------------" = "---------------------"
CPU_slv_mem = 2.8393009155988693 GB
"---------------------" = "---------------------"
fldname = context; fldsize = 0.0 GB
fldname = as; fldsize = 1.2904122546315193 GB
fldname = op; fldsize = 0.12359713017940521 GB
fldname = src_lw; fldsize = 0.41591424494981766 GB
fldname = src_sw; fldsize = 0.3737342208623886 GB
fldname = bcs_lw; fldsize = 0.01569490134716034 GB
fldname = bcs_sw; fldsize = 0.029428035020828247 GB
fldname = fluxb_lw; fldsize = 0.1265399158000946 GB
fldname = fluxb_sw; fldsize = 0.16871988773345947 GB
fldname = flux_lw; fldsize = 0.1265399158000946 GB
fldname = flux_sw; fldsize = 0.16871988773345947 GB
"---------------------" = "---------------------"
fldname = lon; fldsize = 0.0 GB
fldname = lat; fldsize = 0.0 GB
fldname = p_lay; fldsize = 0.041199035942554474 GB
fldname = p_lev; fldsize = 0.04217996448278427 GB
fldname = t_lay; fldsize = 0.041199035942554474 GB
fldname = t_lev; fldsize = 0.04217996448278427 GB
fldname = t_sfc; fldsize = 0.000980965793132782 GB
fldname = col_dry; fldsize = 0.041199035942554474 GB
fldname = vmr; fldsize = 0.7827810272574425 GB
fldname = cld_r_eff_liq; fldsize = 0.041199035942554474 GB
fldname = cld_r_eff_ice; fldsize = 0.041199035942554474 GB
fldname = cld_path_liq; fldsize = 0.041199035942554474 GB
fldname = cld_path_ice; fldsize = 0.041199035942554474 GB
fldname = cld_frac; fldsize = 0.041199035942554474 GB
fldname = random_lw; fldsize = 0.041199035942554474 GB
fldname = random_sw; fldsize = 0.041199035942554474 GB
fldname = cld_mask_lw; fldsize = 0.005149912089109421 GB
fldname = cld_mask_sw; fldsize = 0.005149912089109421 GB
fldname = cld_mask_type; fldsize = 0.0 GB
fldname = ice_rgh; fldsize = 7.450580596923828e-9 GB
fldname = nlay; fldsize = 7.450580596923828e-9 GB
fldname = ncol; fldsize = 7.450580596923828e-9 GB
fldname = ngas; fldsize = 7.450580596923828e-9 GB
"---------------------" = "---------------------"
test %

@sriharshakandala sriharshakandala linked an issue Nov 23, 2023 that may be closed by this pull request
Reduce memory footprint #398
Closed
@sriharshakandala sriharshakandala changed the title Analyze memory footprint Reduce memory footprint Nov 23, 2023
@sriharshakandala
Reduce memory footprint
0d50197 
RRTMGP's memory footprint seems to be primarily dominated by arrays `random_sw`, `random_lw`, `cld_mask_sw` and `cld_mask_lw`.
However, since RRTMGP loops over g-points, the `ngpt` dimension need not
be stored. This significantly reduces the memory footprint.
This PR also contains the following minor changes:
- Add compat entry for Random
- Updates needed by Aqua v0.8.2
@sriharshakandala sriharshakandala marked this pull request as ready for review November 23, 2023 06:14
@sriharshakandala sriharshakandala added the Launch Buildkite Launch Buildkite run label Nov 23, 2023
@sriharshakandala sriharshakandala merged commit b2bc126 into main Nov 23, 2023
8 of 9 checks passed
@sriharshakandala sriharshakandala mentioned this pull request Nov 27, 2023
Closed
@sriharshakandala sriharshakandala removed the Launch Buildkite Launch Buildkite run label Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@szy21 szy21 szy21 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Reduce memory footprint
2 participants
@sriharshakandala @szy21