Improve program operation efficiency

[Alex-jian522]

I use PeleLMeX to run my case with complex chemical mechanism (lu_88sk) on the server.
When I ran the simulation using 512 cores, from the output plt file I found that only 256 cores were actually running. The running speed is also very slow (2 timesteps/min). I would like to ask if there is any way to make full use of the submitted core number?
Below is my input file content.
Thank you very much for your answers !

#----------------------DOMAIN DEFINITION------------------------
geometry.is_periodic = 0 0 0 # For each dir, 0: non-perio, 1: periodic
geometry.coord_sys = 0 # 0 => cart, 1 => RZ
geometry.prob_lo = -0.005 -0.0075 -0.0075 # x_lo y_lo (z_lo)
geometry.prob_hi = 0.005 0.0075 0.0075 # x_hi y_hi (z_hi)

#---------------------- BC FLAGS ----------------------
peleLM.lo_bc = Inflow Outflow Outflow
peleLM.hi_bc = Inflow Outflow Outflow

#-------------------------AMR CONTROL----------------------------
amr.n_cell = 64 96 96 # Level 0 number of cells in each direction
amr.v = 1 # AMR verbose
amr.max_level = 1 # maximum level number allowed
amr.ref_ratio = 2 2 2 2 # refinement ratio
amr.regrid_int = 10 # how often to regrid
amr.n_error_buf = 2 2 2 2 # number of buffer cells in error est
amr.grid_eff = 0.7 # what constitutes an efficient grid
amr.blocking_factor = 8 # block factor in grid generation (min box size)
amr.max_grid_size = 32 # max box size

#--------------------------- Problem -------------------------------
prob.P_mean = 101325.0
prob.T_ox = 725.0
prob.T_fuel = 400.0
prob.U_fuel = 0.04
prob.U_ox = 0.07
prob.pertmag = 0.000
prob.inertVel = 0.05
prob.jet_radius = 0.005
prob.inert_radius = 0.0075
prob.pmf_datafile = "drm_CH4Air_stoich.dat"
prob.do_ignition = 0
prob.ignition_SphRad = 0.005
prob.ignition_SphT = 2200.0

#-------------------------PeleLM CONTROL----------------------------
peleLM.v = 1
peleLM.incompressible = 0
peleLM.rho = 1.17
peleLM.mu = 0.0
peleLM.use_wbar = 1
peleLM.sdc_iterMax = 2
peleLM.floor_species = 1
peleLM.deltaT_verbose = 0

peleLM.gravity = 0.0 0.0 0.0

amr.check_int = 200
amr.plot_int = 100
amr.max_step = 40000
amr.dt_shrink = 0.01
amr.stop_time = 110
amr.cfl = 0.25
#amr.fixed_dt = 1.0e-5
amr.derive_plot_vars = avg_pressure mag_vort mass_fractions

peleLM.chem_integrator = "ReactorCvode"
peleLM.use_typ_vals_chem = 1 # Use species/temp typical values in CVODE
ode.rtol = 1.0e-6 # Relative tolerance of the chemical solve
ode.atol = 1.0e-5 # Absolute tolerance factor applied on typical values
cvode.solve_type = denseAJ_direct # CVODE Linear solve type (for Newton direction)
cvode.max_order = 2 # CVODE max BDF order.

godunov.use_ppm = 0
godunov.use_forceInTrans = 0

nodal_proj.verbose = 0
mac_proj.verbose = 0
peleLM.deltaT_verbose = 1
peleLM.deltaT_tol = 4e-5

peleLM.do_temporals = 1
peleLM.do_mass_balance = 1

#--------------------REFINEMENT CONTROL------------------------
amr.refinement_indicators = heatrelease
amr.heatrelease.max_level = 2
amr.heatrelease.value_greater = 1.50e+3
amr.heatrelease.field_name = HeatRelease

[baperry2]

from the output plt file I found that only 256 cores were actually running

Can you elaborate on this?

Note that the domain decomposition across processors is handled by the AMReX library. The controls below impact how the grid is broken up into blocks and distributed across processors, see the AMReX documentation for more details. The optimal settings are highly dependent on both your problem and the hardware you are running on, so unfortunately optimizing performance can require a bit of experimentation.

amr.n_error_buf = 2 2 2 2 # number of buffer cells in error est
amr.grid_eff = 0.7 # what constitutes an efficient grid
amr.blocking_factor = 8 # block factor in grid generation (min box size)
amr.max_grid_size = 32 # max box size

[Alex-jian522]

Thanks for your reply! I will try it and I think the reason may be the complex chemical mechanism.