Program received signal SIGABRT: Process abort signal.

[jinjiuping]

Describe the bug
I got the problem when I runing a co2 data assimilation experiment in wrf.exe by WRF Version4.1.5. The max_dom of this experiment is 3, the error information shows as below:

Timing for Writing wrfout_d01_2018-01-01_06:00:00 for domain 1: 4.22605 elapsed seconds
d01 2018-01-01_06:00:00 wrf: SUCCESS COMPLETE WRF
corrupted size vs. prev_size

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0 0x7f0779115680 in ???
#1 0x7f0779114885 in ???
#2 0x7f0777fc303f in ???
#3 0x7f0777fc2fb7 in ???
#4 0x7f0777fc4920 in ???
#5 0x7f077800d966 in ???
#6 0x7f07780149d9 in ???
#7 0x7f0778014bdb in ???
#8 0x7f077801c12a in ???
#9 0x7f0776e71bdc in H5MM_xfree
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5MM.c:555
#10 0x7f0776dcc1d7 in H5FL__blk_gc_list
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:1218
#11 0x7f0776dcc2ab in H5FL__blk_gc
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:1266
#12 0x7f0776dccb17 in H5FL_garbage_coll
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:2457
#13 0x7f0776dccd34 in H5FL_term_package
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:184
#14 0x7f0776c5476c in H5_term_library
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:370
#15 0x7f0776c54939 in H5_mpi_delete_cb
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:674
#16 0x7f0778661fc2 in ???
#17 0x7f07786619e5 in ???
#18 0x7f0778661cbe in ???
#19 0x7f0778622713 in ???
#20 0x7f0778b1b708 in ???
#21 0x55b7925685bd in ???
#22 0x55b7926f9fe6 in ???
#23 0x55b7914a6155 in ???
#24 0x7f0777fa5bf6 in ???
#25 0x55b7914a6189 in ???
#26 0xffffffffffffffff in ???

[stacywalters]

jinjiuping, There is little information in the above output. It is important to know what version of WRF you are running. Stacy

…

On Mon, Dec 27, 2021 at 1:24 AM jinjiuping ***@***.***> wrote: *Describe the bug* I got the problem when I runing a co2 data assimilation experiment. The max_dom of this experiment is 3, the error information shows as below: Timing for Writing wrfout_d01_2018-01-01_06:00:00 for domain 1: 4.22605 elapsed seconds d01 2018-01-01_06:00:00 wrf: SUCCESS COMPLETE WRF corrupted size vs. prev_size Program received signal SIGABRT: Process abort signal. Backtrace for this error: #0 0x7f0779115680 in ??? #1 <#1> 0x7f0779114885 in ??? #2 <#2> 0x7f0777fc303f in ??? #3 <#3> 0x7f0777fc2fb7 in ??? #4 <#4> 0x7f0777fc4920 in ??? #5 <#5> 0x7f077800d966 in ??? #6 <#6> 0x7f07780149d9 in ??? #7 <#7> 0x7f0778014bdb in ??? #8 <#8> 0x7f077801c12a in ??? #9 <#9> 0x7f0776e71bdc in H5MM_xfree at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5MM.c:555 #10 <#10> 0x7f0776dcc1d7 in H5FL__blk_gc_list at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:1218 #11 <#11> 0x7f0776dcc2ab in H5FL__blk_gc at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:1266 #12 <#12> 0x7f0776dccb17 in H5FL_garbage_coll at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:2457 #13 <#13> 0x7f0776dccd34 in H5FL_term_package at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:184 #14 <#14> 0x7f0776c5476c in H5_term_library at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:370 #15 <#15> 0x7f0776c54939 in H5_mpi_delete_cb at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:674 #16 <#16> 0x7f0778661fc2 in ??? #17 <#17> 0x7f07786619e5 in ??? #18 <#18> 0x7f0778661cbe in ??? #19 <#19> 0x7f0778622713 in ??? #20 <#20> 0x7f0778b1b708 in ??? #21 <#21> 0x55b7925685bd in ??? #22 <#22> 0x55b7926f9fe6 in ??? #23 <#23> 0x55b7914a6155 in ??? #24 <#24> 0x7f0777fa5bf6 in ??? #25 <#25> 0x55b7914a6189 in ??? #26 <#26> 0xffffffffffffffff in ??? — Reply to this email directly, view it on GitHub <#1620>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACA4BEVVZHVDV2ZW7JJN7JLUTABCZANCNFSM5KZZXXMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jinjiuping]

I run the WRF v4.1.5, and the namelist.input of wrf.exe :

&time_control
run_days = 0,
run_hours = 6,
run_minutes = 0,
run_seconds = 0,
start_year = 2017,2017,2017,
start_month = 01,01,01,
start_day = 02,02,02,
start_hour = 00,00,00,
start_minute = 00,00,00,
start_second = 00,00,00,
end_year = 2017,2017,2017,
end_month = 01,01,01,
end_day = 02,02,02,
end_hour = 06,06,06,
end_minute = 00,00,00,
end_second = 00,00,00,
interval_seconds = 21600,
input_from_file = .true.,.true.,.true.,
fine_input_stream = 0,0,0,
history_interval = 60,60,60,
frames_per_outfile = 1,1,1,
restart = .false.,
restart_interval = 1440,
io_form_history = 2,
io_form_restart = 2,
io_form_input = 2,
io_form_boundary = 2,
auxinput5_inname = 'wrfchemi_d_',
auxinput5_interval_m = 180,180,180,
frames_per_auxinput5 = 1,1,1,
io_form_auxinput5 = 2,
debug_level = 0,
force_use_old_data = true,
/
&domains
time_step = 60,
time_step_fract_num = 0,
time_step_fract_den = 1,
max_dom = 3,
s_we = 1,1,1,
e_we = 240,226,214,
s_sn = 1,1,1,
e_sn = 210,196,184,
s_vert = 1,1,1,
e_vert = 51,51,51,
num_metgrid_levels = 27,
num_metgrid_soil_levels = 4,
dx = 27000,9000,3000,
dy = 27000,9000,3000,
grid_id = 1,2,3,
parent_id = 0,1,2,
i_parent_start = 1,83,78,
j_parent_start = 1,73,68,
parent_grid_ratio = 1,3,3,
parent_time_step_ratio = 1,3,3,
feedback = 0,
smooth_option = 0,
smooth_cg_topo = .true.,
lagrange_order = 2,
interp_type = 2,
extrap_type = 2,
t_extrap_type = 2,
use_surface = .true.,
use_levels_below_ground = .true.,
lowest_lev_from_sfc = .false.,
force_sfc_in_vinterp = 1,
zap_close_levels = 500,
interp_theta = .false.,
hypsometric_opt = 2,
p_top_requested = 5000.,
eta_levels = -1,
/
&physics
mp_physics = 4,4,4,
ra_lw_physics = 1,1,1,
ra_sw_physics = 1,1,1,
radt = 50,9,9,
sf_sfclay_physics = 5,5,5,
sf_surface_physics = 2,2,2,
bl_pbl_physics = 5,5,5,
bldt = 0,0,0,
cu_physics = 1,1,0,
cudt = 5,5,5,
cu_diag = 0,0,
isfflx = 1,
ifsnow = 1,
icloud = 1,
num_soil_layers = 4,
num_land_cat = 21,
sf_urban_physics = 0,0,0,
/
&dfi_control
/
&tc
/
&scm
/
&dynamics
w_damping = 1,
diff_opt = 2,1,1,
diff_6th_opt = 2,0,0,
diff_6th_factor = 0.12,0.12,0.12,
km_opt = 4,4,4,
damp_opt = 0,
zdamp = 5000,5000,5000,
dampcoef = 0.2,0.2,0.2,
epssm = 0.3,0.3,0.3,
non_hydrostatic = .true.,.true.,.true.,
moist_adv_opt = 2,2,2,
scalar_adv_opt = 2,2,2,
chem_adv_opt = 2,2,2,
tke_adv_opt = 2,2,2,
h_mom_adv_order = 5,5,5,
v_mom_adv_order = 3,3,3,
h_sca_adv_order = 5,5,5,
v_sca_adv_order = 3,3,3,

/
&bdy_control
spec_bdy_width = 5,
spec_zone = 1,
relax_zone = 4,
specified = .true.,.false.,.false.,
nested = .false.,.true.,.true.,
/
&grib2
/
&namelist_quilt
nio_tasks_per_group = 0,
nio_groups = 1,
/
&chem
kemit = 1,
chem_opt = 16,16,16,
chemdt = 6,2,2,
io_style_emissions = 2,
emiss_inpt_opt = 16,16,16,
emiss_opt = 16,16,16,
emiss_opt_vol = 0,0,0,
chem_in_opt = 0,0,0,
gas_bc_opt = 1,1,1,
gas_ic_opt = 1,1,1,
vertmix_onoff = 1,1,1,
chem_conv_tr = 0,0,0,
have_bcs_chem = .true.,.true.,.true.,
vprm_opt = 'VPRM_table_US','VPRM_table_US','VPRM_table_US',
/

And when run the wrf.exe, there produce a file "fort.98" in same director, the content of this file is:

**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********
**** OUT OF BOUNDS *********

[jinjiuping]

I used the .ksh to run WRF, and the error happend in wrfchem_initial/ . At the same condition, I tried the max_dom = 1, success; but when max_dom = 2, there happened a same error as max_dom = 3.
when i modify the epssm = 0.5, I got the error :

munmap_chunk(): invalid pointer

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0 0x152956afc680 in ???
#1 0x152956afb885 in ???
#2 0x1529559aa03f in ???
#3 0x1529559a9fb7 in ???
#4 0x1529559ab920 in ???
#5 0x1529559f4966 in ???
#6 0x1529559fb9d9 in ???
#7 0x152955a02fbb in ???
#8 0x152954858bdc in H5MM_xfree
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5MM.c:555
#9 0x1529547b253e in H5FL__reg_gc_list
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:544
#10 0x1529547b27ef in H5FL__reg_gc
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:589
#11 0x1529547b3011 in H5FL_reg_free
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:387
#12 0x1529546ddc45 in H5D__btree_shared_free
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:589
#11 0x1529547b3011 in H5FL_reg_free
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:387
#12 0x1529546ddc45 in H5D__btree_shared_free
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5Dbtree.c:805
#13 0x15295492f705 in H5UC_decr
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5UC.c:118
#14 0x15295466ffcd in H5B__node_dest
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5B.c:2085
#15 0x152954672c6c in H5B__cache_free_icr
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5Bcache.c:382
#16 0x1529546a0ce6 in H5C__flush_single_entry
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5C.c:6373
#17 0x1529546a1a87 in H5C_flush_invalidate_ring
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5C.c:5537
#18 0x1529546a23bb in H5C__flush_invalidate_cache
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5C.c:5131
#19 0x1529546a277d in H5C_dest
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5C.c:852
#20 0x152954662dcb in H5AC_dest
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5AC.c:532
#21 0x152954773363 in H5F__dest
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5Fint.c:1328
#22 0x1529547743b8 in H5F_try_close
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5Fint.c:2229
#23 0x15295477488a in H5F__close
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5Fint.c:2056
#24 0x152954a7baba in H5VL__native_file_close
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5VLnative_file.c:883
#25 0x152954a52070 in H5VL__file_close
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5VLcallback.c:3944
#26 0x152954a60f80 in H5VL_file_close
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5VLcallback.c:3976
#27 0x15295475e574 in H5F__close_cb
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5F.c:242
#28 0x1529548364aa in H5I_dec_ref
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5I.c:1376
#29 0x152954836598 in H5I_dec_app_ref
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5I.c:1421
#30 0x152954761c2e in H5Fclose
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5F.c:885
#31 0x1529556cf776 in ???
#32 0x1529556cf94f in ???
#33 0x1529556d005f in ???
#34 0x15295567308a in ???
#35 0x152956fa56ad in ???
#36 0x557342f6a204 in ???
#37 0x5573410a8f23 in ???
#38 0x557340e852ed in ???
#39 0x557340f8dc0e in ???
#40 0x55733effc6a6 in ???
#41 0x55733effc155 in ???
#42 0x15295598cbf6 in ???
#43 0x55733effc189 in ???
#44 0xffffffffffffffff in ???

[jinjiuping]

When I used the DEBUG mode in WRF, I got this error:

free(): invalid pointer

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0 0x14fe5f7ec680 in ???
#1 0x14fe5f7eb885 in ???
#2 0x14fe5e69a03f in ???
#3 0x14fe5e699fb7 in ???
#4 0x14fe5e69b920 in ???
#5 0x14fe5e6e4966 in ???
#6 0x14fe5e6eb9d9 in ???
#7 0x14fe5e6f2f0b in ???
#8 0x14fe5d548bdc in H5MM_xfree
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5MM.c:555
#9 0x14fe5d4a274e in H5FL__fac_gc_list
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:2265
#10 0x14fe5d4a38c9 in H5FL_fac_term
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5FL.c:2352
#11 0x14fe5d658620 in H5SL_term_package
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5SL.c:658
#12 0x14fe5d32b6a5 in H5_term_library
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:367
#13 0x14fe5d32b939 in H5_mpi_delete_cb
at /home/jinjp/wrf/build/hdf5-1.12.0/src/H5.c:674
#14 0x14fe5ed38fc2 in ???
#15 0x14fe5ed389e5 in ???
#16 0x14fe5ed38cbe in ???
#17 0x14fe5ecf9713 in ???
#18 0x14fe5f1f2708 in ???
#19 0x55e3e1ec92ab in wrf_dm_shutdown_
at /share/nfs2_share6/RUNTIME_WRF/wrf/WRF/frame/module_dm.f90:6699
#20 0x55e3e209fd7d in wrf_shutdown_
at /share/nfs2_share6/RUNTIME_WRF/wrf/WRF/frame/wrf_shutdown.f90:9
#21 0x55e3e0e03076 in __module_wrf_top_MOD_wrf_finalize
at ../main/module_wrf_top.f90:365
#22 0x55e3e0e020e6 in wrf
at /share/nfs2_share6/RUNTIME_WRF/wrf/WRF/main/wrf.f90:33
#23 0x55e3e0e02133 in main
at /share/nfs2_share6/RUNTIME_WRF/wrf/WRF/main/wrf.f90:6

[davegill]

@jinjiuping @stacywalters

The github issues page is for incorrect model behaviors (this field is zero at model start), recently introduced problems (this used to work in v4.1.2), potential bugs (the model should be using total pressure, not dry pressure in this calculation).

This belongs on the WRF Forum, where users get information about problems with the WRF model. Please repost this on www.forum.mmm.ucar.edu, under the WRF Chem section.

What you are going to be asked to do on the forum is what we would do ourselves: simplify the modeling system problem until it works, then step by step add complexity back into the model setup to see what causes the trouble. For example, if you run a single domain with no chemistry, does it work? If yes, then you have a starting position.

I am closing this issue.