I/O error when running calwebb_dark pipeline

[stscijgbot-jp]

Issue JP-1805 was created on JIRA by Misty Cracraft:

I tried to run the file jw02201001001_01101_00001_MIRIMAGE_uncal.fits through calwebb_dark on the command line and ended up with it crashing due to I/O errors.

Here's the command:

$ strun calwebb_dark.cfg jw02201001001_01101_00001_MIRIMAGE_uncal.fits

Here is the error:

2020-12-10 13:55:36,632 - stpipe.DarkPipeline - INFO - Starting calwebb_dark ...
2020-12-10 13:55:37,844 - stpipe.DarkPipeline.group_scale - INFO - Step group_scale running with args (<RampModel(4, 40, 1024, 1032) from jw02201001001_01101_00001_MIRIMAGE_uncal.fits>,).
2020-12-10 13:55:37,845 - stpipe.DarkPipeline.group_scale - INFO - Step group_scale parameters are: \{'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2020-12-10 13:55:37,858 - stpipe.DarkPipeline.group_scale - INFO - NFRAMES=1 is a power of 2; correction not needed
2020-12-10 13:55:37,859 - stpipe.DarkPipeline.group_scale - INFO - Step will be skipped
2020-12-10 13:55:37,859 - stpipe.DarkPipeline.group_scale - INFO - Step group_scale done
2020-12-10 13:55:37,901 - stpipe.DarkPipeline.dq_init - INFO - Step dq_init running with args (<RampModel(4, 40, 1024, 1032) from jw02201001001_01101_00001_MIRIMAGE_uncal.fits>,).
2020-12-10 13:55:37,902 - stpipe.DarkPipeline.dq_init - INFO - Step dq_init parameters are: \{'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2020-12-10 13:55:37,943 - stpipe.DarkPipeline.dq_init - INFO - Using MASK reference file ████████████████████████████████████████████████████████
----------------------------------------------------------------------
ERROR RUNNING STEP 'DarkPipeline':
 I/O operation on closed file
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/Users/cracraft/anaconda3/envs/jwst_b76/bin/strun", line 26, in <module>
 step = Step.from_cmdline(sys.argv[1:])
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/stpipe/step.py", line 163, in from_cmdline
 return cmdline.step_from_cmdline(args)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/stpipe/cmdline.py", line 346, in step_from_cmdline
 step.run(*positional)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/stpipe/step.py", line 399, in run
 step_result = self.process(*args)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/pipeline/calwebb_dark.py", line 64, in process
 input = self.dq_init(input)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/stpipe/step.py", line 399, in run
 step_result = self.process(*args)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/dq_init/dq_init_step.py", line 77, in process
 result = dq_initialization.correct_model(input_model, mask_model)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/dq_init/dq_initialization.py", line 34, in correct_model
 output_model = do_dqinit(input_model, mask_model)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/dq_init/dq_initialization.py", line 60, in do_dqinit
 output_model = input_model.copy()
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/datamodels/model_base.py", line 352, in copy
 self.clone(result, self, deepcopy=True, memo=memo)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/jwst/datamodels/model_base.py", line 331, in clone
 instance = copy.deepcopy(source._instance, memo=memo)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/copy.py", line 180, in deepcopy
 y = _reconstruct(x, memo, *rv)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/copy.py", line 307, in _reconstruct
 value = deepcopy(value, memo)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/copy.py", line 159, in deepcopy
 copier = getattr(x, "__deepcopy__", None)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/asdf/tags/core/ndarray.py", line 362, in __getattr__
 return getattr(self._make_array(), attr)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/asdf/tags/core/ndarray.py", line 260, in _make_array
 self._shape, self._strides, self._dtype, len(block))
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/asdf/fits_embed.py", line 39, in __len__
 return self._hdu.data.nbytes
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/astropy/utils/decorators.py", line 758, in __get__
 val = self.fget(obj)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/astropy/io/fits/hdu/image.py", line 230, in data
 data = self._get_scaled_image_data(self._data_offset, self.shape)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/astropy/io/fits/hdu/image.py", line 701, in _get_scaled_image_data
 raw_data = self._get_raw_data(shape, code, offset)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/astropy/io/fits/hdu/base.py", line 520, in _get_raw_data
 return self._file.readarray(offset=offset, dtype=code, shape=shape)
 File "/Users/cracraft/anaconda3/envs/jwst_b76/lib/python3.7/site-packages/astropy/io/fits/file.py", line 304, in readarray
 filepos = self._file.tell()
ValueError: I/O operation on closed file

 

Here is some feedback from Jonathan on the problem.
That is symptomatic of the datamodel i/o efficiency changes done a few months ago. Please file an issue, with both the pointer to the data and the command which causes the error.In the meantime, to disable some of the I/O efficiencies, set the environmental variable SKIP_FITS_UPDATE to “false”. Then, you should not see the error. In bash, the command would be:
export SKIP_FITS_UPDATE=false
 

That has allowed the step to run.

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Resolved by JWST PR#⁠5692 and STDATAMODELS PR#⁠59

[stscijgbot-jp]

Comment by Misty Cracraft on JIRA:

I ran the command stated in the description first with the released build 7.7 and did see the problem, then tested again with a dev version created today, and the command ran successfully with no errors.

From this test, I will say that the problem was fixed and close out the ticket.