UnicodeDecodeError: 'utf-8' codec can't decode in run_command

[oesteban]

Running FMRIPREP using singularity.

Node: _autorecon30
Working directory: /scratch/03763/oesteban/fmriprep-phase2/work/ds000172/sub-control01/fmriprep_wf/single_subject_control01_wf/anat_preproc_wf/surface_recon_wf/autorecon_resume_wf/autorecon3/mapflow/_autorecon
30

Node inputs:

T1_files = <undefined>
T2_file = <undefined>
args = <undefined>
big_ventricles = <undefined>
brainstem = <undefined>
directive = autorecon3
environ = {}
expert = <undefined>
flags = <undefined>
hemi = lh
hippocampal_subfields_T1 = <undefined>
hippocampal_subfields_T2 = <undefined>
hires = <undefined>
ignore_exception = False
mprage = <undefined>
mri_aparc2aseg = <undefined>
mri_ca_label = <undefined>
mri_ca_normalize = <undefined>
mri_ca_register = <undefined>
mri_edit_wm_with_aseg = <undefined>
mri_em_register = <undefined>
mri_fill = <undefined>
mri_mask = <undefined>
mri_normalize = <undefined>
mri_pretess = <undefined>
mri_remove_neck = <undefined>
mri_segment = <undefined>
mri_segstats = <undefined>
mri_tessellate = <undefined>
mri_watershed = <undefined>
mris_anatomical_stats = <undefined>
mris_ca_label = <undefined>
mris_fix_topology = <undefined>
mris_inflate = <undefined>
mris_make_surfaces = <undefined>
mris_register = <undefined>
mris_smooth = <undefined>
mris_sphere = <undefined>
mris_surf2vol = <undefined>
mrisp_paint = <undefined>
openmp = 8
parallel = <undefined>
subject_id = sub-control01
subjects_dir = /work/03843/crn_plab/stampede2/fmriprep-phase2/derivatives/ds000172/freesurfer
talairach = <undefined>
terminal_output = <undefined>
use_T2 = False
xopts = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 51, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 407, in run
    self._run_interface()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 517, in _run_interface
    self._result = self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 650, in _run_command
    result = self._interface.run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1088, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1654, in _run_interface
    runtime = run_command(runtime, output=self.terminal_output)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base.py", line 1423, in run_command
    result['stdout'] = stdout.decode(default_encoding).split('\n')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc7 in position 9789: invalid continuation byte

[oesteban]

default_encoding:

nid00024(3)$ singularity exec /work/03843/crn_plab/singularity/poldracklab_fmriprep_1.0.0-rc7-747-2017-10-17-cc920f8dcef8.img python -c "import sys; print(sys.getdefaultencoding())"
utf-8

(and from sys)

nid00024(4)$ singularity exec /work/03843/crn_plab/singularity/poldracklab_fmriprep_1.0.0-rc7-747-2017-10-17-cc920f8dcef8.img python -c "import locale; print(locale.getdefaultlocale()[1])"
UTF-8

[oesteban]

We could use chardet as a fallback for these decodings:

python -c "import chardet; print(chardet.detect(bytearray([0xc7])));"
{'encoding': 'ISO-8859-1', 'confidence': 0.73, 'language': ''}

[satra]

@oesteban - do you know where these characters came from?

[oesteban]

Running FreeSurfer's autorecon, but no idea why that character should be there. Does not make much sense. However, a corrupted output is better than no output at all right?

[satra]

yes - decode has an option to take characters that it didn't read and replace them with some properly encoded character.

@yarikoptic - we did this for datalad, do you have a quick pointer?

[oesteban]

that'd be sweet

[oesteban]

This error was indeed fixed. We have some corrupted FreeSurfer runs. They would send this to the standard error:

mri_label2label.bin: could not parse 3880th line 'hB5t���?���}���?`+(^F�F�?�\�;$L�?F^X��N*^G@0{�^WJ^Q�?�^R`h���?C�F�^SO�?�z^Y�|D�?^D^V^?�y�?D^N�����?|4�1a��?' in label file

(thanks to this patch I could get this written to the logs)

[satra]

@oesteban - awesome - i think this will also help with a weird unicode error that pops up from matlab whenever spm crashes.