Resolve segmentation fault in clisops PR 179

[agstephens]

• clisops version: 0.6.5 - branch: regrid-main-ag
• Python version: 3.8
• Operating System: centos7

Description

Trying to debug why we get a segmentation fault in the clisop Pull Request: #179

I did this:

# Install clisops from scratch

## Create VM

Create a VM with 2GB RAM and 2 CPUs

## Install clisops and environment and run tests

--
mkdir ~/clisops-install
cd ~/clisops-install/

wget https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh
bash Miniconda3-py38_4.10.3-Linux-x86_64.sh -b -p ~/clisops-install/miniconda

export PATH=${PWD}/miniconda/bin:$PATH

git clone https://github.com/roocs/clisops
cd clisops
git checkout regrid-main-ag

time conda env create -f environment.yml
# Took 7 minutes
source activate clisops
pip install -e ".[dev]"
### pip install --upgrade git+https://github.com/pangeo-data/xESMF.git
pip install pangeo-xesmf==0.6.0


pytest --cov tests
--

And yes, @sol1105 , I get a segmentation fault here too!
Now, I just need to compare the two environments.

[agstephens]

I have fixed python=3.8. Latest results:

• run all: pytest tests - segfault
• run core tests: pytest tests/core - regrid tests fail (?)
  • can be fixed by referencing .data in:

clisops/core/regrid.py:513: in grid_reformat
ds_ref = xr.Dataset(

- run only core subset tests: tests pass

[agstephens]

The order the tests are run in seems to affect the result:

1. No segfault with:

pytest tests/core/test_average.py tests/core/test_regrid.py tests/core/test_subset.py tests/ops/test_average.py tests/ops/test_subset.py tests/ops/test_xarray_mean.py tests/ops/test_regrid.py tests/test_config.py tests/test_dataset_roll.py tests/test_file_namers.py tests/test_output_utils.py tests/test_roll.py tests/test_utils/test_dataset_utils.py

2. Segfault with:

pytest  # to run all tests

3. Segfaults happen with these combinations:

pytest tests/test_config.py tests/test_dataset_roll.py tests/test_file_namers.py tests/test_output_utils.py tests/test_roll.py tests/core/test_average.py tests/core/test_regrid.py tests/core/test_subset.py

and

pytest tests/test_config.py tests/test_dataset_roll.py tests/test_file_namers.py tests/test_output_utils.py tests/test_roll.py tests/core/test_subset.py

and

pytest tests/test_config.py tests/test_dataset_roll.py tests/test_file_namers.py  tests/core/test_subset.py

pytest tests/test_config.py tests/test_file_namers.py tests/core/test_subset.py

pytest tests/test_file_namers.py tests/core/test_subset.py

But, THESE SUCCEED:

pytest tests/test_config.py tests/test_dataset_roll.py tests/core/test_subset.py

pytest tests/test_config.py tests/core/test_subset.py

pytest tests/core/test_subset.py

--

Does python -m pytest make any difference? No, it doesn't.

[agstephens]

So, to summarise...

FAILS:

pytest tests/test_file_namers.py tests/core/test_subset.py

SUCCEEDS:

pytest tests/test_file_namers.py
pytest tests/core/test_subset.py
pytest tests/core/test_subset.py tests/test_file_namers.py

ASSUMED REASON:

• an interaction is causing the seg fault, between the two test modules when run in this order:
  • tests/test_file_namers.py
  • tests/core/test_subset.py

[agstephens]

@cehbrecht: please can you try to reproduce the problem outlined here. The first part of this issue explains how to set up the environment. Please confirm that you get a segfault that matches my experience above...

FAILS:

pytest tests/test_file_namers.py tests/core/test_subset.py

SUCCEEDS:

pytest tests/test_file_namers.py
pytest tests/core/test_subset.py
pytest tests/core/test_subset.py tests/test_file_namers.py

[cehbrecht]

I can reproduce the segfault error in a Debian docker container. Well, it works on my mac ;)

On my dev branch I can avoid the segfault by removing the regrid import:

clisops/clisops/core/__init__.py

Line 1 in 996bf9e

# from .regrid import Grid, Weights, regrid

and use the direct import of regrid:

clisops/clisops/ops/regrid.py

Lines 7 to 8 in 996bf9e

	from clisops.core.regrid import Grid, Weights
	from clisops.core.regrid import regrid as core_regrid

With these changes, the tests are working:

pytest tests
pytest tests/test_file_namers.py tests/core/test_subset.py

[cehbrecht]

you can also switch subset and regrid imports:

clisops/clisops/core/__init__.py

Lines 1 to 11 in 50a1094

	from .subset import (
	create_mask,
	subset_bbox,
	subset_gridpoint,
	subset_level,
	subset_shape,
	subset_time,
	)
	from .regrid import Grid, Weights, regrid

[agstephens]

This final fix works, as long we use: pangeo-xesmf==0.6.0

But there are tests failing. Need to now look into these...

[agstephens]

Hi @cehbrecht, I've made some progress on this stuff. Can you remember why we/you set the rules for which GitHub Actions are allowed to fail in this workflow:

https://github.com/roocs/clisops/blob/master/.github/workflows/main.yml#L26-L42

[cehbrecht]

I think it is a copy+paste thing. There is no py37-xarray env in tox.ini. We can probably change the env py37-xarray to py37 and it should not fail:

clisops/.github/workflows/main.yml

Lines 34 to 36 in 1280f27

	- python-version: 3.7
	tox-env: py37-xarray
	allowed_to_fail: true

ping @Zeitsperre

[agstephens]

@Zeitsperre I think we got the different CI environments from xclim initially. We now have these various allowed_to_fail options. Do you think it is okay for any of these to fail? I have lost track whether some of them always will because of dependencies used in the subsetting by shape/polygon. Thanks

[Zeitsperre]

@cehbrecht There should be less of a likelihood of segmentation faults now that read operations are separated by worker and cache as of #345. Closing this.