-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Investigate Celery SegFaults on Staging #3089
Comments
I downloaded the Lower Schuylkill HUC-10 for testing, and then created this test script: # test.py
import os
import json
from cStringIO import StringIO
from gwlfe import gwlfe, Parser
filename = 'huc10__1341.json'
filepath = os.path.abspath(filename)
with open(filepath, 'r') as input_json:
mapshed_data = json.load(input_json)
# Round Areas
mapshed_areas = [round(a, 1) for a in mapshed_data['Area']]
mapshed_data['Area'] = mapshed_areas
# Prepare Input GMS
pre_z = Parser.DataModel(mapshed_data)
output = StringIO()
writer = Parser.GmsWriter(output)
writer.write(pre_z)
output.seek(0)
# Read Input GMS
reader = Parser.GmsReader(output)
z = reader.read()
# Run the Model
result, _ = gwlfe.run(z)
# Write to file
outpath = os.path.abspath('output.json')
with open(outpath, 'w') as outfile:
json.dump(result, outfile) which runs fine on my local Worker VM:
But when I run it on Staging I get this:
So the suspicion that it was related to the latest GWLF-E has been confirmed. Proceeding to investigate |
I've destroyed my Worker VM locally and am re-creating it from scratch to see if I can reproduce this. |
🎉 After I destroyed and recreated my Worker locally, I can now reproduce this: $ vagrant ssh worker -c 'cd /vagrant/scratch/celery-segfault/ && python test.py'
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
ImportError: No module named _multiarray_umath
bash: line 1: 14222 Segmentation fault (core dumped) python test.py
Connection to 127.0.0.1 closed. |
Some threads:
indicated that the problem is in one of the underlying libraries, which is building against a more modern version of numpy, which gets replaced later by the one we specify. I tried changing the order of installation: diff --git a/src/mmw/requirements/base.txt b/src/mmw/requirements/base.txt
index 115013e3..c69ffcb5 100644
--- a/src/mmw/requirements/base.txt
+++ b/src/mmw/requirements/base.txt
@@ -14,6 +14,7 @@ django-cors-headers==2.1.0
cryptography==2.1.4
pyOpenSSL==17.4.0
markdown==2.6.9
+numpy==1.14.5
tr55==1.3.0
gwlf-e==2.0.0
requests[security]==2.9.1
@@ -24,7 +25,6 @@ https://bitbucket.org/jurko/suds/get/94664ddd46a6.tar.gz#egg=suds-jurko
django_celery_results==1.0.1
pandas==0.22.0
git+git://github.com/emiliom/ulmo@wml_values_md#egg=ulmo
-numpy==1.14.5
hs_restclient==1.2.10
six==1.11.0
fiona==1.7.11 so that whatever is using the more recent version of numpy in its installation (either But it did not help. |
…ults Circumvent GWLF-E SegFaults due to NumPy Connects #3089
On staging we're seeing segfaults like this:
particularly around GWLF-E execution for MapShed Stage 2. These errors are not happening in local or production. Investigate and resolve.
The text was updated successfully, but these errors were encountered: