awswrangler import error with multiprocessing

[brandonwbrown]

on python:3.7-slim, python:3.7 and python:3.8 import awswrangler suddenly starts failing.

pip3 install --no-cache-dir --trusted-host pypi.python.org awswrangler

Behavior:

import awswrangler as wr

Error:

from awswrangler.pandas import Pandas  # noqa

File "/usr/local/lib/python3.7/site-packages/awswrangler/pandas.py", line 45, in
class Pandas:
File "/usr/local/lib/python3.7/site-packages/awswrangler/pandas.py", line 273, in Pandas
def _read_csv_once_remote(send_pipe: mp.connection.Connection, session_primitives: "SessionPrimitives",
AttributeError: module 'multiprocessing' has no attribute 'connection'

this works with image python:3.6

also posted at: aws/aws-sdk-pandas#151

[wglambert]

Can you give all the commands you ran and any relevant contextual information or files for reproducing the issue

From the line you gave I can't reproduce

$ docker build . -t python:test
Sending build context to Docker daemon  38.15MB
Step 1/2 : FROM python:3.7-slim
 ---> 69afd9568c9d
Step 2/2 : RUN pip3 install --no-cache-dir --trusted-host pypi.python.org awswrangler
 ---> Running in 17d2c87d766c
Collecting awswrangler
  Downloading awswrangler-0.3.2.tar.gz (61 kB)
Collecting numpy~=1.18.1
  Downloading numpy-1.18.2-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
Collecting pandas~=1.0.1
  Downloading pandas-1.0.3-cp37-cp37m-manylinux1_x86_64.whl (10.0 MB)
Collecting pyarrow~=0.16.0
  Downloading pyarrow-0.16.0-cp37-cp37m-manylinux2014_x86_64.whl (63.1 MB)
Collecting botocore>=1.13.34
  Downloading botocore-1.15.31-py2.py3-none-any.whl (6.0 MB)
Collecting boto3>=1.10.34
  Downloading boto3-1.12.31-py2.py3-none-any.whl (128 kB)
Collecting s3fs~=0.4.0
  Downloading s3fs-0.4.1-py3-none-any.whl (19 kB)
Collecting tenacity~=6.0.0
  Downloading tenacity-6.0.0-py2.py3-none-any.whl (24 kB)
Collecting pg8000~=1.13.2
  Downloading pg8000-1.13.2-py3-none-any.whl (26 kB)
Collecting pymysql~=0.9.3
  Downloading PyMySQL-0.9.3-py2.py3-none-any.whl (47 kB)
Collecting pytz>=2017.2
  Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)
Collecting python-dateutil>=2.6.1
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting six>=1.0.0
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3<1.26,>=1.20; python_version != "3.4"
  Downloading urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Collecting docutils<0.16,>=0.10
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.9.5-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
Collecting fsspec>=0.6.0
  Downloading fsspec-0.7.0-py3-none-any.whl (66 kB)
Collecting scramp==1.1.0
  Downloading scramp-1.1.0-py3-none-any.whl (5.9 kB)
Building wheels for collected packages: awswrangler
  Building wheel for awswrangler (setup.py): started
  Building wheel for awswrangler (setup.py): finished with status 'done'
  Created wheel for awswrangler: filename=awswrangler-0.3.2-glue-none-any.whl size=70123 sha256=1519efd521d60a51731e14c7a60b99f0ccea768ba5636eb40c3ae9cb70cd2c64
  Stored in directory: /tmp/pip-ephem-wheel-cache-fw5tkrgi/wheels/27/36/76/9fee91944b8cc13aca5589a9ca5dd1f0d0c01e35d0004ffa2d
Successfully built awswrangler
Installing collected packages: numpy, pytz, six, python-dateutil, pandas, pyarrow, urllib3, docutils, jmespath, botocore, s3transfer, boto3, fsspec, s3fs, tenacity, scramp, pg8000, pymysql, awswrangler
Successfully installed awswrangler-0.3.2 boto3-1.12.31 botocore-1.15.31 docutils-0.15.2 fsspec-0.7.0 jmespath-0.9.5 numpy-1.18.2 pandas-1.0.3 pg8000-1.13.2 pyarrow-0.16.0 pymysql-0.9.3 python-dateutil-2.8.1 pytz-2019.3 s3fs-0.4.1 s3transfer-0.3.3 scramp-1.1.0 six-1.14.0 tenacity-6.0.0 urllib3-1.25.8
Removing intermediate container 17d2c87d766c
 ---> b6331b16ffb9
Successfully built b6331b16ffb9
Successfully tagged python:test

Dockerfile

FROM python:3.7-slim
RUN pip3 install --no-cache-dir --trusted-host pypi.python.org awswrangler

[brandonwbrown]

Sorry I wasn’t more clear. The build happens just fine.

1. Run the container you built
2. python
3. import awswrangler as wr

You get the above behavior regarding multiprocessing

[igorborgest]

Thanks @brandonwbrown and @wglambert! We will fix it in the Wrangler's side and will be released among several others improvements in the version 1.0.0 April 10th!

[brandonwbrown]

Closing pending above