Error running "from google.cloud import storage" command

[ep-tpat]

Thank you for reporting an issue to google-cloud-python!

If you are reporting an issue or requesting a feature, please search the
existing open and closed issues to see if there is already work being done.

• https://github.com/GoogleCloudPlatform/google-cloud-python/issues
• http://stackoverflow.com/questions/tagged/google-cloud-python

If you can provide us with as much of the following information as possible it
will help us identify the cause of your issue more quickly.

---

Hello, I'm hoping you can help. I'm trying to run the Python Client Library for Google Compute and running into issues. See below for my information. Thanks for any help you can provide.

1. OS type and version
$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

2. Python version and virtual environment information python --version
$ python3 -V
Python 3.4.5

3. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
$ pip3 freeze | grep ^google
google-auth==1.1.1
google-cloud==0.27.0
google-cloud-bigquery==0.26.0
google-cloud-bigtable==0.26.0
google-cloud-core==0.26.0
google-cloud-datastore==1.2.0
google-cloud-dns==0.26.0
google-cloud-error-reporting==0.26.0
google-cloud-language==0.27.0
google-cloud-logging==1.2.0
google-cloud-monitoring==0.26.0
google-cloud-pubsub==0.27.0
google-cloud-resource-manager==0.26.0
google-cloud-runtimeconfig==0.26.0
google-cloud-spanner==0.26.0
google-cloud-speech==0.28.0
google-cloud-storage==1.3.2
google-cloud-translate==1.1.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.26.0
google-gax==0.15.15
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2

4. Stacktrace if available
- See below

5. Steps to reproduce
- from a fresh CentOS 7 VM in Google Compute, I ran the following commands:
sudo yum -y install epel-release
sudo yum -y install python34
sudo yum -y install python34-devel
sudo yum -y install python34-setuptools
sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip
sudo pip3 install --upgrade pip
pip3 install --upgrade google-cloud

- from python IDLE, ran the following command:
from google.cloud import storage
- error received:
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in
from google.cloud.storage.batch import Batch
File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in
from google.cloud import _helpers
File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in
from google.protobuf import duration_pb2
ImportError: No module named 'google.protobuf'

6. Code example
N/A
Using GitHub flavored markdown can help make your request clearer.
See: https://guides.github.com/features/mastering-markdown/

[lukesneeringer]

Is protobuf installed? (Your pip freeze command obscures this.)

[ep-tpat]

Hi Luke. Yes, protobuf seems to be installed:

$ pip3 freeze | egrep '^google|protobuf'
google-auth==1.1.1
google-cloud==0.27.0
google-cloud-bigquery==0.26.0
google-cloud-bigtable==0.26.0
google-cloud-core==0.26.0
google-cloud-datastore==1.2.0
google-cloud-dns==0.26.0
google-cloud-error-reporting==0.26.0
google-cloud-language==0.27.0
google-cloud-logging==1.2.0
google-cloud-monitoring==0.26.0
google-cloud-pubsub==0.27.0
google-cloud-resource-manager==0.26.0
google-cloud-runtimeconfig==0.26.0
google-cloud-spanner==0.26.0
google-cloud-speech==0.28.0
google-cloud-storage==1.3.2
google-cloud-translate==1.1.0
google-cloud-videointelligence==0.25.0
google-cloud-vision==0.26.0
google-gax==0.15.15
google-resumable-media==0.2.3
googleapis-common-protos==1.5.2
protobuf==3.4.0
$ python3
Python 3.4.5 (default, May 29 2017, 15:17:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.

from google.cloud import storage
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python3.4/site-packages/google/cloud/storage/init.py", line 37, in
from google.cloud.storage.batch import Batch
File "/usr/lib/python3.4/site-packages/google/cloud/storage/batch.py", line 29, in
from google.cloud import _helpers
File "/usr/lib/python3.4/site-packages/google/cloud/_helpers.py", line 33, in
from google.protobuf import duration_pb2
ImportError: No module named 'google.protobuf'

[lukesneeringer]

Hmm. I just tried on my machine with a new virtualenv and was successful.
Are you able to import google.protobuf directly? (Just trying to think of things to eliminate.)

[dhermes]

We've seen this issue quite a lot and it's almost always caused by a "bad" install, i.e. one where the correct .pth files don't get created.

@ep-tpat Try installing into a virtualenv, so you can verify the install works. Then from your global install (i.e. /usr/lib/python3.4/site-packages) you can work on fixing the issue. (Usually just pip install --ignore-installed protobuf will work.)

I am pre-emptively closing this since it's not an issue in our code (i.e. we can install it on lots of different machines) and it's a common issue we've seen with pip usage in the wild. I'm happy to re-open if this turns out to be something else.

Feel free to keep asking questions in this issue though, we're happy to help.

[ep-tpat]

Thanks for the help, guys. After installing virtualenv, it worked perfectly. So....any chance you can point me in the right direction to get the issue ironed out not using virtualenv? This was a freshly spun up Google VM running CentOS 7. I didn't do anything else to it other than install Python3, Pip and Google Cloud. Also note, I'm new to Python (only about 2-3 months). Commands below:

---

sudo yum -y install epel-release
sudo yum -y install python34
sudo yum -y install python34-devel
sudo yum -y install python34-setuptools
sudo python3 /usr/lib/python3.4/site-packages/easy_install.py pip
sudo pip3 install --upgrade pip
pip3 install --upgrade google-cloud

---

Thanks for any help you can offer.

[dhermes]

@ep-tpat

1. System Python isn't really a great place to set up an environment, pyenv / virtualenv really are good options
2. Did you try pip install --ignore-installed protobuf (will need to be with sudo if you use the yum installed Python)?
3. In an interpreter import google and print out google.__path__, it should give you an idea of where it is installed.
4. Look for "bad" namespace files in your site-packages (e.g. I see .../site-packages/protobuf-3.4.0-py3.4-nspkg.pth, but I want that there, i.e. it isn't bad / dangling from an uninstalled package)
5. Look for directories in your site-packages that start with google, but you didn't expect to be there (just ``.../site-packages/google/`).

[ep-tpat]

@dhermes

I'll stick with your first point. If virtualenv is a good option then I'll just move forward with that. Thanks for everyone's help!

[rsmith49]

@dhermes I'm also running into this problem (but with google-cloud-language), not sure if I should make a new issue or just continue on this thread.

OS Type and Version
Using AWS Elastic Beanstalk, EC2 instance with:
Amazon Linux AMI release 2017.03

Python Version and Virtual Environment Info
Python 3.4.3
Not sure what virtualenv version the EC2 instance is running, Elastic Beanstalk creates its own virtual environments on application deployment

google-cloud-python version
Current pip freeze output:

airbrake==2.1.0
appnope==0.1.0
APScheduler==3.3.1
awscli==1.11.126
awsebcli==3.10.4
backports-abc==0.5
backports.weakref==1.0rc1
bleach==1.5.0
blessed==1.14.2
boto==2.48.0
boto3==1.4.4
botocore==1.5.89
bz2file==0.98
cachetools==2.0.1
cement==2.8.2
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
colorama==0.3.7
cycler==0.10.0
decorator==4.1.2
dill==0.2.7.1
docker-py==1.7.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.13.1
entrypoints==0.2.3
flasgger==0.6.5
Flask==0.12
Flask-API==0.7.1
future==0.16.0
gensim==2.0.0
google-api-python-client==1.6.2
google-auth==1.1.1
google-auth-httplib2==0.0.2
google-cloud-core==0.27.1
google-cloud-language==0.29.0
google-cloud-storage==1.0.0
google-gax==0.15.15
googleapis-common-protos==1.5.3
grpcio==1.4.0
gunicorn==19.7.0
h5py==2.7.0
html5lib==0.9999999
httplib2==0.10.3
idna==2.6
ipykernel==4.6.1
ipython==6.1.0
ipython-genutils==0.2.0
ipywidgets==7.0.0
itsdangerous==0.24
jedi==0.10.2
Jinja2==2.9.5
jmespath==0.9.3
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.1.0
jupyter-console==5.2.0
jupyter-core==4.3.0
Keras==2.0.6
Markdown==2.6.8
MarkupSafe==1.0
matplotlib==2.0.0
mistune==0.7.4
nbconvert==5.2.1
nbformat==4.4.0
nltk==3.2.4
notebook==5.0.0
numpy==1.13.1
oauth2client==3.0.0
pandas==0.20.3
pandocfilters==1.4.2
pathspec==0.5.0
pexpect==4.2.1
pickleshare==0.7.4
ply==3.8
prompt-toolkit==1.0.15
protobuf==3.4.0
ptyprocess==0.5.2
pyasn1==0.3.7
pyasn1-modules==0.1.5
pyenchant==1.6.8
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.0
pytz==2016.10
PyYAML==3.12
pyzmq==16.0.2
qtconsole==4.3.1
requests==2.18.4
rsa==3.4.2
s3transfer==0.1.10
scikit-learn==0.18.1
scipy==0.19.1
semantic-version==2.5.0
simplegeneric==0.8.1
six==1.11.0
smart-open==1.5.3
tabulate==0.7.5
tensorflow==1.2.1
termcolor==1.1.0
terminado==0.6
testpath==0.3.1
Theano==0.9.0
tornado==4.5.1
traitlets==4.3.2
typing==3.6.2
tzlocal==1.4
uritemplate==3.0.0
urllib3==1.22
wcwidth==0.1.7
websocket-client==0.44.0
Werkzeug==0.11.15
widgetsnbextension==3.0.1

Previously had been using:

google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud-core==0.24.1
google-cloud-language==0.24.1
google-cloud-storage==1.0.0
googleapis-common-protos==1.5.2
protobuf==3.0.0b2.post2

which worked fine, but now need to upgrade to the latest google-cloud-language version, and the protobuf==3.0.0b2.post2 workaround is too outdated to work with the latest version.

Stacktrace

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language.py", line 17, in <module>
    from google.cloud.language_v1 import LanguageServiceClient
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/__init__.py", line 17, in <module>
    from google.cloud.language_v1 import types
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/cloud/language_v1/types.py", line 18, in <module>
    from google.gax.utils.messages import get_messages
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/gax/__init__.py", line 44, in <module>
    from google.rpc import code_pb2
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/google/rpc/code_pb2.py", line 6, in <module>
    from google.protobuf.internal import enum_type_wrapper
ImportError: No module named 'google.protobuf'

Steps to Reproduce

Since Elastic Beanstalk automatically installs packages on deployment, I have been ssh'ing into the EC2 instance, running source /opt/python/run/venv/lib/activate, and playing around with different combinations/versions of installing protobuf and google-cloud-language.

eb ssh
cd /opt/python/
source run/venv/bin/activate
python
>>> from google.cloud import language

I have also tried uninstalling all packages and reinstalling using --ignore-installed, which also did not work (same error):

pip freeze | xargs pip uninstall -y
pip install --ignore-installed protobuf
pip install google-cloud-lanuguage
python
>>> from google.cloud import language

[dhermes]

@ep-tpat That is quite a lot of packages! Part of the problem may also be that we don't support Python 3.4, so there may be some unknown issues. What exactly does protobuf==3.0.0b2.post2 work around?

In addition to uninstalling and reinstalling using --ignore-installed, you should for path files (/opt/python/run/venv/local/lib/python3.4/site-packages/*pth) that seem out of place. This google package confusion can happen if there is a mix of installed eggs (you don't want any when dealing with namespace packages) and "normal" (i.e. pip install-ed) packages.

[rsmith49]

@dhermes Not sure what Issue I was reading, but I came across a potential fix where setting the protobuf version to 3.0.0b2.post2 solved the problem, so I had been using that for the past few months. However, I guess the latest google-cloud-language version needs a later version of protobuf.

As far as files that seem out of place, do you have any suggestions on what to look out for? I am not that familiar with the package dependency structure used by pip/python

[rsmith49]

Here is the output of ls /opt/python/run/venv/local/lib/python3.4/site-packages/*pth

run/venv/local/lib/python3.4/site-packages/googleapis_common_protos-1.5.3-py3.4-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_auth-1.1.1-py2.7-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_core-0.27.1-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_language-0.29.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_cloud_storage-1.0.0-py3.6-nspkg.pth
run/venv/local/lib/python3.4/site-packages/google_gax-0.15.15-py3.4-nspkg.pth

[dhermes]

When you uninstall, just make sure all the .pth files are deleted too. (The "bad" one is usually just easy-install.pth when a namespace package comes in as part of an egg.) Similarly, you may want to delete any "dist info" or .egg files that don't seem to belong.

I can confirm the issue isn't Python 3.4 (at least for installing and importing):

>>> # After: 
>>> # $ python3.4 -m virtualenv venv
>>> # $ venv/bin/pip install google-cloud-language
>>> # $ venv/bin/python
>>> import sys
>>> sys.version
'3.4.7 (default, Oct 10 2017, 10:51:21) \n[GCC 5.4.0 20160609]'
>>> sys.executable
'.../venv/bin/python'
>>> import google.cloud.language
>>> google.cloud.language
<module 'google.cloud.language' from '.../venv/lib/python3.4/site-packages/google/cloud/language.py'>
>>> import google.protobuf
>>> google.protobuf
<module 'google.protobuf' from '.../venv/lib/python3.4/site-packages/google/protobuf/__init__.py'>

[rsmith49]

Apparently the issue had to do with Amazon Linux AMI causing pip to import some packages to venv/lib/python3.4/site-packages/ and some to venv/lib64/python3.4/site-packages, which I expect then caused problems because the google/cloud modules were under lib and the google/protobuf module was under lib64.

Here is a relevant link tracking the issue for pip, just in case anyone else runs into this issue with AWS EC2 instances: pypa/pip#4464, and I solved this by including:

/opt/python/run/venv/bin/pip install --target=/opt/python/run/venv/lib/python3.4/site-packages/ --upgrade protobuf
/opt/python/run/venv/bin/pip install --ignore-installed google-cloud

as container commands in .ebextensions