protobuf-3.0.0 is missing _ _init_ _.py, and is not recognized by Python v2.7.6

[zafields]

From Python 2.7.6., the package results in the following error:

ImportError: No module named google.protobuf

In order to get the package to become discoverable, I had to add an empty __init__.py file to the site-packages/google directory.

touch /usr/local/lib/python2.7/site-packages/google/__init__.py

Here is the stackoverflow post that led me to the solution, and it is also the post that directs everyone to use the google package that is incorrectly attributed to you (#4345).

[dhermes]

@zafields There is nothing to "fix" on our end, the issue is twofold:

• the google package puts code in the google package
• we use the google namespace package (see Core library does not include init in root path #4247 for lots of discussion)

So the usage of a non-namespace package (google) colliding with our packages is causing all of the issues.

I'm happy to continue the discussion about namespace packages here, but the only resolution is what is already being discussed in #4345: we need to contact the owners of the google package and hopefully they can do a release that uses the google namespace rather than a non-namespace package.

I'd also note, if you don't need the functionality in that package (google) just uninstall it.

[zafields]

@dhermes This issue is not related to #4345

[zafields]

@dhermes I am creating new environments each time, so there is no need to uninstall google, because google was never installed. I'm looking forward to reading up on the google namespace conversation, to find more clarity about why you have elected not to put __init__.py in the root of the namespace.

[dhermes]

@zafields The issue you've enumerated above is a common issue with "failed" installs of namespace packages.

Can you create a virtualenv and copy the steps here to show what is happening? Maybe I can then reproduce.

why you have elected not to put __init__.py in the root of the namespace.

For an example of the "nuts-and-bolts" of a namespace package see the source of google/__init__.py in the google-cloud-datastore package (you can download the sdist from PyPI and verify it is there if you like). Also see the setuptools declaration of google as a namespace package. The reason google/__init__.py is not present is because pip knows the "right way" to install a namespace package and pip doesn't add the file.

[zafields]

# Filename: Dockerfile

# Base Image
FROM ubuntu:14.04

# Build Arguments
ARG PYTHON_VERSION=2.7.6
ARG PYTHON_MD5=bcf93efa8eaf383c98ed3ce40b763497

# Set execution environment
COPY ./requirements.txt /tmp/requirements.txt
ENV PYTHONPATH=/usr/local/lib/python2.7/dist-packages:$PYTHONPATH
WORKDIR /projects/

# Install Python Dependencies
RUN ["/bin/dash","-c","\
    apt-get update \
 && DEBIAN_FRONTEND=noninteractive \
    apt-get install --no-install-recommends -y \
     ca-certificates \
     curl \
     g++ \
     libssl-dev \
     make \
     python-pip \
     xz-utils \
 && rm -rf /var/lib/apt/lists/* \
"]

# Install Python
# NOTE: Proper format of checksum validation `<md5sum_checksum><space><space><file_name>`
RUN ["/bin/dash","-c","\
    cd /projects/ \
 && curl -sSL \"https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tar.xz\" -o ./python.tar.xz \
 && echo \"$PYTHON_MD5  ./python.tar.xz\" | md5sum -c - \
 && tar -xf python.tar.xz --xz \
 && rm python.tar.xz \
 && cd /projects/Python-$PYTHON_VERSION \
 && ./configure --enable-unicode=ucs4 --prefix=/usr/local/ \
 && make \
 && make install \
 && cd /projects/ \
 && rm -rf Python-$PYTHON_VERSION \
"]

# Install PIP dependencies
RUN ["/bin/bash","-c","\
    pip install pip --upgrade \
 && pip install -r /tmp/requirements.txt \
"]

# Finalize environment
RUN ["/bin/bash", "-c","\
    echo 'Defaults env_keep += \"PYTHONPATH\"' >> /etc/sudoers \
# && touch /usr/local/lib/python2.7/dist-packages/google/__init__.py \
"]

[zafields]

@dhermes Above is the Dockerfile I'm using to reproduce this behavior.

NOTE: I'm installing the GoogleCloudPlatform libraries from my requirements.txt file.

[dhermes]

@zafields For the purposes of me reproducing, only pip install -r /tmp/requirements.txt is "relevant" (though it'd be interesting if it was not reproducible outside of a Docker container).

Can you share what's in requirements.txt?

---

ASIDE: Python 2.7.14 was released about a month ago (Sept. 16, 2017) while 2.7.6 is 4 years old (Nov. 10, 2013). If you are the "master of your container" then I recommend upgrading.

[zafields]

@dhermes According to the discussion on #4247, the final line of the Dockerfile (commented out) should not be necessary, however it is required by the containerized environment I've outlined above (I provided the Dockerfile so you can reproduce the behavior exactly).

The relevant members of my requirements.txt are as follows...

...
google-apitools==0.4.11
google-api-python-client==1.4.2
google-cloud-storage==1.1.1
...

---

I will say, I fully assume this problem originates from the ancient version of Python, which is why I included it in the title of the issue.

However, we have customers we support using Python 2.7.6, so simply upgrading is not an option for us. If you do not intend to support Python 2.7.x, then you should update your documentation and we can inform our management and customers.

[zafields]

@dhermes Can we leave this issue open until you have reproduced it, and confirmed it is not an issue with your integration with Docker or Python 2.7.6?

[dhermes]

I just attempted to reproduce in a 2.7.6 virtualenv and nothing went wrong, i.e. the install (and import after) succeeds:

$ venv/bin/python -V
Python 2.7.6
$ venv/bin/pip install \
> google-apitools==0.4.11 \
> google-api-python-client==1.4.2 \
> google-cloud-storage==1.1.1
$ venv/bin/pip freeze
cachetools==2.0.1
certifi==2017.11.5
chardet==3.0.4
google-api-python-client==1.4.2
google-apitools==0.4.11
google-auth==1.2.0
google-auth-httplib2==0.0.2
google-cloud-core==0.24.1
google-cloud-storage==1.1.1
google-resumable-media==0.0.2
googleapis-common-protos==1.5.3
httplib2==0.10.3
idna==2.6
oauth2client==4.1.2
protobuf==3.4.0
protorpc==0.12.0
pyasn1==0.3.7
pyasn1-modules==0.1.5
requests==2.18.4
rsa==3.4.2
six==1.11.0
uritemplate==3.0.0
urllib3==1.22
$ venv/bin/python -c 'import google.protobuf; print(google.protobuf)'
<module 'google.protobuf' from '.../venv/lib/python2.7/site-packages/google/protobuf/__init__.pyc'>

[zafields]

@dhermes My container installs everything successfully. I have a runtime error. Did you run your unit-tests?

[dhermes]

See the final line, I successfully imported the failure you encountered:

ImportError: No module named google.protobuf

[zafields]

@dhermes Can see it works with virtualenv and Python 2.7.6, but I'm using a Mac (10.12.6) with Docker (17.09.0-ce-mac35). I am curious if you had a fresh install of Python 2.7.6 on a clean machine (similar to the environment in a container), versus a virtualenv environment if the behavior would be different.

Docker is super important to our infrastructure. Are you able to use Docker and the Dockerfile I've provided? Would it be best if I create a new issue with Docker in the title and we try to tackle it there?

[dhermes]

I modified your Dockerfile so that I can run it (since I don't have the full extent of your requirements.txt, also pip was not present on PATH, so I had to add a line to install pip). See my gist, in particular the README, which shows the import works fine within the container.

At this point, I would suggest starting with just protobuf, seeing if the import works after installation, and then adding back packages one-by-one (or group-by-group) until the failure occurs. This way you'll be able to detect which packages are accidentally colliding in the google namespace.

[zafields]

Can you fix your link to the gist? Also, are you using Windows, Linux or Mac?

[dhermes]

Updated with the link (I hit enter a little too early). I am using Ubuntu 16.04 LTS.

[zafields]

Is dhermes/repro-4352 a public container repository (or can it be)? Then I can just do apples and apples.

[dhermes]

@zafields no, it's not hosted (the image is 472MB, hosting doesn't make sense when the build should be reproduce-able).

At this point, I don't have much else to offer. It's clear there is a collision with the packages you listed and some other (as of now unknown) packages. I've offered you my advice: start with something that works and build up.

The issue is not in the Docker container and not with the older version of Python 2.7. The issue is with the package list. (Or possibly you have a broken / old pip that you didn't list in your Dockerfile, see what I did with get-pip.py to use a new version of pip.)

[zafields]

Yup. I've got your container running. I just wanted to make sure I didn't even have a chance of a bad copy/paste. Thank you for your trouble. I'll work through it and report back what I find.

[zafields]

To come full circle, the import failure appears to be a result of the difference in the way we chose to install pip. I was using apt-get install python-pip*, and you elected to download and install it manually.

Interestingly, the manual method results in pip being installed in site-packages, whereas apt-get installs it in dist-packages.

Manual:

root@b1202a1d6987:/projects# pip --version
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

apt-get:

root@4f0ee3bb7480:/projects# pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

Stackoverflow: What's the difference between dist-packages and site-packages? offers a succinct explanation of Python's site-packages vs. Debian's dist-packages?

Lee Mendelowitz: How does python find packages? does great job of detailing the mechanism and shows the source behind Debian's implementation.

Actionable Items

Perhaps if you will have a look at the Debian site package code, it will help you identify the exact point of failure, and you can make accommodation for it in your packaging solution or share an explanation of how it is incompatible with the Google packages. Also, it would be great to know if Debian dist-packages are something Google intends to support going forward, or if Google considers it a non-standard and unsupported utility?

@dhermes Would you mind to reopen this issue until we have fully outlined the source and solution to this packaging discrepancy, or would you like for me to start fresh with dist-packages vs site-packages in order to make the issue more clear/readable/helpful?

---

*typo on the Dockerfile I provided to you - now updated

[dhermes]

(I have updated my gist with your Dockerfile as well.) apt-get install python-pip will install pip for the system Python, not the one you've built. What does this mean?

$ docker run \
>   --rm \
>   --tty \
>   --interactive \
>   zafields/repro-4352:latest \
>   /bin/bash
root@694c911e4c13:/projects# python
Python 2.7.6 (default, Nov  7 2017, 17:21:38) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import google.protobuf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named google.protobuf
>>> 
>>> import sys
>>> sys.executable
'/usr/local/bin/python'
>>>
root@694c911e4c13:/projects# /usr/bin/python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import google.protobuf
>>> google.protobuf
<module 'google.protobuf' from '/usr/local/lib/python2.7/dist-packages/google/protobuf/__init__.pyc'>
>>>

---

Also, apt-get install python-pip will be managed by the OS, which is why it's in dist-packages. You really don't want any of your Python packages managed by the OS, you want them managed by you. I.e. even pip install --upgrade pip is breaking a contract with apt and if bad things happens as a result, it's considered "user error". Notice how in my Dockerfile I've explicitly used python -m pip instead of pip. (One could go even one further and recommend you use /usr/local/bin/python rather than python.)

---

Would you mind to reopen this issue until ...

There is no reason to reopen this issue, there is no problem with how we are using namespace packages. There may be some "issue" with apt-get install python-pip on Ubuntu, but it's likely just the accidental mix of side-by-side Python installs that is the problem. Either way, that is not an issue with this library.

[lukesneeringer]

Would you mind to reopen this issue until we have fully outlined the source and solution to this packaging discrepancy

The reason we closed the issue is because, as best as we can determine, it is not a bug or issue with our libraries. We will be happy to re-open it if we have probable cause to believe there is a bug to be fixed on our side.

[zafields]

@dhermes Thank you for the excellent description of the problem in your last comment. Now that we have root caused the problem and you have identified the solution I consider this issue Closed.

I am sorry for being so persistent, but you had marked this issue closed < 2 mins after it was opened and long before you arrived at your first, incorrect guesses.

Guess 1.

So the usage of a non-namespace package (google) colliding with our packages is causing all of the issues.

Guess 2.

The issue is with the package list.

I think it's reasonable to say, you closed the issue long before you understood my problem. Ultimately, you were correct and your packages were just fine. However, when someone can't consume your packages when doing a reasonable (albeit naive) progression, it is YOUR problem. If people can't consume your code, that's a problem.

Once indexed, I think this long and thorough discussion will help several others who may well run into this EXACT same problem. Again, I really do appreciate your time and effort, thank you for walking me through this problem.

[dhermes]

I think it's reasonable to say, you closed the issue long before you understood my problem

Yes, that's true. But keep in mind I also understand we are packaging our namespace packages correctly and that I've seen a lot of ImportError bugs over the last year plus.