Core library does not include init in root path

[Downchuck]

There is clearly a hierarchy in this package, for example "google.cloud.storage" is a particular package that includes the google.cloud core library. But there is no upstream "google" library, and so there is no associated google/__init__.py file created, which can lead to a painful debugging situation where simple code like "import google.cloud.storage" fails.

In depth:

This is a corner case -- but allowing for that: the wheel packages for Google Cloud Python are setup to use subfolders, which is relatively unique in Python packaging. So these packages are all underneath the "google/" folder. Unfortunately, there is no root package -- because the Google Cloud Core is not actually a root package, it too is a subfolder.

The consequence is that there is no google/__init__.py and there is no package that would own that file as a distributable wheel. This has some unfortunate side effects when attempting to do an import google.cloud.storage or similar when using Spark.

The workaround is simply to touch that file when packaging dependencies.

[dhermes]

@Downchuck It seems you are having an issue with namespace packages in your install.

The real problem is that google is a namespace that many other packages outside of our purview own. And if each of those packages tried to over-write google/__init__.py, bad things would happen.

There should not be a google/__init__.py file, but there should be .pth files, e.g.

$ ls -1 site-packages/*pth
site-packages/googleapis_common_protos-1.5.3-py3.6-nspkg.pth
site-packages/google_auth-1.1.1-py2.7-nspkg.pth
site-packages/google_cloud_core-0.27.1-py3.6-nspkg.pth
site-packages/google_cloud_storage-1.4.0-py3.6-nspkg.pth
site-packages/google_resumable_media-0.3.0-py3.6-nspkg.pth
site-packages/protobuf-3.4.0-py3.6-nspkg.pth
site-packages/pytest-cov.pth

[dhermes]

@Downchuck I am going to pre-emptively close since the namespace packages are:

• Our only choice
• An officially documented (and PEP-accepted) part of Python packaging
• Working "as intended"

---

If you'd like to share some stacktraces, we are happy to help troubleshoot within your environment. (I'm not familiar with what sort of Python interpreters are used for large Spark jobs.)

[Downchuck]

Has PEP closed out the concept of a root package?

[tseaver]

See also: pypa/setuptools#912

[dhermes]

Has PEP closed out the concept of a root package?

I'm sorry I don't understand what you mean.

[Downchuck]

I get that it'd need a root package if a root __init__.py file is going to be created. That's why I'm saying, there ought to be a root package in the dependency chain, even if it's just a single file. This is more an issue of "pain" than it is of anything else. Going with the idea that "With google-cloud-python we try to make authentication as painless as possible" is a worthwhile goal.

You can see in that thread you posted the pits of suffering us poor developers fall into, when backed into the corner of otherwise elegant solutions.

Here's what I'm doing, so I can use Spark's addPyFile

python -m pip wheel --wheel-dir ./google-cloud-storage google-cloud-storage
cd google-cloud-storage
for x in *.whl; do unzip -d all -o -u $x; done;
cd all
touch google/__init__.py
touch google/cloud/__init__.py
zip -r ../google-cloud-storage-py27-none-any.all.zip .

That's not a big deal, but it meant a lot of pain, to figure out that I needed to touch a couple of files.
This is in contrast with every other package I've worked with in this system, where that step wasn't needed.

[dhermes]

There is no need for a "root package". (I'm not sure what you mean when you say "root package".)

---

that I needed to touch a couple of files

But you don't need to touch any files, you need an interpreter that supports namespace packages (i.e. can parse .pth files). This shouldn't be an issue, as namespace packages have been around since 2012.

[dhermes]

It's likely that Spark's addPyFile is only intended for simple modules / self-contained .zip files. Passing in a wheel seems to subvert this usage (i.e. a wheel file assumes it will be installed by pip, which is what will add the .pth files).

Can you point to some documentation for addPyFile?

[duggelz]

addPyFile implementation is here, which appears to be the best documentation as well:

https://github.com/apache/spark/blob/7c7266208a3be984ac1ce53747dc0c3640f4ecac/python/pyspark/context.py#L846

It was written some time ago, before namespace packages and wheels existed. It could be changed to do so, which would be a feature request for the Spark project. Otherwise, we'd have to package the entire transitive closure of google-cloud-python as an .egg or .pyz (or .par).

[tseaver]

[addPyFile] was written some time ago, before namespace packages and wheels existed.

Namespace packages have existed much longer than that: they have been present since setuptools 0.5a9, released in June 2005.

[Downchuck]

Some other related docs, such as zipimport; potentially addressed in a later PEP in Python 3, like importlib:
https://docs.python.org/2/library/zipimport.html
https://docs.python.org/3/library/importlib.html

When I say root package, I mean a python package that represents the root namespace "google" so that only one package is writing to that init file. It seems that the cloud core would write an init to "google/cloud/" for example. So google core would write to "google/".

Regardless of outcome, I hope this conversation helps if someone else hits the same issue I did. It took some time to figure out.

[dhermes]

When I say root package, I mean a python package that represents the root namespace "google" so that only one package is writing to that init file. It seems that the cloud core would write an init to "google/cloud/" for example. So google core would write to "google/".

That's not quite how namespace packages work. google-cloud-core puts code in google.cloud, so there is nothing to place in the google namespace package (other than the google.cloud subpackage).

As @duggelz notes, your best bet is to probably package up all of the files in a self-contained zip. You'll be better off doing this by installing into a virtualenv and then zipping what's installed. (Due to the lack of an obvious way to get Spark to support namespace packages, you will still need to touch the __init__.py files as a workaround).

$ virtualenv venv
$ venv/bin/pip install google-cloud-storage
$ touch venv/lib/python3.6/site-packages/google/__init__.py
$ touch venv/lib/python3.6/site-packages/google/cloud/__init__.py
$ zip -r all_of_it.zip venv/lib/python3.6/site-packages/

(i.e. I'm suggesting you use something other than the wheel for your zip).

Sorry for the pain this has caused you.

[zafields]

@Downchuck I am having a similar experience as you (linked above), and it turns out the root cause is pip and how it is installed. Debian uses dist-packages instead of site-packages as its package folder. For some reason, the Google packages don't seem to recognize this directory without having the __init__.py file.

Are you using Debian? You can see whether you are using dist-packages or site-packages by calling

pip --version

and checking which of the two is reported by the URI in the response.

I'm still trying to figure it out, and I would love to see the details from your pip install/setup.

[dhermes]

@zafields This is not really an accurate description of the issue. The problem is mixing a system installed pip with a non-system installed Python. I recommend you use get-pip with a built Python or you just use the system installed python (which is also the 4 years old 2.7.6 on the 3 years old OS Ubuntu 14.04).

[zafields]

@dhermes That sounds like a really interesting option. Can you demonstrate how to download/use get-pip.py script securely, as opposed to running a script from the internet? Does it have a hash? I didn't see one listed at https://packaging.python.org/tutorials/installing-packages/

[dhermes]

You can just download it locally (e.g. from https://github.com/pypa/get-pip or from the URL) and then manually include that file (rather than downloading) when building your Docker image.