Improve packaging for Google Cloud Python Clients

[thclark]

I'm getting a lot of deprecation warnings, because the explicit declaration of package namespaces via pkg_resources is deprecated.

Rather than make an issue about just that I think it's better to look at the wider problem: that this is a bizarre way to set up a group of very closely related python packages. It causes:

• huge problems with static code analysis like type inference and linting
• unintuitive installation process leading to a really quite poor developer experience, especially for first-timers with the cloud
• fragmented documentation all over the place
• huge overhead burden maintaining a fragile ecosystem of versions for tightly related packages

Please can the google cloud client packaging be architected more straightforwardly? I'd recommend using a single package and specifying optional extras with the [extras] ability - this is exactly the use case for that funcitonality that the python packaging maintainers envisaged.

This would be a revelation in how easy it is to use the python client and manage dependencies, I'm sure it'd lead to stronger uptake of GCP overall from people driving it via python.

Current state of the different platforms

In AWS

poetry add boto3

In GCP

poetry add google-cloud
# wha? No google cloud?
# (googles)
# ok I need pubsub
poetry add google-cloud-pubsub
# yay that worked
python
>>> from google.cloud import pubsub
# linting errors in the IDE; showing no package installed
# wth? I just installed it
poetry add google-cloud-pubsub
# wth? already installed?
python
>>> from google.cloud import pubsub #noqa
        pubsub.DoSomething()
# OK that works just have to noqa everything
# now I need storage
poetry add google-cloud-storage
# wth, version ranges incompatible? 
# figures out compatible versions
# ... 30 minutes ...
poetry add google-cloud-storage@.....

Deprecations that motivated me to start this issue

The messages that led to this particular issue (these arise for the libs that I use, I assume it's the same across the board for all packages):

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:121
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 16 warnings
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: 10 warnings
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.cloud')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2349
../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2349
../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2349
../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2349
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2349: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(parent)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.logging')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.iam')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('ruamel')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
../usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870
  /usr/local/lib/python3.9/site-packages/pkg_resources/__init__.py:2870: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('zope')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    declare_namespace(pkg)

../usr/local/lib/python3.9/site-packages/google/rpc/__init__.py:20
  /usr/local/lib/python3.9/site-packages/google/rpc/__init__.py:20: DeprecationWarning: Deprecated call to `pkg_resources.declare_namespace('google.rpc')`.
  Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
    pkg_resources.declare_namespace(__name__)

[parthea]

Hi @thclark,

Thanks for raising this issue and sharing feedback about providing a better user experience. I'll discuss the feedback internally and create feature requests as we identify them. Please also share feature requests using this link.

Regarding the deprecation notice, there are no usages of pkg_resources in this repository but they exist in dependencies that we use. Python 2.7 support was recently dropped in google-auth so we should be in a position to start dropping usage of pkg_resources in all python packages under the googleapis organization.

[sonja-hiltunen-ivadolabs]

Hi all! Ran into this issue as well, is there a timeline for fixing this issue? I saw that it has priority P2, does that translate to a time span?

[akx]

As a workaround for the deprecation warning, you can set a warning filter (see ibis-project/ibis#5980)

[sonja-hiltunen-ivadolabs]

As a workaround for the deprecation warning, you can set a warning filter (see ibis-project/ibis#5980)

Thanks, I added the following to ignore the warnings

[tool.pytest.ini_options]
filterwarnings = [
    "ignore:Deprecated call to `pkg_resources\\.declare_namespace\\('.*'\\):DeprecationWarning",
    "ignore::DeprecationWarning:google.rpc",
]

[parthea]

Waiting for the next gapic-generator release: googleapis/gapic-generator-python#1860

[spiralman]

Upgrading to google-api-core 2.15.0 seems to have resolved the deprecation warnings in 3.11 for us (although we are not the reporter).