PYTHON-4260 Lazily load optional imports #1550

blink1073 · 2024-03-13T02:27:44Z

pip install -e ".[aws,oscp,encryption,snappy,gssapi,zstd]"
pip install tuna
python -X importtime -c "import pymongo" 2> pymongo.log
tuna pymongo.log

0.209 s without this PR
0.063 s with this PR (about 3X faster).

blink1073 · 2024-03-13T13:40:33Z

@ShaneHarvey I looked at adding this to existing benchmarks, but those run without optional dependencies. We could instead use something like our mongo-arrow checks, adding a pass/fail benchmark on the PR if there is a significant increase in import time.

ShaneHarvey · 2024-03-13T16:00:07Z

pymongo/common.py

+    spec = importlib.util.find_spec(name)
+    if spec is None:
+        # Import the module to trigger an import error.
+        importlib.import_module(name)


Could we raise ModuleNotFound directly?

ShaneHarvey · 2024-03-13T16:00:17Z

pymongo/common.py

+
+    From https://docs.python.org/3/library/importlib.html#implementing-lazy-imports
+    """
+    spec = importlib.util.find_spec(name)


Do we need to guard against this caveat?:

If the module is in sys.modules, then sys.modules[name].spec is returned (unless the spec would be None or is not set, in which case ValueError is raised).

https://docs.python.org/3/library/importlib.html#importlib.util.find_spec

ShaneHarvey · 2024-03-13T18:20:11Z

pymongo/auth_aws.py

        # Clear the cached credentials if we hit a failure in auth.
-        set_cached_credentials(None)
+        pymongo_auth_aws.set_cached_credentials(None)


This behavior is subtly different. set_cached_credentials is now required when before it was optional. Let's raise the min version of pymongo_auth_aws that can be used.

ShaneHarvey · 2024-03-13T18:21:44Z

pymongo/compression_support.py

+from pymongo.monitoring import _SENSITIVE_COMMANDS
+
+
+def lazy_import(name: str) -> ModuleType:


Why do we have two copies of lazy_import?

I hoisted the function to a new private module to avoid import cycles.

ShaneHarvey · 2024-03-13T18:26:22Z

We could instead use something like our mongo-arrow checks, adding a pass/fail benchmark on the PR if there is a significant increase in import time.

Sounds good to me.

…PYTHON-4260-2

blink1073 · 2024-03-20T11:54:09Z

evergreen retry

ShaneHarvey · 2024-03-21T18:58:50Z

.evergreen/run-import-time-test.sh

+fi
+
+# Skip the report of it isn't a PR run.
+if [ "$BASE_SHA" == "$HEAD_SHA" ]; then


Shouldn't we still run this on mainline? The mainline test can check the previous commit (HEAD~) vs the current.

ShaneHarvey · 2024-03-21T19:00:54Z

doc/changelog.rst

@@ -74,6 +74,10 @@ Unavoidable breaking changes
    >>> dict_to_SON(data_as_dict)
    SON([('driver', SON([('name', 'PyMongo'), ('version', '4.7.0.dev0')])), ('os', SON([('type', 'Darwin'), ('name', 'Darwin'), ('architecture', 'arm64'), ('version', '14.3')])), ('platform', 'CPython 3.11.6.final.0')])

+- PyMongo now uses `lazy imports <https://docs.python.org/3/library/importlib.html#implementing-lazy-imports>`_ for external dependencies.
+  If you are relying on any kind of monkey-patching of the standard library, you may need to explicitly import those external libraries in addition


I would add a note that gevent+eventlet style patching still works as expected.

ShaneHarvey · 2024-03-21T19:02:04Z

tools/compare_import_time.py

@@ -0,0 +1,37 @@
+# Copyright 2024-Present MongoDB, Inc.


Should this tool live in the test/ dir instead?

I don't think so, since we won't run it as part of pytest.

Sounds good.

ShaneHarvey · 2024-03-22T18:43:52Z

.evergreen/config.yml

  display_name: "${green-framework} ${python-version} ${platform} ${auth-ssl}"
-  tasks: *all-server-versions
+  tasks: *encryption-server-versions


This doesn't actually test with encryption since it's missing the encryption tag. How about we open a new ticket to add this test coverage rather than add it here?

https://jira.mongodb.org/browse/PYTHON-4296

ShaneHarvey · 2024-03-22T18:44:07Z

pymongo/_lazy_import.py

@@ -0,0 +1,38 @@
+# Copyright 2011-present MongoDB, Inc.


ShaneHarvey · 2024-03-22T18:45:21Z

tools/compare_import_time.py

@@ -0,0 +1,37 @@
+# Copyright 2024-Present MongoDB, Inc.


Sounds good.

…PYTHON-4260-2

ShaneHarvey · 2024-03-25T16:47:45Z

pyproject.toml

@@ -45,7 +45,7 @@ dependencies = [

 [project.optional-dependencies]
 aws = [
-    "pymongo-auth-aws<2.0.0",
+    "pymongo-auth-aws>=1.1.0,<2.0.0",


We should also document this in the changelog.

blink1073 added 8 commits March 12, 2024 21:27

PYTHON-4260 Lazily load optional imports

6cfceb6

fix imports

638b743

fix auth_aws

757d103

fix typing

e3ece41

clean up

1f2cb97

clean up

0a62ea1

clean up

f343dda

delay another import

f30f0db

blink1073 requested a review from ShaneHarvey March 13, 2024 13:38

ShaneHarvey requested changes Mar 13, 2024

View reviewed changes

blink1073 added 3 commits March 20, 2024 05:44

Merge branch 'master' of github.com:mongodb/mongo-python-driver into …

7b07964

…PYTHON-4260-2

add import time check script

2924e70

lint

7137f10

fix function call

92c8abb

blink1073 closed this Mar 20, 2024

blink1073 reopened this Mar 20, 2024

blink1073 added 6 commits March 20, 2024 07:51

try again

0b2e31a

Use a python function for the comparison

25c8098

lint

ff8293a

checkout before comparing time

3bd2191

fix log file name

c5bf32b

fix print statements

ccf44be

blink1073 requested a review from ShaneHarvey March 21, 2024 11:19

blink1073 marked this pull request as ready for review March 21, 2024 18:57

ShaneHarvey requested changes Mar 21, 2024

View reviewed changes

address review

fd0772a

blink1073 requested a review from ShaneHarvey March 22, 2024 01:22

Test green frameworks with encryption

ab5c5dc

ShaneHarvey requested changes Mar 22, 2024

View reviewed changes

blink1073 added 2 commits March 25, 2024 06:44

Merge branch 'master' of github.com:mongodb/mongo-python-driver into …

9af1bff

…PYTHON-4260-2

address review

c487625

blink1073 requested a review from ShaneHarvey March 25, 2024 11:46

ShaneHarvey approved these changes Mar 25, 2024

View reviewed changes

ShaneHarvey reviewed Mar 25, 2024

View reviewed changes

update changelog

f0f5606

blink1073 requested a review from ShaneHarvey March 25, 2024 16:55

ShaneHarvey approved these changes Mar 25, 2024

View reviewed changes

mongodb-drivers-pr-bot bot requested a review from NoahStapp March 25, 2024 17:13

blink1073 merged commit 42a08c4 into mongodb:master Mar 25, 2024

blink1073 deleted the PYTHON-4260-2 branch March 25, 2024 17:55

		from pymongo.monitoring import _SENSITIVE_COMMANDS


		def lazy_import(name: str) -> ModuleType:

PYTHON-4260 Lazily load optional imports #1550

PYTHON-4260 Lazily load optional imports #1550

Uh oh!

Conversation

blink1073 commented Mar 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blink1073 commented Mar 13, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShaneHarvey commented Mar 13, 2024

Uh oh!

blink1073 commented Mar 20, 2024

Uh oh!

ShaneHarvey Mar 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

blink1073 commented Mar 13, 2024 •

edited

Loading

ShaneHarvey Mar 21, 2024 •

edited

Loading