[KED-1888] Enable auto-registration of hooks implementations (kedro-o…

…rg#711)

First iteration, to be revisited next week.

Makefile

@@ -4,7 +4,7 @@ install: build-docs
 	pip install .
 
 clean:
-	rm -rf build dist docs/build kedro/html pip-wheel-metadata .mypy_cache .pytest_cache
+	rm -rf build dist docs/build kedro/html pip-wheel-metadata .mypy_cache .pytest_cache features/steps/test_plugin/test_plugin.egg-info
 	find . -regex ".*/__pycache__" -exec rm -rf {} +
 	find . -regex ".*\.egg-info" -exec rm -rf {} +
 	pre-commit clean || true

RELEASE.md

@@ -10,6 +10,7 @@
 
 # Upcoming 0.16.4 release
 * Fixed a bug for using `ParallelRunner` on Windows.
+* Enabled auto-discovery of hooks implementations coming from installed plugins.
 
 ## Major features and improvements
 

docs/conf.py

@@ -94,7 +94,7 @@
     "README.md",
     "07_extend_kedro/02_transformers.md",
     "07_extend_kedro/03_decorators.md",
-    "11_faq/03_glossary.md"
+    "11_faq/03_glossary.md",
 ]
 
 # The name of the Pygments (syntax highlighting) style to use.

docs/source/07_extend_kedro/04_hooks.md

@@ -79,7 +79,7 @@ class TransformerHooks:
 > * To declare a Hook implementation, use the `@hook_impl` decorator
 > * You only need to make use of a subset of arguments defined in the corresponding specification
 > * Group related Hook implementations under a namespace, preferably a class
-> * You can register more than one implementations for the same specification. They will be called in FIFO (first-in, first-out) order.
+> * You can register more than one implementations for the same specification. They will be called in LIFO (last-in, first-out) order.
 
 
 #### Registering your Hook implementations with Kedro
@@ -109,6 +109,10 @@ class ProjectContext(KedroContext):
 
 This ensures that the `after_data_catalog_created` implementation above will be called automatically after every time a data catalog is created.
 
+Kedro also has auto-discovery on by default, meaning that any installed plugins that declare a hooks entry-point will be registered. To learn more about how to enable this for your custom plugin, see our [plugin development guide](05_plugins.md#Hooks).
+
+>Note: Auto-discovered hooks will run *after* the ones specified in ``ProjectContext.hooks``.
+
 ## Under the hood
 
 Under the hood, we use [pytest's pluggy](https://pluggy.readthedocs.io/en/latest/) to implement Kedro's Hook mechanism. We recommend reading their documentation if you have more questions about the underlying implementation.

docs/source/07_extend_kedro/05_plugins.md

@@ -37,6 +37,36 @@ Global commands use the `entry_point` key `kedro.global_commands`. Project comma
 
 We use the following command convention: `kedro <plugin-name> <command>`. With `kedro <plugin-name>` acting as a top-level command group. Note, this is a suggested way of structuring your plugin and is not necessary for your plugin to work.
 
+### Hooks
+
+You can develop hook implementations and have them automatically registered to the `KedroContext` when the plugin is installed. To enable this for your custom plugin, simply add the following entry in your `setup.py`:
+
+```python
+setup(
+    ...
+    entry_points={"kedro.hooks": ["plugin_name = plugin_name.plugin:hooks"]},
+)
+```
+
+where `plugin.py` is the module where you declare hook implementations:
+
+```python
+import logging
+
+from kedro.framework.hooks import hook_impl
+
+
+class MyHooks:
+    @hook_impl
+    def after_catalog_created(self, catalog):  # pylint: disable=unused-argument
+        logging.info("Reached after_catalog_created hook")
+
+
+hooks = MyHooks()
+```
+
+> Note: Here, `hooks` should be an instance of the class defining the hooks.
+
 ## Working with `click`
 
 Commands must be provided as [`click` `Groups`](https://click.palletsprojects.com/en/7.x/api/#click.Group)

features/load_context.feature

@@ -45,3 +45,10 @@ Feature: Custom Kedro project
         And Source directory is updated to "." in kedro.yml
         And I execute the kedro command "run"
         Then I should get a successful exit code
+
+    Scenario: Hooks from installed plugins are automatically registered
+        Given I have installed the test plugin
+        When I execute the kedro command "run"
+        Then I should get a successful exit code
+        And I should get a message including "Registered hooks from 1 installed plugin(s)"
+        And I should get a message including "Reached after_catalog_created hook"

features/steps/cli_steps.py

@@ -221,7 +221,7 @@ def create_new_env(context, env_name):
     env_path.mkdir()
 
     for config_name in ("catalog", "parameters", "credentials"):
-        path = env_path / "{}.yml".format(config_name)
+        path = env_path / f"{config_name}.yml"
         with path.open("w") as config_file:
             yaml.dump({}, config_file, default_flow_style=False)
 
@@ -265,6 +265,14 @@ def install_project_package_via_pip(context):
     run([context.pip, "install", str(whl_file)], env=context.env)
 
 
+@given("I have installed the test plugin")
+def install_test_plugin(context):
+    """Install a python package using pip."""
+    plugin_dir = Path(__file__).parent / "test_plugin"
+    res = run([context.pip, "install", "-e", str(plugin_dir)], env=context.env)
+    assert res.returncode == OK_EXIT_CODE, res
+
+
 @given("I have initialized a git repository")
 def init_git_repo(context):
     """Init git repo"""
@@ -312,7 +320,7 @@ def add_proj_dir_to_staging(context):
 def commit_changes_to_git(context):
     """Commit changes to git"""
     with util.chdir(context.root_project_dir):
-        check_run("git commit -m 'Change {time}'".format(time=time()))
+        check_run(f"git commit -m 'Change {time()}'")
 
 
 @when('I execute the kedro command "{command}"')
@@ -436,9 +444,8 @@ def udpate_kedro_yml(context: behave.runner.Context, new_source_dir):
 
     kedro_yml_path = context.root_project_dir / ".kedro.yml"
     kedro_yml_path.write_text(
-        "context_path: {}.run.ProjectContext\nsource_dir: {}\n".format(
-            str(context.package_name), new_source_dir
-        )
+        f"context_path: {context.package_name}.run.ProjectContext\n"
+        f"source_dir: {new_source_dir}\n"
     )
 
 
@@ -448,7 +455,7 @@ def update_kedro_req(context: behave.runner.Context):
     that includes all of kedro's dependencies (-r kedro/requirements.txt)
     """
     reqs_path = context.root_project_dir / "src" / "requirements.txt"
-    kedro_reqs = "-r {}\n".format(context.requirements_path.as_posix())
+    kedro_reqs = f"-r {context.requirements_path.as_posix()}\n"
     kedro_with_pandas_reqs = kedro_reqs + "pandas\n"
 
     if reqs_path.is_file():
@@ -518,13 +525,13 @@ def check_pipeline_not_empty(context):
 
 @then("the console log should show that {number} nodes were run")
 def check_one_node_run(context, number):
-    expected_log_line = "Completed {number} out of {number} tasks".format(number=number)
+    expected_log_line = f"Completed {number} out of {number} tasks"
     assert expected_log_line in context.result.stdout
 
 
 @then('the console log should show that "{node}" was run')
 def check_correct_nodes_run(context, node):
-    expected_log_line = "Running node: {node}".format(node=node)
+    expected_log_line = f"Running node: {node}"
     assert expected_log_line in context.result.stdout
 
 
@@ -533,19 +540,24 @@ def check_status_code(context):
     if context.result.returncode != OK_EXIT_CODE:
         print(context.result.stdout)
         print(context.result.stderr)
-        assert False, "Expected exit code {} but got {}".format(
-            OK_EXIT_CODE, context.result.returncode
+
+        error_msg = (
+            f"Expected exit code {OK_EXIT_CODE} but got {context.result.returncode}"
         )
+        assert False, error_msg
 
 
 @then("I should get an error exit code")
 def check_failed_status_code(context):
     if context.result.returncode == OK_EXIT_CODE:
         print(context.result.stdout)
         print(context.result.stderr)
-        assert False, "Expected exit code other than {} but got {}".format(
-            OK_EXIT_CODE, context.result.returncode
+
+        error_msg = (
+            f"Expected exit code other than {OK_EXIT_CODE} "
+            f"but got {context.result.returncode}"
         )
+        assert False, error_msg
 
 
 @then("the relevant packages should be created")
@@ -561,7 +573,7 @@ def check_python_packages_created(context):
 @then('"{env}" environment was used')
 def check_environment_used(context, env):
     env_path = context.root_project_dir / "conf" / env
-    assert env_path.exists(), 'Environment "{}" does not exist'.format(env)
+    assert env_path.exists(), f'Environment "{env}" does not exist'
 
     if isinstance(context.result, ChildTerminatingPopen):
         stdout = context.result.stdout.read().decode()
@@ -570,12 +582,12 @@ def check_environment_used(context, env):
         stdout = context.result.stdout
 
     for config_name in ("catalog", "parameters", "credentials"):
-        path = env_path.joinpath("{}.yml".format(config_name))
+        path = env_path.joinpath(f"{config_name}.yml")
         if path.exists():
-            msg = "Loading: {}".format(str(path.resolve()))
+            msg = f"Loading: {path.resolve()}"
             assert msg in stdout, (
                 "Expected the following message segment to be printed on stdout: "
-                "{0}, but got:\n{1}".format(msg, stdout)
+                f"{msg}, but got:\n{stdout}"
             )
 
 
@@ -591,7 +603,7 @@ def check_message_printed(context, msg):
 
     assert msg in stdout, (
         "Expected the following message segment to be printed on stdout: "
-        "{exp_msg},\nbut got {actual_msg}".format(exp_msg=msg, actual_msg=stdout)
+        f"{msg},\nbut got {stdout}"
     )
 
 
@@ -607,7 +619,7 @@ def check_error_message_printed(context, msg):
 
     assert msg in stderr, (
         "Expected the following message segment to be printed on stderr: "
-        "{exp_msg},\nbut got {actual_msg}".format(exp_msg=msg, actual_msg=stderr)
+        f"{msg},\nbut got {stderr}"
     )
 
 

features/steps/test_plugin/plugin.py

@@ -0,0 +1,42 @@
+# Copyright 2020 QuantumBlack Visual Analytics Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
+# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
+# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
+# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
+# (either separately or in combination, "QuantumBlack Trademarks") are
+# trademarks of QuantumBlack. The License does not grant you any right or
+# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
+# Trademarks or any confusingly similar mark as a trademark for your product,
+# or use the QuantumBlack Trademarks in any other manner that might cause
+# confusion in the marketplace, including but not limited to in advertising,
+# on websites, or on software.
+#
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Dummy plugin with simple hook implementations."""
+import logging
+
+from kedro.framework.hooks import hook_impl
+
+
+class MyPluginHook:
+    @hook_impl
+    def after_catalog_created(
+        self, catalog
+    ):  # pylint: disable=unused-argument,no-self-use
+        logging.info("Reached after_catalog_created hook")
+
+
+hooks = MyPluginHook()

features/steps/test_plugin/setup.py

@@ -0,0 +1,36 @@
+# Copyright 2020 QuantumBlack Visual Analytics Limited
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND
+# NONINFRINGEMENT. IN NO EVENT WILL THE LICENSOR OR OTHER CONTRIBUTORS
+# BE LIABLE FOR ANY CLAIM, DAMAGES, OR OTHER LIABILITY, WHETHER IN AN
+# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF, OR IN
+# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# The QuantumBlack Visual Analytics Limited ("QuantumBlack") name and logo
+# (either separately or in combination, "QuantumBlack Trademarks") are
+# trademarks of QuantumBlack. The License does not grant you any right or
+# license to the QuantumBlack Trademarks. You may not use the QuantumBlack
+# Trademarks or any confusingly similar mark as a trademark for your product,
+# or use the QuantumBlack Trademarks in any other manner that might cause
+# confusion in the marketplace, including but not limited to in advertising,
+# on websites, or on software.
+#
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from setuptools import find_packages, setup
+
+setup(
+    name="test_plugin",
+    version="0.1",
+    description="Dummy plugin with hook implementations",
+    packages=find_packages(),
+    entry_points={"kedro.hooks": ["test_plugin = plugin:hooks"]},
+)

kedro/context/context.py

@@ -54,6 +54,8 @@
 from kedro.utils import load_obj
 from kedro.versioning import Journal
 
+_PLUGIN_HOOKS = "kedro.hooks"  # entry-point to load hooks from for installed plugins
+
 
 def _version_mismatch_error(context_version) -> str:
     return (
@@ -251,7 +253,7 @@ def __init__(
         self._setup_logging()
 
         # setup hooks
-        self._register_hooks()
+        self._register_hooks(auto=True)
 
     @property
     @abc.abstractmethod
@@ -312,10 +314,20 @@ def pipelines(self) -> Dict[str, Pipeline]:
         """
         return self._get_pipelines()
 
-    def _register_hooks(self) -> None:
-        """Register all hooks as specified in ``hooks`` with the global ``hook_manager``.
+    def _register_hooks(self, auto: bool = False) -> None:
+        """Register all hooks as specified in ``hooks`` with the global ``hook_manager``,
+        and, optionally, from installed plugins.
+
+        Args:
+            auto: An optional flag to enable auto-discovery and registration of plugin hooks.
         """
         self._hook_manager = get_hook_manager()
+
+        if auto:
+            found = self._hook_manager.load_setuptools_entrypoints(_PLUGIN_HOOKS)
+            if found:  # pragma: no cover
+                logging.info("Registered hooks from %d installed plugin(s)", found)
+
         for hooks_collection in self.hooks:
             # Sometimes users might create more than one context instance, in which case
             # hooks have already been registered, so we perform a simple check here

kedro/framework/cli/cli.py

@@ -134,7 +134,7 @@ def info():
         click.echo("Installed plugins:")
         for plugin_name, plugin_version in sorted(plugin_versions.items()):
             hooks = ",".join(sorted(plugin_hooks[plugin_name]))
-            click.echo("{}: {} (hooks:{})".format(plugin_name, plugin_version, hooks))
+            click.echo(f"{plugin_name}: {plugin_version} (hooks:{hooks})")
     else:
         click.echo("No plugins installed")
 
@@ -693,7 +693,7 @@ def _init_plugins():
             init_hook = entry_point.load()
             init_hook()
         except Exception:  # pylint: disable=broad-except
-            _handle_exception("Initializing {}".format(str(entry_point)), end=False)
+            _handle_exception(f"Initializing {entry_point}", end=False)
 
 
 def main():  # pragma: no cover