Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make packaged kedro project return session.run() output #4139

Merged
merged 13 commits into from
Sep 5, 2024
Merged
1 change: 1 addition & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Upcoming Release

## Major features and improvements
* Made packaged Kedro projects return `session.run()` output to be used when running it in the interactive environment.
* Enhanced `OmegaConfigLoader` configuration validation to detect duplicate keys at all parameter levels, ensuring comprehensive nested key checking.
## Bug fixes and other changes
* Fixed bug where using dataset factories breaks with `ThreadRunner`.
Expand Down
10 changes: 5 additions & 5 deletions docs/source/deployment/prefect.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This page explains how to run your Kedro pipeline using [Prefect 2.0](https://www.prefect.io/opensource), an open-source workflow management system.

The scope of this documentation is the deployment to a self hosted [Prefect Server](https://docs.prefect.io/2.10.17/host/), which is an open-source backend that makes it easy to monitor and execute your Prefect flows and automatically extends Prefect 2.0. We will use an [Agent that dequeues submitted flow runs from a Work Queue](https://docs.prefect.io/2.10.17/tutorial/deployments/#why-workpools-and-workers).
The scope of this documentation is the deployment to a self hosted [Prefect Server](https://docs-2.prefect.io/latest/guides/host/), which is an open-source backend that makes it easy to monitor and execute your Prefect flows and automatically extends Prefect 2.0. We will use an [Agent that dequeues submitted flow runs from a Work Queue](https://docs-2.prefect.io/latest/concepts/deployments/#workers-and-work-pools).

Check notice on line 5 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L5

[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.
Raw output
{"message": "[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 5, "column": 1}}}, "severity": "INFO"}

Check warning on line 5 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L5

[Kedro.words] Use '' instead of 'easy'.
Raw output
{"message": "[Kedro.words] Use '' instead of 'easy'.", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 5, "column": 179}}}, "severity": "WARNING"}

Check warning on line 5 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L5

[Kedro.toowordy] 'monitor' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'monitor' is too wordy", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 5, "column": 187}}}, "severity": "WARNING"}

Check warning on line 5 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L5

[Kedro.Spellings] Did you really mean 'dequeues'?
Raw output
{"message": "[Kedro.Spellings] Did you really mean 'dequeues'?", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 5, "column": 292}}}, "severity": "WARNING"}

```{note}
This deployment has been tested using Kedro 0.18.10 with Prefect version 2.10.17. If you want to deploy with Prefect 1.0, we recommend you review [earlier versions of Kedro's Prefect deployment documentation](https://docs.kedro.org/en/0.18.9/deployment/prefect.html).
Expand All @@ -12,7 +12,7 @@

To use Prefect 2.0 and Prefect Server, ensure you have the following prerequisites in place:

- [Prefect 2.0 is installed](https://docs.prefect.io/2.10.17/getting-started/installation/#installing-the-latest-version) on your machine
- [Prefect 2.0 is installed](https://docs-2.prefect.io/latest/getting-started/installation/#installing-the-prefect-2-latest-version) on your machine

## Setup

Expand Down Expand Up @@ -40,7 +40,7 @@
prefect server start
```

In a separate terminal, [create a work pool](https://docs.prefect.io/2.10.17/concepts/work-pools/#work-pool-configuration) to organise the work and [create a work queue](https://docs.prefect.io/2.10.17/concepts/work-pools/#work-queues) for your agent to pull from:
In a separate terminal, [create a work pool](https://docs-2.prefect.io/latest/concepts/work-pools/#work-pool-configuration) to organise the work and [create a work queue](https://docs-2.prefect.io/latest/concepts/work-pools/#work-queues) for your agent to pull from:

```bash
prefect work-pool create --type prefect-agent <work_pool_name>
Expand All @@ -57,7 +57,7 @@

### Convert your Kedro pipeline to Prefect 2.0 flow

To build a [Prefect flow](https://docs.prefect.io/core/concepts/flows.html) for your Kedro pipeline programmatically and register it with the Prefect API, use the following Python script, which should be stored in your project’s **root directory**:
To build a [Prefect flow](https://docs-2.prefect.io/latest/concepts/flows/) for your Kedro pipeline programmatically and register it with the Prefect API, use the following Python script, which should be stored in your project’s **root directory**:

Check notice on line 60 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L60

[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.
Raw output
{"message": "[Kedro.sentencelength] Try to keep your sentence length to 30 words or fewer.", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 60, "column": 1}}}, "severity": "INFO"}

```python
# <project_root>/register_prefect_flow.py
Expand Down Expand Up @@ -250,7 +250,7 @@

### Run Prefect flow

Now, having the flow registered, you can use [Prefect Server UI](https://docs.prefect.io/2.10.17/host/) to orchestrate and monitor it.
Now, having the flow registered, you can use [Prefect Server UI](https://docs-2.prefect.io/latest/guides/host/) to orchestrate and monitor it.

Check warning on line 253 in docs/source/deployment/prefect.md

View workflow job for this annotation

GitHub Actions / vale

[vale] docs/source/deployment/prefect.md#L253

[Kedro.toowordy] 'monitor' is too wordy
Raw output
{"message": "[Kedro.toowordy] 'monitor' is too wordy", "location": {"path": "docs/source/deployment/prefect.md", "range": {"start": {"line": 253, "column": 132}}}, "severity": "WARNING"}

Navigate to http://localhost:4200/deployments to see your registered flow.

Expand Down
13 changes: 13 additions & 0 deletions docs/source/tutorial/package_a_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,19 @@ main(

This is equivalent to `python -m <package_name>` at the command line, and you can pass in all the arguments that correspond to the options described by `python -m <package_name> --help`.

```{note}
If you run the packaged project in the interactive environment like IPython or Databricks you can also consume the output of the `main()`
which returns the `session.run()` output.
```

```python
from spaceflights.__main__ import main

def run_kedro_pipeline():
result = main(pipeline_name=<pipeline>)
do_something_with(<result>)
```

### Docker, Airflow and other deployment targets

There are various methods to deploy packaged pipelines via Kedro plugins:
Expand Down
4 changes: 2 additions & 2 deletions kedro/framework/cli/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ def run( # noqa: PLR0913
conf_source: str,
params: dict[str, Any],
namespace: str,
) -> None:
) -> dict[str, Any]:
"""Run the pipeline."""

runner_obj = load_obj(runner or "SequentialRunner", "kedro.runner")
Expand All @@ -225,7 +225,7 @@ def run( # noqa: PLR0913
with KedroSession.create(
env=env, conf_source=conf_source, extra_params=params
) as session:
session.run(
return session.run(
tags=tuple_tags,
runner=runner_obj(is_async=is_async),
node_names=tuple_node_names,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,21 @@
"""
import sys
from pathlib import Path
from typing import Any

from kedro.framework.cli.utils import find_run_command
from kedro.framework.project import configure_project


def main(*args, **kwargs):
def main(*args, **kwargs) -> Any:
package_name = Path(__file__).parent.name
configure_project(package_name)

interactive = hasattr(sys, 'ps1')
kwargs["standalone_mode"] = not interactive

run = find_run_command(package_name)
run(*args, **kwargs)
return run(*args, **kwargs)


if __name__ == "__main__":
Expand Down