Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make packaged kedro project return session.run() output #4139

Merged
merged 13 commits into from
Sep 5, 2024
1 change: 1 addition & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Upcoming Release

## Major features and improvements
* Made packaged kedro project return `session.run()` output when running it in the interactive environment.
ElenaKhaustova marked this conversation as resolved.
Show resolved Hide resolved
ElenaKhaustova marked this conversation as resolved.
Show resolved Hide resolved
* Enhanced `OmegaConfigLoader` configuration validation to detect duplicate keys at all parameter levels, ensuring comprehensive nested key checking.
## Bug fixes and other changes
* Fixed bug where using dataset factories breaks with `ThreadRunner`.
Expand Down
13 changes: 13 additions & 0 deletions docs/source/tutorial/package_a_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,19 @@ main(

This is equivalent to `python -m <package_name>` at the command line, and you can pass in all the arguments that correspond to the options described by `python -m <package_name> --help`.

```{note}
If you run the packaged project in the interactive environment like ipython or Databricks you can also consume the output of the `main()`
ElenaKhaustova marked this conversation as resolved.
Show resolved Hide resolved
ElenaKhaustova marked this conversation as resolved.
Show resolved Hide resolved
which returns the `session.run()` output.
```

```python
from spaceflights.__main__ import main

def run_kedro_pipeline():
result = main(pipeline_name=<pipeline>)
do_something_about(<result>)
ElenaKhaustova marked this conversation as resolved.
Show resolved Hide resolved
```

### Docker, Airflow and other deployment targets

There are various methods to deploy packaged pipelines via Kedro plugins:
Expand Down
4 changes: 2 additions & 2 deletions kedro/framework/cli/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ def run( # noqa: PLR0913
conf_source: str,
params: dict[str, Any],
namespace: str,
) -> None:
) -> dict[str, Any]:
"""Run the pipeline."""

runner_obj = load_obj(runner or "SequentialRunner", "kedro.runner")
Expand All @@ -225,7 +225,7 @@ def run( # noqa: PLR0913
with KedroSession.create(
env=env, conf_source=conf_source, extra_params=params
) as session:
session.run(
return session.run(
tags=tuple_tags,
runner=runner_obj(is_async=is_async),
node_names=tuple_node_names,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,21 @@
"""
import sys
from pathlib import Path
from typing import Any

from kedro.framework.cli.utils import find_run_command
from kedro.framework.project import configure_project


def main(*args, **kwargs):
def main(*args, **kwargs) -> Any:
package_name = Path(__file__).parent.name
configure_project(package_name)

interactive = hasattr(sys, 'ps1')
kwargs["standalone_mode"] = not interactive

run = find_run_command(package_name)
run(*args, **kwargs)
return run(*args, **kwargs)


if __name__ == "__main__":
Expand Down