Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MVP for "add-ons" flow within kedro new CLI command #2987

Merged
merged 92 commits into from
Oct 10, 2023
Merged
Show file tree
Hide file tree
Changes from 78 commits
Commits
Show all changes
92 commits
Select commit Hold shift + click to select a range
3726ed7
Update prompts.yml
SajidAlamQB Aug 30, 2023
592afb5
Update starters.py
SajidAlamQB Aug 31, 2023
dd27d67
add post_gen_project in cookiecutter hooks
SajidAlamQB Sep 4, 2023
ad127d8
add confirmation message for the options selected
SajidAlamQB Sep 4, 2023
868816c
Merge branch 'develop' into feat/add-ons-cli-flow
SajidAlamQB Sep 5, 2023
52e99dd
Update post_gen_project.py
SajidAlamQB Sep 5, 2023
5108223
changes based on review
SajidAlamQB Sep 5, 2023
6cc090c
Merge branch 'develop' into feat/add-ons-cli-flow
lrcouto Sep 7, 2023
a0fe4a2
Lint
Sep 8, 2023
4fb0671
Remove documentation requirements
Sep 11, 2023
c1a1f7d
Remove testing requirements
Sep 11, 2023
ba01bb2
Remove leftover linting requirements
Sep 11, 2023
0f92782
Lint
Sep 11, 2023
bc0c172
Add add-on requirements when an addon is selected
lrcouto Sep 12, 2023
52ddf73
Correct file path
lrcouto Sep 12, 2023
ded026c
Update tests with new default template files number
Sep 12, 2023
f216847
Update tests with add-ons argument
Sep 12, 2023
ee668fd
Make lint
Sep 12, 2023
a500a7a
Lint
Sep 12, 2023
b88f01c
Make tests use all add-ons by default
Sep 12, 2023
fa379a0
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Sep 12, 2023
f39ebad
Make unit tests use no add-ons by default
Sep 12, 2023
4a4164d
Add installing project dependencies to e2e tests
Sep 12, 2023
ea40e0e
Add linting requirements, organize code
lrcouto Sep 12, 2023
d56cc31
Refactor test for all add on options
Sep 13, 2023
6208edb
Add test to check parsing add-ons
Sep 13, 2023
66c9f36
Add scaffolding for add-ons tests
Sep 13, 2023
58f211f
Change name of test class
Sep 13, 2023
119465d
Correct test names
Sep 13, 2023
5434858
Correct tests directory
Sep 13, 2023
225f26a
Clean up success message
Sep 13, 2023
2f866f7
Merge branch 'develop' into feat/add-ons-cli-flow
lrcouto Sep 13, 2023
06509fd
Fix logging option
Sep 13, 2023
e73fe7e
Update lint add-on logic
Sep 13, 2023
bb4e358
Ensure add-ons message only shows when add-ons are configurable
Sep 13, 2023
1a52a3c
Add requirement checks to tests
lrcouto Sep 14, 2023
51d0c12
Refactor unit tests
Sep 14, 2023
39761d0
Add validation to add ons in config file
Sep 14, 2023
da14a74
Refactor add-ons flow script
lrcouto Sep 18, 2023
0160bbe
Pass through correct repo name in test
Sep 19, 2023
8609597
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Sep 19, 2023
603fc93
Clean up and clarify text
Sep 19, 2023
aae48cb
Wrap hook script inside main function
Sep 19, 2023
ca6297a
Revert displayed default
Sep 19, 2023
4a9cdcb
Add range validation
Sep 19, 2023
3711567
Add tests for add-on range validation
Sep 19, 2023
e037441
Apply suggestions from code review
AhdraMeraliQB Sep 19, 2023
08567d3
Update kedro/templates/project/hooks/utils.py
AhdraMeraliQB Sep 19, 2023
7fca19e
Apply suggestions from code review
Sep 19, 2023
47d935e
Remove traceback from add-on validation - review suggestion
Sep 19, 2023
e3c7de5
Output add-on names when selected (via CLI)
Sep 19, 2023
fd0dabd
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Sep 19, 2023
f8b8920
Merge branch 'develop' into feat/add-ons-cli-flow
lrcouto Sep 20, 2023
57cf29c
Revert 47d935e and fix tests
Sep 21, 2023
951015d
Fix test errors
Sep 21, 2023
d280307
Try remove validation
Sep 21, 2023
8a51b26
Add validation back
Sep 21, 2023
abd467f
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Sep 22, 2023
47d063f
Add config file input validation
Sep 22, 2023
488eca9
Add /site-packages/ to coverage report omit
Sep 22, 2023
c5e3050
Lint
Sep 22, 2023
472a57c
Remove duplicate error message
Sep 22, 2023
e0e5077
Lint
Sep 22, 2023
e89ed0c
Merge develop into feat/add-ons-cli-flow
Sep 22, 2023
406393d
Remove suppression
Sep 27, 2023
36d72dc
Merge develop into fead/add-ons-flow
Oct 4, 2023
006555e
Merge develop into fead/add-ons-flow
Oct 4, 2023
49b72b0
Prep for merge
Oct 6, 2023
e4138d3
Remove suppression
Sep 27, 2023
a3403de
Merge develop into feat/add-ons-cli-flow
Oct 6, 2023
3579326
Apply changes from code review
Oct 6, 2023
2ad5078
Add clarification to error messages
Oct 9, 2023
608af72
Remove suppression
Sep 27, 2023
78c89e4
Merge develop into feat/add-ons-cli-flow
Oct 9, 2023
01ad5c0
Correct files in starter template
Oct 9, 2023
574cf01
Fix broken link
Oct 9, 2023
902b858
Add project teardown
Oct 9, 2023
c819148
Merge develop into feat/add-ons-cli-flow
Oct 9, 2023
618d4c0
Try be more direct
Oct 9, 2023
1b9c23f
Try be more direct pt 2
Oct 9, 2023
566462c
Try be more direct pt 3
Oct 9, 2023
b5439d0
Try be more direct pt 4
Oct 9, 2023
a8c8f86
Fix clean-up
Oct 9, 2023
b679b6a
Fix type
Oct 9, 2023
90aad98
Apply suggestions from code review
AhdraMeraliQB Oct 9, 2023
4314f23
Lint
Oct 9, 2023
e3800ec
Add docstring
Oct 9, 2023
20a8e61
Add changes to RELEASE.md
Oct 9, 2023
c31a9f4
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Oct 9, 2023
f7518a1
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Oct 10, 2023
c292d0d
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Oct 10, 2023
1332139
Merge branch 'develop' into feat/add-ons-cli-flow
AhdraMeraliQB Oct 10, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions features/steps/cli_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ def create_config_file(context):
context.root_project_dir = context.temp_dir / context.project_name
context.package_name = context.project_name.replace("-", "_")
config = {
"add_ons": "all",
"project_name": context.project_name,
"repo_name": context.project_name,
"output_dir": str(context.temp_dir),
Expand Down
58 changes: 54 additions & 4 deletions kedro/framework/cli/starters.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
import re
import shutil
import stat
import sys
import tempfile
from collections import OrderedDict
from itertools import groupby
Expand All @@ -30,6 +31,7 @@
_safe_load_entry_point,
command_with_verbosity,
)
from kedro.templates.project.hooks.utils import parse_add_ons_input

KEDRO_PATH = Path(kedro.__file__).parent
TEMPLATE_PATH = KEDRO_PATH / "templates" / "project"
Expand Down Expand Up @@ -226,9 +228,12 @@ def new(config_path, starter_alias, checkout, directory, **kwargs):
config = {}
if config_path:
config = _fetch_config_from_file(config_path)
_validate_config_file_inputs(config)

elif config_path:
config = _fetch_config_from_file(config_path)
_validate_config_file(config, prompts_required)
_validate_config_file_against_prompts(config, prompts_required)
_validate_config_file_inputs(config)
else:
config = _fetch_config_from_user_prompts(prompts_required, cookiecutter_context)

Expand Down Expand Up @@ -333,6 +338,22 @@ def _make_cookiecutter_args(
return cookiecutter_args


def _get_add_ons_text(add_ons):
add_ons_dict = {
"1": "Linting",
"2": "Testing",
"3": "Custom Logging",
"4": "Documentation",
"5": "Data structure",
}
add_ons_list = parse_add_ons_input(add_ons)
add_ons_text = [add_ons_dict[add_on] for add_on in add_ons_list]
return (
" ".join(str(add_on) + "," for add_on in add_ons_text[:-1])
+ f" and {add_ons_text[-1]}"
)


def _create_project(template_path: str, cookiecutter_args: dict[str, Any]):
"""Creates a new kedro project using cookiecutter.

Expand Down Expand Up @@ -363,6 +384,17 @@ def _create_project(template_path: str, cookiecutter_args: dict[str, Any]):
python_package = extra_context.get(
"python_package", project_name.lower().replace(" ", "_").replace("-", "_")
)
add_ons = extra_context.get("add_ons")

# Only non-starter projects have configurable add-ons
if template_path == str(TEMPLATE_PATH):
if add_ons == "none":
click.secho("\nYou have selected no add-ons")
else:
click.secho(
f"\nYou have selected the following add-ons: {_get_add_ons_text(add_ons)}"
)

click.secho(
f"\nThe project name '{project_name}' has been applied to: "
f"\n- The project title in {result_path}/README.md "
Expand Down Expand Up @@ -504,10 +536,10 @@ def __str__(self) -> str:
def validate(self, user_input: str) -> None:
"""Validate a given prompt value against the regex validator"""
if self.regexp and not re.match(self.regexp, user_input):
message = f"'{user_input}' is an invalid value for {self.title}."
message = f"'{user_input}' is an invalid value for {(self.title).lower()}."
click.secho(message, fg="red", err=True)
click.secho(self.error_message, fg="red", err=True)
raise ValueError(message, self.error_message)
sys.exit(1)


def _get_available_tags(template_path: str) -> list:
Expand All @@ -530,7 +562,9 @@ def _get_available_tags(template_path: str) -> list:
return sorted(unique_tags)


def _validate_config_file(config: dict[str, str], prompts: dict[str, Any]):
def _validate_config_file_against_prompts(
config: dict[str, str], prompts: dict[str, Any]
):
"""Checks that the configuration file contains all needed variables.

Args:
Expand All @@ -553,3 +587,19 @@ def _validate_config_file(config: dict[str, str], prompts: dict[str, Any]):
f"'{config['output_dir']}' is not a valid output directory. "
"It must be a relative or absolute path to an existing directory."
)


def _validate_config_file_inputs(config: dict[str, str]):
AhdraMeraliQB marked this conversation as resolved.
Show resolved Hide resolved
AhdraMeraliQB marked this conversation as resolved.
Show resolved Hide resolved
project_name_reg_ex = "^[\\w -]{2,}$"
AhdraMeraliQB marked this conversation as resolved.
Show resolved Hide resolved
input_project_name = config["project_name"]
AhdraMeraliQB marked this conversation as resolved.
Show resolved Hide resolved
if not re.match(project_name_reg_ex, input_project_name):
message = f"'{input_project_name}' is an invalid value for project name. It must contain only alphanumeric symbols, spaces, underscores and hyphens and be at least 2 characters long"
click.secho(message, fg="red", err=True)
sys.exit(1)

add_on_reg_ex = "^(all|none|(\\d(,\\d)*|(\\d-\\d)))$"
AhdraMeraliQB marked this conversation as resolved.
Show resolved Hide resolved
input_add_ons = config.get("add_ons", "none")
if not re.match(add_on_reg_ex, input_add_ons):
message = f"'{input_add_ons}' is an invalid value for project add-ons. Please select valid options for add-ons using comma-separated values, ranges, or 'all/none'."
click.secho(message, fg="red", err=True)
sys.exit(1)
3 changes: 2 additions & 1 deletion kedro/templates/project/cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@
"project_name": "New Kedro Project",
"repo_name": "{{ cookiecutter.project_name.strip().replace(' ', '-').replace('_', '-').lower() }}",
"python_package": "{{ cookiecutter.project_name.strip().replace(' ', '_').replace('-', '_').lower() }}",
"kedro_version": "{{ cookiecutter.kedro_version }}"
"kedro_version": "{{ cookiecutter.kedro_version }}",
"add_ons": "none"
}
27 changes: 27 additions & 0 deletions kedro/templates/project/hooks/post_gen_project.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
from pathlib import Path

from kedro.templates.project.hooks.utils import (
parse_add_ons_input,
setup_template_add_ons,
sort_requirements,
)

def main():
current_dir = Path.cwd()
requirements_file_path = current_dir / "requirements.txt"
pyproject_file_path = current_dir / "pyproject.toml"

# Get the selected add-ons from cookiecutter
selected_add_ons = "{{ cookiecutter.add_ons }}"

# Parse the add-ons to get a list
selected_add_ons_list = parse_add_ons_input(selected_add_ons)

# Handle template directories and requirements according to selected add-ons
setup_template_add_ons(selected_add_ons_list, requirements_file_path, pyproject_file_path)

# Sort requirements.txt file in alphabetical order
sort_requirements(requirements_file_path)

if __name__ == "__main__":
main()
157 changes: 157 additions & 0 deletions kedro/templates/project/hooks/utils.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
from pathlib import Path
import shutil
import sys
import click

current_dir = Path.cwd()

lint_requirements = "black~=22.12.0\nruff~=0.0.290\n"
lint_pyproject_requirements = """
[tool.ruff]
select = [
"F", # Pyflakes
"E", # Pycodestyle
"W", # Pycodestyle
"UP", # pyupgrade
"I", # isort
"PL", # Pylint
]
ignore = ["E501"] # Black takes care of line-too-long
"""

test_requirements = "pytest-cov~=3.0\npytest-mock>=1.7.1, <2.0\npytest~=7.2"
test_pyproject_requirements = """
[tool.pytest.ini_options]
addopts = \"\"\"
--cov-report term-missing \\
--cov src/{{ cookiecutter.python_package }} -ra
\"\"\"

[tool.coverage.report]
fail_under = 0
show_missing = true
exclude_lines = ["pragma: no cover", "raise NotImplementedError"]
"""

docs_pyproject_requirements = """
docs = [
"docutils<0.18.0",
"sphinx~=3.4.3",
"sphinx_rtd_theme==0.5.1",
"nbsphinx==0.8.1",
"sphinx-autodoc-typehints==1.11.1",
"sphinx_copybutton==0.3.1",
"ipykernel>=5.3, <7.0",
"Jinja2<3.1.0",
"myst-parser~=0.17.2",
]
"""

def _validate_range(start, end):
if int(start) > int(end):
message = f"'{start}-{end}' is an invalid range for project add-ons.\nPlease ensure range values go from smaller to larger."
click.secho(message, fg="red", err=True)
sys.exit(1)

def _validate_selection(add_ons):
for add_on in add_ons:
if int(add_on) < 1 or int(add_on) > 5:
message = f"'{add_on}' is not a valid selection.\nPlease select from the available add-ons: 1, 2, 3, 4, 5."
click.secho(message, fg="red", err=True)
sys.exit(1)


def parse_add_ons_input(add_ons_str):
"""Parse the add-ons input string.

Args:
add_ons_str: Input string from prompts.yml.

Returns:
list: List of selected add-ons as strings.
"""
if add_ons_str == "all":
return ["1", "2", "3", "4", "5"]
if add_ons_str == "none":
return []

# Split by comma
add_ons_choices = add_ons_str.split(",")
selected = []

for choice in add_ons_choices:
if "-" in choice:
start, end = choice.split("-")
_validate_range(start, end)
selected.extend(str(i) for i in range(int(start), int(end) + 1))
else:
selected.append(choice.strip())

_validate_selection(selected)
return selected


def setup_template_add_ons(selected_add_ons_list, requirements_file_path, pyproject_file_path):
"""Removes directories and files related to unwanted addons from
a Kedro project template. Adds the necessary requirements for
the addons that were selected.

Args:
selected_add_ons_list: a list containing numbers from 1 to 5,
representing specific add-ons.
requirements_file_path: the path to the requirements.txt file.
pyproject_file_path: the path to the pyproject.toml file
located on the the root of the template.
"""
if "1" not in selected_add_ons_list: # If Linting not selected
pass
else:
with open(requirements_file_path, 'a') as file:
file.write(lint_requirements)
with open(pyproject_file_path, 'a') as file:
file.write(lint_pyproject_requirements)

if "2" not in selected_add_ons_list: # If Testing not selected
tests_path = current_dir / "tests"
if tests_path.exists():
shutil.rmtree(str(tests_path))
else:
with open(requirements_file_path, 'a') as file:
file.write(test_requirements)
with open(pyproject_file_path, 'a') as file:
file.write(test_pyproject_requirements)

if "3" not in selected_add_ons_list: # If Logging not selected
logging_yml_path = current_dir / "conf/logging.yml"
if logging_yml_path.exists():
logging_yml_path.unlink()

if "4" not in selected_add_ons_list: # If Documentation not selected
docs_path = current_dir / "docs"
if docs_path.exists():
shutil.rmtree(str(docs_path))
else:
with open(pyproject_file_path, 'a') as file:
file.write(docs_pyproject_requirements)

if "5" not in selected_add_ons_list: # If Data Structure not selected
data_path = current_dir / "data"
if data_path.exists():
shutil.rmtree(str(data_path))


def sort_requirements(requirements_file_path):
"""Sort the requirements.txt file in alphabetical order.

Args:
requirements_file_path: the path to the requirements.txt file.
"""
with open(requirements_file_path, 'r') as requirements:
lines = requirements.readlines()

lines = [line.strip() for line in lines]
lines.sort()
sorted_content = '\n'.join(lines)

with open(requirements_file_path, 'w') as requirements:
requirements.write(sorted_content)
18 changes: 18 additions & 0 deletions kedro/templates/project/prompts.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,21 @@
add_ons:
title: "Project Add-Ons"
text: |
Here you can select which add-ons you'd like to include. By default, none are included.
To read more about these add-ons and what they do visit: kedro.org/{insert-documentation}

Add-Ons
1) Linting : Provides a basic linting set up with Black and ruff
2) Testing : Provides basic testing set up with pytest
3) Custom Logging : Provides more logging options
4) Documentation: Provides basic documentations setup with Sphinx
5) Data Structure: Provides a directory structure for storing data

Which add-ons would you like to include in your project? [1-5/1,3/all/none]:
regex_validator: "^(all|none|(\\d(,\\d)*|(\\d-\\d)))$"
error_message: |
Invalid input. Please select valid options for add-ons using comma-separated values, ranges, or 'all/none'.

project_name:
title: "Project Name"
text: |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
version: 1

disable_existing_loggers: False

formatters:
simple:
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: simple
stream: ext://sys.stdout

info_file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: simple
filename: info.log
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
delay: True

rich:
class: kedro.logging.RichHandler
rich_tracebacks: True
# Advance options for customisation.
# See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration
# tracebacks_show_locals: False

loggers:
kedro:
level: INFO

{{ cookiecutter.python_package }}:
level: INFO

root:
handlers: [rich, info_file_handler]
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,3 @@ namespaces = false
package_name = "{{ cookiecutter.python_package }}"
project_name = "{{ cookiecutter.project_name }}"
kedro_init_version = "{{ cookiecutter.kedro_version }}"

[tool.pytest.ini_options]
addopts = """
--cov-report term-missing \
--cov src/{{ cookiecutter.python_package }} -ra"""

[tool.coverage.report]
fail_under = 0
show_missing = true
exclude_lines = ["pragma: no cover", "raise NotImplementedError"]
Loading
Loading