MVP for "add-ons" flow within kedro new CLI command (#2987)

* Update prompts.yml Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update starters.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add post_gen_project in cookiecutter hooks Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * add confirmation message for the options selected Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Update post_gen_project.py Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * changes based on review Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove documentation requirements Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove testing requirements Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove leftover linting requirements Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add add-on requirements when an addon is selected Signed-off-by: lrcouto <laurarccouto@gmail.com> * Correct file path Signed-off-by: lrcouto <laurarccouto@gmail.com> * Update tests with new default template files number Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Update tests with add-ons argument Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make tests use all add-ons by default Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Make unit tests use no add-ons by default Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add installing project dependencies to e2e tests Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add linting requirements, organize code Signed-off-by: lrcouto <laurarccouto@gmail.com> * Refactor test for all add on options Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add test to check parsing add-ons Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add scaffolding for add-ons tests Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Change name of test class Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Correct test names Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Correct tests directory Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Clean up success message Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix logging option Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Update lint add-on logic Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Ensure add-ons message only shows when add-ons are configurable Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add requirement checks to tests Signed-off-by: lrcouto <laurarccouto@gmail.com> * Refactor unit tests Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add validation to add ons in config file Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Refactor add-ons flow script Signed-off-by: lrcouto <laurarccouto@gmail.com> * Pass through correct repo name in test Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Clean up and clarify text Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Wrap hook script inside main function Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Revert displayed default Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add range validation Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add tests for add-on range validation Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Apply suggestions from code review Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Update kedro/templates/project/hooks/utils.py Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> * Apply suggestions from code review Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove traceback from add-on validation - review suggestion Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Output add-on names when selected (via CLI) Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Revert 47d935e and fix tests Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix test errors Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Try remove validation Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add validation back Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add config file input validation Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add /site-packages/ to coverage report omit Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove duplicate error message Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove suppression Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Merge develop into fead/add-ons-flow Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Prep for merge Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove suppression Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Apply changes from code review Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add clarification to error messages Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Remove suppression Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Correct files in starter template Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix broken link Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add project teardown Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Try be more direct Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Try be more direct pt 2 Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Try be more direct pt 3 Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Try be more direct pt 4 Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix clean-up Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Fix type Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Apply suggestions from code review Co-authored-by: Sajid Alam <90610031+SajidAlamQB@users.noreply.github.com> * Lint Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add docstring Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> * Add changes to RELEASE.md Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> --------- Signed-off-by: SajidAlamQB <90610031+SajidAlamQB@users.noreply.github.com> Signed-off-by: Ahdra Merali <ahdra.merali@quantumblack.com> Signed-off-by: lrcouto <laurarccouto@gmail.com> Co-authored-by: L. R. Couto <57910428+lrcouto@users.noreply.github.com> Co-authored-by: Ahdra Merali <ahdra.merali@quantumblack.com> Co-authored-by: lrcouto <laurarccouto@gmail.com> Co-authored-by: Ahdra Merali <90615669+AhdraMeraliQB@users.noreply.github.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>
kedro-org · Oct 10, 2023 · ab576bc · ab576bc
1 parent 2297d23
commit ab576bc
Show file tree

Hide file tree

Showing 12 changed files with 615 additions and 26 deletions.
diff --git a/RELEASE.md b/RELEASE.md
@@ -2,6 +2,7 @@
 
 ## Major features and improvements
 * Dropped Python 3.7 support.
+* Introduced add-ons to the `kedro new` CLI flow.
 
 ## Bug fixes and other changes
 

diff --git a/features/steps/cli_steps.py b/features/steps/cli_steps.py
@@ -151,6 +151,7 @@ def create_config_file(context):
     context.root_project_dir = context.temp_dir / context.project_name
     context.package_name = context.project_name.replace("-", "_")
     config = {
+        "add_ons": "all",
         "project_name": context.project_name,
         "repo_name": context.project_name,
         "output_dir": str(context.temp_dir),

diff --git a/kedro/framework/cli/starters.py b/kedro/framework/cli/starters.py
@@ -9,6 +9,7 @@
 import re
 import shutil
 import stat
+import sys
 import tempfile
 from collections import OrderedDict
 from itertools import groupby
@@ -30,6 +31,7 @@
     _safe_load_entry_point,
     command_with_verbosity,
 )
+from kedro.templates.project.hooks.utils import parse_add_ons_input
 
 KEDRO_PATH = Path(kedro.__file__).parent
 TEMPLATE_PATH = KEDRO_PATH / "templates" / "project"
@@ -226,9 +228,12 @@ def new(config_path, starter_alias, checkout, directory, **kwargs):
         config = {}
         if config_path:
             config = _fetch_config_from_file(config_path)
+            _validate_config_file_inputs(config)
+
     elif config_path:
         config = _fetch_config_from_file(config_path)
-        _validate_config_file(config, prompts_required)
+        _validate_config_file_against_prompts(config, prompts_required)
+        _validate_config_file_inputs(config)
     else:
         config = _fetch_config_from_user_prompts(prompts_required, cookiecutter_context)
 
@@ -333,6 +338,22 @@ def _make_cookiecutter_args(
     return cookiecutter_args
 
 
+def _get_add_ons_text(add_ons):
+    add_ons_dict = {
+        "1": "Linting",
+        "2": "Testing",
+        "3": "Custom Logging",
+        "4": "Documentation",
+        "5": "Data structure",
+    }
+    add_ons_list = parse_add_ons_input(add_ons)
+    add_ons_text = [add_ons_dict[add_on] for add_on in add_ons_list]
+    return (
+        " ".join(str(add_on) + "," for add_on in add_ons_text[:-1])
+        + f" and {add_ons_text[-1]}"
+    )
+
+
 def _create_project(template_path: str, cookiecutter_args: dict[str, Any]):
     """Creates a new kedro project using cookiecutter.
 
@@ -363,6 +384,17 @@ def _create_project(template_path: str, cookiecutter_args: dict[str, Any]):
     python_package = extra_context.get(
         "python_package", project_name.lower().replace(" ", "_").replace("-", "_")
     )
+    add_ons = extra_context.get("add_ons")
+
+    # Only non-starter projects have configurable add-ons
+    if template_path == str(TEMPLATE_PATH):
+        if add_ons == "none":
+            click.secho("\nYou have selected no add-ons")
+        else:
+            click.secho(
+                f"\nYou have selected the following add-ons: {_get_add_ons_text(add_ons)}"
+            )
+
     click.secho(
         f"\nThe project name '{project_name}' has been applied to: "
         f"\n- The project title in {result_path}/README.md "
@@ -504,10 +536,10 @@ def __str__(self) -> str:
     def validate(self, user_input: str) -> None:
         """Validate a given prompt value against the regex validator"""
         if self.regexp and not re.match(self.regexp, user_input):
-            message = f"'{user_input}' is an invalid value for {self.title}."
+            message = f"'{user_input}' is an invalid value for {(self.title).lower()}."
             click.secho(message, fg="red", err=True)
             click.secho(self.error_message, fg="red", err=True)
-            raise ValueError(message, self.error_message)
+            sys.exit(1)
 
 
 def _get_available_tags(template_path: str) -> list:
@@ -530,7 +562,9 @@ def _get_available_tags(template_path: str) -> list:
     return sorted(unique_tags)
 
 
-def _validate_config_file(config: dict[str, str], prompts: dict[str, Any]):
+def _validate_config_file_against_prompts(
+    config: dict[str, str], prompts: dict[str, Any]
+):
     """Checks that the configuration file contains all needed variables.
 
     Args:
@@ -553,3 +587,27 @@ def _validate_config_file(config: dict[str, str], prompts: dict[str, Any]):
             f"'{config['output_dir']}' is not a valid output directory. "
             "It must be a relative or absolute path to an existing directory."
         )
+
+
+def _validate_config_file_inputs(config: dict[str, str]):
+    """Checks that variables provided through the config file are of the expected format.
+
+    Args:
+        config: The config as a dictionary.
+
+    Raises:
+        SystemExit: If the provided variables are not properly formatted.
+    """
+    project_name_reg_ex = r"^[\w -]{2,}$"
+    input_project_name = config.get("project_name", "New Kedro Project")
+    if not re.match(project_name_reg_ex, input_project_name):
+        message = f"'{input_project_name}' is an invalid value for project name. It must contain only alphanumeric symbols, spaces, underscores and hyphens and be at least 2 characters long"
+        click.secho(message, fg="red", err=True)
+        sys.exit(1)
+
+    add_on_reg_ex = r"^(all|none|(\d(,\d)*|(\d-\d)))$"
+    input_add_ons = config.get("add_ons", "none")
+    if not re.match(add_on_reg_ex, input_add_ons):
+        message = f"'{input_add_ons}' is an invalid value for project add-ons. Please select valid options for add-ons using comma-separated values, ranges, or 'all/none'."
+        click.secho(message, fg="red", err=True)
+        sys.exit(1)
diff --git a/kedro/templates/project/cookiecutter.json b/kedro/templates/project/cookiecutter.json
@@ -2,5 +2,6 @@
     "project_name": "New Kedro Project",
     "repo_name": "{{ cookiecutter.project_name.strip().replace(' ', '-').replace('_', '-').lower() }}",
     "python_package": "{{ cookiecutter.project_name.strip().replace(' ', '_').replace('-', '_').lower() }}",
-    "kedro_version": "{{ cookiecutter.kedro_version }}"
+    "kedro_version": "{{ cookiecutter.kedro_version }}",
+    "add_ons": "none"
 }
diff --git a/kedro/templates/project/hooks/post_gen_project.py b/kedro/templates/project/hooks/post_gen_project.py
@@ -0,0 +1,27 @@
+from pathlib import Path
+
+from kedro.templates.project.hooks.utils import (
+    parse_add_ons_input,
+    setup_template_add_ons,
+    sort_requirements,
+)
+
+def main():
+    current_dir = Path.cwd()
+    requirements_file_path = current_dir / "requirements.txt"
+    pyproject_file_path = current_dir / "pyproject.toml"
+
+    # Get the selected add-ons from cookiecutter
+    selected_add_ons = "{{ cookiecutter.add_ons }}"
+
+    # Parse the add-ons to get a list
+    selected_add_ons_list = parse_add_ons_input(selected_add_ons)
+
+    # Handle template directories and requirements according to selected add-ons
+    setup_template_add_ons(selected_add_ons_list, requirements_file_path, pyproject_file_path)
+
+    # Sort requirements.txt file in alphabetical order
+    sort_requirements(requirements_file_path)
+
+if __name__ == "__main__":
+    main()
diff --git a/kedro/templates/project/hooks/utils.py b/kedro/templates/project/hooks/utils.py
@@ -0,0 +1,157 @@
+from pathlib import Path
+import shutil
+import sys
+import click
+
+current_dir = Path.cwd()
+
+lint_requirements = "black~=22.12.0\nruff~=0.0.290\n"
+lint_pyproject_requirements = """
+[tool.ruff]
+select = [
+    "F",  # Pyflakes
+    "E",  # Pycodestyle
+    "W",  # Pycodestyle
+    "UP",  # pyupgrade
+    "I",  # isort
+    "PL", # Pylint
+]
+ignore = ["E501"]  # Black takes care of line-too-long
+"""
+
+test_requirements = "pytest-cov~=3.0\npytest-mock>=1.7.1, <2.0\npytest~=7.2"
+test_pyproject_requirements = """
+[tool.pytest.ini_options]
+addopts = \"\"\"
+--cov-report term-missing \\
+--cov src/{{ cookiecutter.python_package }} -ra
+\"\"\"
+
+[tool.coverage.report]
+fail_under = 0
+show_missing = true
+exclude_lines = ["pragma: no cover", "raise NotImplementedError"]
+"""
+
+docs_pyproject_requirements = """
+docs = [
+    "docutils<0.18.0",
+    "sphinx~=3.4.3",
+    "sphinx_rtd_theme==0.5.1",
+    "nbsphinx==0.8.1",
+    "sphinx-autodoc-typehints==1.11.1",
+    "sphinx_copybutton==0.3.1",
+    "ipykernel>=5.3, <7.0",
+    "Jinja2<3.1.0",
+    "myst-parser~=0.17.2",
+]
+"""
+
+def _validate_range(start, end):
+    if int(start) > int(end):
+        message = f"'{start}-{end}' is an invalid range for project add-ons.\nPlease ensure range values go from smaller to larger."
+        click.secho(message, fg="red", err=True)
+        sys.exit(1)
+
+def _validate_selection(add_ons):
+    for add_on in add_ons:
+        if int(add_on) < 1 or int(add_on) > 5:
+            message = f"'{add_on}' is not a valid selection.\nPlease select from the available add-ons: 1, 2, 3, 4, 5."
+            click.secho(message, fg="red", err=True)
+            sys.exit(1)
+
+
+def parse_add_ons_input(add_ons_str):
+    """Parse the add-ons input string.
+
+    Args:
+        add_ons_str: Input string from prompts.yml.
+
+    Returns:
+        list: List of selected add-ons as strings.
+    """
+    if add_ons_str == "all":
+        return ["1", "2", "3", "4", "5"]
+    if add_ons_str == "none":
+        return []
+
+    # Split by comma
+    add_ons_choices = add_ons_str.split(",")
+    selected = []
+
+    for choice in add_ons_choices:
+        if "-" in choice:
+            start, end = choice.split("-")
+            _validate_range(start, end)
+            selected.extend(str(i) for i in range(int(start), int(end) + 1))
+        else:
+            selected.append(choice.strip())
+
+    _validate_selection(selected)
+    return selected
+
+
+def setup_template_add_ons(selected_add_ons_list, requirements_file_path, pyproject_file_path):
+    """Removes directories and files related to unwanted addons from
+    a Kedro project template. Adds the necessary requirements for
+    the addons that were selected.
+
+    Args:
+        selected_add_ons_list: a list containing numbers from 1 to 5,
+            representing specific add-ons.
+        requirements_file_path: the path to the requirements.txt file.
+        pyproject_file_path: the path to the pyproject.toml file
+            located on the the root of the template.
+    """
+    if "1" not in selected_add_ons_list:  # If Linting not selected
+        pass
+    else:
+        with open(requirements_file_path, 'a') as file:
+            file.write(lint_requirements)
+        with open(pyproject_file_path, 'a') as file:
+            file.write(lint_pyproject_requirements)
+
+    if "2" not in selected_add_ons_list:  # If Testing not selected
+        tests_path = current_dir / "tests"
+        if tests_path.exists():
+            shutil.rmtree(str(tests_path))
+    else:
+        with open(requirements_file_path, 'a') as file:
+            file.write(test_requirements)
+        with open(pyproject_file_path, 'a') as file:
+            file.write(test_pyproject_requirements)
+
+    if "3" not in selected_add_ons_list:  # If Logging not selected
+        logging_yml_path = current_dir / "conf/logging.yml"
+        if logging_yml_path.exists():
+            logging_yml_path.unlink()
+
+    if "4" not in selected_add_ons_list:  # If Documentation not selected
+        docs_path = current_dir / "docs"
+        if docs_path.exists():
+            shutil.rmtree(str(docs_path))
+    else:
+        with open(pyproject_file_path, 'a') as file:
+            file.write(docs_pyproject_requirements)
+
+    if "5" not in selected_add_ons_list:  # If Data Structure not selected
+        data_path = current_dir / "data"
+        if data_path.exists():
+            shutil.rmtree(str(data_path))
+
+
+def sort_requirements(requirements_file_path):
+    """Sort the requirements.txt file in alphabetical order.
+
+    Args:
+        requirements_file_path: the path to the requirements.txt file.
+    """
+    with open(requirements_file_path, 'r') as requirements:
+        lines = requirements.readlines()
+
+    lines = [line.strip() for line in lines]
+    lines.sort()
+    sorted_content = '\n'.join(lines)
+
+    with open(requirements_file_path, 'w') as requirements:
+        requirements.write(sorted_content)
diff --git a/kedro/templates/project/prompts.yml b/kedro/templates/project/prompts.yml
@@ -1,3 +1,21 @@
+add_ons:
+  title: "Project Add-Ons"
+  text: |
+    Here you can select which add-ons you'd like to include. By default, none are included.
+    To read more about these add-ons and what they do visit: kedro.org/{insert-documentation}
+
+    Add-Ons
+    1) Linting : Provides a basic linting set up with Black and ruff
+    2) Testing : Provides basic testing set up with pytest
+    3) Custom Logging : Provides more logging options
+    4) Documentation: Provides basic documentations setup with Sphinx
+    5) Data Structure: Provides a directory structure for storing data
+
+    Which add-ons would you like to include in your project? [1-5/1,3/all/none]:
+  regex_validator: "^(all|none|(\\d(,\\d)*|(\\d-\\d)))$"
+  error_message: |
+    Invalid input. Please select valid options for add-ons using comma-separated values, ranges, or 'all/none'.
+
 project_name:
   title: "Project Name"
   text: |

diff --git a/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/logging.yml b/kedro/templates/project/{{ cookiecutter.repo_name }}/conf/logging.yml
@@ -0,0 +1,41 @@
+version: 1
+
+disable_existing_loggers: False
+
+formatters:
+  simple:
+    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+
+handlers:
+  console:
+    class: logging.StreamHandler
+    level: INFO
+    formatter: simple
+    stream: ext://sys.stdout
+
+  info_file_handler:
+    class: logging.handlers.RotatingFileHandler
+    level: INFO
+    formatter: simple
+    filename: info.log
+    maxBytes: 10485760 # 10MB
+    backupCount: 20
+    encoding: utf8
+    delay: True
+
+  rich:
+    class: kedro.logging.RichHandler
+    rich_tracebacks: True
+    # Advance options for customisation.
+    # See https://docs.kedro.org/en/stable/logging/logging.html#project-side-logging-configuration
+    # tracebacks_show_locals: False
+
+loggers:
+  kedro:
+    level: INFO
+
+  {{ cookiecutter.python_package }}:
+    level: INFO
+
+root:
+  handlers: [rich, info_file_handler]