Can't run spaceflights-pandas starter via tools example_pipeline

[SajidAlamQB]

Description

A user who was trying to follow our First Steps guide ran into a an issue whilst executing kedro run as specified in the Run the new project step. See slack for more detail.

I was able to recreate this bug after installing kedro and running kedro new with no tools selected and wanting an example project. This combination pulls the spaceflights-pandas starter and after installing its requirements when trying to run the pipelines I get the same error as the user, RuntimeError: Found unexpected keys in 'pyproject.toml'.

Context

New users are unable to follow the documentation and run the spaceflights-pandas starter.

Steps to Reproduce

1. pip install kedro
2. kedro new -> none -> yes
3. cd into project
4. pip install -r requirements.txt
5. kedro run

Expected Result

spaceflights-pandas pipelines should run successfully

Actual Result

TypeError: ProjectMetadata.__new__() got an unexpected keyword argument 'example_pipeline'
RuntimeError: Found unexpected keys in 'pyproject.toml'. Make sure it only contains the following keys: ['package_name', 'project_name', 'kedro_init_version', 'source_dir', 'tools'].

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.19.1
• Python version used (python -V): 3.11
• Operating system and version: MacOS

[SajidAlamQB]

After some digging I found that in our _make_cookiecutter_args_and_fetch_template function we fetch the main branch of kedro-starters, this has the potential to make our starters go out of sync with the current released version of Kedro as the example above has shown. Instead, as we do with --starters flag, we should pass the checkout argument with the kedro version to cookiecutter to make sure it uses the same version of kedro_starters as kedro.

I have opened a PR for this fix here:

#3504

Unfortunately this won't fix this issue immediately but prevent it happening again.

@deepyaman has made a PR with a temporary solution that reverts a change in kedro-starters that will keep kedro 0.19.1 and the kedro-starters on main in sync until we are ready to do the next release.

kedro-org/kedro-starters#206

So there is a discussion to be had if we should go ahead with this.

[stichbury]

This was a really nice write up and summary of the issue @SajidAlamQB -- thanks for taking the time! 🌟

[SajidAlamQB]

Discussed in backlog grooming, we will do a release after #3504 is merged.

[SajidAlamQB]

Issue pending on the following PRs being merged:

• Ensure the tools example selection uses the same starter version as kedro #3504
• Make e2e tests pull main branch of kedro-starters when using tools #3518
• Move tools e2e tests to kedro-starters kedro-starters#209
• Release 0.19.2 #3517