Fix ci - Use --resolver=backtracking with kedro build-reqs in e2e tests

[ankatiyar]

NOTE: Kedro datasets are moving from kedro.extras.datasets to a separate kedro-datasets package in
kedro-plugins repository. Any changes to the dataset implementations
should be done by opening a pull request in that repository.

Description

e2e tests failing for python 3.7 -> https://github.com/kedro-org/kedro/actions/runs/6077082782/job/16486164324?pr=3008
The problem is incompatible dependencies -

WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.
Could not find a version that matches importlib-metadata<4.3,<5.0,>=0.12,>=3.6,>=4.6 (from flake8==4.0.1->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 2))
Tried: 0.0, 0.1, 0.1, 0.2, 0.2, 0.3, 0.3, 0.4, 0.4, 0.5, 0.5, 0.6, 0.6, 0.7, 0.7, 0.8, 0.8, 0.9, 0.9, 0.10, 0.10, 0.11, 0.11, 0.12, 0.12, 0.13, 0.13, 0.14, 0.14, 0.15, 0.15, 0.16, 0.16, 0.17, 0.17, 0.18, 0.18, 0.19, 0.19, 0.20, 0.20, 0.21, 0.21, 0.22, 0.22, 0.23, 0.23, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.1.3, 1.1.3, 1.2.0, 1.2.0, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.6.1, 1.6.1, 1.7.0, 1.7.0, 2.0.0, 2.0.0, 2.1.0, 2.1.0, 2.1.1, 2.1.1, 2.1.2, 2.1.2, 2.1.3, 2.1.3, 3.0.0, 3.0.0, 3.1.0, 3.1.0, 3.1.1, 3.1.1, 3.2.0, 3.2.0, 3.3.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0, 3.5.0, 3.6.0, 3.6.0, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.7.2, 3.7.2, 3.7.3, 3.7.3, 3.8.0, 3.8.0, 3.8.1, 3.8.1, 3.8.2, 3.8.2, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.10.0, 3.10.0, 3.10.1, 3.10.1, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.1.0, 4.1.0, 4.2.0, 4.2.0, 4.3.0, 4.3.0, 4.3.1, 4.3.1, 4.4.0, 4.4.0, 4.5.0, 4.5.0, 4.6.0, 4.6.0, 4.6.1, 4.6.1, 4.6.2, 4.6.2, 4.6.3, 4.6.3, 4.6.4, 4.6.4, 4.7.0, 4.7.0, 4.7.1, 4.7.1, 4.8.0, 4.8.0, 4.8.1, 4.8.1, 4.8.2, 4.8.2, 4.8.3, 4.8.3, 4.9.0, 4.9.0, 4.10.0, 4.10.0, 4.10.1, 4.10.1, 4.11.0, 4.11.0, 4.11.1, 4.11.1, 4.11.2, 4.11.2, 4.11.3, 4.11.3, 4.11.4, 4.11.4, 4.12.0, 4.12.0, 4.13.0, 4.13.0, 5.0.0, 5.0.0, 5.1.0, 5.1.0, 5.2.0, 5.2.0, 6.0.0, 6.0.0, 6.0.1, 6.0.1, 6.1.0, 6.1.0, 6.2.0, 6.2.0, 6.2.1, 6.2.1, 6.3.0, 6.3.0, 6.4.0, 6.4.0, 6.4.1, 6.4.1, 6.5.0, 6.5.0, 6.5.1, 6.5.1, 6.6.0, 6.6.0, 6.7.0, 6.7.0
There are incompatible versions in the resolved dependencies:
  importlib-metadata>=0.12 (from pytest==7.4.1->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 13))
  importlib-metadata (from attrs==23.1.0->kedro==0.18.13->kedro-telemetry==0.2.5->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 9))
  importlib-metadata (from click==8.1.7->black==22.12.0->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 1))
  importlib-metadata (from jsonschema==4.17.3->jupyterlab_server==2.15.2->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 7))
  importlib-metadata>=3.6 (from jupyterlab_server==2.15.2->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 7))
  importlib-metadata>=4.6 (from build==1.0.0->kedro==0.18.13->kedro-telemetry==0.2.5->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 9))
  importlib-metadata<5.0,>=3.6 (from kedro==0.18.13->kedro-telemetry==0.2.5->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 9))
  importlib-metadata>=3.6 (from nbconvert==7.6.0->jupyter==1.0.0->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 6))
  importlib-metadata<4.3 (from flake8==4.0.1->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 2))
  importlib-metadata>=0.12 (from pluggy==1.2.0->pytest==7.4.1->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 13))
  importlib-metadata>=3.6 (from nbformat==5.8.0->nbstripout==0.6.1->-r /tmp/tmpbdu3jw1h/project-dummy/src/requirements.txt (line 10))

Development notes

OUTDATED

• build has a new release 1.0.0 which is causing this
• Simply pinning build version in the pyproject.toml is not working because the requirement also comes from kedro-telemetry -> kedro 0.18.13 -> build
• Remove kedro-telemetry as a requirement from the test starter project used for e2e tests?

UPDATE:

• The issue was happening because of kedro build-reqs -> incompatible dependencies because of the new build version & flake8
• Updated e2e test to use pip-compile with --resolver=backtracking

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[astrojuanlu]

Maybe worth restoring this to see if the tests pass with the conditional pinning

[ankatiyar]

They don't, I tried it yesterday on this PR https://github.com/kedro-org/kedro/pull/2993/files

[astrojuanlu]

Thanks, hadn't seen it 👍

[noklam]

It's worth to mention we cannot reproduce the behavior locally, build==1.0.0 is not a requirement and pip should be able to resolve it to 0.10.0 if this is conflicted. Either a bug in pip or pip-tools

I think it's important we ensure the flow of creating a new project and run is important. Right now we don't have any e2e test in the kedro-telemetry. It may be the more appropriate place to do this and with the nightly build it may make more sense to have it there?

I am ok remove it in kedro now to unblock CI (it's default to false now in the template), but we should then have the test in kedro-telemetry

[astrojuanlu]

Simply pinning build version in the pyproject.toml is not working because the requirement also comes from kedro-telemetry -> kedro 0.18.13 -> build

Okay I think I understand now. What about constraining the installation?

$ echo "build>=0.7.0,<1.0.0 ; python_version < '3.8'" > constraints.txt
$ echo "build>=0.7.0 ; python_version >= '3.8'" >> constraints.txt
$ export PIP_CONSTRAINT=constraints.txt
$ # run the tests

https://pip.pypa.io/en/stable/user_guide/#constraints-files

That way we can control that pip install "kedro==0.18.13" (triggered by the installation of kedro-telemetry) gets the proper version of build.

[astrojuanlu]

Or better, use the new resolver in pip-tools:

WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. The default resolver will be changed to 'backtracking' in pip-tools 7.0.0. Specify --resolver=backtracking to silence this warning.
Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`.

[noklam]

@astrojuanlu we tried that too.

[ankatiyar]

@noklam @astrojuanlu passing --resolver=backtracking to build-reqs in the e2e tests worked! 🥳

[astrojuanlu]

🎉

[AhdraMeraliQB]

Thanks for unblocking this!! ⭐

Just a QQ, are all CircleCI checks to be ignored in favour of GH Actions?

[noklam]

@noklam @astrojuanlu passing --resolver=backtracking to build-reqs in the e2e tests worked! 🥳

Thought we tried this yesterday but glad we have a better solution. :)

[astrojuanlu]

are all CircleCI checks to be ignored in favour of GH Actions?

Yep, we're effectively migrating !