Refactor of Pipeline Tests #3203

adamkells · 2023-10-19T10:25:06Z

Description

To improve the fixtures and docstrings in the test_pipeline.py as described in this issue

Development notes

Have mostly changed function names and included docstrings for readability as well as some minor refactoring of code.

Have tested this by running tests locally.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

Read the contributing guidelines
Signed off each commit with a Developer Certificate of Origin (DCO)
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes
Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

merelcht

Thank you so much @adamkells ! Your changes have made it a lot easier to understand these tests and fixtures ⭐ ⭐

Don't forget to add your name to the contributors in the release notes. You can add a new section like "Community contributions" under upcoming release.

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

…_source (kedro-org#3145) * Fix bug for config path containing dots Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update logic to work with lower python versions Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Co-authored-by: IngerMathilde <42641352+IngerMathilde@users.noreply.github.com> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Remove os Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: IngerMathilde <42641352+IngerMathilde@users.noreply.github.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Move project setup to advanced section Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> * Update index.rst 🤦‍♀️ --------- Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* feat(datasets): create custom `DeprecationWarning` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * feat(datasets): use the custom deprecation warning Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * chore: del unneeded `_filter_deprecation_warnings` Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Use a simple filter inside `sys.warnoptions` block Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> * Also change FutureWarnings to KedroDeprecationWarnings Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Skip jupyter convert all e2e test(deprecated) Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Skip jupyter convert all e2e test(deprecated) Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Deepyaman Datta <deepyaman.datta@utexas.edu> Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Co-authored-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

…set` (kedro-org#3170) * Add deprecation warnings to Partitioned and Incremental Datasets Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update release notes Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Make settings.py optional Signed-off-by: Miguel Ortiz <miguel_ortiz_marin@mckinsey.com> * Fix unit tests Signed-off-by: Miguel Ortiz <miguel_ortiz_marin@mckinsey.com> * Fix coverage Signed-off-by: Miguel Ortiz <miguel_ortiz_marin@mckinsey.com> * Update RELEASE.md and logging Signed-off-by: Miguel Ortiz <miguel_ortiz_marin@mckinsey.com> --------- Signed-off-by: Miguel Ortiz <miguel_ortiz_marin@mckinsey.com> Co-authored-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Update makefile to pin pip version to fix nightly CI * Add more comments Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Update pip for RTD Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * update RTD build Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * hijack the pre_install steps Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * Fix linting Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Add deprecation notice to archived starters Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Add '(deprecated)' to the names of the starters Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Co-authored-by: Merel Theisen <49397448+merelcht@users.noreply.github.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

kedro-org#3186) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Remove 3.7 from CI Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Separate notifications into separate jobs Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update links Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update link Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Mustapha Abdullahi <mustious@gmail.com> Signed-off-by: Mustapha Abdullahi <mustapha.abdullahi@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

…o-spark` (kedro-org#3226) * Add condition for develop branch Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update if statement Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> * Update if for make test-no-spark' Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> --------- Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

* Change the order of CI, run windows test before ubuntu Signed-off-by: Nok <nok.lam.chan@quantumblack.com> * try to optimise job Signed-off-by: Nok <nok.lam.chan@quantumblack.com> --------- Signed-off-by: Nok <nok.lam.chan@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Signed-off-by: Adam Kells <adam.kells@artefact.com>

ankatiyar

Thank you @adamkells! 💯

Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

adamkells force-pushed the refactor_test_pipeline branch from 6b5811a to cba6cf0 Compare October 19, 2023 10:27

adamkells marked this pull request as ready for review October 24, 2023 13:29

adamkells requested a review from merelcht as a code owner October 24, 2023 13:29

merelcht added the Hacktoberfest label Oct 25, 2023

merelcht approved these changes Oct 27, 2023

View reviewed changes

adamkells and others added 21 commits October 30, 2023 09:43

First pass refactor

15a5590

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Update release notes for docs changes (kedro-org#3176)

ea58e1d

Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Ankita Katiyar <110245118+ankatiyar@users.noreply.github.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Mute deprecation warnings coming from io.core that a user can't change (

9d1dec8

kedro-org#3186) Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Release Kedro 0.18.14 (kedro-org#3192)

da18634

Signed-off-by: Merel Theisen <merel.theisen@quantumblack.com> Signed-off-by: Jo Stichbury <jo_stichbury@mckinsey.com> Co-authored-by: Jo Stichbury <jo_stichbury@mckinsey.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Improving docstrings of Filters

2ad43a6

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Improving docstrings of Fixtures

fc89953

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Fix bugs in tests

90e2dfa

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Add nightly build to develop branch (kedro-org#3215)

61040aa

Signed-off-by: Ankita Katiyar <ankitakatiyar2401@gmail.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Pre-commit code quality fixes

f795094

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

add contribution

6bd8c8b

Signed-off-by: Adam Kells <adamjkells93@gmail.com>

Debugging setup of vscode section update (kedro-org#3181)

68d4b45

Signed-off-by: Mustapha Abdullahi <mustious@gmail.com> Signed-off-by: Mustapha Abdullahi <mustapha.abdullahi@quantumblack.com> Signed-off-by: Adam Kells <adamjkells93@gmail.com>

adamkells force-pushed the refactor_test_pipeline branch from 19db6b6 to c2184e7 Compare October 30, 2023 09:43

adamkells requested review from yetudada, astrojuanlu and stichbury as code owners October 30, 2023 09:43

Merge branch 'main' into refactor_test_pipeline

d24ccde

Signed-off-by: Adam Kells <adam.kells@artefact.com>

merelcht requested review from AhdraMeraliQB and ankatiyar and removed request for astrojuanlu, stichbury and yetudada October 30, 2023 10:47

merelcht added the Community Issue/PR opened by the open-source community label Oct 30, 2023

ankatiyar approved these changes Oct 30, 2023

View reviewed changes

Merge branch 'main' into refactor_test_pipeline

f8ee297

Signed-off-by: Merel Theisen <49397448+merelcht@users.noreply.github.com>

merelcht enabled auto-merge (squash) October 30, 2023 11:30

merelcht merged commit f89a504 into kedro-org:main Oct 30, 2023
44 of 65 checks passed

merelcht mentioned this pull request Nov 3, 2023

Refactor fixtures in test_pipeline.py #1808

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor of Pipeline Tests #3203

Refactor of Pipeline Tests #3203

adamkells commented Oct 19, 2023 •

edited

Loading

merelcht left a comment

ankatiyar left a comment

Refactor of Pipeline Tests #3203

Refactor of Pipeline Tests #3203

Conversation

adamkells commented Oct 19, 2023 • edited Loading

Description

Development notes

Developer Certificate of Origin

Checklist

merelcht left a comment

Choose a reason for hiding this comment

ankatiyar left a comment

Choose a reason for hiding this comment

adamkells commented Oct 19, 2023 •

edited

Loading