[KED-2961] Move scripts from private-kedro-sync #1107

antonymilne · 2021-12-09T13:02:50Z

Description

Builds on #1102 - take a look at that first!

I've migrated over the workflows we want from private-kedro-sync.

Added all the files from private-kedro-sync to our repo
These commits show what I changed/removed from them
These commits show how it's integrated into our CircleCI config

What's changed compared to previous jobs on private-kedro-sync:

clone_repo is no longer needed: we just do checkout. So this repo's root should be the cwd
following chat with @limdauto, sync job (which includes merging main to develop) now happens every time there's a push to main rather than at midnight. This should mean there's fewer conflicts
removed weekly_updates_checks job that sent an email no one ever looked at (or even received?) containing pip list --outdated)
removed SSH keys that are not needed any more
all bash scripts etc. in tools/circleci are exactly the same as they were in private-kedro-sync apart from the deleted files and this change to the merge PR message. All that's changed is the CircleCI config

Development notes

Still to do:

Testing
Make sure all the file paths like ./kedro are correct 😬
Figure out credentials and tokens - will need some help on this
Figure out SSH stuff - will need some help on this
Turn off private-kedro-sync - maybe rename directory at same time as this merges so it doesn't get triggered

After merging:

Make sure nightly_build and hourly_pr_merge work
Follow up PR to develop: look at https://github.com/quantumblacklabs/private-kedro-sync/pull/37

Checklist

Read the contributing guidelines
Opened this PR as a 'Draft Pull Request' if it is work-in-progress
Updated the documentation to reflect the code changes
Added a description of this change in the RELEASE.md file
Added tests to cover my changes

…-2961' into feature/tidy-circleci-config#ked-2961

…-2961' into feature/tidy-circleci-config#ked-2961 # Conflicts: # .circleci/config.yml

lorenabalan

Generally looks good to me but I'll have another closer look after some of the comments are answered.

tools/circleci/github_scripts/merge.sh

tools/circleci/docker_build_img/build.sh

.circleci/config.yml

tools/circleci/requirements.txt

.circleci/config.yml

Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com>

lorenabalan

LGTM!

tools/circleci/github_scripts/merge.sh

tools/circleci/requirements.txt

# Conflicts: # .circleci/config.yml

.circleci/config.yml

antonymilne · 2021-12-17T12:00:50Z

Following much testing in #1121, I have made the following changes in addition to the original PR description:

76b10d1: Make two different CIRCLE_TOKENs. Needed because we now have tokens to trigger workflows in both kedro (release workflow) and kedro-viz (build workflow) in the same repo
f86874e: Move automatic merging to hourly rather than main_updated. The automatic PR should be raised when we merge to main, but we should still try and merge it hourly rather than waiting till the next time main is updated
34a0674: Move merge to above trigger release workflow. This solves the problem that triggering a release interrupted (and hence cancelled/failed) the currently running sync job so that the rest of it (merging to develop) would not complete. This change also necessitated b85cf09 so that "Maybe trigger the release workflow" worked on the right branch

SajidAlamQB

Awesome! 💥

… feature/move-sync#ked-2961

* Pre-tidy * Simplify linux e2e_tests * Simplify linux e2e_tests * Move e2e_tests from command to job * String python version * Comment out unwanted tests for now * Try parametrised command name * Try conditionals * Indent * Indent * Urgh yaml * Pass arguments * Try win_e2e_tests separately * Try win_e2e_tests separately * Oops * Fix windows hopefully * URgh yaml * Alter lint to matrix * De-parametrise executor * Rename executors * Oops * Add win_pip_compile * Add win unit tests * Add viz build * Cosmetic changes * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Back to parametrised executor * Back to parametrised executor * Add unit_tests * Clarify comment * Add all files to tools/sync * Remove unwanted workflows * Alter file paths and remove clone_repo * Replace build_kedro with one job * Replace build_kedro with one job * Trigger sync on merge to main * Trigger on merge to main rather than nightly * Remove old private-kedro ssh * Merge into main config.yml file * Add unless * Correct to match schema * Lint * Tidy * Hopefully correct a filepath? * Alter PR message * Update tools/circleci/github_scripts/merge.sh Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> * Update requirements * Remove SSH and pre-commit install * Post-merge fix * Update text * Make two different CIRCLE_TOKENs * Update .circleci/config.yml * Move automatic merging to hourly rather than main_updated * Move merge to above trigger release workflow * Add git checkout after merge.sh * Add git checkout after merge.sh * New line! Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> Signed-off-by: Rashida Kanchwala <rashida.kanchwala@quantumblack.com>

* Pre-tidy * Simplify linux e2e_tests * Simplify linux e2e_tests * Move e2e_tests from command to job * String python version * Comment out unwanted tests for now * Try parametrised command name * Try conditionals * Indent * Indent * Urgh yaml * Pass arguments * Try win_e2e_tests separately * Try win_e2e_tests separately * Oops * Fix windows hopefully * URgh yaml * Alter lint to matrix * De-parametrise executor * Rename executors * Oops * Add win_pip_compile * Add win unit tests * Add viz build * Cosmetic changes * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Back to parametrised executor * Back to parametrised executor * Add unit_tests * Clarify comment * Add all files to tools/sync * Remove unwanted workflows * Alter file paths and remove clone_repo * Replace build_kedro with one job * Replace build_kedro with one job * Trigger sync on merge to main * Trigger on merge to main rather than nightly * Remove old private-kedro ssh * Merge into main config.yml file * Add unless * Correct to match schema * Lint * Tidy * Hopefully correct a filepath? * Alter PR message * Update tools/circleci/github_scripts/merge.sh Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> * Update requirements * Remove SSH and pre-commit install * Post-merge fix * Update text * Make two different CIRCLE_TOKENs * Update .circleci/config.yml * Move automatic merging to hourly rather than main_updated * Move merge to above trigger release workflow * Add git checkout after merge.sh * Add git checkout after merge.sh * New line! Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> Signed-off-by: Yetunde Dada <yetudada@users.noreply.github.com>

* Pre-tidy * Simplify linux e2e_tests * Simplify linux e2e_tests * Move e2e_tests from command to job * String python version * Comment out unwanted tests for now * Try parametrised command name * Try conditionals * Indent * Indent * Urgh yaml * Pass arguments * Try win_e2e_tests separately * Try win_e2e_tests separately * Oops * Fix windows hopefully * URgh yaml * Alter lint to matrix * De-parametrise executor * Rename executors * Oops * Add win_pip_compile * Add win unit tests * Add viz build * Cosmetic changes * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Back to parametrised executor * Back to parametrised executor * Add unit_tests * Clarify comment * Add all files to tools/sync * Remove unwanted workflows * Alter file paths and remove clone_repo * Replace build_kedro with one job * Replace build_kedro with one job * Trigger sync on merge to main * Trigger on merge to main rather than nightly * Remove old private-kedro ssh * Merge into main config.yml file * Add unless * Correct to match schema * Lint * Tidy * Hopefully correct a filepath? * Alter PR message * Update tools/circleci/github_scripts/merge.sh Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> * Update requirements * Remove SSH and pre-commit install * Post-merge fix * Update text * Make two different CIRCLE_TOKENs * Update .circleci/config.yml * Move automatic merging to hourly rather than main_updated * Move merge to above trigger release workflow * Add git checkout after merge.sh * Add git checkout after merge.sh * New line! Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> Signed-off-by: Laurens Vijnck <laurens_vijnck@mckinsey.com>

antonymilne added 30 commits December 8, 2021 11:20

Pre-tidy

596afcc

Simplify linux e2e_tests

41559c3

Simplify linux e2e_tests

cd084c2

Merge remote-tracking branch 'origin/feature/tidy-circleci-config#ked…

e608704

…-2961' into feature/tidy-circleci-config#ked-2961

Move e2e_tests from command to job

d04e4be

String python version

06d11c5

Comment out unwanted tests for now

7454095

Try parametrised command name

57bd647

Try conditionals

a6a5c51

Indent

44a6065

Indent

ed08221

Indent

3549480

Urgh yaml

ad325a6

Pass arguments

7dc51cb

Try win_e2e_tests separately

19372a1

Merge remote-tracking branch 'origin/feature/tidy-circleci-config#ked…

dd7e884

…-2961' into feature/tidy-circleci-config#ked-2961 # Conflicts: # .circleci/config.yml

Try win_e2e_tests separately

081b3b0

Oops

bf8a5e0

Fix windows hopefully

f42354d

URgh yaml

2b3b7c6

Alter lint to matrix

f4e8231

De-parametrise executor

f0a8506

Rename executors

93d8fd5

Oops

28f4002

Add win_pip_compile

796cbf9

Add win unit tests

9053479

Add viz build

71032d6

Cosmetic changes

eec510c

Refactor into setup and win_setup

6bfb8c2

Refactor into setup and win_setup

58e19b2

Alter PR message

e397cb4

lorenabalan reviewed Dec 13, 2021

View reviewed changes

antonymilne and others added 2 commits December 15, 2021 14:30

Update tools/circleci/github_scripts/merge.sh

b38dff8

Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com>

Update requirements

a2f0ff9

Base automatically changed from feature/tidy-circleci-config#ked-2961 to main December 16, 2021 09:55

lorenabalan approved these changes Dec 16, 2021

View reviewed changes

tools/circleci/github_scripts/merge.sh Outdated Show resolved Hide resolved

tools/circleci/requirements.txt Show resolved Hide resolved

antonymilne added 5 commits December 16, 2021 12:14

Merge branch 'main' into feature/move-sync#ked-2961

047c3ec

# Conflicts: # .circleci/config.yml

Remove SSH and pre-commit install

38f203e

Post-merge fix

bfe25b5

Update text

4b64f0b

Make two different CIRCLE_TOKENs

76b10d1

antonymilne commented Dec 16, 2021

View reviewed changes

.circleci/config.yml Show resolved Hide resolved

antonymilne and others added 3 commits December 16, 2021 17:34

Update .circleci/config.yml

d4318fe

Move automatic merging to hourly rather than main_updated

f86874e

Move merge to above trigger release workflow

34a0674

SajidAlamQB approved these changes Dec 17, 2021

View reviewed changes

antonymilne and others added 5 commits December 17, 2021 15:25

Add git checkout after merge.sh

8fc1738

Merge branch 'main' into feature/move-sync#ked-2961

55976e0

Add git checkout after merge.sh

92d9055

Merge remote-tracking branch 'origin/feature/move-sync#ked-2961' into…

b85cf09

… feature/move-sync#ked-2961

New line!

2db6778

antonymilne mentioned this pull request Dec 17, 2021

[KED-2961] Move scripts from private-kedro-sync to develop #1125

Merged

5 tasks

datajoely merged commit 55b38b3 into main Dec 17, 2021

antonymilne deleted the feature/move-sync#ked-2961 branch December 17, 2021 17:03

Galileo-Galilei pushed a commit to Galileo-Galilei/kedro that referenced this pull request Feb 19, 2022

[KED-1176] Document why "load_version" is not allowed (kedro-org#1107)

ee8313a

antonymilne mentioned this pull request Apr 5, 2022

Heavily refactor CircleCI config, part 1 of 2 kedro-org/kedro-viz#812

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KED-2961] Move scripts from private-kedro-sync #1107

[KED-2961] Move scripts from private-kedro-sync #1107

antonymilne commented Dec 9, 2021 •

edited

Loading

lorenabalan left a comment

lorenabalan left a comment

antonymilne commented Dec 17, 2021 •

edited

Loading

SajidAlamQB left a comment

[KED-2961] Move scripts from private-kedro-sync #1107

[KED-2961] Move scripts from private-kedro-sync #1107

Conversation

antonymilne commented Dec 9, 2021 • edited Loading

Description

Development notes

Checklist

lorenabalan left a comment

Choose a reason for hiding this comment

lorenabalan left a comment

Choose a reason for hiding this comment

antonymilne commented Dec 17, 2021 • edited Loading

SajidAlamQB left a comment

Choose a reason for hiding this comment

antonymilne commented Dec 9, 2021 •

edited

Loading

antonymilne commented Dec 17, 2021 •

edited

Loading