-
Notifications
You must be signed in to change notification settings - Fork 906
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[KED-2961] Move scripts from private-kedro-sync (#1107)
* Pre-tidy * Simplify linux e2e_tests * Simplify linux e2e_tests * Move e2e_tests from command to job * String python version * Comment out unwanted tests for now * Try parametrised command name * Try conditionals * Indent * Indent * Urgh yaml * Pass arguments * Try win_e2e_tests separately * Try win_e2e_tests separately * Oops * Fix windows hopefully * URgh yaml * Alter lint to matrix * De-parametrise executor * Rename executors * Oops * Add win_pip_compile * Add win unit tests * Add viz build * Cosmetic changes * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Refactor into setup and win_setup * Back to parametrised executor * Back to parametrised executor * Add unit_tests * Clarify comment * Add all files to tools/sync * Remove unwanted workflows * Alter file paths and remove clone_repo * Replace build_kedro with one job * Replace build_kedro with one job * Trigger sync on merge to main * Trigger on merge to main rather than nightly * Remove old private-kedro ssh * Merge into main config.yml file * Add unless * Correct to match schema * Lint * Tidy * Hopefully correct a filepath? * Alter PR message * Update tools/circleci/github_scripts/merge.sh Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com> * Update requirements * Remove SSH and pre-commit install * Post-merge fix * Update text * Make two different CIRCLE_TOKENs * Update .circleci/config.yml * Move automatic merging to hourly rather than main_updated * Move merge to above trigger release workflow * Add git checkout after merge.sh * Add git checkout after merge.sh * New line! Co-authored-by: Lorena Bălan <lorena.balan@quantumblack.com>
- Loading branch information
1 parent
0ce3ecc
commit 55b38b3
Showing
11 changed files
with
560 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| KEDRO_VERSION=$1 | ||
|
|
||
| PYPI_ENDPOINT="https://pypi.org/pypi/kedro/${KEDRO_VERSION}/json/" | ||
|
|
||
| STATUS_CODE=$(curl --location --silent \ | ||
| --output /dev/null \ | ||
| --write-out "%{http_code}\n" \ | ||
| "${PYPI_ENDPOINT}") | ||
|
|
||
| [ "${STATUS_CODE}" == "404" ] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Exit script if you try to use an uninitialized variable. | ||
| set -o nounset | ||
|
|
||
| # Exit script if a statement returns a non-true return value. | ||
| set -o errexit | ||
|
|
||
| PROJECT_SLUG=$1 | ||
| CIRCLE_ENDPOINT="https://circleci.com/api/v2/project/${PROJECT_SLUG}/pipeline" | ||
|
|
||
| PAYLOAD=$(cat <<-END | ||
| { | ||
| "branch": "${CIRCLE_BRANCH}", | ||
| "parameters": {"release_kedro": true} | ||
| } | ||
| END | ||
| ) | ||
|
|
||
| curl -X POST \ | ||
| --silent --show-error --fail --retry 3 \ | ||
| --output /dev/null \ | ||
| --header "Content-Type: application/json" \ | ||
| --header "Circle-Token: ${CIRCLE_RELEASE_TOKEN}" \ | ||
| --data "${PAYLOAD}" \ | ||
| "${CIRCLE_ENDPOINT}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| FROM cimg/python:3.8 | ||
|
|
||
| WORKDIR /home/circleci | ||
|
|
||
| RUN sudo apt-get update && \ | ||
| sudo apt-get install curl pandoc openjdk-8-jdk-headless -y && \ | ||
| sudo apt-get clean && \ | ||
| sudo update-alternatives --set java /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java | ||
|
|
||
| # Update cacerts: https://stackoverflow.com/a/50103533/1684058 | ||
| RUN sudo rm /etc/ssl/certs/java/cacerts && \ | ||
| sudo update-ca-certificates -f | ||
|
|
||
| RUN curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \ | ||
| > miniconda.sh && bash miniconda.sh -b -p /home/circleci/miniconda | ||
|
|
||
| RUN sudo rm -rf ~/.pyenv/ /opt/circleci/.pyenv/ | ||
|
|
||
| ARG PIP_REQS | ||
| ARG PY_VERSION | ||
| ARG CONDA_ENV_NAME=kedro_builder | ||
|
|
||
| # Install/Setup anaconda env | ||
| RUN bash -c "source /home/circleci/miniconda/etc/profile.d/conda.sh && \ | ||
| echo \"$PIP_REQS\" > /tmp/requirements.txt && \ | ||
| conda create --name=$CONDA_ENV_NAME python=$PY_VERSION && \ | ||
| conda activate $CONDA_ENV_NAME && \ | ||
| pip install --no-cache-dir --prefer-binary --upgrade -r /tmp/requirements.txt" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
| cd "$(dirname "$0")" | ||
|
|
||
| KEDRO_REPO=$1 | ||
| ECR_IMAGE_URL=$2 | ||
|
|
||
| get_pip_reqs() { | ||
| local project_path=$1 | ||
| cat $project_path/*requirements.txt | grep -v requirements | ||
| } | ||
|
|
||
| docker_build() { | ||
| local pip_reqs="$1" | ||
| local py_version=$2 | ||
| local image=$ECR_IMAGE_URL:$py_version | ||
| echo "Building docker image: $image" | ||
| docker build -t $image \ | ||
| --build-arg PIP_REQS="$pip_reqs" \ | ||
| --build-arg PY_VERSION=$py_version \ | ||
| . | ||
| } | ||
|
|
||
| docker_push() { | ||
| local py_version=$1 | ||
| local image=$ECR_IMAGE_URL:$py_version | ||
| echo "Pushing docker image: $image" | ||
| docker push $image | ||
| } | ||
|
|
||
| main() { | ||
| local pip_reqs="$(get_pip_reqs $KEDRO_REPO)" | ||
|
|
||
| # Image for python 3.6 | ||
| docker_build "$pip_reqs" 3.6 | ||
| docker_push 3.6 | ||
|
|
||
| # Image for python 3.7 | ||
| docker_build "$pip_reqs" 3.7 | ||
| docker_push 3.7 | ||
|
|
||
| # Image for python 3.8 | ||
| docker_build "$pip_reqs" 3.8 | ||
| docker_push 3.8 | ||
|
|
||
| } | ||
|
|
||
| main |
Oops, something went wrong.