Skip to content

Automate release #32

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 14 commits into from
May 5, 2023
Merged
152 changes: 127 additions & 25 deletions .github/workflows/whl-build-ec2.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,53 @@
name: CI
on:
workflow_dispatch:
inputs:
tag:
description: 'Release Tag. This is in the form x.y.z'
required: true

concurrency:
group: ${{ github.workflow }}-${{ github.head_ref }}
cancel-in-progress: true

jobs:
create-release-branch:
name: Creates a release branch and updates the version number
runs-on: ubuntu-latest
env:
CI_COMMIT_AUTHOR: CentML
CI_COMMIT_EMAIL: centml-machine-user@users.noreply.github.com
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Create release branch
run: |
git checkout -b release/${{ github.event.inputs.tag }}
git fetch
git branch --set-upstream-to=origin/main release/${{ github.event.inputs.tag }}

- name: Update version
run: |
pip install incremental
pushd analyzer
python3 -m incremental.update habitat --newversion=${{ github.event.inputs.tag }}
popd

- name: Commit updated version number and tag it
run: |
git config --global user.name "${{ env.CI_COMMIT_AUTHOR }}"
git config --global user.email "${{ env.CI_COMMIT_EMAIL }}"
git add .
git commit -am "Release version ${{ github.event.inputs.tag }}"
git push -u origin release/${{ github.event.inputs.tag }}
git tag ${{ github.event.inputs.tag }}

start-runner:
name: Start self-hosted EC2 runner
runs-on: ubuntu-latest
needs:
- create-release-branch
outputs:
label: ${{ steps.start-ec2-runner.outputs.label }}
ec2-instance-id: ${{ steps.start-ec2-runner.outputs.ec2-instance-id }}
Expand All @@ -30,54 +68,44 @@ jobs:
a10g: false
v100: false

whl-build-cu117:
build-release:
runs-on:
group: organization/t4
labels: [cu117]
needs:
- start-runner
steps:
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.HABITAT_AWS_ACCESS_KEY }}
aws-region: ${{ secrets.HABITAT_AWS_REGION }}
aws-secret-access-key: ${{ secrets.HABITAT_AWS_SECRET_ACCESS_KEY }}

- name: Display host information
run: |
uname -a
hostname
pwd
id
ls -la

- name: Fetch repository
uses: actions/checkout@v3
with:
ref: release/${{ github.event.inputs.tag }}

- name: Build Python3.7 wheel
- name: Build Python3.7 distribution
run: |
./build_scripts/build_wheel.sh python3.7

- name: Build Python3.8 wheel
run: |
./build_scripts/build_wheel.sh python3.8

- name: Build Python3.9 wheel
run: |
./build_scripts/build_wheel.sh python3.9

- name: Build Python3.10 wheel
run: |
./build_scripts/build_wheel.sh python3.10

- name: Upload wheels to S3
run: |
aws s3 cp analyzer/dist/ s3://centml-releases/habitat/wheels-cu117/ --recursive --include "*"


- uses: actions/upload-artifact@v3
with:
name: ${{ github.event.inputs.tag }}
path: analyzer/dist/*.whl

stop-runner:
name: Stop self-hosted EC2 runner
needs:
- start-runner # required to get output from the start-runner job
- whl-build-cu117 # required to wait when the main job is done
- build-release # required to wait when the main job is done
runs-on: ubuntu-latest
if: ${{ always() }} # required to stop the runner even if the error happened in the previous jobs
steps:
Expand All @@ -96,3 +124,77 @@ jobs:
t4: true # required to match the list above - otherwise the runners will not stop
a10g: false # required to match the list above - otherwise the runners will not stop
v100: false

publish-to-github:
name: Publish to Github
needs:
- build-release
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
with:
ref: release/${{ github.event.inputs.tag }}

- name: Download artifact
uses: actions/download-artifact@v3
with:
name: ${{ github.event.inputs.tag }}
path: dist

- name: Publish a release
run: |
RELEASE_NOTES="$(git log $(git describe --abbrev=0 --tags --always).. --merges --pretty=format:"%s %b" | cut -f 4,7- -d ' ')"
echo "Autogenerated Release Notes:"
echo "$RELEASE_NOTES"
RELEASE_ARTIFACTS=$(find ./dist -name "*${{ github.event.inputs.tag }}*" -type f | paste -s -d ' ' - )
VERSION_TAG="v${{ github.event.inputs.tag }}"
gh auth login --with-token <<< "${{ secrets.GH_TOKEN }}"
gh release create "$VERSION_TAG" \
--title "$VERSION_TAG" \
--notes "$RELEASE_NOTES" \
--target "$GITHUB_SHA" \
$RELEASE_ARTIFACTS
gh pr create --title "Release $VERSION_TAG" --body "$RELEASE_NOTES"

publish-to-test-pypi:
name: Publish to Test PyPI
needs: build-release
runs-on: ubuntu-latest
environment: Test
concurrency: Test
permissions:
id-token: write

steps:
- name: Download artifact
uses: actions/download-artifact@v3
with:
name: ${{ github.event.inputs.tag }}
path: dist

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository_url: https://test.pypi.org/legacy/

publish-to-pypi:
name: Publish to PyPI
needs: publish-to-test-pypi
runs-on: ubuntu-latest
environment: Production
concurrency: Production
permissions:
id-token: write

steps:
- name: Download artifact
uses: actions/download-artifact@v3
with:
name: ${{ github.event.inputs.tag }}
path: dist

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository_url: https://upload.pypi.org/legacy/
18 changes: 18 additions & 0 deletions RELEASE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# How to release a new version of DeepView.Predict
- Go to Github repo and run the action `CI`. You will be prompted to specify the version number.

- This runs a GitHub Action that will take the following steps:
1. Fetches the repo and its dependencies
2. Creates a release branch
3. Updates the version number to the user-specified version using `incremental`
4. Commits the changes and tag the commit with the version number
5. Launches an EC2 instance with a T4
6. The EC2 instance builds the Python build artifacts for Python versions 3.7-3.10
7. Publishes a release to Github
8. Create a PR to merge back into main
9. Publishes to Test PyPI
10. Publishes to PyPI

- This GitHub action is defined under `.github/workflows/whl-build-ec2.yaml`

- This release process follows the release process outlined in [OneFlow](https://www.endoflineblog.com/oneflow-a-git-branching-model-and-workflow).
3 changes: 2 additions & 1 deletion analyzer/habitat/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
traceback.print_exc()
print(MISSING_LIBRARY_MESSAGE)

__version__ = '0.1.0'
from ._version import __version__

__description__ = 'Cross-GPU performance predictions for PyTorch neural network training.'

__author__ = 'CentML'
Expand Down
11 changes: 11 additions & 0 deletions analyzer/habitat/_version.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
"""
Provides habitat version information.
"""

# This file is auto-generated! Do not edit!
# Use `python -m incremental.update habitat` to change this file.

from incremental import Version

__version__ = Version("habitat", 0, 1, 0)
__all__ = ["__version__"]
8 changes: 5 additions & 3 deletions analyzer/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@
PYTHON_VERSION = sysconfig.get_python_version().replace('.', '')

SETUP_REQUIRES = [
"patchelf"
"patchelf",
"incremental"
]

PACKAGE_DATA = {
Expand All @@ -47,7 +48,8 @@
"pandas>=1.1.2",
"tqdm>=4.49.0",
"nvidia-cuda-cupti-cu11==11.7.101",
"nvidia-cuda-runtime-cu11==11.7.99"
"nvidia-cuda-runtime-cu11==11.7.99",
"incremental"
]

KEYWORDS = [
Expand Down Expand Up @@ -128,12 +130,12 @@ def find_meta(meta):
name=NAME,
description=find_meta("description"),
license=find_meta("license"),
version=find_meta("version"),
author=find_meta("author"),
author_email=find_meta("email"),
maintainer=find_meta("author"),
maintainer_email=find_meta("email"),
long_description=read(README_PATH),
use_incremental=True,
long_description_content_type="text/markdown",
cmdclass= {
"build": CustomBuildCommand
Expand Down
2 changes: 1 addition & 1 deletion build_scripts/build_wheel.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ git lfs pull

pushd analyzer
./install-dev.sh --build
python3 setup.py sdist bdist_wheel
python3 setup.py bdist_wheel
popd

deactivate
Expand Down