Skip to content

Add Automation of Python SDK Release Process. #168

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 67 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
88398fb
add wheel building action
MOmarMiraj Mar 4, 2025
8eac2b7
fix manylinux image and dont build wheels for PyPi
MOmarMiraj Mar 4, 2025
c2627e6
go to 2_34 manylinux image as theres no 2_32 image
MOmarMiraj Mar 4, 2025
ea82680
fix linux image name
MOmarMiraj Mar 4, 2025
9742677
add full url for linux image
MOmarMiraj Mar 4, 2025
16a9cb4
update setup.py to build in correct folder
MOmarMiraj Mar 4, 2025
cd8b06f
update test cmd path
MOmarMiraj Mar 4, 2025
6f659f6
checkout only the example folder
MOmarMiraj Mar 4, 2025
deaeb15
checkout example folder correctly
MOmarMiraj Mar 4, 2025
efe387e
edit path
MOmarMiraj Mar 4, 2025
670175f
add support for shared libs packaging
MOmarMiraj Mar 5, 2025
3ad27e9
add cryptography as a wheel
MOmarMiraj Mar 5, 2025
5e84ed0
make it manual
MOmarMiraj Mar 5, 2025
35d2f0d
don't stop if fail
MOmarMiraj Mar 5, 2025
67edcbb
debug working dir
MOmarMiraj Mar 5, 2025
145758d
check file permissions
MOmarMiraj Mar 5, 2025
f316654
change to python instead of python3
MOmarMiraj Mar 5, 2025
881497d
remove other jobs and see files path
MOmarMiraj Mar 5, 2025
f7a1083
try hardcoded path
MOmarMiraj Mar 5, 2025
1cc21ce
debug
MOmarMiraj Mar 5, 2025
f94d10b
see python version and go to 3.12
MOmarMiraj Mar 5, 2025
c777afd
try
MOmarMiraj Mar 5, 2025
521efd0
check here
MOmarMiraj Mar 5, 2025
bacb8c7
try bilding wheels now
MOmarMiraj Mar 5, 2025
efc4ed4
fix path of test and removei nstallation of python
MOmarMiraj Mar 5, 2025
8f19e0d
add python3
MOmarMiraj Mar 5, 2025
1a3e9bb
revert path
MOmarMiraj Mar 5, 2025
e3ffc3e
add script
MOmarMiraj Mar 5, 2025
ea0c161
add permission
MOmarMiraj Mar 5, 2025
fce070e
fix name
MOmarMiraj Mar 5, 2025
8bba2eb
delete script
MOmarMiraj Mar 5, 2025
028d32b
fix test command
MOmarMiraj Mar 5, 2025
2e6941f
pass sa token to linux
MOmarMiraj Mar 5, 2025
4e944bd
add morgan change
MOmarMiraj Mar 5, 2025
cd9b562
fix rules of when this job wil run
MOmarMiraj Mar 5, 2025
98f4832
update sa token
MOmarMiraj Mar 5, 2025
8bf9ccb
test on test_client instead of example.py
MOmarMiraj Mar 5, 2025
e88f750
skip musllinux wheels
MOmarMiraj Mar 5, 2025
976c276
fix rules for when the job is ran
MOmarMiraj Mar 5, 2025
2a09d16
check if its manual and its a release branch
MOmarMiraj Mar 5, 2025
1fc3895
add input
MOmarMiraj Mar 5, 2025
c3f0211
remove the push
MOmarMiraj Mar 5, 2025
cc5cf76
fix up yml file
MOmarMiraj Mar 5, 2025
5690c32
revert example file
MOmarMiraj Mar 5, 2025
2345ad1
clean up setup.py
MOmarMiraj Mar 5, 2025
a16e535
add build sdist and fix EOL
MOmarMiraj Mar 5, 2025
94fe548
add test job for source distro
MOmarMiraj Mar 6, 2025
1a487ca
test sdist job
MOmarMiraj Mar 6, 2025
f7b1d8c
install python dep for source distro
MOmarMiraj Mar 6, 2025
3593dfb
clean up .yml file
MOmarMiraj Mar 6, 2025
212f8a6
add publishing of pypi to workflow
MOmarMiraj Mar 6, 2025
fa1c3ec
fix wheels to correctly build and publish wheels to pypi and add pypr…
MOmarMiraj Mar 8, 2025
314cdc7
add new line at EOF
MOmarMiraj Mar 8, 2025
a004bf2
Add prep-release and release and update wheels according to fork of P…
MOmarMiraj Apr 3, 2025
43548d1
remove release notes and add comments
MOmarMiraj Apr 3, 2025
2474110
Merge branch 'main' into omar/automate-wheel-build
MOmarMiraj Apr 3, 2025
5750b30
remove unnecessary c omments
MOmarMiraj Apr 3, 2025
95c3aee
update setting up git user to use person who commited to be the one w…
MOmarMiraj Apr 4, 2025
0a8e17a
Apply suggestions from code review
MOmarMiraj May 9, 2025
55c7792
Merge branch 'main' into omar/automate-wheel-build
MOmarMiraj May 9, 2025
c63f59b
Clean up files and change version.txt to .VERSION
MOmarMiraj May 9, 2025
6891a34
remove wheels.yml and add building+publishing to release
MOmarMiraj May 9, 2025
62c6f61
add passphrase and token env variables for signing commits
MOmarMiraj May 9, 2025
2e6d5e5
Remove prep release action and combine it with the release action and…
MOmarMiraj May 12, 2025
1debd81
add new line at EOF of version file and fix script to trim it. Addres…
MOmarMiraj May 20, 2025
6b1d7eb
fail prep release if on CI to avoid inf loop
MOmarMiraj May 20, 2025
4699a2e
update wheels script to match CI/CD incase of error
MOmarMiraj May 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.3.1
172 changes: 172 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
name: Release Python SDKs

on:
workflow_dispatch:
inputs:
version:
description: "New version for the SDKs in the format of M.m.p"
required: true
type: string
build_number:
description: "New build number for the SDKs in the format of Mmmppbb "
required: true
type: string

jobs:
prepare-release:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/') # Only run on branches that start with sdk-core/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: are we able to enforce this more? E.g. with a regex, also matching on the part that comes after sdk-core/?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can but we Github Actions don't support regex so we would have to create a whole job to check this which I don't think its necessary. In our case, we want to ensure the branches are in forms of the sdk-core/yyyy-mm-dd-hash but regardless we don't have any checks to ensure that the RC branch are in this form or we verify the commit sha so this additional regex check can add additional verification but for very little upside.

env:
SDK_VERSION: ${{ github.event.inputs.version }}
SDK_BUILD_NUMBER: ${{ github.event.inputs.build_number }}
steps:
- name: Checkout the code
uses: actions/checkout@v4

- name: Import GPG key
uses: crazy-max/ghaction-import-gpg@v6
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.PASSPHRASE }}
git_user_signingkey: true
git_commit_gpgsign: true

- name: Run the Prep Release Script
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SDK_CI: 1
run: |
make prep-release VERSION=${{ env.SDK_VERSION }} BUILD_NUMBER=${{ env.SDK_BUILD_NUMBER }}
shell: bash

build-wheels:
name: Build wheels for Python SDK on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
if: startsWith(github.ref, 'refs/heads/sdk-core/')
needs: [prepare-release]
strategy:
# we don't want all of them failing if one fails
fail-fast: false
matrix:
# macOS 13 is an Intel runner and macOS 14 is an Apple Silicon runner
os: [ubuntu-22.04, ubuntu-22.04-arm, windows-latest, macos-13, macos-14]
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}

- name: Upgrade build dependencies
run: python -m pip install --upgrade pip setuptools wheel

# Need to grab the SDK version for the wheel name
- name: Extract SDK Version
run: echo "SDK_VERSION=$(cat .VERSION)" >> "$GITHUB_ENV"
shell: bash

- name: Install cibuildwheel
run: |
python -m pip install cibuildwheel

- name: Build wheels
env:
# Skip all the unneeded images like PyPy and musllinux images
CIBW_SKIP: pp* *-musllinux_*
# Set the manylinux X86_64/aarch64 image to this specific 2.34 image
CIBW_MANYLINUX_X86_64_IMAGE: "quay.io/pypa/manylinux_2_34_x86_64"
CIBW_MANYLINUX_AARCH64_IMAGE: "quay.io/pypa/manylinux_2_34_aarch64"
# Set the wheel to the native archtiecture (output of platform.machine() which we use in the setup.py script)
CIBW_ARCHS: "native"
# Windows reparing of wheels are not supported so manually install it.
CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel"
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}"
# Dependencies required for testing each wheel
CIBW_TEST_REQUIRES: "pydantic pytest pytest-asyncio"
# The minimum MacOS version for darwin wheels (matches OPH)
MACOSX_DEPLOYMENT_TARGET: "12.0"
# The command to test every wheel
CIBW_TEST_COMMAND: "python -m pytest {project}/src/onepassword/test_client.py"
# Pass the service account token in all wheel buildings for testing of the wheels.
CIBW_ENVIRONMENT: OP_SERVICE_ACCOUNT_TOKEN=${{ secrets.TEST_SERVICE_ACCOUNT_TOKEN }}
run: |
python -m cibuildwheel --output-dir dist

- uses: actions/upload-artifact@v4
with:
name: onepassword-sdk-${{ env.SDK_VERSION }}-${{ matrix.os }}
path: ./dist/*.whl

build-sdist:
name: Build source distribution for Python SDK
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/')
needs: [prepare-release]
steps:
- uses: actions/checkout@v4
with:
ref: ${{ github.ref }}
# Need to grab the SDK version for the wheel name
- name: Extract SDK Version
run: echo "SDK_VERSION=$(cat .VERSION)" >> "$GITHUB_ENV"
shell: bash

- name: Install dependencies
run: pip3 install build pydantic pytest pytest-asyncio

- name: Build source distribution
run: python3 -m build --sdist

- name: Test Source Distribution
env:
OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.TEST_SERVICE_ACCOUNT_TOKEN }}
run: |
python3 -m pip install dist/*.tar.gz
python3 -m pytest src/onepassword/test_client.py

- uses: actions/upload-artifact@v4
with:
name: onepassword-sdk-${{ env.SDK_VERSION }}
path: ./dist/*.tar.gz

Release-SDK:
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/') # Only run on branches that start with sdk-core/
needs: [build-wheels, build-sdist]
steps:
- name: Checkout the code
uses: actions/checkout@v4
with:
ref: ${{ github.ref }}

- name: Import GPG key
uses: crazy-max/ghaction-import-gpg@v6
with:
gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
passphrase: ${{ secrets.PASSPHRASE }}
git_user_signingkey: true
git_commit_gpgsign: true
git_tag_gpgsign: true

- name: Run the Release Script
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: make release
shell: bash

publish-to-pypi:
name: Publish to PyPI
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/heads/sdk-core/')
environment:
name: pypi
url: https://pypi.org/project/onepassword-sdk/
permissions:
id-token: write # IMPORTANT: this permission is mandatory for trusted publishing
needs: [Release-SDK]
steps:
- uses: actions/download-artifact@v4
with:
pattern: onepassword-sdk-*
path: ./dist
merge-multiple: true
- name: Publish package distributions to PyPi
uses: pypa/gh-action-pypi-publish@release/v1.12
33 changes: 33 additions & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
[build-system]
requires = ["setuptools>=66", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "onepassword-sdk"
dynamic = ["version"]
description = "The 1Password Python SDK offers programmatic read access to your secrets in 1Password in an interface native to Python."
authors = [{ name = "1Password" }]
license = { file = "LICENSE" }
readme = "README.md"
requires-python = ">=3.9"
classifiers = [
"Development Status :: 5 - Production/Stable",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"License :: OSI Approved :: MIT License",
]
dependencies = [
"pydantic>=2.5",
]

[project.urls]
Homepage = "https://github.com/1Password/onepassword-sdk-python"

[tool.setuptools.dynamic]
version = {file = "./.VERSION"}
33 changes: 5 additions & 28 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
from pathlib import Path
from setuptools import setup, find_packages
from setuptools import setup, find_packages, Distribution
from sysconfig import get_platform
from version import SDK_VERSION
import platform
import os

Expand All @@ -19,6 +17,9 @@ def finalize_options(self):
except ImportError:
bdist_wheel = None

class BinaryDistribution(Distribution):
def has_ext_modules(self):
return True

def get_shared_library_data_to_include():
# Return the correct uniffi C shared library extension for the given platform
Expand Down Expand Up @@ -46,35 +47,11 @@ def get_shared_library_data_to_include():


setup(
name="onepassword-sdk",
version=SDK_VERSION,
author="1Password",
long_description=(Path(__file__).parent / "README.md").read_text(),
long_description_content_type="text/markdown",
description="The 1Password Python SDK offers programmatic read access to your secrets in 1Password in an interface native to Python.",
url="https://github.com/1Password/onepassword-sdk-python",
packages=find_packages(
where="src",
),
license="MIT",
license_files="LICENSE",
distclass=BinaryDistribution,
package_dir={"": "src"},
python_requires=">=3.9",
classifiers=[
"Development Status :: 5 - Production/Stable",
"Operating System :: MacOS",
"Operating System :: POSIX :: Linux",
"Operating System :: Microsoft :: Windows",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"License :: OSI Approved :: MIT License",
],
cmdclass={"bdist_wheel": bdist_wheel},
package_data={"": get_shared_library_data_to_include()},
install_requires=[
"pydantic>=2.5", # Minimum Pydantic version to run the Python SDK
],
)
14 changes: 12 additions & 2 deletions src/release/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
## How to Prepare a Release for the Python SDK
# How to Release the Python SDK

## Release off an Release Candidate Branch in Github Actions
To release the Python SDK via Github Action, you must do the following:
1. SDK core opens a new PR with the latest generated code and latest core. This branch should start off with `sdk-core/...`
2. Add the release notes for the RC as well as update the examples if needed.
3. Run the `Release Python SDKs` action and input the correct build and version number while referencing the RC branch.
4. After the action is completed, the Python SDK is released on Github and PyPi, you can merge the PR branch.

If the Github Action isn't working, you can follow the manual steps below to release the Python SDK.

## Manual Steps to release a Python SDK
Before running this script, the user must make sure that they have the write permissions to the Python SDK repository.

Run this make command to install all dependencies required for the Python SDK release process.
Expand All @@ -10,7 +20,7 @@ release/install-dependencies
Step 1. Make any changes to the SDK as required on a feature branch or main branch.
NOTE: If ran on a main branch, a release branch will be created.

Step 2. Go to the root of the repo and run
Step 2. Go to the root of the repo and run
```
make prep-release
```
Expand Down
30 changes: 5 additions & 25 deletions src/release/scripts/build-wheels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,21 @@

# Helper script to build the required wheels for the Python SDK

output_version_file="version.py"

# The list of python verisons the SDKs release for
python_versions=("$@")

# Minimum glibc version we support
glibc_version=2-32

# These versions are being supported due to the SDKs supporting Python 3.9+
macOS_version_x86_64=10.9
macOS_version_arm64=11.0
glibc_version=2-34

# Extracts the current verison number for cleanup function
current_version=$(awk -F "['\"]" '/SDK_VERSION =/{print $2}' "$output_version_file")
current_version=$(cat .VERSION)

# Function to execute upon exit
cleanup() {
echo "Performing cleanup tasks..."
# Remove dist and egg-info and the potential release candidate if created
rm -r dist src/*.egg-info/ onepassword_sdk-"${current_version}"
exit 1
exit 1
}

# Set the trap to call the cleanup function on exit
Expand All @@ -44,23 +38,9 @@ build_wheels() {
export PYTHON_OS_PLATFORM=$os_platform
export PYTHON_MACHINE_PLATFORM=$machine_platform

case "$os_platform" in
case "$os_platform" in
Darwin)
macos_version=
# Min MacOS version for Python 3.13+ is 10.13
python_version=$(pyenv exec python3 --version 2>&1)

if [[ "$machine_platform" == "x86_64" ]]; then
if [[ "$python_version" == "Python 3.13"* ]]; then
macos_version="10.13"
else
macos_version=$macOS_version_x86_64
fi
else
macos_version=$macOS_version_arm64
fi

export _PYTHON_HOST_PLATFORM="macosx-${macos_version}-${PYTHON_MACHINE_PLATFORM}"
export _PYTHON_HOST_PLATFORM="macosx-12.0-${PYTHON_MACHINE_PLATFORM}"
;;
Linux)
export _PYTHON_HOST_PLATFORM="manylinux-${glibc_version}-${PYTHON_MACHINE_PLATFORM}"
Expand Down
Loading
Loading