Publish To PyPi

Goal

Publishing a Python package to PyPi and Test-PyPi.

Requirements

1. dist/ directory - contains the artifacts of sdist and bdist_wheel, see Build A Package
2. PyPi account - to host the Python package
3. twine - A CLI for publishing Python packages to PyPi

   $ pip install -U twine

Getting Started

Publish To PyPi

1. Generate an API token for PyPi
   • Login to PyPi > Account Settings > Add API Token
   • Token name: temp-global
   • Scope: Entire account (all projects)
   • Click Add token
   • Save the token in a safe place; we'll use it in the next step
2. Publish to PyPi with twine - the username should be hardcoded to __token__, do NOT use your username

   $ twine upload --username __token__ dist/*
   password: pypi-your_token...

3. Create an API token and set its scope to the newly published package - replace appy with the package name
   • Login to PyPi > Account Settings > Add API Token
   • Token name: appy
   • Scope: appy
   • Click Add token
   • Save the token in a safe place; we'll get to that
4. Remove temp-global API token - we don't need it anymore
   • Login to PyPi > Account Settings
   • Next to temp global > Click Options > Click Remove token
5. Install the package from PyPi

   $ pip install --upgrade --force-reinstall --no-deps YOUR_PACKAGE_NAME

   NOTE: If you have previously installed the package in development mode, you might want to add --upgrade to upgrade to the latest version, --force-reinstall to re-install version 0.0.1 or any other existing version, and --no-deps will prevent re-installing dependencies

Publish To Test-PyPi

How do you upload your package to PyPi, for testing purposes, without publishing it as an official release?

Test-PyPi to the rescue!

a separate instance of the Python Package Index allows you to try distribution tools and processes without affecting the real index.

1. Register for Test-PyPi
2. Do the same process as in Publish to PyPi, and add to twine's upload command the --repository testpypi flag

   $ twine upload --username __token__ --repository testpypi dist/*
   password: pypi-your_token...

3. Install the package from Test PyPi

   $ pip install --upgrade --force-reinstall --no-deps --index-url https://test.pypi.org/simple/ YOUR_PACKAGE_NAME

   NOTE: Don't let it fool you; the string simple/ is part of the URL. It's not part of the example, so don't omit it.

Using A CI/CD Service

In the previous sections, the upload was done by using prompts. We should skip prompts when using an automated process such as CI/CD. There are several ways to do so. Here's how I do it.

1. Create two secrets in your CI/CD service (GitHub Actions, drone.io, CircleCi, etc.)
   • PIP_USERNAME: __token__
   • PIP_PASSWORD: pypi-your_token...
2. Set the trigger of the CI/CD pipeline to release (edited or published)
3. Add the Publish to PyPi step to your pipeline. Here's a snippet from GitHub Actions

      - name: Publish to PyPi
        env:
          TWINE_USERNAME: ${{ secrets.PIP_USERNAME }}
          TWINE_PASSWORD: ${{ secrets.PIP_PASSWORD }}
          TWINE_NON_INTERACTIVE: true
        run: |
          twine upload ./dist/*

Release Version Validation

Following PEP 440 -- Version Identification and Dependency Specification, it's important to check the semantics of the release version before publishing to PyPi.

I've created scripts/version_validation.sh, which makes sure that the release version is according to PEP 440.

The required pattern

[N!]N(.N)*[{a|b|rc}N][.postN][.devN]

To check whether the provided version matches this pattern, with a regular expression.

^[0-9]+(\.[0-9]*)*(\.[0-9]+(a|b|rc)|(\.post)|(\.dev))*[0-9]+$

An online example, including tests, is available at https://regexr.com/5fb9q.

Explaining The Regular Expression

We'll go over it bit by bit.

• ^[0-9]+ - String must start ^ with at least one or more + digits [0-9]. Matching patterns: 0, 23, 200
• (...)* - This group () can repeat zero to infinite times *
• \.[0-9]* - The previous group must start with . (\ backslash escapes .), and zero to infinite number * of digits [0-9]. Matching patterns: ., .1, .23
• (...)* - This group () can repeat zero to infinite times *
• \.[0-9]+(a|b|rc) or \.post or \.dev - The previous group must match one of the following patterns: .1a, .1b, .1rc, .post, .dev
• [0-9]+$ - String must end ($) with at least one digit. Matching patterns: 3, 03, 92

Matching Patterns

PyPa - Packaging and distributing projects

1.2.0.dev1  # Development release
1.2.0a1     # Alpha Release
1.2.0b1     # Beta Release
1.2.0rc1    # Release Candidate
1.2.0       # Final Release
1.2.0.post1 # Post Release
15.10       # Date based release
23          # Serial release

Thoughts

• Why have I used [0-9] instead of \d? - I found out the hard way that some versions of Bash don't support \d, so sticking with [0-9] is better
• Initially, I used (?<=[0-9]) at the end of the string, instead of [0-9]+$. And again, I found out the hard way that positive lookbehind is not supported in some versions of Bash. Also, it's better to keep it simple. Using positive lookahead might look weird to future you

References

• Using Test-PyPi