Migrate from Travis to GitHub Actions (#1218)

* Rename build and publish scripts to STRTA conventions

* Create CI workflow

* Remove Travis env file sourcing from cibuild

* Move CI tests into cibuild script

* Move codecov into cibuild

* Stop running test containers with interactive TTY

* Create test script

* Add help documentation to cibuild

* Enable pseudo TTY for Docker commands

* Fix coverage configuration

An upgrade for the coverage package was necessary, along with fixing
some path references and volume mounts in the unit test container.

* Use Codecov Action instead of script

* Create release workflow to run cipublish

* Remove check for pytorch image type, interpolate IMAGE_TYPE

* Use Raster Vision Quay credentials for release

* Remove Travis config

* Run test script commands in Docker containers when on CI

* Remove obsolete release instructions

* Restrict Quay.io credentials to cipublish steps

* Add help documentation to cipublish

* Determine how to tag image in cipublish script

* Use double quotes consistently for shell conditional clauses

.coveragerc

@@ -1,10 +1,7 @@
 [run]
 source =
-    rastervision/
+    rastervision
 omit =
-    rastervision/protos/**.py
-    rastervision/utils/filter_geojson.py
-    rastervision/utils/geojson.py
-    rastervision/utils/misc.py
-    rastervision/data/vector_source/label_maker/**.py
-    rastervision/data/label/tfod_utils/**.py
+    */rastervision_core/rastervision/core/data/vector_source/label_maker/*.py
+    */rastervision_core/rastervision/core/data/label/tfod_utils/*.py
+    */rastervision_core/rastervision/core/utils/*.py

.github/workflows/continuous_integration.yml

@@ -0,0 +1,21 @@
+name: CI
+
+on:
+  pull_request:
+
+jobs:
+  build:
+    name: build
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        image_type: [pytorch]
+    env:
+      DOCKER_BUILDKIT: 1
+      IMAGE_TYPE: ${{ matrix.image_type }}
+    steps:
+      - uses: actions/checkout@v2
+
+      - run: ./scripts/cibuild
+
+      - uses: codecov/codecov-action@v2

.github/workflows/release.yml

@@ -0,0 +1,34 @@
+name: Release
+
+on:
+  push:
+    branches:
+      - master
+      - "[0-9]+" # Major version only
+      - "[0-9]+.[0-9]+" # Major and minor version
+      - "[0-9]+.[0-9]+.[0-9]+" # Major, minor, and patch version
+    tags:
+      - "*"
+
+jobs:
+  release:
+    name: release
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        image_type: [pytorch]
+    env:
+      DOCKER_BUILDKIT: 1
+      IMAGE_TYPE: ${{ matrix.image_type }}
+    steps:
+      - uses: actions/checkout@v2
+
+      - run: ./scripts/cibuild
+
+      - uses: codecov/codecov-action@v2
+
+      - run: ./scripts/cipublish
+        env:
+          QUAY_USER: ${{ secrets.QUAY_RASTERVISION_USER }}
+          QUAY_PASSWORD: ${{ secrets.QUAY_RASTERVISION_PASSWORD }}
+

README.md

@@ -5,7 +5,7 @@
 [![Docker Repository on Quay](https://quay.io/repository/azavea/raster-vision/status "Docker Repository on Quay")](https://quay.io/repository/azavea/raster-vision)
 [![Join the chat at https://gitter.im/azavea/raster-vision](https://badges.gitter.im/azavea/raster-vision.svg)](https://gitter.im/azavea/raster-vision?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
-[![Build Status](https://api.travis-ci.org/azavea/raster-vision.svg?branch=master)](http://travis-ci.org/azavea/raster-vision)
+[![Build Status](https://github.com/azavea/raster-vision/actions/workflows/release.yml/badge.svg)](https://github.com/azavea/raster-vision/actions/workflows/release.yml)
 [![codecov](https://codecov.io/gh/azavea/raster-vision/branch/master/graph/badge.svg)](https://codecov.io/gh/azavea/raster-vision)
 [![Documentation Status](https://readthedocs.org/projects/raster-vision/badge/?version=latest)](https://docs.rastervision.io/en/latest/?badge=latest)
 

docs/release.rst

@@ -35,7 +35,7 @@ Minor or Major Version Release
 #. Make a PR to the ``master`` branch with the preceding updates. In the PR, there should be a link to preview the docs. Check that they are building and look correct.
 #. Make a git branch with the version as the name, and push to Github.
 #. Ensure that the docs are building correctly for the new version branch on `readthedocs <https://readthedocs.org/projects/raster-vision/>`_. You will need to have admin access on your RTD account. Once the branch is building successfully, Under Versions -> Activate a Version, you can activate the version to add it to the sidebar of the docs for the latest version. (This might require manually triggering a rebuild of the docs.) Then, under Admin -> Advanced Settings, change the default version to the new version.
-#. Travis CI is supposed to publish an image whenever there is a push to a branch with a version number as the name. If this doesn't work or you want to publish it immediately, then you can manually make a Docker image for the new version and push to Quay. For this you will need an account on Quay.io under the Azavea organization.
+#. GitHub Actions is supposed to publish an image whenever there is a push to a branch with a version number as the name. If this doesn't work or you want to publish it immediately, then you can manually make a Docker image for the new version and push to Quay. For this you will need an account on Quay.io under the Azavea organization.
 
     .. code-block:: terminal
 
@@ -98,16 +98,6 @@ Bug Fix Release
 This describes how to create a new bug fix release, using incrementing from 0.8.0 to 0.8.1 as an example. This assumes that there is already a branch for a minor release called ``0.8``.
 
 #. To create a bug fix release (version 0.8.1), we need to backport all the bug fix commits on the ``master`` branch that have been added since the last bug fix release onto the ``0.8`` branch. For each bug fix PR on ``master``, we need to create a PR against the ``0.8`` branch based on a branch of ``0.8`` that has cherry-picked the commits from the original PR. The title of the PR should start with [BACKPORT].
-#. Make and merge a PR against ``0.8`` (but not ``master``) that increments the version in each ``setup.py`` file to ``0.8.1``. Then wait for the ``0.8`` branch to be built by Travis and the ``0.8`` Docker images to be published to Quay. If that is successful, we can proceed to the next steps of actually publishing a release.
+#. Make and merge a PR against ``0.8`` (but not ``master``) that increments the version in each ``setup.py`` file to ``0.8.1``. Then wait for the ``0.8`` branch to be built by GitHub Actions and the ``0.8`` Docker images to be published to Quay. If that is successful, we can proceed to the next steps of actually publishing a release.
 #. Using the Github UI, make a new release. Use ``0.8.1`` as the tag, and the ``0.8`` branch as the target.
-#. The image for ``0.8`` is created automatically by Travis, but we need to manually create images for ``0.8.1``. For this you will need an account on Quay  under the Azavea organization.
-
-    .. code-block:: terminal
-
-        docker login quay.io
-
-        docker pull quay.io/azavea/raster-vision:pytorch-0.8
-        docker tag quay.io/azavea/raster-vision:pytorch-0.8 quay.io/azavea/raster-vision:pytorch-0.8.1
-        docker push quay.io/azavea/raster-vision:pytorch-0.8.1
-
 #. Publish the new version to PyPI. Follow the same instructions for PyPI that are listed above for minor/major version releases.

requirements-dev.txt

@@ -1,7 +1,7 @@
 flake8==3.5.*
 moto==2.0.10
-coverage==4.5.1
-codecov==2.0.15
+coverage==5.5
+codecov==2.1.11
 yapf==0.22.*
 unify==0.4
 sphinx==4.0.*

scripts/cibuild

@@ -0,0 +1,28 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Build library for integration or a release.
+"
+}
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+    if [[ "${1:-}" == "--help" ]]; then
+        usage
+    else
+        docker build \
+            --build-arg CUDA_VERSION="10.2" \
+            -t "raster-vision-${IMAGE_TYPE}" \
+            -f Dockerfile \
+            .
+
+        ./scripts/test
+    fi
+fi

scripts/cipublish

@@ -0,0 +1,50 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Publish container images to Quay.io.
+"
+}
+
+if [[ -n "${GITHUB_SHA}" ]]; then
+    GITHUB_SHA="${GITHUB_SHA:0:7}"
+else
+    GITHUB_SHA="$(git rev-parse --short HEAD)"
+fi
+
+if [[ -z "${GITHUB_REF}" ]]; then
+    # We don't know where GitHub sources the value of GITHUB_REF. git-describe
+    # is close enough to be valid for publishing.
+    GITHUB_REF="$(git describe --all)"
+fi
+
+if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
+    if [[ "${1:-}" == "--help" ]]; then
+        usage
+    else
+        docker login -u="${QUAY_USER}" -p="${QUAY_PASSWORD}" quay.io
+
+        if [[ "${GITHUB_REF}" == "refs/heads/master" ]]; then
+            # On master, use the current SHA and latest
+            docker tag "raster-vision-${IMAGE_TYPE}" \
+                "quay.io/azavea/raster-vision:${IMAGE_TYPE}-${GITHUB_SHA}"
+            docker push "quay.io/azavea/raster-vision:${IMAGE_TYPE}-${GITHUB_SHA}"
+
+            docker tag "raster-vision-${IMAGE_TYPE}" \
+                "quay.io/azavea/raster-vision:${IMAGE_TYPE}-latest"
+            docker push "quay.io/azavea/raster-vision:${IMAGE_TYPE}-latest"
+        else
+            # For everything else, use the branch or tag name
+            docker tag "raster-vision-${IMAGE_TYPE}" \
+                "quay.io/azavea/raster-vision:${IMAGE_TYPE}-${GITHUB_REF##*/}"
+            docker push "quay.io/azavea/raster-vision:${IMAGE_TYPE}-${GITHUB_REF##*/}"
+        fi
+    fi
+fi

scripts/test

@@ -0,0 +1,60 @@
+#!/bin/bash
+
+set -e
+
+if [[ -n "${RASTER_VISION_DEBUG}" ]]; then
+    set -x
+fi
+
+function usage() {
+    echo -n \
+        "Usage: $(basename "$0")
+Run linters and tests.
+"
+}
+
+if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
+    if [[ "${1:-}" == "--help" ]]; then
+        usage
+    else
+        if [[ -n "${CI}" ]]; then
+            # Execute test suites
+            docker run \
+                --rm -t \
+                "raster-vision-${IMAGE_TYPE}" \
+                /opt/src/scripts/style_tests
+            docker run \
+                -w "$(pwd)" \
+                -v "$(pwd):$(pwd)" \
+                --rm -t \
+                "raster-vision-${IMAGE_TYPE}" \
+                /opt/src/scripts/unit_tests
+            docker run \
+                --rm -t \
+                "raster-vision-${IMAGE_TYPE}" \
+                /opt/src/scripts/integration_tests
+
+            # Create new coverage reports
+            docker run \
+                -w "$(pwd)" \
+                -v "$(pwd):$(pwd)" \
+                --rm -t \
+                "raster-vision-${IMAGE_TYPE}" \
+                coverage xml
+        else
+            # Local test suite runs against pytorch image by default
+            IMAGE_TYPE=${IMAGE_TYPE:-pytorch}
+
+            # Delete old coverage reports
+            rm -f ./.coverage ./coverage.xml
+
+            # Execute test suites
+            ./scripts/style_tests
+            ./scripts/unit_tests
+            ./scripts/integration_tests
+
+            # Create new coverage reports
+            coverage xml
+        fi
+    fi
+fi