Skip to content

Run Continuous Integration tests on Github Actions #475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Jul 10, 2020
Merged

Run Continuous Integration tests on Github Actions #475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4c0fd39
Run Continuous Integration tests on Github Actions
weiji14 May 19, 2020
31340ee
Install make earlier
weiji14 May 19, 2020
9a4e7bc
Set Linux/MacOS environment variables properly
weiji14 May 20, 2020
51472dc
Install pytest and other dev dependencies
weiji14 May 20, 2020
9c2e393
Don't fail-fast and cancel all jobs
weiji14 May 20, 2020
2220702
Update setup-miniconda Github Action from v1 to v1.3.1
weiji14 May 20, 2020
6936b3c
Don't set any environment variables for all OSes
weiji14 May 21, 2020
6906426
Cache GMT tutorial files in ${{ github.workspace }}/.gmt on 20200605
weiji14 Jun 5, 2020
d2ce09b
Fix typo python_version to python-version
weiji14 Jun 7, 2020
e407081
Fix python_version to python-version
seisman Jun 9, 2020
841f937
Disable tests to enable caches
seisman Jun 10, 2020
9fd73f0
Re-enable tests to check caches
seisman Jun 10, 2020
1a23b55
Change github action name
seisman Jun 10, 2020
0ba3d7e
Simplify job name
seisman Jun 10, 2020
c744db9
The versioneer pacakge need tags to determine the PyGMT version
seisman Jun 10, 2020
3a06afc
Change runner.os to matrix.os
seisman Jun 10, 2020
05c225f
Improve the docs
seisman Jun 10, 2020
8bd8a66
Add a cron job for daily tests
seisman Jun 10, 2020
d4408c0
Merge branch 'master' into gh-actions-ci
seisman Jun 29, 2020
c12cfff
Change earth_relief_60m to earth_relief_01d
seisman Jun 29, 2020
9179bc2
Let conda install packages listed in requirements.txt
seisman Jun 29, 2020
8662be0
Single conda command to install all packages
seisman Jun 29, 2020
a806efa
Add github.ref to the cache key
seisman Jun 29, 2020
895e871
Disable the cache temporarily to cache the GMT files
seisman Jun 29, 2020
e0c79e6
Enable tests again
seisman Jun 29, 2020
4d271c0
Set fetch-depth to 0
seisman Jun 29, 2020
0c1ba71
Create an environment using environment.yml
seisman Jun 29, 2020
80c6508
Still slow using environment.yml
seisman Jun 29, 2020
bf2126f
Cache conda packages
seisman Jun 30, 2020
9c97722
Disable tests to cache conda packages
seisman Jun 30, 2020
37f2a53
Update cache keys
seisman Jun 30, 2020
8ff30d9
Enable tests
seisman Jun 30, 2020
1b947d3
Run Github Actions CI tests on GMT 6.1.0
weiji14 Jul 7, 2020
cd7d7da
Merge branch 'master' into gh-actions-ci
seisman Jul 9, 2020
098e03d
Revert "Run Github Actions CI tests on GMT 6.1.0"
weiji14 Jul 7, 2020
b4c7492
Disable Azure Pipeline CI on Windows and macOS, but keep Style Checks on
weiji14 Jul 9, 2020
b6445d0
Do not cache the conda packages
seisman Jul 9, 2020
b731f0c
Add the status badge to README
seisman Jul 10, 2020
5b65455
Test Python 3.6-3.8
seisman Jul 10, 2020
36fa209
Let the github actions run in all PRs
seisman Jul 10, 2020
a5af2dc
Use PYTEST_EXTRA in Makefile
seisman Jul 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ jobs:
########################################################################################
- job:
displayName: 'Mac'
condition: False

pool:
vmImage: 'macOS-10.14'
Expand Down Expand Up @@ -171,6 +172,7 @@ jobs:
########################################################################################
- job:
displayName: 'Windows'
condition: False

pool:
vmImage: 'vs2017-win2016'
Expand Down
98 changes: 98 additions & 0 deletions .github/workflows/ci_tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# This workflow will install Python dependencies, run tests and lint with a single version of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Tests

on:
push:
branches: [ master ]
pull_request:
# Schedule daily tests
schedule:
- cron: '0 0 * * *'

jobs:
test:
name: ${{ matrix.os }} - Python ${{ matrix.python-version }}
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
python-version: [3.6, 3.7, 3.8]
os: [ubuntu-latest, macOS-latest, windows-latest]

steps:
# Checkout current git repository
- name: Checkout
uses: actions/checkout@v2.3.1
with:
# fecth all history so that versioneer works
fetch-depth: 0

# Setup Miniconda
- name: Setup Miniconda
uses: goanpeca/setup-miniconda@v1.6.0
with:
python-version: ${{ matrix.python-version }}
channels: conda-forge

# Install GMT and other required dependencies from conda-forge
- name: Install GMT and required dependencies
shell: bash -l {0}
run: |
requirements_file=full-conda-requirements.txt
cat requirements.txt requirements-dev.txt > $requirements_file
cat << EOF >> $requirements_file
gmt=6.0.0
make
EOF
conda install --yes --file $requirements_file
Comment on lines +32 to +49
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can speed up the dependency install using caching too, see https://github.com/goanpeca/setup-miniconda#caching. I've used it for one of my projects, the relevant commit is at weiji14/deepicedrain@391632b.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I saw that. But I'm not sure how the caches are updated. For example, we may cache numpy v1.18.0 now. When new numpy versions are released, conda will download and install the latest version (e.g., numpy v1.19.0). What will happen to the caches? Do the caches contains the tarballs of numpy v1.18.0, v1.19.0 or both? Do we have to manually update the cache key?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache would hash the environment.yml/requirement.txt files, and unless the hash changes, the cache will stay at v1.18.0 (unless we manually update the cache key). Maybe we should start thinking about a minimum supported dependency policy, as I suggested at #340 (comment).


# Show installed pkg information for postmortem diagnostic
- name: List installed packages
shell: bash -l {0}
run: conda list

# Cache the ${HOME}/.gmt directory, for docs and testing
- name: Cache GMT directory
uses: actions/cache@v2
id: cache
with:
path: |
~/.gmt/cache
~/.gmt/server
key: cache-gmt-${{ github.ref }}-${{ runner.os }}-20200609
restore-keys: cache-gmt-refs/heads/master-

# Workaround for the timeouts of 'gmt which' on Linux and Windows
- name: Download remote data using wget (Linux & Windows)
shell: bash -l {0}
run: |
if [ "$RUNNER_OS" == "Windows" ]; then choco install wget; fi # install wget on Windows
mkdir ~/.gmt ~/.gmt/cache ~/.gmt/server
wget --no-check-certificate https://oceania.generic-mapping-tools.org/gmt_hash_server.txt -P ~/.gmt/server/
for data in earth_relief_01d.grd earth_relief_30m.grd earth_relief_10m.grd; do
wget --no-check-certificate https://oceania.generic-mapping-tools.org/${data} -P ~/.gmt/server/
done
Comment on lines +70 to +76
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to change these lines when bumping to 6.1. In 6.1, the earth relief data are stored in ~/.gmt/server/earth/earth_relief/ directory.

for data in ridge.txt Table_5_11.txt tut_bathy.nc tut_quakes.ngdc tut_ship.xyz usgs_quakes_22.txt; do
wget --no-check-certificate https://oceania.generic-mapping-tools.org/cache/${data} -P ~/.gmt/cache/
done
if: steps.cache.outputs.cache-hit != 'true' && runner.os != 'macOS'

# Download remote files, if not already cached
- name: Download remote data (macOS)
shell: bash -l {0}
run: gmt which -Gu @earth_relief_01d @earth_relief_30m @earth_relief_10m @ridge.txt @Table_5_11.txt @tut_bathy.nc @tut_quakes.ngdc @tut_ship.xyz @usgs_quakes_22.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to change gmt which -Ga in GMT 6.1.

if: steps.cache.outputs.cache-hit != 'true' && runner.os == 'macOS'

# Install the package that we want to test
- name: Install the package
shell: bash -l {0}
run: |
python setup.py sdist --formats=zip
pip install dist/*

# Run the tests
- name: Test with pytest
shell: bash -l {0}
run: make test PYTEST_EXTRA="-r P"
2 changes: 1 addition & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Build, package, test, and clean
PROJECT=pygmt
TESTDIR=tmp-test-dir-with-unique-name
PYTEST_ARGS=--cov-config=../.coveragerc --cov-report=term-missing --cov=$(PROJECT) --doctest-modules -v --mpl --mpl-results-path=results --pyargs
PYTEST_ARGS=--cov-config=../.coveragerc --cov-report=term-missing --cov=$(PROJECT) --doctest-modules -v --mpl --mpl-results-path=results --pyargs ${PYTEST_EXTRA}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't work, because we didn't declare the $PYTEST_EXTRA environment variable. Also need to remove the $PYTEST_EXTRA string from the ci_tests.yaml file.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it works. It doesn't have to be an environmental variable. make test and make test PYTEST_EXTRA="-r P" are using different pytext arguments.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now although the tests all pass, we actually can see more "errors" by using -r P:

______________________ test_call_module_invalid_arguments ______________________
----------------------------- Captured stderr call -----------------------------
gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla)
________________________ test_call_module_error_message ________________________
----------------------------- Captured stderr call -----------------------------
gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla)
__________________________ test_extract_region_fails ___________________________
----------------------------- Captured stderr call -----------------------------
pygmt-session [ERROR]: No hidden PS file found
____________________________ test_get_default_fails ____________________________
----------------------------- Captured stderr call -----------------------------
pygmt-session [ERROR]: Syntax error: Unrecognized keyword NOT_A_VALID_NAME
________________________ test_text_nonexistent_filename ________________________
----------------------------- Captured stderr call -----------------------------
text [ERROR]: Error for input file: No such file (notexist.txt)
_______________________________ test_which_fails _______________________________
----------------------------- Captured stderr call -----------------------------
gmtwhich [ERROR]: File 5a9f5f5cb31c4bb99525d0d82eaf0fc6 not found!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it because the CI (Travis and GH Actions) still uses make test PYTEST_EXTRA="-r P"? If we run make test locally after checking out this branch, it probably won't work.

Copy link
Member

@seisman seisman Jul 10, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Look at the captured stderr above:

gmtinfo [ERROR]: Error for input file: No such file (bogus-data.bla)

This is a GMT error message, but I believe it's already corrected captured in the pygmt test.

I believe the initial idea of PYTEST_EXTRA is that, when we or other users run make test locally, we won't see these "error" messages, as they're a little confusing (tests pass but there are "errors"). But in CI, make test PYTEXT_EXTRA="-r P" gives more verbose messages, useful for debugging.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then we don't actually need to change the Makefile. You'll see that PROJECT is declared before using $(PROJECT), but we don't declare PYTEST_EXTRA before using $(PYTEST_EXTRA), it is effectively blank.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you run make test, then PYTEST_EXTRA is blank; if you run make test PYTEST_EXTRA="-r P", then variable PYTEST_EXTRA is defined in shell, and passed into Makefile.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make test runs:

cd tmp-test-dir-with-unique-name; pytest --cov-config=../.coveragerc --cov-report=term-missing --cov=pygmt --doctest-modules -v --mpl --mpl-results-path=results --pyargs  pygmt

and make test PYTEST_EXTRA="-r P" runs:

cd tmp-test-dir-with-unique-name; pytest --cov-config=../.coveragerc --cov-report=term-missing --cov=pygmt --doctest-modules -v --mpl --mpl-results-path=results --pyargs -r P pygmt

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I think I get it now. It wasn't working properly before, but now it does.

BLACK_FILES=$(PROJECT) setup.py doc/conf.py examples
FLAKE8_FILES=$(PROJECT) setup.py
LINT_FILES=$(PROJECT) setup.py
Expand Down
3 changes: 3 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ PyGMT
.. image:: https://img.shields.io/azure-devops/build/GenericMappingTools/7682ad4e-76bb-4775-849e-7c4f8dce4e1a/3/master.svg?label=Mac|Windows&style=flat-square
:alt: Azure Pipelines build status
:target: https://dev.azure.com/GenericMappingTools/PyGMT/_build
.. image:: https://github.com/GenericMappingTools/pygmt/workflows/Tests/badge.svg
:alt: GitHub Actions build status
:target: https://github.com/GenericMappingTools/pygmt/actions
.. image:: https://img.shields.io/codecov/c/github/GenericMappingTools/pygmt/master.svg?style=flat-square
:alt: Test coverage status
:target: https://codecov.io/gh/GenericMappingTools/pygmt
Expand Down