Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FIX-#1867: establish CI #1868

Merged
merged 47 commits into from
Aug 3, 2020
Merged

FIX-#1867: establish CI #1868

Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
2a2f3ce
FIX-#1867: add '--no-cache-dir' option
anmyachev Jul 30, 2020
2119aeb
FIX-#1867: use 'pip cache purge'
anmyachev Jul 30, 2020
bc685cd
FIX-#1867: use pip cache list
anmyachev Jul 30, 2020
5ea4819
FIX-#1867: remove purge usage
anmyachev Jul 30, 2020
3024acf
FIX-#1867: upgrade pip
anmyachev Jul 30, 2020
b82c283
FIX-#1867: upgrade setuptools and wheel also
anmyachev Jul 30, 2020
c896e55
FIX-#1867: remove '--no-cache-dir' option; upgrade pip; pin pytest<6
anmyachev Jul 30, 2020
4ed57db
FIX-#1867: use setup-python@2; use checkout@v2
anmyachev Jul 30, 2020
4bd5102
FIX-#1867: add pip list
anmyachev Jul 30, 2020
bfa8aa1
FIX-#1867: use 2020-resolver feature
anmyachev Jul 30, 2020
880bc8f
FIX-#1867: remove extra command
anmyachev Jul 30, 2020
18207ff
FIX-#1867: install pyarrow explicitly
anmyachev Jul 30, 2020
a415d60
FIX-#1867: add verbose mode
anmyachev Jul 31, 2020
9e6e289
FIX-#1867: try to use setup-miniconda action
anmyachev Jul 31, 2020
d0b6ee8
FIX-#1867: remove setup-python action
anmyachev Jul 31, 2020
8d7ce4a
FIX-#1867: fix test-api
anmyachev Jul 31, 2020
52e4ba5
FIX-#1867: add test-all, test-windows, test-pyarrow with conda usage
anmyachev Jul 31, 2020
2fad256
FIX-#1867: try to use cache
anmyachev Jul 31, 2020
67e378b
FIX-#1867: add some options for proper cache usage
anmyachev Jul 31, 2020
86a4b8d
FIX-#1867: add pip cache
anmyachev Jul 31, 2020
970ebbf
FIX-#1867: work with coverage inside modin environment
anmyachev Jul 31, 2020
bb79dcd
FIX-#1867: skip failed tests
anmyachev Jul 31, 2020
aee03e9
FIX-#1867: remove CODECOV_TOKEN
anmyachev Jul 31, 2020
fb15fae
FIX-#1867: remove pytest-testmon usage
anmyachev Jul 31, 2020
2671880
FIX-#1867: fix CACHE_NUMBER setup for 'Cache pip' step
anmyachev Jul 31, 2020
663db73
FIX-#1867: return 'requirements.txt' for teamcity tests
anmyachev Jul 31, 2020
87102e6
FIX-#1867: fix coverage report name on windows
anmyachev Aug 1, 2020
d6edee1
FIX-#1867: add tests for python 3.8
anmyachev Aug 1, 2020
031b10d
FIX-#1867: use unix-like separator for codecov on Windows
anmyachev Aug 1, 2020
7b7e563
FIX-#1867: try to use 'github.run_id' for cache key
anmyachev Aug 2, 2020
ed5fdeb
FIX-#1867: add include option for coverage
anmyachev Aug 2, 2020
d18daa1
FIX-#1867: update push.yml
anmyachev Aug 2, 2020
ae33bf6
FIX-#1867: test if condition
anmyachev Aug 2, 2020
23e0da7
FIX-#1867: fix cache pip
anmyachev Aug 2, 2020
9a64063
FIX-#1867: remove auto-activate-base option
anmyachev Aug 2, 2020
ba4797d
FIX-#1867: remove pytest_custom-exit-code
anmyachev Aug 2, 2020
84c9592
Merge branch 'master' of https://github.com/modin-project/modin into …
anmyachev Aug 3, 2020
95ac6b6
FIX-#1867: update test-experimental
anmyachev Aug 3, 2020
7609a07
FIX-#1867: remove auto-update-conda option for ubuntu
anmyachev Aug 3, 2020
af23f3b
FIX-#1867: add prepare-cache job to ci.yml
anmyachev Aug 3, 2020
7fd6919
FIX-#1867: fix needs
anmyachev Aug 3, 2020
7a6383e
FIX-#1867: fix yml syntax
anmyachev Aug 3, 2020
ab60ee5
FIX-#1867: change if condition
anmyachev Aug 3, 2020
1b30bbb
FIX-#1867: return python3.7 for test API
anmyachev Aug 3, 2020
8f72b9e
FIX-#1867: use runner.os for key name
anmyachev Aug 3, 2020
9a92f9d
FIX-#1867: skip tests only on Windows
anmyachev Aug 3, 2020
8ae3eae
FIX-#1867: remove Cache conda step
anmyachev Aug 3, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
262 changes: 204 additions & 58 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,172 +19,318 @@ jobs:
- run: git remote add upstream https://github.com/modin-project/modin.git
- run: git fetch upstream
- run: npx commitlint --from upstream/master --to HEAD --verbose

lint-black:
name: lint (black)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- uses: actions/setup-python@v2
with:
python-version: "3.7.x"
architecture: "x64"
- run: pip install black
- run: black --check --diff modin/

lint-flake8:
name: lint (flake8)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- uses: actions/setup-python@v2
with:
python-version: "3.7.x"
architecture: "x64"
- run: pip install flake8 flake8-print
- run: flake8 --enable=T modin

test-api:
runs-on: ubuntu-latest
name: test api
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
python-version: "3.7.x"
architecture: "x64"
path: ~/conda_pkgs_dir
# github.run_id using for create new cache entry for each run
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whose job is to clean the cache?

Copy link
Collaborator Author

@anmyachev anmyachev Aug 3, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really, the old cache will be deleted only if the total amount of caches stored under different keys exceeds the 5GB limit.

https://github.com/actions/cache#cache-limits

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we have so many concurrent CI runs that cache is out of space?..
And the question still stays - what is going to clean the cache?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So - how many different variants of the cache (keys) we have - 12 (2 - OS, 3 - python versions, 2 managers - pip and conda). At the moment, one cache write for pep is <50MB, one cache for conda is <300MB. Less than 3GB in total for one CI run. So part of cache will be clean up by 2 concurrent CI runs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may not use the Conda cache, only pip. Then problems should not arise up to 10 parallel CI runs. Need to look at performance.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep in mind this restriction then, and use cache for all runs for now, as I don't think we have enough resources for a lot of concurrent CI runs anyway.

Maybe dropping a comment about that in the .yml file would be good.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the conda cache doesn't make performance worse. So let's cache only pip packages.

# this allows us to use actual packages
key: ${{ runner.os }}-python-3.7-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-3.7-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
with:
activate-environment: modin
environment-file: environment.yml
python-version: "3.7"
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
# `shell: bash -l {0}` - special way to activate modin environment
shell: bash -l {0}
run: |
conda info
conda list
- run: sudo apt update && sudo apt install -y libhdf5-dev
- run: pip install -r requirements.txt
- run: python -m pytest modin/pandas/test/test_api.py
- name: Api tests
shell: bash -l {0}
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
run: python -m pytest modin/pandas/test/test_api.py

test-headers:
runs-on: ubuntu-latest
name: test-headers
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
python-version: "3.7.x"
architecture: "x64"
- run: pip install -r requirements.txt
- run: python -m pytest modin/test/test_headers.py
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-python-3.8-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-3.8-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
with:
activate-environment: modin
environment-file: environment.yml
python-version: "3.8"
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
shell: bash -l {0}
run: |
conda info
conda list
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
- name: Headers tests
shell: bash -l {0}
run: python -m pytest modin/test/test_headers.py

test-internals:
runs-on: ubuntu-latest
name: test-internals
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
python-version: "3.6.x"
architecture: "x64"
- run: pip install -r requirements.txt
- run: python -m pytest modin/test/test_publisher.py modin/data_management/test/test_dispatcher.py
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-python-3.6-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-3.6-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
with:
activate-environment: modin
environment-file: environment.yml
python-version: "3.6"
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
shell: bash -l {0}
run: |
conda info
conda list
- name: Internals tests
shell: bash -l {0}
run: python -m pytest modin/test/test_publisher.py modin/data_management/test/test_dispatcher.py

test-all:
needs: [lint-commit, lint-flake8, lint-black, test-api, test-headers]
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.6.x", "3.7.x"]
python-version: ["3.6", "3.7", "3.8"]
engine: ["python", "ray", "dask"]
part: ["Reduction_A", "Reduction_B", "Binary", "MapMetadata", "UDF", "Default", "Window", "Indexing", "Iter", "JoinSort", 3]
env:
MODIN_ENGINE: ${{matrix.engine}}
MODIN_MEMORY: 1000000000
name: test (${{matrix.engine}}, part ${{matrix.part}}, python ${{matrix.python-version}})
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
path: ~/conda_pkgs_dir
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
key: ${{ runner.os }}-python-${{ matrix.python-version }}-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-${{ matrix.python-version }}-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
with:
activate-environment: modin
environment-file: environment.yml
python-version: ${{matrix.python-version}}
architecture: "x64"
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
shell: bash -l {0}
run: |
conda info
conda list
- name: Install HDF5
if: matrix.part == 3
run: sudo apt update && sudo apt install -y libhdf5-dev
- run: pip install -r requirements.txt
- run: bash run-tests.sh ${{matrix.engine}} -k "TestDataFrame${{matrix.part}}"
- shell: bash -l {0}
run: pytest modin/pandas/test/ -k "TestDataFrame${{matrix.part}}"
if: matrix.part != 3
- run: python -m pytest modin/pandas/test/test_series.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_series.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_rolling.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_rolling.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_concat.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_concat.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_groupby.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_groupby.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_reshape.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_reshape.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_general.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_general.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_io.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_io.py
if: matrix.part == 3
- run: python -m pytest modin/experimental/pandas/test/test_io_exp.py
- shell: bash -l {0}
run: python -m pytest modin/experimental/pandas/test/test_io_exp.py
if: matrix.part == 3
- run: bash <(curl -s https://codecov.io/bash)
- shell: bash -l {0}
run: bash <(curl -s https://codecov.io/bash)

test-windows:
needs: [lint-commit, lint-flake8, lint-black, test-api, test-headers]
runs-on: windows-latest
strategy:
matrix:
python-version: ["3.6.x", "3.7.x"]
python-version: ["3.6", "3.7", "3.8"]
engine: ["ray", "dask"]
part: ["Reduction_A", "Reduction_B", "Binary", "MapMetadata", "UDF", "Default", "Window", "Indexing", "Iter", "JoinSort", 3]
env:
MODIN_ENGINE: ${{matrix.engine}}
MODIN_MEMORY: 1000000000
name: test-windows
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
path: ~/conda_pkgs_dir
vnlitvinov marked this conversation as resolved.
Show resolved Hide resolved
key: ${{ runner.os }}-python-${{ matrix.python-version }}-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~\AppData\Local\pip\Cache
anmyachev marked this conversation as resolved.
Show resolved Hide resolved
key: ${{ runner.os }}-python-${{ matrix.python-version }}-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
with:
activate-environment: modin
environment-file: environment.yml
python-version: ${{matrix.python-version}}
architecture: "x64"
- run: pip install -r requirements.txt
- run: python -m pytest modin/pandas/test/test_dataframe.py::TestDataFrame${{matrix.part}}
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
shell: bash -l {0}
run: |
conda info
conda list
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_dataframe.py::TestDataFrame${{matrix.part}}
if: matrix.part != 3
- run: python -m pytest modin/pandas/test/test_series.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_series.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_rolling.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_rolling.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_concat.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_concat.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_groupby.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_groupby.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_reshape.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_reshape.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_general.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_general.py
if: matrix.part == 3
- run: python -m pytest modin/pandas/test/test_io.py
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_io.py
if: matrix.part == 3
- run: choco install codecov
- run: codecov -f .\coverage.xml -t ${{secrets.CODECOV_TOKEN}}
- shell: bash -l {0}
run: choco install codecov
- shell: bash -l {0}
run: codecov -f ./coverage.xml

test-pyarrow:
needs: [lint-commit, lint-flake8, lint-black, test-api, test-headers]
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.6.x", "3.7.x"]
python-version: ["3.6", "3.7", "3.8"]
env:
MODIN_BACKEND: pyarrow
MODIN_EXPERIMENTAL: "True"
name: test (pyarrow, python ${{matrix.python-version}})
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
with:
fetch-depth: 1
- uses: actions/setup-python@v1
- name: Cache conda
uses: actions/cache@v1
with:
path: ~/conda_pkgs_dir
key: ${{ runner.os }}-python-${{ matrix.python-version }}-conda-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- name: Cache pip
uses: actions/cache@v1
with:
path: ~/.cache/pip
key: ${{ runner.os }}-python-${{ matrix.python-version }}-pip-${{ github.run_id }}-${{ hashFiles('environment.yml') }}
- uses: goanpeca/setup-miniconda@v1.6.0
with:
activate-environment: modin
environment-file: environment.yml
python-version: ${{matrix.python-version}}
architecture: "x64"
channel-priority: strict
use-only-tar-bz2: true # IMPORTANT: This needs to be set for caching to work properly!
auto-update-conda: true
- name: Conda environment
shell: bash -l {0}
run: |
conda info
conda list
- run: sudo apt update && sudo apt install -y libhdf5-dev
- run: pip install -r requirements.txt
- run: python -m pytest modin/pandas/test/test_io.py::test_from_csv
- shell: bash -l {0}
run: python -m pytest modin/pandas/test/test_io.py::test_from_csv
Loading