Skip to content

MAINT: Actually use all caches #8702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 6, 2021
Merged

MAINT: Actually use all caches #8702

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c7b38a1
MAINT: Actually use all caches [circle full] [skip azp] [skip github]
larsoner Jan 6, 2021
587b47a
FIX: Key [circle full] [skip azp] [skip github]
larsoner Jan 6, 2021
1e6ed7c
FIX: New keys, split for size [circle full]
larsoner Jan 6, 2021
bba0bc6
ENH: One more [circle front]
larsoner Jan 6, 2021
f8cd655
MAINT: One more [circle front] [skip azp] [skip github]
larsoner Jan 6, 2021
4cda52d
FIX: One more [circle front] [skip github] [skip azp]
larsoner Jan 6, 2021
4866e0f
FIX: Probably final tweak [circle front] [skip azp] [skip github]
larsoner Jan 6, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 137 additions & 61 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,21 @@ jobs:
docker:
- image: circleci/python:3.8.5-buster
steps:
- restore_cache:
keys:
- source-cache
- checkout
- run:
name: Complete checkout
command: |
if ! git remote -v | grep upstream; then
git remote add upstream git://github.com/mne-tools/mne-python.git
fi
git fetch upstream
- save_cache:
key: source-cache
paths:
- ".git"
- run:
name: Set BASH_ENV
command: |
Expand All @@ -21,6 +35,8 @@ jobs:
echo "export PATH=~/.local/bin/:${MNE_ROOT}/bin:$PATH" >> $BASH_ENV
echo "BASH_ENV:"
cat $BASH_ENV
mkdir -p ~/mne_data
touch pattern.txt;
- run:
name: check neuromag2ft
command: |
Expand All @@ -33,25 +49,9 @@ jobs:
echo ${CI_PULL_REQUEST//*pull\//} | tee merge.txt
if [[ $(cat merge.txt) != "" ]]; then
echo "Merging $(cat merge.txt)";
git remote add upstream git://github.com/mne-tools/mne-python.git;
git pull --ff-only upstream "refs/pull/$(cat merge.txt)/merge";
git fetch upstream master;
fi

# Load our data
- restore_cache:
keys:
- data-cache-0
- data-cache-1
- data-cache-2
- data-cache-3
- data-cache-4
- data-cache-5
- data-cache-6
- data-cache-7
- data-cache-8
- pip-cache

- run:
name: Install 3D rendering libraries \ PyQt5 dependencies \ graphviz \ optipng (for optimized images)
command: |
Expand All @@ -76,26 +76,44 @@ jobs:
curl https://codeload.github.com/adobe-fonts/source-sans-pro/tar.gz/3.006R | tar xz -C $HOME/.fonts
fc-cache -f

# Load pip cache
- restore_cache:
keys:
- pip-cache
- restore_cache:
keys:
- user-install-bin-cache

- run:
name: Get Python running
command: |
python -m pip install --user --upgrade --progress-bar off pip setuptools
python -m pip install --user --upgrade --progress-bar off --pre sphinx
python -m pip install --user --upgrade --progress-bar off -r requirements.txt -r requirements_testing.txt -r requirements_doc.txt
python -m pip install --user --progress-bar off https://github.com/pyvista/pyvista/zipball/master
python -m pip install --user --progress-bar off https://github.com/pyvista/pyvistaqt/zipball/master
python -m pip install --user --upgrade --progress-bar off -r requirements.txt -r requirements_testing.txt -r requirements_doc.txt
python -m pip uninstall -yq pysurfer mayavi
python -m pip install --user -e .

- save_cache:
key: pip-cache
paths:
- ~/.cache/pip
- save_cache:
key: user-install-bin-cache
paths:
- ~/.local/lib/python3.8/site-packages
- ~/.local/bin

- run:
name: Check PyQt5
command: LD_DEBUG=libs python -c "from PyQt5.QtWidgets import QApplication, QWidget; app = QApplication([])"

# Load tiny cache so that ~/.mne does not need to be created below
- restore_cache:
keys:
- data-cache-tiny-0

# Look at what we have and fail early if there is some library conflict
- run:
name: Check installation
Expand All @@ -109,14 +127,55 @@ jobs:
python -c "import mne; level = mne.get_config('MNE_LOGGING_LEVEL'); assert level.lower() == 'info', repr(level)"

# Figure out if we should run a full, pattern, or noplot version
- restore_cache:
keys:
- data-cache-multimodal
- restore_cache:
keys:
- data-cache-limo
- restore_cache:
keys:
- data-cache-bst-phantom-ctf
- restore_cache:
keys:
- data-cache-bst-raw
- restore_cache:
keys:
- data-cache-bst-phantom-elekta
- restore_cache:
keys:
- data-cache-bst-auditory
- restore_cache:
keys:
- data-cache-bst-resting
- restore_cache:
keys:
- data-cache-fieldtrip
- restore_cache:
keys:
- data-cache-somato
- restore_cache:
keys:
- data-cache-hf-sef
- restore_cache:
keys:
- data-cache-opm
- restore_cache:
keys:
- data-cache-sample
- restore_cache:
keys:
- data-cache-spm-face
- restore_cache:
keys:
- data-cache-testing
- restore_cache:
keys:
- data-cache-visual
- run:
name: Get data
command: |
python setup.py develop --user
mkdir -p ~/mne_data
touch pattern.txt;
./tools/circleci_download.sh

- run:
name: Get data (again)
when: on_fail
Expand Down Expand Up @@ -165,15 +224,11 @@ jobs:
name: Reduce artifact upload time
command: |
if grep -q html_dev-pattern-memory build.txt || grep -q html_dev-noplot build.txt; then
tar czf doc/_build/html/_downloads.tgz doc/_build/html/_downloads
rm -Rf doc/_build/html/_downloads
rm -f doc/auto_*/*/*.pickle
rm -f doc/auto_*/*/*.codeobj
rm -f doc/auto_*/*/*.md5
rm -f doc/auto_*/*/*.py
rm -f doc/auto_*/*/*.ipynb
rm -f doc/generated/*.examples
zip -rm doc/_build/html/_downloads.zip doc/_build/html/_downloads
fi
for NAME in generated auto_tutorials auto_examples; do
zip -rm doc/${NAME}.zip doc/${NAME}
done

# Save the JUnit file
- store_test_results:
Expand All @@ -183,14 +238,11 @@ jobs:
destination: test-results
# Save the SG RST
- store_artifacts:
path: doc/auto_examples
destination: auto_examples
path: doc/auto_examples.zip
- store_artifacts:
path: doc/auto_tutorials
destination: auto_tutorials
path: doc/auto_tutorials.zip
- store_artifacts:
path: doc/generated
destination: generated
path: doc/generated.zip
# Save the HTML
- store_artifacts:
path: doc/_build/html/
Expand All @@ -206,50 +258,74 @@ jobs:

# Keep these separate, maybe better in terms of size limitations (?)
- save_cache:
key: data-cache-0
key: data-cache-tiny-0 # < 100 M, might as well combine
paths:
- ~/.mne
- ~/mne_data/mTRF_1.5
- ~/mne_data/MNE-kiloword-data # (28 M)
- ~/mne_data/MNE-eegbci-data # (35 M)
- ~/mne_data/MNE-misc-data # (39 M)
- ~/mne_data/mTRF_1.5 # (56 M)
- ~/mne_data/MNE-phantom-4DBTi # (77 M)
- save_cache:
key: data-cache-1
key: data-cache-multimodal
paths:
- ~/mne_data/HF_SEF
- ~/mne_data/MEGSIM
- ~/mne_data/MNE-multimodal-data # (240 M)
- save_cache:
key: data-cache-2
key: data-cache-limo
paths:
- ~/mne_data/MNE-brainstorm-data
- ~/mne_data/MNE-eegbci-data
- ~/mne_data/MNE-limo-data # (244 M)
- save_cache:
key: data-cache-3
key: data-cache-bst-phantom-ctf
paths:
- ~/mne_data/MNE-fieldtrip_cmc-data
- ~/mne_data/MNE-kiloword-data
- ~/mne_data/MNE-brainstorm-data/bst_phantom_ctf # (177 M)
- save_cache:
key: data-cache-4
key: data-cache-bst-bst-raw
paths:
- ~/mne_data/MNE-misc-data
- ~/mne_data/MNE-multimodal-data
- ~/mne_data/MNE-brainstorm-data/bst_raw # (830 M)
- save_cache:
key: data-cache-5
key: data-cache-bst-phantom-elekta
paths:
- ~/mne_data/MNE-OPM-data
- ~/mne_data/MNE-phantom-4DBTi
- ~/mne_data/MNE-brainstorm-data/bst_phantom_elekta # (1.4 G)
- save_cache:
key: data-cache-6
key: data-cache-bst-auditory
paths:
- ~/mne_data/MNE-sample-data
- ~/mne_data/MNE-somato-data
- ~/mne_data/MNE-brainstorm-data/bst_auditory # (2.9 G)
- save_cache:
key: data-cache-7
key: data-cache-bst-resting
paths:
- ~/mne_data/MNE-spm-face
- ~/mne_data/MNE-testing-ata
- ~/mne_data/MNE-brainstorm-data/bst_resting # (4.5 G)
- save_cache:
key: data-cache-8
key: data-cache-fieldtrip
paths:
- ~/mne_data/MNE-visual_92_categories-data
- ~/mne_data/MNE-limo-data
- ~/mne_data/MNE-fieldtrip_cmc-data # (699 M)
- save_cache:
key: data-cache-somato
paths:
- ~/mne_data/MNE-somato-data # (750 M)
- save_cache:
key: data-cache-hf-sef
paths:
- ~/mne_data/HF_SEF # (1.3 G)
- save_cache:
key: data-cache-opm
paths:
- ~/mne_data/MNE-OPM-data # (1.9 G)
- save_cache:
key: data-cache-sample
paths:
- ~/mne_data/MNE-sample-data # (3.2 G)
- save_cache:
key: data-cache-spm-face
paths:
- ~/mne_data/MNE-spm-face # (1.5 G)
- save_cache:
key: data-cache-testing
paths:
- ~/mne_data/MNE-testing-data # (2.5 G)
- save_cache:
key: data-cache-visual
paths:
- ~/mne_data/MNE-visual_92_categories-data # (6 G)


linkcheck:
Expand Down
7 changes: 6 additions & 1 deletion mne/utils/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,12 @@ def sys_info(fid=None, show_paths=False):
elif mod_name in ('mayavi', 'vtk'):
has_3d = True
if mod_name == 'vtk':
version = getattr(mod, 'VTK_VERSION', 'VTK_VERSION missing')
version = mod.vtkVersion()
# 9.0 dev has VersionFull but 9.0 doesn't
for attr in ('GetVTKVersionFull', 'GetVTKVersion'):
if hasattr(version, attr):
version = getattr(version, attr)()
break
elif mod_name == 'PyQt5':
version = _check_pyqt5_version()
else:
Expand Down
4 changes: 2 additions & 2 deletions requirements_doc.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
sphinx
https://github.com/numpy/numpydoc/archive/master.zip
numpydoc
sphinx_fontawesome
sphinx_bootstrap_theme
https://github.com/sphinx-gallery/sphinx-gallery/archive/master.zip
sphinx-gallery
sphinxcontrib-bibtex==2.0.0
memory_profiler
neo
Expand Down
6 changes: 3 additions & 3 deletions requirements_testing.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ pytest-timeout
pytest-harvest
flake8
flake8-array-spacing
https://github.com/sphinx-gallery/sphinx-gallery/archive/master.zip
https://github.com/numpy/numpydoc/archive/master.zip
https://github.com/codespell-project/codespell/archive/master.zip
sphinx-gallery
numpydoc
codespell
pydocstyle
check-manifest
twine
Expand Down
2 changes: 0 additions & 2 deletions tools/circleci_download.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ elif [ "$CIRCLE_BRANCH" == "maint/0.22" ]; then
python -c "import mne; mne.datasets._download_all_example_data()";
else
echo "Doing a partial build";
if ! git remote -v | grep upstream ; then git remote add upstream git://github.com/mne-tools/mne-python.git; fi
git fetch upstream
FNAMES=$(git diff --name-only $(git merge-base $CIRCLE_BRANCH upstream/master) $CIRCLE_BRANCH);
if [[ $(cat gitlog.txt) == *"[circle front]"* ]]; then
FNAMES="tutorials/source-modeling/plot_mne_dspm_source_localization.py tutorials/machine-learning/plot_receptive_field.py examples/connectivity/plot_mne_inverse_label_connectivity.py tutorials/machine-learning/plot_sensors_decoding.py tutorials/stats-source-space/plot_stats_cluster_spatio_temporal.py tutorials/evoked/plot_20_visualize_evoked.py "${FNAMES};
Expand Down