Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pre fetch openml data for pytest #112

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
"codecov",
"pep8",
"mypy",
"openml"
],
"examples": [
"matplotlib",
Expand Down
38 changes: 38 additions & 0 deletions test/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

import numpy as np

import openml

import pandas as pd

import pytest
Expand All @@ -23,6 +25,42 @@
from autoPyTorch.utils.pipeline import get_dataset_requirements


@pytest.fixture(scope="session")
def callattr_ahead_of_alltests(request):
"""
This procedure will run at the start of the pytest session.
It will prefetch several task that are going to be used by
the testing face, and it does so in a robust way, until the openml
API provides the desired resources
"""
tasks_used = [
146818, # Australian
2295, # cholesterol
2075, # abalone
2071, # adult
3, # kr-vs-kp
9981, # cnae-9
146821, # car
146822, # Segment
2, # anneal
53, # vehicle
5136, # tecator
4871, # sensory
4857, # boston
3916, # kc1
]

# Populate the cache
# This will make the test fail immediately rather than
# Waiting for a openml fetch timeout
openml.populate_cache(task_ids=tasks_used)
# Also the bunch
for task in tasks_used:
fetch_openml(data_id=openml.tasks.get_task(task).dataset_id,
return_X_y=True)
return


def slugify(text):
return re.sub(r'[\[\]]+', '-', text.lower())

Expand Down