Skip to content

CI: Fail when a test-gating dependency is installed nowhere - #67294

Open
jbrockmendel wants to merge 3 commits into
pandas-dev:mainfrom
jbrockmendel:bug-26890
Open

CI: Fail when a test-gating dependency is installed nowhere#67294
jbrockmendel wants to merge 3 commits into
pandas-dev:mainfrom
jbrockmendel:bug-26890

Conversation

@jbrockmendel

@jbrockmendel jbrockmendel commented Aug 29, 2026

Copy link
Copy Markdown
Member

pytest.importorskip and td.skip_if_no skip silently. When a dependency is installed in no CI environment, the tests behind that gate never run anywhere, and nothing reports it: a local run can't see it either, because a development environment installs more than CI does. That's GH-26890, which has been open since 2019.

This adds a pre-commit check that every module gated by importorskip/skip_if_no under pandas/ is installed in at least one pixi environment that unit-tests.yml runs pytest in. Environments are read out of the workflow matrix (including include: entries) rather than hardcoded, so it follows the workflow automatically.

What it found

ipython is declared only in [feature.typing.dependencies]. Confirmed in pixi.lock: the py313 environment resolves 379 packages, none of them ipython; typing has ipython-9.16.1. This isn't pixi-migration fallout — the pre-pixi ci/deps/actions-313.yaml had no ipython either. It is in environment.yml and requirements-dev.txt, which is exactly why the gap was invisible to anyone running the suite locally.

So the nine tests behind the ip fixture (pandas/conftest.py) have never run in CI:

file tests
io/formats/test_ipython_compat.py test_publishes, test_publishes_not_implemented, test_enable_data_resource_formatter
frame/test_api.py test_tab_complete_warning (×2 params)
arrays/categorical/test_warnings.py test_tab_complete_warning
indexes/test_base.py test_tab_complete_warning
resample/test_resampler_grouper.py test_tab_complete_ipython6_warning
io/formats/test_to_html.py test_repr_html_ipython_config

The three in test_ipython_compat.py are the only coverage of enable_data_resource_formatter / _repr_data_resource_ in pandas/io/formats/printing.py.

The fix adds ipython to [feature.test-base.dependencies], so it lands in every environment that runs the suite — not just py311py314, but also minimum-versions, downstream, no-pyarrow and the nightlies. It solves cleanly everywhere, including py313-freethreading (ipython 9.17.0 against python 3.13.15 cp313t). The pixi.lock update is ipython's dependency tree, including psutil>=7, which conda-forge's ipython requires. All nine tests pass locally against ipython 9.17.0, the version the new lock resolves.

Turning the tests on surfaced one latent failure, fixed here too. test_publishes asserts a text/latex key in the mimebundle, but _repr_latex_ calls to_latex, which renders through Styler and so needs jinja2 — which the environments built from test-base alone (numpy-nightly, pyarrow-nightly, py313-freethreading) don't install. That half is now a separate test behind an importorskip, so the table-schema and html coverage keeps running in the minimal environments. It illustrates the point nicely: the assertion has been wrong for as long as it has existed, and no run anywhere could tell you.

Allowlist

One module is gated but legitimately absent from pixi.toml, with the reason in the source: moto, which CI serves from a container via PANDAS_MOTO_URL.

Two others started out on that list and shouldn't have been. botocore is now declared explicitly next to boto3 rather than relying on it arriving transitively — pandas/io/common.py imports it as a real optional dependency, so leaning on boto3's dependency tree is exactly the invisible coupling this check exists to catch. The toolz gate in test_dask is gone: toolz is a hard dependency of dask, so the extra importorskip could never fire on its own.

Scope

This is deliberately the cheap half of GH-26890. It catches the failure at the point where someone forgets to add a dependency to the env files, costs no CI time, and needs no per-job flag plumbing. It does not catch a dependency that's declared but fails to install, or a skip caused by something other than a missing import.

Two things I'd rather raise than fold in here:

  1. A runtime --strict-optional-deps (a pytest_runtest_makereport hookwrapper turning import-caused skips into failures) would cover the declared-but-absent case. I prototyped it and it works, but it can only be enabled on the four full-dependency environments — no-pyarrow, minimum-versions, py313-freethreading, downstream and the nightlies intentionally lack dependencies. Happy to open it separately if there's appetite.
  2. The silent-skip problem isn't only about optional dependencies. td.skip_if_not_us_locale tests locale.getlocale()[0] != "en_US", and the ubuntu matrix sets LANG: C.UTF-8 (the two locale jobs set it_IT/zh_CN), so all 27 usages across 10 files skip on every ubuntu job. Same failure mode, no dependency involved.

For context on how invisible this class is: a local full-suite run skips 7653 tests, of which only 47 are import-caused, and the ipython ones aren't among them — because ipython is installed locally.

pytest.importorskip and td.skip_if_no skip silently, so a dependency that
is in no CI environment produces tests that never run anywhere. A local
run cannot see it, because a development environment installs more than
CI does.

Add a pre-commit check that every module gated this way is installed in
at least one environment the unit test workflow runs pytest in. It found
ipython, which is only in the typing environment, so the nine tests
behind the `ip` fixture have never run in CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZH7J4sFBwdKdbtuVdMysc
@jbrockmendel jbrockmendel added CI Continuous Integration Testing pandas testing functions or related to the test suite labels Aug 29, 2026
@jbrockmendel
jbrockmendel marked this pull request as ready for review August 30, 2026 19:54
Comment thread scripts/validate_test_dependencies.py Outdated
Comment thread .pre-commit-config.yaml Outdated
Comment thread pixi.toml Outdated
jbrockmendel and others added 2 commits August 31, 2026 07:37
Declare botocore explicitly rather than relying on it arriving with
boto3, and drop the redundant toolz gate in test_dask -- toolz is a
hard dependency of dask, so the extra importorskip never fired on its
own.  ALLOWED_UNDECLARED is now just moto, which CI serves from a
container.

Move ipython from its own feature into test-base, so the tests behind
the `ip` fixture also run under minimum-versions, downstream,
no-pyarrow and the nightly environments.

Narrow the pre-commit trigger to the files that actually hold gates.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Now that ipython is in test-base, test_publishes runs everywhere, and its
latex assertion fails in the environments built from test-base alone --
numpy-nightly, pyarrow-nightly and py313-freethreading. `_repr_latex_`
calls `to_latex`, which renders via Styler and so needs jinja2; without
it the mimebundle has no text/latex key.

Split that half into its own test behind an importorskip, so the table
schema and html coverage keeps running in the minimal environments.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhQUdt2tN1H4PFfPS3ahxm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Continuous Integration Testing pandas testing functions or related to the test suite

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants