Skip to content

test: make the suite hermetic against inherited PYTHONPATH - #22

Open
jmvbambico wants to merge 1 commit into
mainfrom
fix/tests-pythonpath-isolation
Open

jmvbambico wants to merge 1 commit into
mainfrom
fix/tests-pythonpath-isolation

Conversation

@jmvbambico

Copy link
Copy Markdown
Owner

Two tests fail for anyone who has actually installed og, and pass in CI. This
makes the suite hermetic instead.

FAILED tests/test_og_install.py::test_install_pth_lands_in_omnigent_site_packages_and_imports
FAILED tests/test_og_install.py::test_og_start_guard_restores_a_pth_stranded_by_an_upgrade

Root cause

bin/og launches the host daemon with
PYTHONPATH="$OG_POLICY_PATH${PYTHONPATH:+:$PYTHONPATH}" — deliberately, as a
fallback for when the installer's .pth goes stale. Everything that daemon
spawns inherits it, including the coding agents it runs. So running pytest
from inside an og-managed session executes the suite with
PYTHONPATH=~/.omnigent/policies set.

PYTHONPATH lands on sys.path before anything a site-packages .pth
appends. So install_pth's verification subprocess imported the developer's
real 22.6 KB ~/.omnigent/policies/omnigent_local_policies.py instead of the
fixture's stand-in. That file does
from omnigent.policies.builtins import orchestration, the test's fake venv
(venv.create(..., with_pip=False)) has no omnigent, so the import raised
ModuleNotFoundError, die() turned it into SystemExit(1), and the test
failed.

CI never sees it: no ~/.omnigent, no PYTHONPATH.

Measured on main, no code changes:

pytest tests/ -q                     ->  2 failed, 297 passed
env -u PYTHONPATH pytest tests/ -q   ->  299 passed

Worth noting it is not only a red-build problem: on a contaminated machine the
.pth tests could also pass for the wrong reason, with the import they
verify resolving through PYTHONPATH rather than through the .pth under
test.

The fix

An autouse fixture in the repo-root conftest.py strips PYTHONPATH for every
test. Suite-wide rather than scoped to the two known failures: no test
references PYTHONPATH, so nothing wants the inherited value, and any test
that spawns a subprocess is exposed whether or not it fails today.

tests/test_env_isolation.py guards it — in-process, in a spawned subprocess,
and (the load-bearing one, since the first two are vacuous when the ambient
value is already unset) by re-execing a nested pytest with a deliberately
poisoned PYTHONPATH and asserting the inner run still comes back clean.

The product is deliberately untouched

Neither bin/og nor install_pth is wrong:

  • bin/og's export is intentional and documented in place.
  • install_pth verifying in the environment the daemon really runs in is
    more faithful, not less — its comment says "this is exactly what the server
    will see", and that is true.

The contamination is a property of the test harness, so the fix lives there.

Verification

PYTHONPATH=~/.omnigent/policies pytest tests/ -q   ->  302 passed
env -u PYTHONPATH             pytest tests/ -q     ->  302 passed

Proven non-vacuous: with the fixture flipped to autouse=False and
PYTHONPATH set, all three isolation tests fail and the two original
failures return.

bash -n bin/og install.sh and shellcheck -S error bin/og install.sh clean
(untouched).

Known residual

The scrub fixes the environment, not this interpreter's own sys.path: Python
applies PYTHONPATH at boot, so an entry the pytest process started with is
already baked in and monkeypatch.delenv cannot un-apply it. Retroactively
purging it would mean mutating a global every test shares. The guarantee that
matters is the environment, because that is what a subprocess launches from —
and that is the direction the install_pth failure actually travelled. The
probe test says so in a comment.

🤖 Generated with Claude Code

`bin/og` exports PYTHONPATH=$OG_POLICY_PATH when it launches the host
daemon, so every coding agent it runs inherits it. Anyone running pytest
from inside an og-managed session therefore executes the suite with
PYTHONPATH=~/.omnigent/policies set.

PYTHONPATH is applied before any site-packages .pth, so it shadowed the
test fixtures' stand-ins. install_pth's verification subprocess imported
the developer's real policies module, which imports `omnigent.policies`
that the tests' fake venv does not have, and died with SystemExit(1).
Two tests failed -- but only for developers who have actually installed
og, since CI has no ~/.omnigent and no PYTHONPATH.

The product is correct as-is and is untouched: bin/og's export is
deliberate and documented, and install_pth checking in the environment
the daemon really runs in is more faithful, not less. The contamination
was a property of the harness, so the fix belongs here.

An autouse fixture strips PYTHONPATH for every test. tests/
test_env_isolation.py asserts it holds in-process, that a subprocess
spawned from a test does not see it, and -- the part that matters, since
the other two are vacuous without an ambient value -- that a nested
pytest run handed a poisoned PYTHONPATH still comes back clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant