Skip to content

tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters #109

Closed
#129
Closed
tests/integration/ is unmarked, so the documented unit-test command provisions billable AWS EKS clusters#109
#129
Labels
P0-criticalBlocks everything; safety or correctness landminebugtestingTest suite, CI, coverage

Description

@jeremymanning
jeremymanning
opened on Aug 17, 2026

Part of #108 · Phase 0 · Do this before running the test suite again.

Problem

Every file in tests/integration/ has zero pytest markers (verified: grep -rc "pytest.mark" tests/integration/ returns 0 for all 20+ files). The documented "unit test" command therefore selects them:

pytest tests/ -m "not real_world"     # <- selects tests/integration/

One of the files it selects is tests/integration/test_aws_eks_real_provision.py, whose own docstring reads:

REAL AWS EKS provisioning test - this WILL create resources and incur costs!
Only run this if you're ready to pay for AWS EKS cluster.

This was confirmed empirically, not inferred. During an audit run of pytest -m "not real_world", lsof showed the pytest process holding ESTABLISHED TCP connections to ec2-*.compute-1.amazonaws.com:443, and stderr emitted Error during cluster cleanup in destructor: Cleanup failed.

Anyone who has AWS credentials in their environment and runs the documented command starts billing real EKS infrastructure — and the cleanup path is itself failing, so resources may be orphaned.

This is also what makes the suite hang: clustrix/kubernetes/aws_provisioner.py:763,771 sits in a time.sleep(30) x max_attempts retry loop, which is why a full coverage run reproducibly stalls at 98%.

Acceptance criteria

  • Every file under tests/integration/ carries an explicit marker; anything touching a billable API is marked real_world (or a new costly marker) and is excluded by default.
  • pyproject.toml addopts excludes billable markers by default, so the bare pytest command is safe.
  • Tests that provision cloud resources require an explicit opt-in env var (e.g. CLUSTRIX_ALLOW_BILLABLE=1) and pytest.skip without it — belt and braces with the marker.
  • --strict-markers is already set (pyproject.toml:189); every new marker is registered.
  • The orphaned-resource risk is addressed: the destructor cleanup failure is fixed or the test refuses to start unless cleanup is verifiable.
  • Regression guard: a CI check that fails if any file under tests/integration/ lacks a marker.

Verification

grep -rL "pytest.mark" tests/integration/     # must return nothing
pytest tests/ -m "not real_world" --collect-only -q | grep -c eks_real_provision   # must be 0
# with AWS creds exported, run the documented command and confirm via lsof that
# no connection to amazonaws.com is ever established

Activity

  1. jeremymanning
    added
    bug
    P0-criticalBlocks everything; safety or correctness landmine
    testingTest suite, CI, coverage
    on Aug 17, 2026
  2. jeremymanning
    added a parent issue on Aug 17, 2026

  3. jeremymanning commented on Aug 17, 2026

    @jeremymanning
    jeremymanning
    on Aug 17, 2026
    MemberAuthor

    Fix opened: PR #129.

    The root cause turned out to be worse than this issue described, in a way that changes the fix.

    Markers would not have been sufficient. pytest imports a module in order to collect it, and two files here are standalone scripts rather than test modules — test_eks_permissions.py and test_aws_eks_debug.py fetch credentials, call boto3, and call exit() at module scope. So the AWS calls happen before any marker or skip is consulted, and the module-level sys.exit(1) crashes the run outright:

    INTERNALERROR> File ".../tests/integration/test_aws_eks_debug.py", line 63, in <module>
INTERNALERROR>   sys.exit(1)
INTERNALERROR> SystemExit: 1
mainloop: caught unexpected SystemExit!

    The gate therefore stops collection (collect_ignore_glob), not execution. It is directory-wide default-deny rather than a per-file allowlist: misclassifying 1 file of 48 costs money, and a file added later must not run for free by default. Opt in with CLUSTRIX_ALLOW_BILLABLE=1.

    Also worth recording: the credentials to make this real are present on a normal dev machine. ~/.clustrix/.env ships uncommented AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY and FlexibleCredentialManager loads exactly that file (credential_manager.py:304-305), with a 1Password CLI fallback. The only reason the audit run did not reach AWS was that boto3 was absent from that venv.

    Verification

    Full suite run with a socket-blocking sitecustomize logging every attempt with a stack trace:

    Before After
    Collection crashed (SystemExit, exit 3) 2067 tests collected
    AWS/EC2/EKS connection attempts live connections confirmed via lsof 0
    boto3/botocore frames in any stack present 0

    Two non-billable network sources remain (not fixed here)

    The same instrumentation found 3,078 total connection attempts. Neither is billable:

    Both have been noted on those issues.

  4. jeremymanning
    added 2 commits that reference this issue on Aug 17, 2026

    Issue #109: Fix guard test on Windows (cp1252 decode, None stdout)

    6175fc7

    Issue #109: Close bypasses found by red-teaming the guard

    3073f0e
  5. jeremymanning
    closed this as completedin #129on Aug 17, 2026
  6. jeremymanning
    added a commit that references this issue on Aug 17, 2026

    Merge pull request #129 from ContextLab/fix/109-billable-test-guard

    a9393b7

  7. jeremymanning commented on Aug 17, 2026

    @jeremymanning
    jeremymanning
    on Aug 17, 2026
    MemberAuthor

    Closed by the merge of PR #129 (merge commit a9393b7).

    The suite is now free to run. Verified on merged master with the network hard-blocked and every connection attempt logged with a stack trace:

    Vector Result
    pytest tests/ -m "not real_world" (the documented command) 0 AWS/EC2/EKS connection attempts
    explicit file / node-id / directory targeting refused with an actionable message
    direct import of the two unguarded scripts IMPORTED CLEANLY, no side effects (previously fetched real AWS credentials)
    gate ON vs OFF 1592 vs 1657 collected — withholds exactly the 65 integration tests, nothing else

    An adversarial review before merge found 5 defects in the first version of the fix, two serious:

    1. Explicitly-named paths bypassed collect_ignore_glob entirely and imported the modules — pytest tests/integration/test_aws_eks_debug.py printed Got credentials for account: .... Only a missing boto3 stopped it.
    2. The marker hook was tagging the whole suite expensive (pytest passes that hook every item in the session), so -m "not expensive" would have selected nothing.

    Both fixed; full details on PR #129.

    Two residual limits, documented rather than fixed:

  8. jeremymanning
    added 5 commits that reference this issue on Aug 17, 2026

    Issue #130: Add handoff plan for the pytest config-shadowing fix

    de09f1a

    Issue #130: Close the guard bypasses red-teaming found, and fix my ow…

    a5b665f

    Issue #130: pytest.ini was dead config shadowing pyproject.toml (#134)

    760432c

    Update notes for handoff to a fresh session (#136)

    ed42375

    Fix three fabricated-result bugs, close the RCE, and overhaul the doc…

    0e3490e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0-criticalBlocks everything; safety or correctness landminebugtestingTest suite, CI, coverage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions