Skip to content

Mutation testing: dependabot_automerge live-execution and merge-state survivors #337

Description

@leynos

Context

Local mutation-testing run against main at 3c1fa01bcf4c26dac79975c18b74c01d577dcf95 (mutmut 3.6.0, committed [tool.mutmut] configuration: source_paths = ["workflow_scripts/"], test selection workflow_scripts/tests/, act-gated workflow tests excluded). Run totals: 1463 mutants — 979 killed, 478 survived, 5 no-tests, 1 timeout.

These scripts power the estate's CI, so mutation gaps here matter most. This issue covers one coherent area; sibling issues cover the rest. Equivalent mutants are recorded in the triage worklist and deliberately omitted here. Inspect any survivor with uv run --with mutmut==3.6.0 mutmut show <name>; names below are workflow_scripts.<module>.x_<function>__mutmut_<n> with the numbers listed per line.

Analysis

The live-execution path of dependabot_automerge.py (_handle_live_execution, _fetch_pull_request, _classify_merge_state, _refresh_merge_state, _enable_automerge, _merge_pull_request, and the _extract_* response mappers) is covered only for coarse outcomes. Tests never assert the arguments flowing into request_graphql (query variables, mutation payloads, the token), the decision emitted on the skipped/enabled/retry paths, the refresh loop's sleep schedule, or the fallback behaviour of the response mappers. Argument-to-None mutants, GraphQL variable-key mutants, Decision string mutants, and classification-table mutants all survive. All 117 survivors here are real test gaps.

One related mutant timed out rather than survived: x__refresh_merge_state__mutmut_10 (workflow_scripts/dependabot_automerge.py:712) inverts if state != "retry" so the loop sleeps on every attempt; the 30 s pytest timeout kills it slowly. A monkeypatched time.sleep in the suggested refresh tests would kill it promptly.

Suggested tests

  1. Stub request_graphql with a recorder and drive _handle_live_execution end-to-end for each path (skipped, already-enabled before and after refresh, skip on DIRTY/BEHIND/MERGED/CONFLICTING, retry-exhausted UNKNOWN, direct merge on CLEAN/HAS_HOOKS/UNSTABLE, and arm-auto-merge on BLOCKED). Assert both the recorded GraphQL calls (token, query, owner/name/number variables; pullRequestId/mergeMethod payload keys and values for both mutations) and the exact emitted decision lines via capsys. This kills the bulk: argument-None, variable-key, and Decision status/reason mutants.
  2. Table-test _classify_merge_state: notably one dimension UNKNOWN while the other is known must classify as retry (kills the orand mutant), plus the exact reason strings (already-mergeable, merge-state-unknown) and the ("ok", None) fall-through.
  3. Test _refresh_merge_state with a monkeypatched time.sleep and a stubbed _fetch_pull_request: assert the exponential sleep sequence (base × 2^attempt, capped at max_sleep), the fetch arguments, and that a non-retry state returns immediately. Kills the sleep-arithmetic and argument mutants and the timeout mutant above.
  4. Test _fetch_pull_request against fixture payloads: a draft PR (isDraft: true) must produce is_draft=True (kills the key-case and default-True/bool(None) mutants), missing author/labels/merge-state fields must fall back to ""/()/UNKNOWN (kills _extract_* default mutants), and a missing repository or pull request must fail with the exact message.
  5. _handle_dry_run without an event: assert the exact failure message text.

Surviving mutants

  • workflow_scripts/dependabot_automerge.py:358 in _extract_author_login (mutants 6, 12; 1-test-gap)
    • original: return
    • 6: return "XXXX
    • 12: return "XXXX
  • workflow_scripts/dependabot_automerge.py:424 in _extract_labels (mutants 13; 1-test-gap)
    • original: continue
    • 13: break
  • workflow_scripts/dependabot_automerge.py:445 in _fetch_pull_request (mutants 42; 1-test-gap)
    • original: owner=owner,
    • 42: owner=None,
  • workflow_scripts/dependabot_automerge.py:446 in _fetch_pull_request (mutants 43; 1-test-gap)
    • original: repo=repo,
    • 43: repo=None,
  • workflow_scripts/dependabot_automerge.py:573 in _extract_merge_state_status (mutants 4; 1-test-gap)
    • original: MergeStateStatus.UNKNOWN,
    • 4: None,
  • workflow_scripts/dependabot_automerge.py:583 in _extract_mergeable_state (mutants 4; 1-test-gap)
    • original: MergeableState.UNKNOWN,
    • 4: None,
  • workflow_scripts/dependabot_automerge.py:592 in _enable_automerge (mutants 1; 1-test-gap)
    • original: token,
    • 1: None,
  • workflow_scripts/dependabot_automerge.py:592 in _merge_pull_request (mutants 1; 1-test-gap)
    • original: token,
    • 1: None,
  • workflow_scripts/dependabot_automerge.py:592 in _fetch_pull_request (mutants 2; 1-test-gap)
    • original: token,
    • 2: None,
  • workflow_scripts/dependabot_automerge.py:594 in _fetch_pull_request (mutants 4, 8, 9, 10, 11, 12, 13; 1-test-gap)
    • original: {"owner": owner, "name": repo, "number": number},
    • 4: None,
    • 8: {"XXownerXX": owner, "name": repo, "number": number},
    • 9: {"OWNER": owner, "name": repo, "number": number},
    • 10: {"owner": owner, "XXnameXX": repo, "number": number},
    • 11: {"owner": owner, "NAME": repo, "number": number},
    • 12: {"owner": owner, "name": repo, "XXnumberXX": number},
    • 13: {"owner": owner, "name": repo, "NUMBER": number},
  • workflow_scripts/dependabot_automerge.py:598 in _fetch_pull_request (mutants 19, 26; 1-test-gap)
    • original: fail(f"Pull request {owner}/{repo}#{number} was not found.")
    • 19: fail(None)
    • 26: fail(None)
  • workflow_scripts/dependabot_automerge.py:607 in _fetch_pull_request (mutants 41; 1-test-gap)
    • original: number=number,
    • 41: number=None,
  • workflow_scripts/dependabot_automerge.py:611 in _fetch_pull_request (mutants 45, 61, 62, 63, 64, 65, 66, 67, 68, 69; 1-test-gap/2-equivalent)
    • original: is_draft=bool(pull_request.get("isDraft", False)),
    • 45: is_draft=None,
    • 61: is_draft=bool(None),
    • 62: is_draft=bool(pull_request.get(None, False)),
    • 63: is_draft=bool(pull_request.get("isDraft", None)), — bool(None) == bool(False) for missing isDraft
    • 64: is_draft=bool(pull_request.get(False)),
    • 65: is_draft=bool(pull_request.get("isDraft", )), — .get default None is falsy like False
    • 66: is_draft=bool(pull_request.get("XXisDraftXX", False)),
    • 67: is_draft=bool(pull_request.get("isdraft", False)),
    • 68: is_draft=bool(pull_request.get("ISDRAFT", False)),
    • 69: is_draft=bool(pull_request.get("isDraft", True)),
  • workflow_scripts/dependabot_automerge.py:635 in _classify_merge_state (mutants 10, 11; 1-test-gap)
    • original: return "merge", "already-mergeable
    • 10: return "merge", "XXalready-mergeableXX
    • 11: return "merge", "ALREADY-MERGEABLE
  • workflow_scripts/dependabot_automerge.py:636 in _classify_merge_state (mutants 12; 1-test-gap)
    • original: if merge_state in MERGE_STATE_RETRYABLE or mergeable_state in MERGEABLE_RETRYABLE:
    • 12: if merge_state in MERGE_STATE_RETRYABLE and mergeable_state in MERGEABLE_RETRYABLE:
  • workflow_scripts/dependabot_automerge.py:637 in _classify_merge_state (mutants 17, 18; 1-test-gap)
    • original: return "retry", "merge-state-unknown
    • 17: return "retry", "XXmerge-state-unknownXX
    • 18: return "retry", "MERGE-STATE-UNKNOWN
  • workflow_scripts/dependabot_automerge.py:638 in _classify_merge_state (mutants 19, 20; 1-test-gap)
    • original: return "ok", None
    • 19: return "XXokXX", None
    • 20: return "OK", None
  • workflow_scripts/dependabot_automerge.py:709 in _refresh_merge_state (mutants 6; 1-test-gap)
    • original: current.merge_state_status,
    • 6: None,
  • workflow_scripts/dependabot_automerge.py:710 in _refresh_merge_state (mutants 7; 1-test-gap)
    • original: current.mergeable_state,
    • 7: None,
  • workflow_scripts/dependabot_automerge.py:712 in _refresh_merge_state (mutants 10; timeout; timeout)
    • original: if state != "retry":
    • 10: if state == "retry":
  • workflow_scripts/dependabot_automerge.py:714 in _refresh_merge_state (mutants 13; 1-test-gap)
    • original: sleep_seconds = min( / retry_config.base_sleep * (2**attempt), / retry_config.max_sleep, / )
    • 13: sleep_seconds = None
  • workflow_scripts/dependabot_automerge.py:715 in _refresh_merge_state (mutants 18, 19, 20; 1-test-gap)
    • original: retry_config.base_sleep * (2**attempt),
    • 18: retry_config.base_sleep / (2**attempt),
    • 19: retry_config.base_sleep * (2 * attempt),
    • 20: retry_config.base_sleep * (3**attempt),
  • workflow_scripts/dependabot_automerge.py:718 in _refresh_merge_state (mutants 21; 1-test-gap)
    • original: time.sleep(sleep_seconds)
    • 21: time.sleep(None)
  • workflow_scripts/dependabot_automerge.py:720 in _refresh_merge_state (mutants 23, 24, 25, 26; 1-test-gap)
    • original: token, current.owner, current.repo, current.number
    • 23: None, current.owner, current.repo, current.number
    • 24: token, None, current.repo, current.number
    • 25: token, current.owner, None, current.number
    • 26: token, current.owner, current.repo, None
  • workflow_scripts/dependabot_automerge.py:730 in _enable_automerge (mutants 3, 7, 8, 9, 10, 11, 12; 1-test-gap)
    • original: {"pullRequestId": pull_request_id, "mergeMethod": merge_method},
    • 3: None,
    • 7: {"XXpullRequestIdXX": pull_request_id, "mergeMethod": merge_method},
    • 8: {"pullrequestid": pull_request_id, "mergeMethod": merge_method},
    • 9: {"PULLREQUESTID": pull_request_id, "mergeMethod": merge_method},
    • 10: {"pullRequestId": pull_request_id, "XXmergeMethodXX": merge_method},
    • 11: {"pullRequestId": pull_request_id, "mergemethod": merge_method},
    • 12: {"pullRequestId": pull_request_id, "MERGEMETHOD": merge_method},
  • workflow_scripts/dependabot_automerge.py:730 in _merge_pull_request (mutants 3, 7, 8, 9, 10, 11, 12; 1-test-gap)
    • original: {"pullRequestId": pull_request_id, "mergeMethod": merge_method},
    • 3: None,
    • 7: {"XXpullRequestIdXX": pull_request_id, "mergeMethod": merge_method},
    • 8: {"pullrequestid": pull_request_id, "mergeMethod": merge_method},
    • 9: {"PULLREQUESTID": pull_request_id, "mergeMethod": merge_method},
    • 10: {"pullRequestId": pull_request_id, "XXmergeMethodXX": merge_method},
    • 11: {"pullRequestId": pull_request_id, "mergemethod": merge_method},
    • 12: {"pullRequestId": pull_request_id, "MERGEMETHOD": merge_method},
  • workflow_scripts/dependabot_automerge.py:751 in _handle_dry_run (mutants 3; 1-test-gap)
    • original: fail("Dry-run mode requires GITHUB_EVENT_PATH with pull_request data.")
    • 3: fail("XXDry-run mode requires GITHUB_EVENT_PATH with pull_request data.XX")
  • workflow_scripts/dependabot_automerge.py:756 in _handle_live_execution (mutants 26, 29; 1-test-gap)
    • original: decision,
    • 26: None,
    • 29: ``
  • workflow_scripts/dependabot_automerge.py:757 in _handle_live_execution (mutants 27, 52, 92; 1-test-gap)
    • original: config=config,
    • 27: config=None,
    • 52: config=None,
    • 92: config=None,
  • workflow_scripts/dependabot_automerge.py:757 in _handle_live_execution (mutants 30, 55, 95; 1-test-gap)
    • original: config=config, / )
    • 30: )
    • 55: )
    • 95: )
  • workflow_scripts/dependabot_automerge.py:769 in _handle_live_execution (mutants 3; 1-test-gap)
    • original: pr_number = _resolve_pull_request_number( / context.pull_request_number, / context.event, / )
    • 3: pr_number = None
  • workflow_scripts/dependabot_automerge.py:771 in _handle_live_execution (mutants 5; 1-test-gap)
    • original: context.event,
    • 5: None,
  • workflow_scripts/dependabot_automerge.py:774 in _handle_live_execution (mutants 9, 10, 11, 12; 1-test-gap)
    • original: pr = _fetch_pull_request(github_token, owner, repo, pr_number)
    • 9: pr = _fetch_pull_request(None, owner, repo, pr_number)
    • 10: pr = _fetch_pull_request(github_token, None, repo, pr_number)
    • 11: pr = _fetch_pull_request(github_token, owner, None, pr_number)
    • 12: pr = _fetch_pull_request(github_token, owner, repo, None)
  • workflow_scripts/dependabot_automerge.py:775 in _handle_live_execution (mutants 19; 1-test-gap)
    • original: decision = _evaluate(pr, config.required_label)
    • 19: decision = _evaluate(pr, None)
  • workflow_scripts/dependabot_automerge.py:778 in _handle_live_execution (mutants 25, 28, 50, 53, 90, 93; 1-test-gap)
    • original: pr,
    • 25: None,
    • 28: ``
    • 50: None,
    • 53: ``
    • 90: None,
    • 93: ``
  • workflow_scripts/dependabot_automerge.py:787 in _handle_live_execution (mutants 51, 54, 56, 57, 58, 59, 60, 61, 62, 63; 1-test-gap)
    • original: Decision(status="enabled", reason="already-enabled"),
    • 51: None,
    • 54: ``
    • 56: Decision(status=None, reason="already-enabled"),
    • 57: Decision(status="enabled", reason=None),
    • 58: Decision(reason="already-enabled"),
    • 59: Decision(status="enabled", ),
    • 60: Decision(status="XXenabledXX", reason="already-enabled"),
    • 61: Decision(status="ENABLED", reason="already-enabled"),
    • 62: Decision(status="enabled", reason="XXalready-enabledXX"),
    • 63: Decision(status="enabled", reason="ALREADY-ENABLED"),
  • workflow_scripts/dependabot_automerge.py:792 in _handle_live_execution (mutants 46; 1-test-gap)
    • original: pr = _refresh_merge_state(github_token, pr)
    • 46: pr = _refresh_merge_state(None, pr)
  • workflow_scripts/dependabot_automerge.py:808 in _handle_live_execution (mutants 85, 86; 1-test-gap)
    • original: Decision(status="skipped", reason=reason or "merge-state-unknown"),
    • 85: Decision(status="skipped", reason=reason or "XXmerge-state-unknownXX"),
    • 86: Decision(status="skipped", reason=reason or "MERGE-STATE-UNKNOWN"),
  • workflow_scripts/dependabot_automerge.py:812 in _handle_live_execution (mutants 88, 89; 1-test-gap)
    • original: if state == "retry":
    • 88: if state == "XXretryXX":
    • 89: if state == "RETRY":
  • workflow_scripts/dependabot_automerge.py:815 in _handle_live_execution (mutants 91, 94, 96, 97, 98, 99, 100, 101, 102, 103; 1-test-gap)
    • original: Decision(status="skipped", reason="merge-state-unknown"),
    • 91: None,
    • 94: ``
    • 96: Decision(status=None, reason="merge-state-unknown"),
    • 97: Decision(status="skipped", reason=None),
    • 98: Decision(reason="merge-state-unknown"),
    • 99: Decision(status="skipped", ),
    • 100: Decision(status="XXskippedXX", reason="merge-state-unknown"),
    • 101: Decision(status="SKIPPED", reason="merge-state-unknown"),
    • 102: Decision(status="skipped", reason="XXmerge-state-unknownXX"),
    • 103: Decision(status="skipped", reason="MERGE-STATE-UNKNOWN"),
  • workflow_scripts/dependabot_automerge.py:821 in _handle_live_execution (mutants 105, 106, 107, 108; 1-test-gap)
    • original: fail("Pull request node ID missing from GitHub response.")
    • 105: fail(None)
    • 106: fail("XXPull request node ID missing from GitHub response.XX")
    • 107: fail("pull request node id missing from github response.")
    • 108: fail("PULL REQUEST NODE ID MISSING FROM GITHUB RESPONSE.")
  • workflow_scripts/dependabot_automerge.py:824 in _handle_live_execution (mutants 112, 113, 114; 1-test-gap)
    • original: _merge_pull_request(github_token, pr.node_id, config.merge_method)
    • 112: _merge_pull_request(None, pr.node_id, config.merge_method)
    • 113: _merge_pull_request(github_token, None, config.merge_method)
    • 114: _merge_pull_request(github_token, pr.node_id, None)
  • workflow_scripts/dependabot_automerge.py:832 in _handle_live_execution (mutants 132, 133, 134; 1-test-gap)
    • original: _enable_automerge(github_token, pr.node_id, config.merge_method)
    • 132: _enable_automerge(None, pr.node_id, config.merge_method)
    • 133: _enable_automerge(github_token, None, config.merge_method)
    • 134: _enable_automerge(github_token, pr.node_id, None)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmediumRoadmap items to schedule within the current quarter. Clear scope, normal review cycles.pythontestingTest coverage, test infrastructure, and verification tooling work.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions