perf: Optimize resolver hash option intersection

[KRRT7]

Fixes #14105.

Constraint.__and__() intersects per-algorithm hash options by iterating other.hash_options[alg] and checking each value against self.hash_options[alg]. Since both are lists, that membership check is linear, making the intersection O(n*m) for each shared hash algorithm.

This keeps the existing result order from other.hash_options[alg], but builds a temporary set from self.hash_options[alg] so membership checks are O(1), reducing the intersection to O(n+m).

Tests run:

uv run ruff check src/pip/_internal/resolution/resolvelib/base.py tests/unit/resolution_resolvelib/test_base.py
uv run pytest tests/unit/resolution_resolvelib/test_base.py -q
uv run pytest tests/unit/resolution_resolvelib -q

[KRRT7]

I added a targeted benchmark for the changed resolver path. It constructs real Constraint and InstallRequirement objects with hash_options, then repeatedly executes constraint & ireq.

Environment: macOS, Python 3.14.6.

case	main median	PR median	speedup
large existing constraint, small incoming hash set	1.392277s	0.137044s	10.2x
similarly sized hash sets	3.473684s	0.036912s	94.1x

The benchmark is intentionally scoped to the resolver merge operation this PR changes. It models hash-checking scenarios where requirements/constraints carry many --hash values for the same algorithm.

Benchmark script:

from __future__ import annotations

import gc
import statistics
import time

from pip._vendor.packaging.requirements import Requirement
from pip._vendor.packaging.specifiers import SpecifierSet

from pip._internal.req.req_install import InstallRequirement
from pip._internal.resolution.resolvelib.base import Constraint
from pip._internal.utils.hashes import Hashes


def make_ireq(values: list[str]) -> InstallRequirement:
    return InstallRequirement(
        Requirement("example"),
        comes_from=None,
        hash_options={"sha256": values},
    )


def run_case(name: str, *, base_size: int, incoming_size: int, overlap_start: int, rounds: int, repeat: int) -> None:
    base_values = [f"{i:064x}" for i in range(base_size)]
    incoming_values = [f"{i:064x}" for i in range(overlap_start, overlap_start + incoming_size)]
    constraint = Constraint(SpecifierSet(), Hashes(), {"sha256": base_values}, frozenset())
    incoming = [make_ireq(incoming_values) for _ in range(rounds)]

    expected = [value for value in incoming_values if value in set(base_values)]
    assert (constraint & incoming[0]).hash_options == {"sha256": expected}

    samples = []
    gc.disable()
    try:
        for _ in range(repeat):
            start = time.perf_counter()
            for ireq in incoming:
                constraint & ireq
            samples.append(time.perf_counter() - start)
    finally:
        gc.enable()

    print(f"{name}: min={min(samples):.6f}s median={statistics.median(samples):.6f}s rounds={rounds}")


run_case(
    "large-constraint-small-incoming",
    base_size=5000,
    incoming_size=20,
    overlap_start=4500,
    rounds=2000,
    repeat=7,
)
run_case(
    "similar-sized-overlap",
    base_size=1000,
    incoming_size=1000,
    overlap_start=500,
    rounds=500,
    repeat=7,
)

[notatallshaw]

The change is behavior preserving, but the benchmark doesn't reflect real world scenarios.

Constraint.__and__ is only called from collect_root_requirements, once per pip invocation, and only when the same project appears on multiple constraint lines with hashes on both. Realistic hash counts are one per release file: requests has 2, numpy 44, pydantic-core 107. The benchmark used 1000 and 5000.

The benchmark also passes an empty Hashes(), a state Constraint.from_ireq can't produce: real constraints carry the same digests in hashes, and the line above the changed code runs Hashes.__and__, which has the same O(n*m) scan and is untouched here. Building the benchmark constraint with Constraint.from_ireq instead, I measure 1.6x at 5000x20 and 2.0x at 1000x1000, not 10x and 94x. At realistic sizes the saving is tens of microseconds once per run, and I measured slightly negative at 2 hashes.

I'm not sure this is worth the churn, in general I would like to see benchmarks that reflect real world scenarios if you are going to make many performance PRs.

[KRRT7]

Thanks, you’re right. The original benchmark bypassed the real Constraint.from_ireq path and overstated the benefit because it avoided the preceding Hashes.__and__ intersection.

I pushed 18801da52 to address that. The resolver tests now build constraints via Constraint.from_ireq, and the optimization now covers both Hashes.__and__ and Constraint.hash_options. I also kept the old list-membership path for small intersections, only switching to a temporary set past a small threshold.

With the updated uv run benchmark on the real construction path, I’m seeing the tiny 2x2 case remain effectively neutral/slightly slower, while more realistic larger hash sets improve: roughly 3.1x for 44x44 and 7.4x for 107x107 in my local run.