Performance Optimization: Resolver, Benchmarking & Batch Operations

[matteius]

Fix for #6415

Performance Optimization: Resolver, Benchmarking & Batch Operations

Performance Improvements

Resolver Optimizations

• Increased resolution depth limit from 200K to 1M for complex dependency graphs
• Updated ResolveLib to 1.2.0 with optimistic backjumping (10-30% faster resolution)
• Added intelligent resolution caching with 10-minute TTL
• Fixed session management to prevent stale authentication

Batch Operations

• New add_packages_to_pipfile_batch() for bulk Pipfile updates
• Optimized requirements import pipeline (60-80% faster for large files)
• Single I/O operations instead of per-package writes

Benchmarking Infrastructure

Added comprehensive benchmark suite based on python-package-manager-shootout:

• Tests import, lock, install (cold/warm), update, add-package operations
• Uses real-world Sentry dependencies (50+ packages)
• Runs in CI with results in GitHub job summaries
• Available locally via make benchmark

Expected Performance Gains

• Requirements imports: 60-80% faster for 50+ packages
• Complex resolution: 10-30% faster with new ResolveLib
• Repeated operations: Up to 90% faster with caching
• Enterprise projects: Handles deeper dependency trees without limits

Backward Compatibility

Fully backward compatible - no breaking changes to APIs, Pipfiles, or CLI commands.

Key Files

• pipenv/utils/resolver.py - Caching and session fixes
• pipenv/project.py - Batch operations
• pipenv/utils/requirements.py - Import optimization
• benchmarks/ - Complete benchmark infrastructure
• ResolveLib patches for 1.2.0 update

The checklist

• Associated issue
• A news fragment in the news/ directory to describe this fix with the extension .bugfix.rst, .feature.rst, .behavior.rst, .doc.rst. .vendor.rst. or .trivial.rst (this will appear in the release changelog). Use semantic line breaks and name the file after the issue number or the PR #.

[Copilot]

Pull Request Overview

This PR optimizes pipenv’s dependency resolution performance, introduces intelligent caching and batch operations, and establishes a comprehensive benchmarking suite. Key changes include resolver optimizations (increased limits, caching, and optimistic backjumping), a new batch package addition function for Pipfile updates, and benchmark infrastructure updates.

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pyproject.toml	Added a new lint rule comment for module-level imports
pipenv/utils/resolver.py	Removed cached_property for session; added caching for resolution
pipenv/utils/requirements.py	Updated return type of redact_netloc and adjusted auth redaction logic
pipenv/project.py	Introduced a new batch add function for Pipfile packages
pipenv/patched/pip/_vendor/resolvelib/resolution.py	Enhanced optimistic backjumping strategy and state rollback mechanisms
Other files (environment, CLI, benchmarks, etc.)	Various minor improvements and benchmark suite additions

Comments suppressed due to low confidence (4)

pipenv/utils/requirements.py:18

• Changing the return type of redact_netloc from a tuple to a string requires careful review of its downstream usage; verify that _redact_netloc_wrapper correctly adapts to this new return type.

def redact_netloc(netloc: str) -> str:

pipenv/project.py:1314

• [nitpick] When multiple categories are provided, generating the package entry using only the first category may not be ideal; consider revisiting this design to ensure consistency across all specified categories.

                )

pipenv/patched/pip/_vendor/resolvelib/resolvers/resolution.py:358

• [nitpick] Consider adding inline comments to clarify the logic behind skipping optimistic backjumping in this condition, which will aid future maintainability and understanding of the algorithm.

                ):

pipenv/utils/resolver.py:309

• Review the removal of the cached_property decorator to ensure the session property still meets performance expectations and doesn’t introduce unintended repeated session builds.

    @property  # Remove cached_property to prevent stale sessions and authentication issues