Skip to content

Conversation

@chrisjsewell
Copy link
Member

@chrisjsewell chrisjsewell commented Nov 25, 2025

Have standard test, to make sure the build works, and separate benchmark test.

The standard test runs during standard testing, e.g. running tox,
and the benchmark test can be run using e.g. tox -e py312-benchmark -- tests/benchmarks/test_schema_benchmark.py --benchmark-columns=min,max,mean


Current results on my MacBook:

--------------------------------------- benchmark: 4 tests --------------------------------------
Name (time in ms)                       Min                   Max                  Mean          
-------------------------------------------------------------------------------------------------
test_schema_benchmark[10]            3.7598 (1.0)          3.8571 (1.0)          3.8040 (1.0)    
test_schema_benchmark[100]          34.5185 (9.18)        36.2754 (9.40)        35.2756 (9.27)   
test_schema_benchmark[1000]        359.5095 (95.62)      404.7844 (104.95)     375.7929 (98.79)  
test_schema_benchmark[10000]     3,766.5854 (>1000.0)  4,034.7065 (>1000.0)  3,875.5382 (>1000.0)
-------------------------------------------------------------------------------------------------

Run standard test, to make sure the build works, and separate benchmark tests
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR restructures the schema benchmark testing to separate functional validation from performance benchmarking. The benchmark project setup is extracted into a reusable fixture (schema_benchmark_app) in conftest.py, enabling both standard tests that verify build correctness and dedicated benchmark tests that measure schema processing performance.

Key changes:

  • Extracted benchmark project setup into a reusable fixture
  • Added standard test in test_schema.py that runs during normal testing
  • Simplified benchmark test to focus on performance measurement
  • Moved snapshot assertions to the standard test

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/schema/test_schema.py Adds standard test that validates the benchmark project builds successfully using snapshot assertions
tests/schema/snapshots/test_schema.ambr Contains new snapshot data for the standard benchmark test validation
tests/conftest.py Introduces schema_benchmark_app fixture that dynamically generates benchmark projects with configurable need counts
tests/benchmarks/test_schema_benchmark.py Simplified to focus on performance benchmarking using pytest-benchmark, removing project setup logic
tests/benchmarks/__snapshots__sphinx_lt_8/test_schema_benchmark.ambr Removed as snapshot assertions moved to standard test
tests/benchmarks/__snapshots__sphinx_ge_8/test_schema_benchmark.ambr Removed as snapshot assertions moved to standard test

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codecov
Copy link

codecov bot commented Nov 25, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.94%. Comparing base (4e10030) to head (242ff54).
⚠️ Report is 198 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1579      +/-   ##
==========================================
+ Coverage   86.87%   87.94%   +1.06%     
==========================================
  Files          56       70      +14     
  Lines        6532     9630    +3098     
==========================================
+ Hits         5675     8469    +2794     
- Misses        857     1161     +304     
Flag Coverage Δ
pytests 87.94% <ø> (+1.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Member

@ubmarco ubmarco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@chrisjsewell chrisjsewell merged commit 8817754 into master Nov 25, 2025
20 checks passed
@chrisjsewell chrisjsewell deleted the schema-benchmarking branch November 25, 2025 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants