Skip to content

Run py-fuzzer with --profile=profiling locally and in CI #21266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 3, 2025
Merged

Run py-fuzzer with --profile=profiling locally and in CI #21266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 23 additions & 19 deletions .github/workflows/ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,8 @@ jobs:
workspaces: "fuzz -> target"
- name: "Install Rust toolchain"
run: rustup show
- name: "Install mold"
uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- name: "Install cargo-binstall"
uses: cargo-bins/cargo-binstall@b3f755e95653da9a2d25b99154edfdbd5b356d0a # v1.15.10
- name: "Install cargo-fuzz"
Expand Down Expand Up @@ -645,45 +647,45 @@ jobs:
name: "Fuzz for new ty panics"
runs-on: ${{ github.repository == 'astral-sh/ruff' && 'depot-ubuntu-22.04-16' || 'ubuntu-latest' }}
needs:
- cargo-test-linux
- determine_changes
# Only runs on pull requests, since that is the only we way we can find the base version for comparison.
if: ${{ !contains(github.event.pull_request.labels.*.name, 'no-test') && github.event_name == 'pull_request' && (needs.determine_changes.outputs.ty == 'true' || needs.determine_changes.outputs.py-fuzzer == 'true') }}
timeout-minutes: ${{ github.repository == 'astral-sh/ruff' && 10 || 20 }}
steps:
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
with:
fetch-depth: 0
persist-credentials: false
- uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
name: Download new ty binary
id: ty-new
with:
name: ty
path: target/debug
- uses: dawidd6/action-download-artifact@20319c5641d495c8a52e688b7dc5fada6c3a9fbc # v8
name: Download baseline ty binary
with:
name: ty
branch: ${{ github.event.pull_request.base.ref }}
workflow: "ci.yaml"
check_artifacts: true
- uses: astral-sh/setup-uv@85856786d1ce8acfbcc2f13a5f3fbd6b938f9f41 # v7.1.2
- uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
- name: "Install Rust toolchain"
run: rustup show
- name: "Install mold"
uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- name: Fuzz
env:
FORCE_COLOR: 1
NEW_TY: ${{ steps.ty-new.outputs.download-path }}
run: |
# Make executable, since artifact download doesn't preserve this
chmod +x "${PWD}/ty" "${NEW_TY}/ty"
echo "new commit"
git rev-list --format=%s --max-count=1 "$GITHUB_SHA"
cargo build --profile=profiling --bin=ty
mv target/profiling/ty ty-new

MERGE_BASE="$(git merge-base "$GITHUB_SHA" "origin/$GITHUB_BASE_REF")"
git checkout -b old_commit "$MERGE_BASE"
echo "old commit (merge base)"
git rev-list --format=%s --max-count=1 old_commit
cargo build --profile=profiling --bin=ty
mv target/profiling/ty ty-old
Comment on lines +669 to +679
Copy link
Member Author

@AlexWaygood AlexWaygood Nov 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is all basically copied-and-pasted from

ruff/.github/workflows/typing_conformance.yaml

Lines 63 to 78 in 79a0271

# Build the executable for the old and new commit
(
cd ruff
echo "new commit"
git rev-list --format=%s --max-count=1 "$GITHUB_SHA"
cargo build --bin ty
mv target/debug/ty ty-new
MERGE_BASE="$(git merge-base "$GITHUB_SHA" "origin/$GITHUB_BASE_REF")"
git checkout -b old_commit "$MERGE_BASE"
echo "old commit (merge base)"
git rev-list --format=%s --max-count=1 old_commit
cargo build --bin ty
mv target/debug/ty ty-old
)

and is also similar to what mypy_primer does.


(
uv run \
--python="${PYTHON_VERSION}" \
--project=./python/py-fuzzer \
--locked \
fuzz \
--test-executable="${NEW_TY}/ty" \
--baseline-executable="${PWD}/ty" \
--test-executable=ty-new \
--baseline-executable=ty-old \
--only-new-bugs \
--bin=ty \
0-1000
Expand Down Expand Up @@ -715,6 +717,8 @@ jobs:
- uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
- name: "Install Rust toolchain"
run: rustup show
- name: "Install mold"
uses: rui314/setup-mold@725a8794d15fc7563f59595bd9556495c0564878 # v1
- name: "Run ty completion evaluation"
run: cargo run --release --package ty_completion_eval -- all --threshold 0.4 --tasks /tmp/completion-evaluation-tasks.csv
- name: "Ensure there are no changes"
Expand Down
5 changes: 3 additions & 2 deletions python/py-fuzzer/fuzz.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,13 +395,14 @@ def parse_args() -> ResolvedCliArgs:

if not args.test_executable:
print(
"Running `cargo build --release` since no test executable was specified...",
"Running `cargo build --profile=profiling` since no test executable was specified...",
flush=True,
)
cmd: list[str] = [
"cargo",
"build",
"--release",
"--profile",
"profiling",
"--locked",
"--color",
"always",
Expand Down