Add lexer benchmark

[dhruvmanila]

Summary

This PR adds a separate lexer benchmark. The parser benchmark might not be
sufficient to detect any regressions in the lexer performance.

Test Plan

cargo bench -p ruff_benchmark --bench lexer

[dhruvmanila]

Current dependencies on/for this PR:

• main
  • PR Add lexer benchmark #7132 👈

This comment was auto-generated by Graphite.

[MichaReiser]

Thank you

[codspeed-hq]

CodSpeed Performance Report

Merging #7132 will degrade performances by 2.49%

_{Comparing dhruv/lexer-benchmark (39c91ef) with main (ece30e7)}

Summary

❌ 1 regressions
✅ 19 untouched benchmarks

🆕 5 new benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	main	dhruv/lexer-benchmark	Change
🆕	lexer[unicode/pypinyin.py]	N/A	700.4 µs	N/A
🆕	lexer[numpy/globals.py]	N/A	257.7 µs	N/A
🆕	lexer[numpy/ctypeslib.py]	N/A	2.3 ms	N/A
🆕	lexer[pydantic/types.py]	N/A	4.8 ms	N/A
❌	linter/all-rules[numpy/ctypeslib.py]	32.9 ms	33.8 ms	-2.49%
🆕	lexer[large/dataset.py]	N/A	11.6 ms	N/A

[github-actions]

PR Check Results

Ecosystem

✅ ecosystem check detected no changes.