-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use pyperformance to run the benchmarks. #3
Merged
Merged
Changes from 31 commits
Commits
Show all changes
33 commits
Select commit
Hold shift + click to select a range
18ef346
Move the benchmarks into individual projects.
ericsnowcurrently 3b1fbc9
Fix the manifest.
ericsnowcurrently 431d5a3
Add a version to the base metadata.
ericsnowcurrently 440762d
Add basic networking-related utils to share.
ericsnowcurrently 5a683f7
Drop a buch of unnecessary code in netutils.
ericsnowcurrently 1959205
Clean up waitUntilUp().
ericsnowcurrently e5f8861
Finish updating the benchmarks to the pyperformance format.
ericsnowcurrently 7e73fe0
Introduce metadata for "pre" and "post" scripts.
ericsnowcurrently 50bae09
Drop "pre" and "post" script support.
ericsnowcurrently 4c85a42
Use symlinks instead of the "libsdir" metadata.
ericsnowcurrently b131693
Don't rely on an internal Runner attr.
ericsnowcurrently e3a3970
Be clear about what is getting benchmarked.
ericsnowcurrently 0919748
Update for changes to pyperformance.
ericsnowcurrently ef83537
metabase -> inherits
ericsnowcurrently 8db13db
Ignore data generated during benchmark runs.
ericsnowcurrently 8a6a487
Run pyperformance more uniformly.
ericsnowcurrently c1bbc79
Make the script better.
ericsnowcurrently 3fd8188
Fix a numeric test in the script.
ericsnowcurrently 18988a9
Print dividers.
ericsnowcurrently 0061184
Prepare mypy even if not resetting it.
ericsnowcurrently 094e170
Add --skip-setup CLI to the runner script.
ericsnowcurrently 52a0858
Make the script executable.
ericsnowcurrently f810156
Run with mypyc 50 times.
ericsnowcurrently c71124f
Add --with-mypyc CLI to the runner script.
ericsnowcurrently c6072d6
Add --clean CLI to the runner script.
ericsnowcurrently 30fe453
Merge branch 'main' into pyperformance
ericsnowcurrently 08c9183
Make kinto_bench a pyperformance benchmark.
ericsnowcurrently 1d82993
Show how to run the benchmarks.
ericsnowcurrently e6164a8
Allow running the benchmarks the old way.
ericsnowcurrently c533adb
Fix bm_kinto.
ericsnowcurrently 7523c3f
Run all the benchmarks.
ericsnowcurrently 9828f98
Use an absolute path in the README.
ericsnowcurrently 766642a
Add a missing import.
ericsnowcurrently File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -129,3 +129,4 @@ dmypy.json | |
.pyre/ | ||
|
||
results | ||
benchmarks/bm_pytorch_alexnet_inference/data/dog.jpg |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,21 @@ | ||
# python-macrobenchmarks | ||
A collection of macro benchmarks for the Python programming language | ||
|
||
|
||
## usage | ||
|
||
```shell | ||
# Run the default benchmarks: | ||
python3 -m pyperformance run --manifest ./benchmarks/MANIFEST | ||
``` | ||
|
||
The benchmarks can still be run without pyperformance. This will produce | ||
the old results format. | ||
|
||
```shell | ||
# Run the benchmarks: | ||
sh ./run_all.sh | ||
|
||
# Run the mypy benchmark using mypyc: | ||
sh ./run_mypy.sh | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
import sys | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like this is missing an |
||
|
||
|
||
def maybe_handle_legacy(bench_func, *args, loopsarg='loops', legacyarg=None): | ||
if '--legacy' not in sys.argv: | ||
return | ||
argv = list(sys.argv[1:]) | ||
argv.remove('--legacy') | ||
|
||
kwargs = {} | ||
if legacyarg: | ||
kwargs[legacyarg] = True | ||
if argv: | ||
assert loopsarg | ||
kwargs[loopsarg] = int(argv[0]) | ||
|
||
_, times = bench_func(*args, **kwargs) | ||
if len(argv) > 1: | ||
json.dump(times, open(argv[1], 'w')) | ||
|
||
sys.exit(0) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
import contextlib | ||
import ipaddress | ||
import os.path | ||
import socket | ||
import subprocess | ||
import time | ||
|
||
|
||
@contextlib.contextmanager | ||
def serving(argv, sitedir, addr, *, | ||
pause=None, | ||
kill=False, | ||
quiet=True, | ||
): | ||
if os.path.exists(addr): | ||
sock = addr | ||
addr = None | ||
try: | ||
os.remove(sock) | ||
except FileNotFoundError: | ||
pass | ||
else: | ||
sock = None | ||
|
||
p = subprocess.Popen( | ||
argv, | ||
cwd=sitedir, | ||
stdout=subprocess.DEVNULL if quiet else None, | ||
stderr=subprocess.STDOUT if quiet else None, | ||
) | ||
try: | ||
if pause: | ||
time.sleep(pause) | ||
if not sock: | ||
try: | ||
waitUntilUp(addr) | ||
except NotImplementedError: | ||
sock = addr | ||
addr = None | ||
if sock: | ||
while not os.path.exists(sock): | ||
time.sleep(0.001) | ||
assert p.poll() is None, p.poll() | ||
yield | ||
assert p.poll() is None, p.poll() | ||
finally: | ||
p.terminate() | ||
if kill: | ||
p.kill() | ||
p.wait() | ||
|
||
|
||
def waitUntilUp(addr, timeout=10.0): | ||
end = time.time() + timeout | ||
addr = parse_socket_addr(addr) | ||
started = False | ||
current = time.time() | ||
while not started or current <= end: | ||
try: | ||
with socket.create_connection(addr) as sock: | ||
return | ||
except ConnectionRefusedError: | ||
time.sleep(0.001) | ||
started = True | ||
current = time.time() | ||
raise Exception('Timeout reached when trying to connect') | ||
|
||
|
||
def parse_socket_addr(addr, *, resolve=True): | ||
if not isinstance(addr, str): | ||
raise NotImplementedError(addr) | ||
host, _, port = addr.partition(':') | ||
|
||
if not host: | ||
raise NotImplementedError(addr) | ||
try: | ||
host = ipaddress.ip_address(host) | ||
except ValueError: | ||
raise NotImplementedError(addr) | ||
host = str(host) | ||
|
||
if not port: | ||
raise NotImplementedError(addr) | ||
if not port.isdigit(): | ||
raise NotImplementedError(addr) | ||
port = int(port) | ||
|
||
return (host, port) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
[benchmarks] | ||
|
||
name metafile | ||
aiohttp <local> | ||
djangocms <local> | ||
flaskblogging <local> | ||
gevent_hub <local> | ||
gunicorn <local> | ||
json <local> | ||
kinto <local> | ||
mypy <local> | ||
mypyc <local:mypy> | ||
pycparser <local> | ||
pylint <local> | ||
pytorch_alexnet_inference <local> | ||
thrift <local> | ||
|
||
[group default] | ||
-mypyc |
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../pyproject.toml |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../.libs/legacyutils.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../.libs/netutils.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
[project] | ||
name = "bm_aiohttp" | ||
dependencies = [ | ||
"aiohttp", | ||
"gunicorn", | ||
"requests", | ||
"uvloop", | ||
] | ||
dynamic = ["version"] | ||
|
||
[tool.pyperformance] | ||
inherits = ".." |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
import os.path | ||
import requests | ||
import sys | ||
|
||
import pyperf | ||
import netutils | ||
|
||
|
||
DATADIR = os.path.join( | ||
os.path.dirname(__file__), | ||
"data", | ||
) | ||
ARGV = [sys.executable, "serve.py"] | ||
|
||
|
||
############################# | ||
# benchmarks | ||
|
||
def bench_aiohttp_requests(loops=3000): | ||
elapsed, _ = _bench_aiohttp_requests(loops) | ||
return elapsed | ||
|
||
|
||
def _bench_aiohttp_requests(loops=3000, legacy=False): | ||
"""Measure N HTTP requests to a local server. | ||
Note that the server is freshly started here. | ||
Only the time for requests is measured here. The following are not: | ||
* preparing the site the server will serve | ||
* starting the server | ||
* stopping the server | ||
Hence this should be used with bench_time_func() | ||
insted of bench_func(). | ||
""" | ||
start = pyperf.perf_counter() | ||
elapsed = 0 | ||
times = [] | ||
with netutils.serving(ARGV, DATADIR, "127.0.0.1:8080"): | ||
requests_get = requests.get | ||
for i in range(loops): | ||
# This is a macro benchmark for a Python implementation | ||
# so "elapsed" covers more than just how long a request takes. | ||
t0 = pyperf.perf_counter() | ||
requests_get("http://localhost:8080/blog/").text | ||
t1 = pyperf.perf_counter() | ||
|
||
elapsed += t1 - t0 | ||
times.append(t0) | ||
if legacy and (i % 100 == 0): | ||
print(i, t0 - start) | ||
times.append(pyperf.perf_counter()) | ||
if legacy: | ||
total = times[-1] - start | ||
print("%.2fs (%.3freq/s)" % (total, loops / total)) | ||
return elapsed, times | ||
|
||
|
||
############################# | ||
# the script | ||
|
||
if __name__ == "__main__": | ||
from legacyutils import maybe_handle_legacy | ||
maybe_handle_legacy(_bench_aiohttp_requests, legacyarg='legacy') | ||
|
||
runner = pyperf.Runner() | ||
runner.metadata['description'] = "Test the performance of aiohttp" | ||
runner.bench_time_func("aiohttp", bench_aiohttp_requests) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../.libs/legacyutils.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
../.libs/netutils.py |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
[project] | ||
name = "bm_djangocms" | ||
dependencies = [ | ||
"Django", | ||
"django-cms", | ||
"djangocms-bootstrap4", | ||
"djangocms-file", | ||
"djangocms-googlemap", | ||
"djangocms-installer", | ||
"djangocms-snippet", | ||
"djangocms-style", | ||
"djangocms-video", | ||
"requests", | ||
] | ||
dynamic = ["version"] | ||
|
||
[tool.pyperformance] | ||
inherits = ".." |
File renamed without changes.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks like pyperformance main doesn't support manifests at relative directories:
But
seems to work
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FWIW, that's a bug in pyperformance. For now I've updated the README so the command at least works.