Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add perf metrics for 2.43.0 #50864

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add perf metrics for 2.43.0 #50864

wants to merge 1 commit into from

Conversation

khluu
Copy link
Contributor

@khluu khluu commented Feb 24, 2025

REGRESSION 47.69%: tasks_per_second (THROUGHPUT) regresses from 427.6684054263659 to 223.72283336753296 in benchmarks/many_nodes.json
REGRESSION 39.45%: pgs_per_second (THROUGHPUT) regresses from 22.74259799982883 to 13.769920950065924 in benchmarks/many_pgs.json
REGRESSION 32.47%: tasks_per_second (THROUGHPUT) regresses from 588.1874987887217 to 397.2166130464684 in benchmarks/many_tasks.json
REGRESSION 28.32%: multi_client_put_gigabytes (THROUGHPUT) regresses from 47.38817873082456 to 33.968763685249776 in microbenchmark.json
REGRESSION 24.93%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 7.56687497857023 to 5.680486486924976 in microbenchmark.json
REGRESSION 15.78%: single_client_wait_1k_refs (THROUGHPUT) regresses from 5.424179804481537 to 4.568303345800756 in microbenchmark.json
REGRESSION 12.61%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 12.987017792446045 to 11.349304359077122 in microbenchmark.json
REGRESSION 10.56%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 4669.450952977499 to 4176.255856784519 in microbenchmark.json
REGRESSION 9.48%: n_n_actor_calls_with_arg_async (THROUGHPUT) regresses from 2732.074477927061 to 2473.163327924083 in microbenchmark.json
REGRESSION 8.44%: single_client_tasks_sync (THROUGHPUT) regresses from 1013.1673399687909 to 927.6195505025511 in microbenchmark.json
REGRESSION 8.24%: 1_1_async_actor_calls_with_args_async (THROUGHPUT) regresses from 2953.9546990180343 to 2710.6889492331006 in microbenchmark.json
REGRESSION 6.69%: single_client_tasks_async (THROUGHPUT) regresses from 8032.409007811969 to 7495.427972759755 in microbenchmark.json
REGRESSION 5.90%: multi_client_tasks_async (THROUGHPUT) regresses from 22745.167201851888 to 21404.22305204687 in microbenchmark.json
REGRESSION 3.88%: 1_1_async_actor_calls_sync (THROUGHPUT) regresses from 1474.7069154202795 to 1417.4714984406974 in microbenchmark.json
REGRESSION 3.69%: 1_1_actor_calls_sync (THROUGHPUT) regresses from 1985.8445748508243 to 1912.6174326068551 in microbenchmark.json
REGRESSION 3.63%: n_n_async_actor_calls_async (THROUGHPUT) regresses from 23390.23156817461 to 22542.16458297063 in microbenchmark.json
REGRESSION 3.53%: 1_1_actor_calls_concurrent (THROUGHPUT) regresses from 5218.943213086157 to 5034.662033195711 in microbenchmark.json
REGRESSION 3.40%: client__1_1_actor_calls_sync (THROUGHPUT) regresses from 522.5361491166506 to 504.79511779286827 in microbenchmark.json
REGRESSION 2.70%: client__tasks_and_put_batch (THROUGHPUT) regresses from 14255.363968554979 to 13870.502654856362 in microbenchmark.json
REGRESSION 2.43%: multi_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 15931.811977493457 to 15543.88478060201 in microbenchmark.json
REGRESSION 1.77%: single_client_put_calls_Plasma_Store (THROUGHPUT) regresses from 4866.041059585032 to 4780.133410691611 in microbenchmark.json
REGRESSION 1.58%: client__put_gigabytes (THROUGHPUT) regresses from 0.15408690555366225 to 0.1516460375456968 in microbenchmark.json
REGRESSION 0.87%: n_n_actor_calls_async (THROUGHPUT) regresses from 26441.672940245888 to 26212.485337497925 in microbenchmark.json
REGRESSION 0.22%: single_client_get_calls_Plasma_Store (THROUGHPUT) regresses from 10611.609624248378 to 10588.032610459411 in microbenchmark.json
REGRESSION 190.22%: stage_4_spread (LATENCY) regresses from 0.20731175121781154 to 0.6016603078317008 in stress_tests/stress_test_many_tasks.json
REGRESSION 64.03%: avg_iteration_time (LATENCY) regresses from 0.7432218527793885 to 1.2191412186622619 in stress_tests/stress_test_dead_actors.json
REGRESSION 55.33%: stage_0_time (LATENCY) regresses from 4.678402662277222 to 7.266809701919556 in stress_tests/stress_test_many_tasks.json
REGRESSION 52.14%: stage_3_creation_time (LATENCY) regresses from 1.1666333675384521 to 1.7748794555664062 in stress_tests/stress_test_many_tasks.json
REGRESSION 29.29%: dashboard_p99_latency_ms (LATENCY) regresses from 3492.239 to 4515.016 in benchmarks/many_actors.json
REGRESSION 16.55%: avg_pg_remove_time_ms (LATENCY) regresses from 1.2978452747749973 to 1.5126715450448436 in stress_tests/stress_test_placement_group.json
REGRESSION 16.10%: dashboard_p50_latency_ms (LATENCY) regresses from 4.297 to 4.989 in benchmarks/many_nodes.json
REGRESSION 8.46%: 10000_args_time (LATENCY) regresses from 17.283816822999995 to 18.745566793000002 in scalability/single_node.json
REGRESSION 4.86%: dashboard_p95_latency_ms (LATENCY) regresses from 2727.324 to 2859.782 in benchmarks/many_actors.json
REGRESSION 4.18%: 10000_get_time (LATENCY) regresses from 23.877336194999998 to 24.874795295 in scalability/single_node.json
REGRESSION 3.53%: avg_pg_create_time_ms (LATENCY) regresses from 1.470244198198109 to 1.522115139639421 in stress_tests/stress_test_placement_group.json
REGRESSION 3.36%: 1000000_queued_time (LATENCY) regresses from 192.979472547 to 199.46889562500002 in scalability/single_node.json
REGRESSION 2.39%: stage_3_time (LATENCY) regresses from 1823.5409452915192 to 1867.0339968204498 in stress_tests/stress_test_many_tasks.json
REGRESSION 0.27%: stage_1_avg_iteration_time (LATENCY) regresses from 12.529016280174256 to 12.563214778900146 in stress_tests/stress_test_many_tasks.json

Signed-off-by: kevin <kevin@anyscale.com>
@khluu khluu requested review from aslonnie and jjyao February 24, 2025 22:01
@jjyao
Copy link
Collaborator

jjyao commented Feb 25, 2025

REGRESSION 47.69%: tasks_per_second (THROUGHPUT) regresses from 427.6684054263659 to 223.72283336753296 in benchmarks/many_nodes.json

Expected due to release test runtime change.

REGRESSION 39.45%: pgs_per_second (THROUGHPUT) regresses from 22.74259799982883 to 13.769920950065924 in benchmarks/many_pgs.json

Expected due to release test runtime change.

REGRESSION 32.47%: tasks_per_second (THROUGHPUT) regresses from 588.1874987887217 to 397.2166130464684 in benchmarks/many_tasks.json

Expected due to release test runtime change.

REGRESSION 28.32%: multi_client_put_gigabytes (THROUGHPUT) regresses from 47.38817873082456 to 33.968763685249776 in microbenchmark.json

Expected due to release test runtime change.

REGRESSION 24.93%: single_client_tasks_and_get_batch (THROUGHPUT) regresses from 7.56687497857023 to 5.680486486924976 in microbenchmark.json

Expected due to release test runtime change.

REGRESSION 15.78%: single_client_wait_1k_refs (THROUGHPUT) regresses from 5.424179804481537 to 4.568303345800756 in microbenchmark.json

Noise + small regression that doesn't block the release

REGRESSION 12.61%: single_client_get_object_containing_10k_refs (THROUGHPUT) regresses from 12.987017792446045 to 11.349304359077122 in microbenchmark.json

Noise

REGRESSION 10.56%: 1_1_async_actor_calls_async (THROUGHPUT) regresses from 4669.450952977499 to 4176.255856784519 in microbenchmark.json

Noise

REGRESSION 190.22%: stage_4_spread (LATENCY) regresses from 0.20731175121781154 to 0.6016603078317008 in stress_tests/stress_test_many_tasks.json

Expected due to release test runtime change.

REGRESSION 64.03%: avg_iteration_time (LATENCY) regresses from 0.7432218527793885 to 1.2191412186622619 in stress_tests/stress_test_dead_actors.json

REGRESSION 55.33%: stage_0_time (LATENCY) regresses from 4.678402662277222 to 7.266809701919556 in stress_tests/stress_test_many_tasks.json

Expected due to release test runtime change.

REGRESSION 52.14%: stage_3_creation_time (LATENCY) regresses from 1.1666333675384521 to 1.7748794555664062 in stress_tests/stress_test_many_tasks.json

Expected due to release test runtime change.

REGRESSION 16.55%: avg_pg_remove_time_ms (LATENCY) regresses from 1.2978452747749973 to 1.5126715450448436 in stress_tests/stress_test_placement_group.json

Noise.

@aslonnie
Copy link
Collaborator

will prefer #50894 which is more up to date.

numbers look similar (at least to me)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants