Skip to content

Latest commit

 

History

History
110 lines (92 loc) · 5.75 KB

README.md

File metadata and controls

110 lines (92 loc) · 5.75 KB
Raw

s5cmd performance regression tests

bench.py allow us to compare two different build (from either version tag, PR number or commit tag) performance under various scenarios. These scenarios include:

  1. Upload, Download, Remove many small sized file
  2. Upload, Download, Remove large file
  3. Upload, Download, Remove very large file

To change the scenarios, you should edit it inside the bench.py for now. In the future, this could be read from a file. From each scenario, user should not forget to change the file size and file count keeping in mind the restrictions of their system.

required tools

This script is dependent on the following tools. Make sure you install them to your system before running bench.py.

  • git
  • go
  • hyperfine
  • truncate

To run use the following syntax:

usage: bench.py [-h] [-s OLD NEW] [-w WARMUP] [-r RUNS] [-o OUTPUT_FILE_NAME] -b BUCKET [-l LOCAL_PATH] [-p PREFIX] [-hf HYPERFINE_EXTRA_FLAGS] [-sf S5CMD_EXTRA_FLAGS]

Compare performance of two different builds of s5cmd.

optional arguments:
  -h, --help            show this help message and exit
  -s OLD NEW, --s5cmd OLD NEW
                        Reference to old and new s5cmd.It can be a decimal indicating PR number,any of the version tags like v2.0.0 or any commit tag. Additionally it can be 'latest_release' or
                        'master'. (default: ('latest_release', 'master'))
  -w WARMUP, --warmup WARMUP
                        Number of program executions before the actual benchmark: (default: 2)
  -r RUNS, --runs RUNS  Number of runs to perform for each command (default: 10)
  -o OUTPUT_FILE_NAME, --output_file_name OUTPUT_FILE_NAME
                        Name of the output file (default: summary.md)
  -b BUCKET, --bucket BUCKET
                        Name of the bucket in remote (default: None)
  -l LOCAL_PATH, --local-path LOCAL_PATH
                        specify a local path for temporary files to be loaded. (default: None)
  -p PREFIX, --prefix PREFIX
                        Key prefix to be used while uploading to a specified bucket (default: s5cmd-benchmarks-)
  -hf HYPERFINE_EXTRA_FLAGS, --hyperfine-extra-flags HYPERFINE_EXTRA_FLAGS
                        hyperfine global extra flags. Write in between quotation marks and start with a space to avoid bugs. (default: None)
  -sf S5CMD_EXTRA_FLAGS, --s5cmd-extra-flags S5CMD_EXTRA_FLAGS
                        s5cmd global extra flags. Write in between quotation marks and start with a space to avoid bugs. (default: None)

Examples

./bench.py --bucket tempbucket

Above command will compare latest-release to master with 2 warmup runs and 10 benchmark runs.

./bench.py --bucket tempbucket --s5cmd v2.0.0 456 --warmup 2 --runs 10

Above command will compare v2.0.0 to PR:456 with 2 warmup runs and 10 benchmark runs.

./bench.py --bucket tempbucket --s5cmd v2.0.0 456 --warmup 2 --runs 10 -sf " --log error" -hf " --show-output"

When using -hf and -sf flags, use quotes like above and start with an empty space. If not started with an empty space, it might give an error. This is a known issue with argparse and this discussion can be useful to understand the problem deeper.

Example Output

./bench.py --bucket tempbucket --s5cmd master 478 --warmup 2 --runs 15

Benchmark summary:

Scenarios File Size File Count
small files 1M 10000
large file 10G 1
very large file 300G 1
Scenario Summary
upload small files 'PR:478' ran 1.01 ± 0.02 times faster than 'master'
download small files 'PR:478' ran 1.00 ± 0.01 times faster than 'master'
remove small files 'master' ran 1.05 ± 0.41 times faster than 'PR:478'
upload large file 'PR:478' ran 1.18 ± 0.23 times faster than 'master'
download large file 'master' ran 1.05 ± 0.08 times faster than 'PR:478'
remove large file 'master' ran 1.21 ± 0.39 times faster than 'PR:478'
upload very large file 'PR:478' ran 1.01 times faster than 'master'
download very large file 'master' ran 1.02 times faster than 'PR:478'
remove very large file 'PR:478' ran 1.13 times faster than 'master'

Detailed summary:

Scenario Command Mean [s] Min [s] Max [s] Relative
upload small files PR:478 9.117 ± 0.155 8.848 9.337 1.00
upload small files master 9.252 ± 0.160 9.084 9.483 1.01 ± 0.02
download small files PR:478 79.992 ± 0.091 79.879 80.177 1.00
download small files master 79.993 ± 0.462 79.096 81.028 1.00 ± 0.01
remove small files PR:478 2.603 ± 0.435 2.308 3.245 1.05 ± 0.41
remove small files master 2.470 ± 0.878 2.012 3.787 1.00
upload large file PR:478 10.093 ± 1.491 9.043 14.222 1.00
upload large file master 11.876 ± 1.486 10.730 15.787 1.18 ± 0.23
download large file PR:478 27.689 ± 1.378 25.979 30.803 1.05 ± 0.08
download large file master 26.452 ± 1.667 24.891 29.375 1.00
remove large file PR:478 0.157 ± 0.029 0.122 0.210 1.21 ± 0.39
remove large file master 0.130 ± 0.034 0.090 0.220 1.00
upload very large file PR:478 270.462 270.462 270.462 1.00
upload very large file master 272.473 272.473 272.473 1.01
download very large file PR:478 2538.727 2538.727 2538.727 1.02
download very large file master 2501.010 2501.010 2501.010 1.00
remove very large file PR:478 1.011 1.011 1.011 1.00
remove very large file master 1.145 1.145 1.145 1.13