performance.md

Performance

These are benchmarks for sake. The implementation is in a separate repository at github.com/alajmo/sake-performance.

sake is between 6 and 8 faster than pyinfra and between 4 and 18 times faster than Ansible, depending on the number of hosts.

The benchmarks are generated by running each test 20 times and taking an average for each of the following metrics:

• Elapsed Time: The elapsed real (wall clock) time (seconds) used by the process
• CPU: Percentage of the CPU that this job got ((user + sys time) / tot time
• Memory: Maximum memory (Megabyte) usage of the process during its lifetime

Ping

Modules

Test Case 1

A simple ping.

Graphs

Data

Elapsed Time (seconds)

name	sake	pyinfra	ansible
1	0.143	0.888	0.602
3	0.125	0.948	0.621
5	0.131	0.951	0.637
8	0.157	0.972	0.671
10	0.137	0.968	0.701
25	0.158	1.117	0.957
50	0.175	1.364	1.419
100	0.320	1.879	2.463
200	0.559	2.914	4.308
300	0.826	4.050	6.240
400	1.112	5.148	8.137
500	1.400	6.332	10.152

CPU (%)

name	sake	pyinfra	ansible
1	16	89	86
3	19	87	95
5	28	87	103
8	34	90	113
10	34	92	120
25	48	96	183
50	48	103	236
100	53	113	311
200	52	126	332
300	51	136	357
400	49	144	361
500	49	152	377

Memory (MB)

name	sake	pyinfra	ansible
1	15	56	55
3	16	56	55
5	18	56	55
8	21	58	56
10	21	58	55
25	25	60	55
50	27	61	56
100	29	63	58
200	40	68	61
300	48	72	65
400	59	76	70
500	66	80	76

Test Case 2

The following tasks are ran:

1. Add a user
2. Add a file
3. Copy a file

Note, after the first command is ran, the subsequent commands won't do anything since the user and files already exists, so all tasks are idempotent.

Graphs

Data

Elapsed Time (seconds)

name	sake	pyinfra	ansible
1	0.156	1.568	1.174
3	0.159	1.487	1.210
5	0.175	1.456	1.268
8	0.203	1.489	1.436
10	0.246	1.489	1.535
25	0.274	2.180	2.742
50	0.439	2.982	4.717
100	0.664	4.711	8.947
200	1.006	8.371	17.410
300	1.425	12.108	26.110
400	1.911	15.797	34.888
500	2.443	19.418	44.075

CPU (%)

name	sake	pyinfra	ansible
1	14	54	67
3	27	65	86
5	39	70	104
8	53	74	124
10	61	76	138
25	70	77	236
50	111	86	258
100	168	95	305
200	147	101	324
300	138	105	339
400	135	109	352
500	157	112	365

Memory (MB)

name	sake	pyinfra	ansible
1	15	55	55
3	19	57	55
5	20	57	56
8	21	58	55
10	22	59	55
25	25	61	56
50	29	63	56
100	34	67	59
200	45	74	63
300	55	80	68
400	66	89	73
500	76	96	79