Support performance tests

[ligurio]

We have a need to run performance tests using luatest in, for example, CRUD module. Now CRUD has a single test that report results on each iteration to console log and then someone aggregates these results and analyze manually (or in spreadsheets). To be honest it is not convenient and is not usable for regular automated runs on CI for tracking regressions.

Performance tests are similar to unit tests, but requires at least two additional things: results aggregation and analyzing with math. I propose to support in luatest such type of tests.

How it can look like:

• luatest already has an option --repeat NUM, that Execute all tests NUM times, e.g. to trig the JIT. We can reuse it for set a number of test iterations for results stability.
• we need to somehow store test results for each test run and then aggregate it in a single place (table?) for analysis.
• we need to add functions that will calculate average, stdev, max, +/- stdev (anything else?).

For better results stability we need to use taskset with luatest. Is it possible to use it automatically once performance tests run?

It would be interesting to hear ideas and opinion from @Totktonada and @VitaliyaIoffe.

[Totktonada]

Is there some standard way to report perf. test results?

[olegrok]

There is wrk that accepts Lua scripts and we could provide similar interface. But I'm not sure that it's good suggestion. Workload engine written in Lua potentially could be a bottleneck in the test.
Popular workload engines are written in C/C++/Go/Java but not script languages.

[ligurio]

Is there some standard way to report perf. test results?

As far as I know - no.

[ligurio]

@olegrok there is a k6, that scriptable with JS.
https://habr.com/ru/company/vk/blog/563446/

[VitaliyaIoffe]

Is there some standard way to report perf. test results?

I think it would depend on our database-visualization system.

[olegrok]

@olegrok there is a k6, that scriptable with JS. https://habr.com/ru/company/vk/blog/563446/

It's written in Go. JS is just frontend. And seems it supports more than one thread for workload. To achieve something similar in Tarantool you need to start several instances of Tarantool. Otherwise TX thread could be a bottleneck. Or you need to use coio-threads.

[ligurio]

I would split performance testing for two parts: system testing (when a target is a whole project or Lua module, when we need a special environment and good workload) and unit testing, aka microbenchmarks (when a target is a single function and a couple of nested functions, no need special environment and Tarantool with single core is more than enough to load a target), and here consider the second one. Imagine we have a Lua function that need a performance test. In case of C/C++ we will use Google Benchmark, but for Lua there is no such library, so let's add support for it to luatest.