Remove performance tests iterations logic as it is now handled natively in k6

[alopezari]

This PR aims to remove the logic around performance tests iterations on QIT in favor of this other PR, which leveraged running multiple iterations to k6.

This drastically reduces performance tests duration from 1h with 7 iterations (baseline and SUT each) to 15m with 10 iterations.

Testing instructions

1. Ensure you use a QIT version with the changes from this PR.
2. Ensure you use a CD version with the changes from this other PR.
3. Publish the latest version of performance tests package with php src/qit-cli.php package:publish ../compatibility-dashboard/ci/tests/performance/test-package/ latest.
4. Run performance tests locally with php src/qit-cli.php run:performance automatewoo -vvv --test_packages=latest --local.
5. Check the logs and ensure it k6 tests only runs twice on the QIT side (one for the baseline and another one for SUT), and check in the k6 execution logs that it runs 10 times there.
6. Ensure tests pass and flow durations are shown in the k6 logs.
7. Check that test artifacts are properly generated by QIT and no files or folders make any reference to iterations.
8. Repeat steps 3-7 on staging while running the tests on CI with php src/qit-cli.php run:performance automatewoo -vvv --test_packages=latest.

[zhongruige]

This might be just from me running locally but I did hit some memory exhaustion errors, did you see this at all?

running (03m38.0s), 0/1 VUs, 10 complete and 0 interrupted iterations
frontend ✓ [ 100% ] 1 VUs  03m36.1s/10m0s  10/10 iters, 10 per VU
✓ k6 performance test passed
k6 results saved to: /var/folders/mz/zl1q1bdd6c16qkgsw4c7r3sm0000gn/T/qit-results/qitenv12e21c30e90ccad8/result.json
PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /Users/gregbell/workspace/qit-cli/src/src/Performance/Result/PerformanceTestResult.php on line 157

[alopezari]

Hi @zhongruige!

I ran the tests on my local environment and they finished properly. It might be related to the memory allocated in your local env 🤔

[zhongruige]

That's my guess, here's what I had mine set at:

I'll bump it up and give it a try.

[zhongruige]

Overall tested great for me, thanks @alopezari! I did notice a weird issue when trying to switch back using the binary I have in my path:

qit switch production

                                                                                                                        
 [ERROR] Invalid environment type.   

If I use the one built on the PR it works fine, do you see the same when toggling between them?

[alopezari]

@zhongruige hmm I can't reproduce the issue you mentioned when switching environments using the manager from the branch remove/performance-tests-iterations-logic and my global qit binary. I started from the qit-cli remove/performance-tests-iteration-logic branch and then switched to trunk, as you can see below:

qit switch production
Backend switched to "production".
Sync complete.

qit switch staging
Backend switched to "staging".
Sync complete.

git checkout trunk
Switched to branch 'trunk'
Your branch is up to date with 'origin/trunk'.

qit switch local
Backend switched to "local".
Sync complete.

qit switch production
Backend switched to "production".
Sync complete.

qit switch local
Backend switched to "local".
Sync complete.

Did you perform any specific steps to reproduce this issue?

[zhongruige]

Weird! I may have messed something up by accident, possibly when doing the utility package testing without realizing it. I'll get things set up again, but thanks for confirming this is working for you!

[zhongruige]

Happy to add my approval, nice work here!