Add performance testing setup to Kedro

[ankatiyar]

Description

Part of #4128

Development notes

I've managed to set the performance testing framework using asv on my fork of Kedro: https://github.com/ankatiyar/kedro

The setup is -

• benchmarks/ folder is where the performance/stress tests for various components such as DataCatalog or OmegaConfigLoader will go. Right now I've left a dummy test in that we can replace or leave in. I suggest we leave this in so we will know if some regressions in the future are happening because of the runners on Github Actions and not because of our changes.
• The workflow in this repository - A copy of https://github.com/ankatiyar/kedro/blob/main/.github/workflows/asv.yml - will run everytime there is a commit to the main repo. The results are then committed to a separate repo -> https://github.com/ankatiyar/benchmark-kedro (example). I made a similar repo in kedro-org -> https://github.com/kedro-org/kedro-benchmark-results
• There is an actions workflow that runs on the results repo that publishes the result as github pages.
  • Example GH Page: https://ankatiyar.github.io/benchmark-kedro/
  • Workflow (on my fork): https://github.com/ankatiyar/benchmark-kedro/blob/main/.github/workflows/publish-benchmark.yml

❗ Disclaimer ❗ : This might not immediately work and might require tweaks with permission issues as follow up PRs so I would like to get a first draft in to test the action with the dummy test.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[ElenaKhaustova]

@ankatiyar , this looks great!

I have a short question just for my understanding.

How publishing results is happening for the newly created repo: https://github.com/kedro-org/kedro-benchmark-results?

I see in your example it's happening every time the results are committed to the main: https://github.com/ankatiyar/benchmark-kedro/blob/c4971ae3d47f26b32dc43b4fe356ed629cc93f19/.github/workflows/publish-benchmark.yml#L6

But in the new repo with the benchmark results the workflow looks different: https://github.com/kedro-org/kedro-benchmark-results/blob/7bd633596c3289ae18a70818099d7ef27b86539f/.github/workfllows/publish-results.yml#L4

Is that on purpose?

[ankatiyar]

@ElenaKhaustova Thanks for taking a look! Github actions are not currently enabled on kedro-org/kedro-benchmark-results and will have to be enabled by an admin so I've just got a basic version of the workflow on there. It'll need to be updated later. With this PR I wanted to get the setup on the kedro repo set up first and make sure the results are being committed to kedro-benchmark-results first. As long as the results show up there, they will show up on the report when we enable actions there.

[ElenaKhaustova]

@ElenaKhaustova Thanks for taking a look! Github actions are not currently enabled on kedro-org/kedro-benchmark-results and will have to be enabled by an admin so I've just got a basic version of the workflow on there. It'll need to be updated later. With this PR I wanted to get the setup on the kedro repo set up first and make sure the results are being committed to kedro-benchmark-results first. As long as the results show up there, they will show up on the report when we enable actions there.

That totally makes sense, thanks for explaining!

[ElenaKhaustova]

Great work, us usual 🚀

[merelcht]

This looks really great! My main question is how this would work with a full Kedro project, like the one @lrcouto has been building?

Would it maybe make sense to have a separate repo for all benchmarking tests to not pollute the main repo?

[ankatiyar]

This looks really great! My main question is how this would work with a full Kedro project, like the one @lrcouto has been building?

Would it maybe make sense to have a separate repo for all benchmarking tests to not pollute the main repo?
@merelcht

I've left a comment here #4128 (comment) with a proposal for the performance test setup over-all. Would like to get your thoughts.

I think it is possible to also have the benchmark tests in a separate repo but I think this is the simplest setup and I'm worried that they might end up being neglected if they are. It will also help to have these tests here to execute the second part of the proposal, to run performance tests for PRs against main to see if any regression is being added by a particular PR before merging.

[noklam]

My question are:

• How do we store the result and how big are those files? Is is small enough to keep it in Git?
• Is it possible to backfill historical commits? There have been few refactor that may affect performance, will be interesting to see at least in 0.19 how this change.

[ankatiyar]

@noklam

• The results are just small json files, they'll be committed in https://github.com/kedro-org/kedro-benchmark-results (similar to my repo https://github.com/ankatiyar/benchmark-kedro/tree/main/.asv/results/github-actions)
• It is possible to get results for previous commits but we'll need to set up tests first

[noklam]

Thanks for getting this work nicely!