Alternative solution to removal of test on optional-test-infra

[annajung]

[UPDATE June 14th, 2022] This is no longer a blocker for the 1.6 release as all WG have pivoted to using GitHub Actions as a short-term solution

This is a blocker for the 1.6 release

This issue is to track an alternative solution to the recent removal of existing presubmit and postsubmit tests on optional-test-infra.

As of May 31st,

• Testing clusters are deleted and not available for use
• All WGs who leveraged AWS optional-test-infra are blocked by the removal and need to find an alternative until a long term solution is found
• KServe WG started working on migrating to GitHub actions with KinD clusters and shared as a possible alternative. However, this may not work for other WG due to resource constraints
• AWS folks are willing to provide credits but need to work out logistics on creating a non-personal AWS account and how to get AWS credit associated with it. (Previously was using a personal AWS account)

References

• Optional Test Infra Deprecation Notice #993
• Remove existing presubmit and postsubmit on optional-test-infra #1003
• Basic tests for manifests manifests#2099

@kubeflow/wg-automl-leads @kubeflow/wg-training-leads @kubeflow/wg-notebooks-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-manifests-leads @pvaneck @yuzisun @kubeflow/release-team @akartsky @surajkota

[annajung]

Hi WG leads, there is a discussion happening in parallel with AWS, but I also like to kick off a discussion to see if you would be transitioning to a different CI/CD pipeline before the release. If so, do you have any timelines in mind?

[jlewi]

Hi folks; its been a minute.

As a bit of additional context. The issue of a sustainable and scalable approach to test infrastructure was identified almost two years ago in #737.

What is the current thinking in terms of making each WG responsible for its own test-infra?

[thesuperzapper]

@annajung I have created a new private slack channel with all the working group leads to organize the creation of separate AWS accounts for each Working Group, and get AWS credits applied to them.

[surajkota]

As discussed in the Community Meeting 05/31, each Working Groups will need to setup their own testing infrastructure, as the optional-test-infra has been deleted.

AWS is willing to provide credits to the Working Groups for testing, to facilitate this please do the following:

• Each WG must delegate a single “testing manager” who will be the point of contact between AWS and their WG about credits/etc
• Each WG creates their own AWS account for any tests they may need to run
• Each “testing manager” must send a AWS Pricing Calculator estimate of the costs for their testing infra (in this slack channel)
• Each “testing manager” will work with AWS to get credits applied to their AWS account (in this slack channel)

The above is based on the assumption that a decentralised infra is more scalable and sustainable approach in the long term

[jbottum]

@johnugeorge @kimwnasptd @yuzisun @james-jwu @annajung This seems like a very generous offer from @surajkota. Would you please reply in a timely manner with your perspective / response? Thanks!

[yuzisun]

Thanks @surajkota !!