[WIP] Optimize CI

[Stevenjin8]

We spend a good amount of money on CI and with our AWS migration coming up, I would like to run fewer jobs. See istio/istio#61500

Using the last 48 hours as a representative samples, I think we are using ~48% fewer CPU hours.

This PR keeps our test coverage the same. It just happens that a lot of it moves to postsubmit rather than presubmit. The only postsubmit chagnes we make is to stop testing k8s versions <= 1.31

[fjglira]

I'm in favor of moving jobs to postsubmit to save cost. My only concern is how it's going to check when those postubmit fails. We still have a lot of postsubmit tests failing, and there is a lot of noise in the #test-failures channel to be able to catch this when it fails.

Also +1 on removing older jobs than 1.32

[dhawton]

I don't think we can just remove k8s jobs like that. We document we test k8s versions going fairly far back, if we turn those off we can't say what's "supported" anymore beyond the version we do test: https://istio.io/latest/docs/releases/supported-releases/

[Stevenjin8]

I don't think we can just remove k8s jobs like that. We document we test k8s versions going fairly far back, if we turn those off we can't say what's "supported" anymore beyond the version we do test: https://istio.io/latest/docs/releases/supported-releases/

@dhawton yep we would need to update our docs. 1.31 support ended 9 months ago. 1.24 ended 3 years ago. The oldest version. We should just drop those from our docs

[dhawton]

I don't think we can just remove k8s jobs like that. We document we test k8s versions going fairly far back, if we turn those off we can't say what's "supported" anymore beyond the version we do test: https://istio.io/latest/docs/releases/supported-releases/

@dhawton yep we would need to update our docs. 1.31 support ended 9 months ago. 1.24 ended 3 years ago. The oldest version. We should just drop those from our docs

The "Tested but not supported" column is what I am talking about. That's quite a change that probably would be worth discussing in the workgroup meeting as we have, for many years, had this column. It's more than just turning off and updating docs imho. I know we have users who use outside of the supported that rely on the tests to help validate it can still run on the slightly older kubernetes versions.

[fjglira]

If we want to keep those older jobs and still save money there is another alternative, set periodic jobs and execute those jobs for example once a week. Users can still monitor periodic jobs to check that nothing is broken in those older k8s version and we skip executing on every PR running only once a week. Running periodic jobs like this is a regular practice to keep testing but avoiding over test when is not needed

[fjglira]

To avoid more discussion in the PR, I just added anothers options in the issue istio/istio#61500

[howardjohn]

I would prob keep at least the oldest one

[istio-testing]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[geomachine]

learned so many things just reading this PR comments.