[feature] Option to disable Caching in V2 at the KFP, Pipeline, and Run level

[HumairAK]

Feature Area

/area frontend
/area backend

What feature would you like to see?

Allow flexibility in being able to enable/disable caching at multiple levels in KFP.
A user should be able to:

• Disable caching for a single run of a pipeline
• Disable caching for all runs for a pipeline (toggle?)
• Disable caching entirely for a KFP deployment

The user should be able to do this VIA the UI, or API. I don't think this requires changes from SDK side, but if it does, we should point that out.

As part of this acceptance criteria, the implementer should also update the configurations to clearly indicate where the old v1 caching behavior is for v1 only and not for v2. Example here.

What is the use case or pain point?

Currently the only way to disable caching in KFP V2 is via the SDK , by setting .set_caching_options.

This can be annoying to do for every single step, every single time. In some cases a user may not be satisfied with the current caching capabilities and want to opt to not use it, the user should be easily enabled to do so.

This feature seems like it used to exist with kfp v1 cache server to some degree, but no longer exists in kfpv2.

Is there a workaround currently?

The only workaround is to disable caching via sdk on a per component basis.

---

Love this idea? Give it a 👍.

[HumairAK]

It looks like there was a lot of work on this done by @TobiasGoerke in the past here: #8177

It seems like it didn't end up getting merged, but could serve as a useful reference.

[HumairAK]

Related: #8104

[gregsheremeta]

I will begin working on this

edit: scratch that, I'm not going to work on (all) of this quite yet

[HumairAK]

/assign gregsheremeta

[gregsheremeta]

partial duplicate of #6578

[gregsheremeta]

/unassign @gregsheremeta