[feature] Ability to disable caching for a particular pipeline run via the UI

[jackwhelpton]

Feature Area

/area frontend

What feature would you like to see?

The ability to enable or disable (v2) caching for a pipeline run via the UI.

What is the use case or pain point?

When developing pipelines, it is possible to end up in a position where the component "succeeds" (does not error), but returns erroneous results. Once this has happened, correcting the component code and re-executing does not suffice, as the previous (incorrect) results are cached.

Is there a workaround currently?

We separate authoring pipelines from executing them, so typically our executions are via the Vertex Pipelines UI, by uploading the compiled JSON.

One workaround would be to extend the pipeline code to support executing the pipeline from code, which does allow the caching to be disabled. This change would have to be made to all our pipelines, and is onerous.

Either move the input files and change the components to bust the cache, or execute the pipeline using Python.

Neither of these approaches are ideal, and neither would be easily available to non-coders when they execute the pipeline (people who can upload a given JSON file and set parameters, but not make code changes).

Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[capri-xiyue]

@zijianjoy
I talked with sdk side and vertex side, the agreement is to implement disable/enable caching level at client side first.
Later when we have a clear goal of caching CUJ, we can talk about implementing this in server side which means change pipeline job proto.
SDK already implemented such logic in

pipelines/sdk/python/kfp/v2/google/client/client.py

Lines 140 to 152 in 55a2fb5

	def _set_enable_caching_value(pipeline_spec: Dict[str, Any],
	enable_caching: bool) -> None:
	"""Sets pipeline tasks caching options.
	Args:
	pipeline_spec: The dictionary of pipeline spec.
	enable_caching: Whether to enable caching.
	"""
	for component in [pipeline_spec['root']] + list(
	pipeline_spec['components'].values()):
	if 'dag' in component:
	for task in component['dag']['tasks'].values():
	task['cachingOptions'] = {'enableCache': enable_caching}

, you can follow such logic in UI.

[zijianjoy]

@capri-xiyue Sounds good on providing the ability to disable pipeline level caching on UI side, once we updated pipeline job proto to support this field. But question: currently when you create a run from pipeline template, UI doesn't use the PipelineJob payload itself, instead UI will create a run using pipeline_version_id for PIPELINE_VERSION resource_reference. So I guess it should be backend which exposes caching configuration first?

[capri-xiyue]

@capri-xiyue Sounds good on providing the ability to disable pipeline level caching on UI side, once we updated pipeline job proto to support this field. But question: currently when you create a run from pipeline template, UI doesn't use the PipelineJob payload itself, instead UI will create a run using pipeline_version_id for PIPELINE_VERSION resource_reference. So I guess it should be backend which exposes caching configuration first?

Can UI manipulate the template before UI call backend?

[capri-xiyue]

Discussed offline, backend needs to expose cache configuration first and then front end can disable caching for a particular pipeline run via the UI

[capri-xiyue]

Reassigned it to James Wu to further discuss the priority and the assignee

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

We will cover this and more in #8177 hopefully we can present it tomorrow in the KFP meeting

[thesuperzapper]

@zijianjoy it would be great to disable caching for a specific run via the UI, what do you think?

Many users don't understand what the cache does, and having that option in the UI would help them understand that sometimes their runs will pull from a cache.

Note, we already have a feature to do this when submitting runs from the SDK:

https://www.kubeflow.org/docs/components/pipelines/v2/caching/#how-to-use-caching

[juliusvonkohout]

There are also some older issues and PRs for exactly that. For example #8177

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[juliusvonkohout]

/lifecycle frozen