feat(backend): add the option to enable/disable cache globally #11831

hbelmiro · 2025-04-15T18:18:11Z

Description of your changes:
Resolves: #11840
This PR adds the option to enable/disable cache globally.

How to test

Deploy KFP with the CACHEENABLED environment variable set to "false" in the API Server deployment.

spec:
  template:
    spec:
      containers:
      - env:
        - name: CACHEENABLED
          value: "false"

Run a pipeline twice. The second run should not use the cache.

Checklist:

You have signed off your commits
The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

google-oss-prow · 2025-04-15T18:18:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

backend/src/v2/driver/driver.go

kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc.py

kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc.yaml

...etes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc_from_task_output.py

...es_platform/python/test/snapshot/data/create_mount_delete_existing_pvc_from_task_output.yaml

backend/src/apiserver/resource/resource_manager.go

hbelmiro · 2025-04-25T17:31:00Z

@rimolive regarding the changes in the PVC name, I added them because I ran all the tests with cache disabled just to see if I had done all the needed changes. When cache is disabled the PVC names conflict with each other.
I thought it would be better to leave them with different names. Not a strong opinion though. That's totally optional.

rimolive · 2025-04-25T21:28:06Z

/lgtm

gmfrasca

Tested as per PR instructions, including running a Pipeline and then cloning the run, with CACHEENABLED env var in API Server Deployment set to false, true, and unset (defaults to caching enabled). Verified expected behavior in all cases.

/lgtm

hbelmiro · 2025-05-07T18:00:32Z

@mprahl all comments addressed/answered.
Can you please take another look?

mprahl

/lgtm

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Co-authored-by: Giulio Frasca <giulio.m.frasca@gmail.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

mprahl

/lgtm
/approve

google-oss-prow · 2025-05-07T20:02:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mprahl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~.github/OWNERS~~ [mprahl]
~~backend/OWNERS~~ [mprahl]
~~kubernetes_platform/OWNERS~~ [mprahl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

hbelmiro · 2025-05-08T12:19:57Z

/retest

…low#11831) * feat(backend): add the option to enable/disable cache globally Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chore(backend): added logging when cache is disabled globally Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * Update backend/src/apiserver/resource/resource_manager.go Co-authored-by: Giulio Frasca <giulio.m.frasca@gmail.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chore(backend): Fixed Scheduled Workflows Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chrore(backend): Added integration tests Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chrore(backend): Removed container arg when --cache_enabled is true Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * Update backend/src/v2/cmd/driver/main.go Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * Update backend/src/v2/cmd/compiler/main.go Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chrore(backend): Renamed to CacheDisabled and removed pointers Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> * chrore(backend): Added Argo compiler unit test with the cache disabled Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> --------- Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com> Co-authored-by: Giulio Frasca <giulio.m.frasca@gmail.com> Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> (cherry picked from commit 9aebb62) Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

google-oss-prow bot added the do-not-merge/work-in-progress label Apr 15, 2025

google-oss-prow bot added the size/XL label Apr 15, 2025

google-oss-prow bot requested review from gkcalat and rimolive April 15, 2025 18:18

hbelmiro force-pushed the cache-config branch 5 times, most recently from 7e2b253 to d4957a3 Compare April 21, 2025 14:08

hbelmiro changed the title ~~WIP: Add an option to enable/disable cache globally~~ WIP: feat(backend): add the option to enable/disable cache globally Apr 21, 2025

hbelmiro changed the title ~~WIP: feat(backend): add the option to enable/disable cache globally~~ feat(backend): add the option to enable/disable cache globally Apr 21, 2025

hbelmiro marked this pull request as ready for review April 21, 2025 14:44

google-oss-prow bot removed the do-not-merge/work-in-progress label Apr 21, 2025

google-oss-prow bot requested review from DharmitD and HumairAK April 21, 2025 14:44

gregsheremeta reviewed Apr 22, 2025

View reviewed changes

backend/src/v2/driver/driver.go Outdated Show resolved Hide resolved

hbelmiro force-pushed the cache-config branch 2 times, most recently from 774c968 to 0683e55 Compare April 22, 2025 12:24

rimolive reviewed Apr 25, 2025

View reviewed changes

kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc.py Show resolved Hide resolved

rimolive reviewed Apr 25, 2025

View reviewed changes

kubernetes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc.yaml Show resolved Hide resolved

rimolive reviewed Apr 25, 2025

View reviewed changes

...etes_platform/python/test/snapshot/data/create_mount_delete_existing_pvc_from_task_output.py Show resolved Hide resolved

rimolive reviewed Apr 25, 2025

View reviewed changes

...es_platform/python/test/snapshot/data/create_mount_delete_existing_pvc_from_task_output.yaml Show resolved Hide resolved

gmfrasca reviewed Apr 25, 2025

View reviewed changes

backend/src/apiserver/resource/resource_manager.go Outdated Show resolved Hide resolved

hbelmiro force-pushed the cache-config branch from c4c7e6e to 9ad60c9 Compare April 25, 2025 20:46

google-oss-prow bot assigned rimolive Apr 25, 2025

google-oss-prow bot added the lgtm label Apr 25, 2025

gmfrasca approved these changes Apr 29, 2025

View reviewed changes

hbelmiro force-pushed the cache-config branch from 6f30c90 to 3a253f8 Compare May 7, 2025 15:04

google-oss-prow bot added size/XXL and removed size/XL labels May 7, 2025

hbelmiro marked this pull request as ready for review May 7, 2025 17:59

google-oss-prow bot removed the do-not-merge/work-in-progress label May 7, 2025

mprahl approved these changes May 7, 2025

View reviewed changes

google-oss-prow bot assigned mprahl May 7, 2025

google-oss-prow bot added the lgtm label May 7, 2025

hbelmiro and others added 10 commits May 7, 2025 16:46

feat(backend): add the option to enable/disable cache globally

dd9308e

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chore(backend): added logging when cache is disabled globally

2fd486a

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Update backend/src/apiserver/resource/resource_manager.go

a12571c

Co-authored-by: Giulio Frasca <giulio.m.frasca@gmail.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chore(backend): Fixed Scheduled Workflows

e461753

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chrore(backend): Added integration tests

9b3b9b5

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chrore(backend): Removed container arg when --cache_enabled is true

78be32a

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Update backend/src/v2/cmd/driver/main.go

4ec55d0

Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

Update backend/src/v2/cmd/compiler/main.go

0c7337c

Co-authored-by: Matt Prahl <mprahl@users.noreply.github.com> Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chrore(backend): Renamed to CacheDisabled and removed pointers

47d568e

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

chrore(backend): Added Argo compiler unit test with the cache disabled

e497ea5

Signed-off-by: Helber Belmiro <helber.belmiro@gmail.com>

hbelmiro force-pushed the cache-config branch from 72f8665 to e497ea5 Compare May 7, 2025 19:48

google-oss-prow bot removed the lgtm label May 7, 2025

mprahl approved these changes May 7, 2025

View reviewed changes

google-oss-prow bot added the lgtm label May 7, 2025

google-oss-prow bot added the approved label May 7, 2025

google-oss-prow bot merged commit 9aebb62 into kubeflow:master May 8, 2025
68 of 75 checks passed

hbelmiro deleted the cache-config branch May 8, 2025 12:45

VaniHaripriya mentioned this pull request May 19, 2025

Add CacheEnabled field to APIServer for configurable caching in DSPA opendatahub-io/data-science-pipelines-operator#838

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(backend): add the option to enable/disable cache globally #11831

feat(backend): add the option to enable/disable cache globally #11831

Uh oh!

hbelmiro commented Apr 15, 2025 •

edited

Loading

Uh oh!

google-oss-prow bot commented Apr 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hbelmiro commented Apr 25, 2025 •

edited

Loading

Uh oh!

rimolive commented Apr 25, 2025

Uh oh!

gmfrasca left a comment

Uh oh!

hbelmiro commented May 7, 2025

Uh oh!

mprahl left a comment

Uh oh!

mprahl left a comment

Uh oh!

google-oss-prow bot commented May 7, 2025

Uh oh!

hbelmiro commented May 8, 2025

Uh oh!

Uh oh!

Uh oh!

feat(backend): add the option to enable/disable cache globally #11831

feat(backend): add the option to enable/disable cache globally #11831

Uh oh!

Conversation

hbelmiro commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to test

Uh oh!

google-oss-prow bot commented Apr 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hbelmiro commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rimolive commented Apr 25, 2025

Uh oh!

gmfrasca left a comment

Choose a reason for hiding this comment

Uh oh!

hbelmiro commented May 7, 2025

Uh oh!

mprahl left a comment

Choose a reason for hiding this comment

Uh oh!

mprahl left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented May 7, 2025

Uh oh!

hbelmiro commented May 8, 2025

Uh oh!

Uh oh!

Uh oh!

hbelmiro commented Apr 15, 2025 •

edited

Loading

hbelmiro commented Apr 25, 2025 •

edited

Loading