[Backend] Cache - Update max_cache_staleness logic

[rui5i]

This PR updates max_cache_staleness logic to:

1. get all_unexpired_cache_entry with target cache_key
2. from all_unexpired_cache_entry get valid cache entry:
   (1) if cache_entry is too old for target pod, ignore
   (2) if pod.maxCacheStaleness != -1, return latest cache_entry with entry.maxCacheStaleness == pod.maxCacheStaleness
   (3) return overall latest cache_entry if pod.maxCacheStaleness == -1

Part of #2904

[kubeflow-bot]

This change is 

[rui5i]

/assign @Ark-kun

[rmgogogo]

/lgtm

[rmgogogo]

@Ark-kun will leave approval to you.
I'm OK with this PR.

[Ark-kun]

Hmm. Initially I had the impression that the code logic in this PR was different from the stated logic. However, now I see that the logic was later revised and now it looks correct.

I'd like to add some tests to verify the logic before checking this PR in though.
/hold

[rmgogogo]

Hmm. Initially I had the impression that the code logic in this PR was different from the stated logic. However, now I see that the logic was later revised and now it looks correct.

I'd like to add some tests to verify the logic before checking this PR in though.
/hold

kindly ping. Alexy, will you continue this PR? Assume it won't block 0.5.1 release.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign ark-kun
You can assign the PR to them by writing /assign @ark-kun in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• backend/src/cache/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rmgogogo]

(sorry, miss clicked close)

[Ark-kun]

I think the check needs to be

Suggested change

	if maxCacheStaleness == -1 || s.time.Now().UTC().Unix()-startedAtInSec <= maxCacheStaleness {
	if podMaxCacheStaleness == -1 || s.time.Now().UTC().Unix()-startedAtInSec <= podMaxCacheStaleness {

(with podMaxCacheStaleness parameter added back to the function).
The task cannot/should not control whether its data can be reused in the future. It can only control whether it can be skipped.
So, when we do lookup, we only care about the preferences of the current pod/task.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.