Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove unnecessary needUpdate to avoid unexpected skipping updating before cache synced #3994

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

lxtywypc
Copy link
Contributor

What type of PR is this?
/kind bug

What this PR does / why we need it:
Consider if some error occured in execution-controller, it would update the condition of work from true to false, and requeue in a short time(5ms for first time as default now), but during this reconciling, the condition of got work might be still true because the cache of work might not be synced yet. If no error occurs this time, when calling needUpdateCondition, we would find the condition in both "old" and "new" work status is true, that would cause unexpected skipping updating work status with the condition is false in cluster in fact.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:
I'm sorry for bringing this in #3808 , I intended to reduce the attempts to update work because now there are much more chances to update work status than before. But it seems quite a potential risk here. Maybe we could consider about a better way later to reduce the attempts to do unnecessary updating.

Does this PR introduce a user-facing change?:


…efore cache synced

Signed-off-by: lxtywypc <lxtywypc@gmail.com>
@karmada-bot karmada-bot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 24, 2023
@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign garrybest after the PR has been reviewed.
You can assign the PR to them by writing /assign @garrybest in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 24, 2023
@RainbowMango RainbowMango added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 24, 2023
@RainbowMango RainbowMango added this to the v1.7 milestone Aug 24, 2023
@RainbowMango
Copy link
Member

I'm sorry for bringing this in #3808 , I intended to reduce the attempts to update work because now there are much more chances to update work status than before. But it seems quite a potential risk here. Maybe we could consider about a better way later to reduce the attempts to do unnecessary updating.

I can't tell how bad it is right now. Please @XiShanYongYe-Chang take a look.

@lxtywypc Should we consider revert #3808?

@RainbowMango
Copy link
Member

/assign @XiShanYongYe-Chang

@XiShanYongYe-Chang
Copy link
Member

Ok, I will take a look ASAP.

@XiShanYongYe-Chang
Copy link
Member

Hi @lxtywypc, thank you for your update. I indeed did not consider this point when reviewing it earlier. How did you discover it during testing?

Copy link
Member

@XiShanYongYe-Chang XiShanYongYe-Chang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't tell how bad it is right now. Please @XiShanYongYe-Chang take a look.

@lxtywypc Should we consider revert #3808?

We don't need to revert the previously merged pull request. The purpose of this modification was originally to reduce the number of work updates, but due to possible delayed cache updates, it could result in an incorrect work status. After removing this logic, the work status can be correctly updated, although there may be some unnecessary update operations.

/lgtm

Ask an again review from @chaunceyjiang
/cc @chaunceyjiang

@karmada-bot karmada-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 24, 2023
@lxtywypc
Copy link
Contributor Author

How did you discover it during testing?

Due to that our testing environment sometimes need a little long time(for example 1s) for cache syncing. Once we noticed a work failed applied for a long time and tried to figure out what happened, we found that it had been applied successfully after once retry through the log of karmada-controller-manager. Then we found the potential risk of this part.

@chaunceyjiang
Copy link
Member

/assign

@lxtywypc lxtywypc marked this pull request as draft August 31, 2023 07:49
@karmada-bot karmada-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 31, 2023
@lxtywypc
Copy link
Contributor Author

Draft for reverting #3808 as comment #3959 (comment)

@RainbowMango RainbowMango modified the milestones: v1.7, v1.8 Sep 11, 2023
@RainbowMango RainbowMango modified the milestones: v1.8, v1.9 Jan 9, 2024
@RainbowMango RainbowMango modified the milestones: v1.9, v1.10 Feb 29, 2024
@RainbowMango RainbowMango modified the milestones: v1.10, v1.11 May 29, 2024
@RainbowMango RainbowMango removed this from the v1.11 milestone Aug 28, 2024
@RainbowMango RainbowMango added this to the v1.12 milestone Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants