Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: specify node for rebuild instance ops #8151

Merged
merged 10 commits into from
Sep 18, 2024
Merged

Conversation

cjc7373
Copy link
Contributor

@cjc7373 cjc7373 commented Sep 14, 2024

Note: I add nodeSelector to pod in a way that won't affect pod's revision hash, because adding nodeSelector happens after calculating hash.

@cjc7373 cjc7373 requested review from wangyelei and a team as code owners September 14, 2024 03:14
@github-actions github-actions bot added the size/L Denotes a PR that changes 100-499 lines. label Sep 14, 2024
@cjc7373 cjc7373 added this to the Release 0.9.1 milestone Sep 14, 2024
Copy link

codecov bot commented Sep 14, 2024

Codecov Report

Attention: Patch coverage is 54.90196% with 46 lines in your changes missing coverage. Please review.

Project coverage is 61.56%. Comparing base (197596a) to head (e38f13a).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
pkg/controller/instanceset/instance_util.go 59.61% 15 Missing and 6 partials ⚠️
controllers/apps/operations/rebuild_instance.go 58.33% 7 Missing and 3 partials ⚠️
...ollers/apps/operations/rebuild_instance_inplace.go 40.00% 6 Missing and 3 partials ⚠️
pkg/controller/instanceset/reconciler_status.go 25.00% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8151      +/-   ##
==========================================
+ Coverage   61.11%   61.56%   +0.45%     
==========================================
  Files         359      360       +1     
  Lines       41299    41786     +487     
==========================================
+ Hits        25240    25726     +486     
+ Misses      13812    13792      -20     
- Partials     2247     2268      +21     
Flag Coverage Δ
unittests 61.56% <54.90%> (+0.45%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@zjx20
Copy link
Contributor

zjx20 commented Sep 14, 2024

I suspect there's a potential data race in this implementation: The rebuild workflow adds an annotation to the InstanceSet and then deletes the pod, prompting the InstanceSet to recreate the pod. The InstanceSet controller clears the annotation after determining that the pod has been created. If the InstanceSet controller enters reconciliation and completes successfully before the rebuild workflow deletes the pod (which is possible because the InstanceSet CR's annotation has been modified), the annotation would be cleared prematurely (since the pod hasn't been deleted yet), causing this feature to fail.

I suggest that only clear the annotation of a pod if pod.spec.nodeName equals the one specified in the annotation.

@apecloud-bot apecloud-bot added the approved PR Approved Test label Sep 18, 2024
@apecloud-bot apecloud-bot removed the approved PR Approved Test label Sep 18, 2024
@apecloud-bot apecloud-bot added the approved PR Approved Test label Sep 18, 2024
@apecloud-bot apecloud-bot removed the approved PR Approved Test label Sep 18, 2024
@apecloud-bot apecloud-bot added the approved PR Approved Test label Sep 18, 2024
@cjc7373 cjc7373 merged commit 87e27f5 into main Sep 18, 2024
35 checks passed
@cjc7373 cjc7373 deleted the feature/specify-node-rebuild branch September 18, 2024 09:50
@wangyelei
Copy link
Contributor

/cherry-pick release-0.9

Copy link

🤖 says: Error cherry-picking.

Auto-merging apis/apps/v1alpha1/opsrequest_types.go
Auto-merging config/crd/bases/apps.kubeblocks.io_opsrequests.yaml
Auto-merging controllers/apps/operations/rebuild_instance.go
CONFLICT (content): Merge conflict in controllers/apps/operations/rebuild_instance.go
Auto-merging controllers/apps/operations/rebuild_instance_test.go
CONFLICT (content): Merge conflict in controllers/apps/operations/rebuild_instance_test.go
Auto-merging deploy/helm/crds/apps.kubeblocks.io_opsrequests.yaml
Auto-merging docs/developer_docs/api-reference/cluster.md
Auto-merging pkg/constant/annotations.go
Auto-merging pkg/controller/instanceset/instance_util.go
Auto-merging pkg/controller/instanceset/instance_util_test.go
Auto-merging pkg/controller/instanceset/reconciler_instance_alignment_test.go
Auto-merging pkg/controller/instanceset/reconciler_status.go
Auto-merging pkg/controller/instanceset/reconciler_status_test.go
CONFLICT (content): Merge conflict in pkg/controller/instanceset/reconciler_status_test.go
error: could not apply 87e27f5... feat: specify node for rebuild instance ops (#8151)
hint: After resolving the conflicts, mark them with
hint: "git add/rm ", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config advice.mergeConflict false"

Copy link

🤖 says: ‼️ cherry pick action failed.
See: https://github.com/apecloud/kubeblocks/actions/runs/10933182583

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved PR Approved Test size/L Denotes a PR that changes 100-499 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants