Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]cluster is always updating after vscale #7579

Closed
ahjing99 opened this issue Jun 20, 2024 · 0 comments · Fixed by #7592
Closed

[BUG]cluster is always updating after vscale #7579

ahjing99 opened this issue Jun 20, 2024 · 0 comments · Fixed by #7592
Assignees
@free6om
Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.9.0

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Jun 20, 2024 •
edited
Loading

➜ ~ kbcli version
Kubernetes: v1.29.4-gke.1043002
KubeBlocks: 0.9.0-beta.34
kbcli: 0.9.0-beta.27

All apecloud and redis and foxlake have this problem

  1. Create cluster
  `kbcli cluster create  mysql-gukzbk --termination-policy=Halt --cluster-definition=apecloud-mysql --enable-all-logs=false --cluster-version=ac-mysql-8.0.30-1 --set cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi  --namespace default `

Cluster mysql-gukzbk created

➜  ~ kbcli cluster describe mysql-gukzbk
Name: mysql-gukzbk	 Created Time: Jun 20,2024 16:36 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION             STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30-1   Running   Halt

Endpoints:
COMPONENT   MODE        INTERNAL                                            EXTERNAL
mysql       ReadWrite   mysql-gukzbk-mysql.default.svc.cluster.local:3306   <none>

Topology:
COMPONENT   INSTANCE               ROLE       STATUS    AZ              NODE                                                  CREATED-TIME
mysql       mysql-gukzbk-mysql-0   follower   Running   us-central1-c   gke-yjtest-default-pool-36251504-z5l7/10.128.15.197   Jun 20,2024 16:36 UTC+0800
mysql       mysql-gukzbk-mysql-1   leader     Running   us-central1-c   gke-yjtest-default-pool-36251504-xw1f/10.128.15.202   Jun 20,2024 16:36 UTC+0800
mysql       mysql-gukzbk-mysql-2   follower   Running   us-central1-c   gke-yjtest-default-pool-36251504-r78c/10.128.15.238   Jun 20,2024 16:36 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mysql       false       100m / 100m          512Mi / 512Mi           data:1Gi       kb-default-sc

Images:
COMPONENT   TYPE    IMAGE
mysql       mysql   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.14

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION   RECOVERABLE-TIME

Show cluster events: kbcli cluster list-events -n default mysql-gukzbk
  1. vscale
➜  ~  kbcli cluster vscale mysql-gukzbk --auto-approve --force=true                 --components mysql                 --cpu 200m                 --memory 0.6Gi
OpsRequest mysql-gukzbk-verticalscaling-j5jhc created successfully, you can view the progress:
	kbcli cluster describe-ops mysql-gukzbk-verticalscaling-j5jhc -n default
  1. Cluster is always updating for pod is always running
➜  ~ k describe ops mysql-gukzbk-verticalscaling-j5jhc
Name:         mysql-gukzbk-verticalscaling-j5jhc
Namespace:    default
Labels:       app.kubernetes.io/instance=mysql-gukzbk
              app.kubernetes.io/managed-by=kubeblocks
              ops.kubeblocks.io/ops-type=VerticalScaling
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         OpsRequest
Metadata:
  Creation Timestamp:  2024-06-20T08:39:43Z
  Finalizers:
    opsrequest.kubeblocks.io/finalizer
  Generate Name:  mysql-gukzbk-verticalscaling-
  Generation:     2
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName:
        f:labels:
          .:
          f:app.kubernetes.io/instance:
          f:app.kubernetes.io/managed-by:
      f:spec:
        .:
        f:clusterName:
        f:force:
        f:preConditionDeadlineSeconds:
        f:type:
        f:verticalScaling:
          .:
          k:{"componentName":"mysql"}:
            .:
            f:componentName:
            f:limits:
              .:
              f:cpu:
            f:requests:
              .:
              f:cpu:
    Manager:      kbcli
    Operation:    Update
    Time:         2024-06-20T08:39:43Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"opsrequest.kubeblocks.io/finalizer":
        f:labels:
          f:ops.kubeblocks.io/ops-type:
        f:ownerReferences:
          .:
          k:{"uid":"851984fa-8ff3-4901-a949-18c32a9a9861"}:
      f:spec:
        f:verticalScaling:
          k:{"componentName":"mysql"}:
            f:limits:
              f:memory:
            f:requests:
              f:memory:
    Manager:      manager
    Operation:    Update
    Time:         2024-06-20T08:39:43Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterGeneration:
        f:components:
          .:
          f:mysql:
            .:
            f:phase:
            f:progressDetails:
        f:conditions:
          .:
          k:{"type":"Validated"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"VerticalScaling"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
          k:{"type":"WaitForProgressing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:reason:
            f:status:
            f:type:
        f:lastConfiguration:
          .:
          f:components:
            .:
            f:mysql:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
        f:phase:
        f:progress:
        f:startTimestamp:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2024-06-20T08:39:44Z
  Owner References:
    API Version:     apps.kubeblocks.io/v1alpha1
    Kind:            Cluster
    Name:            mysql-gukzbk
    UID:             851984fa-8ff3-4901-a949-18c32a9a9861
  Resource Version:  280683
  UID:               037a017f-084d-47c3-a911-e79344691fe3
Spec:
  Cluster Name:                    mysql-gukzbk
  Force:                           true
  Pre Condition Deadline Seconds:  0
  Type:                            VerticalScaling
  Vertical Scaling:
    Component Name:  mysql
    Limits:
      Cpu:     200m
      Memory:  644245094400m
    Requests:
      Cpu:     200m
      Memory:  644245094400m
Status:
  Cluster Generation:  2
  Components:
    Mysql:
      Phase:  Updating
      Progress Details:
        Object Key:  Pod/mysql-gukzbk-mysql-1
        Status:      Pending
        Object Key:  Pod/mysql-gukzbk-mysql-0
        Status:      Pending
        Object Key:  Pod/mysql-gukzbk-mysql-2
        Status:      Pending
  Conditions:
    Last Transition Time:  2024-06-20T08:39:43Z
    Message:               wait for the controller to process the OpsRequest: mysql-gukzbk-verticalscaling-j5jhc in Cluster: mysql-gukzbk
    Reason:                WaitForProgressing
    Status:                True
    Type:                  WaitForProgressing
    Last Transition Time:  2024-06-20T08:39:43Z
    Message:               OpsRequest: mysql-gukzbk-verticalscaling-j5jhc is validated
    Reason:                ValidateOpsRequestPassed
    Status:                True
    Type:                  Validated
    Last Transition Time:  2024-06-20T08:39:43Z
    Message:               Start to vertical scale resources in Cluster: mysql-gukzbk
    Reason:                VerticalScalingStarted
    Status:                True
    Type:                  VerticalScaling
  Last Configuration:
    Components:
      Mysql:
        Limits:
          Cpu:     100m
          Memory:  512Mi
        Requests:
          Cpu:      100m
          Memory:   512Mi
  Phase:            Running
  Progress:         0/3
  Start Timestamp:  2024-06-20T08:39:43Z
Events:
  Type    Reason                    Age   From                    Message
  ----    ------                    ----  ----                    -------
  Normal  WaitForProgressing        14m   ops-request-controller  wait for the controller to process the OpsRequest: mysql-gukzbk-verticalscaling-j5jhc in Cluster: mysql-gukzbk
  Normal  ValidateOpsRequestPassed  14m   ops-request-controller  OpsRequest: mysql-gukzbk-verticalscaling-j5jhc is validated
  Normal  VerticalScalingStarted    14m   ops-request-controller  Start to vertical scale resources in Cluster: mysql-gukzbk
➜  ~
W0620 08:49:11.021135       1 warnings.go:70] The ClusterVersion CRD has been deprecated since 0.9.0
2024-06-20T08:50:41.100Z	ERROR	Reconciler error	{"controller": "instanceset", "controllerGroup": "workloads.kubeblocks.io", "controllerKind": "InstanceSet", "InstanceSet": {"name":"mysql-gukzbk-mysql","namespace":"default"}, "namespace": "default", "name": "mysql-gukzbk-mysql", "reconcileID": "7338cf94-1e72-4940-bc8c-da5a6a06b36a", "error": "Pod \"mysql-gukzbk-mysql-2\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`,`spec.initContainers[*].image`,`spec.activeDeadlineSeconds`,`spec.tolerations` (only additions to existing tolerations),`spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)\n  core.PodSpec{\n  \tVolumes:        {{Name: \"annotations\", VolumeSource: {DownwardAPI: &{Items: {{Path: \"leader\", FieldRef: &{APIVersion: \"v1\", FieldPath: \"metadata.annotations['cs.apps.kubeblocks.io/leader']\"}}, {Path: \"component-replicas\", FieldRef: &{APIVersion: \"v1\", FieldPath: \"metadata.annotations['apps.kubeblocks.io/component-replicas']\"}}}, DefaultMode: &420}}}, {Name: \"scripts\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-apecloud-mysql-scripts\"}, DefaultMode: &365}}}, {Name: \"mysql-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-mysql-consensusset-config\"}, DefaultMode: &292}}}, {Name: \"mysql-scale-config\", VolumeSource: {ConfigMap: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-vttablet-config\"}, DefaultMode: &292}}}, ...},\n  \tInitContainers: {{Name: \"init-data\", Image: \"infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mys\"..., Command: {\"bash\", \"-c\", \"cp /usr/lib/mysql/plugin/ /data/mysql/plugin -r\\n\"}, EnvFrom: {{ConfigMapRef: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-env\"}, Optional: &false}}}, ...}},\n  \tContainers: []core.Container{\n  \t\t{\n  \t\t\t... // 6 identical fields\n  \t\t\tEnvFrom: {{ConfigMapRef: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-env\"}, Optional: &false}}, {ConfigMapRef: &{LocalObjectReference: {Name: \"mysql-gukzbk-mysql-rsm-env\"}, Optional: &false}}},\n  \t\t\tEnv:     {{Name: \"KB_POD_NAME\", ValueFrom: &{FieldRef: &{APIVersion: \"v1\", FieldPath: \"metadata.name\"}}}, {Name: \"KB_POD_UID\", ValueFrom: &{FieldRef: &{APIVersion: \"v1\", FieldPath: \"metadata.uid\"}}}, {Name: \"KB_NAMESPACE\", ValueFrom: &{FieldRef: &{APIVersion: \"v1\", FieldPath: \"metadata.namespace\"}}}, {Name: \"KB_SA_NAME\", ValueFrom: &{FieldRef: &{APIVersion: \"v1\", FieldPath: \"spec.serviceAccountName\"}}}, ...},\n  \t\t\tResources: core.ResourceRequirements{\n  \t\t\t\tLimits: core.ResourceList{\n- \t\t\t\t\ts\"cpu\":    {i: resource.int64Amount{value: 100, scale: -3}, s: \"100m\", Format: \"DecimalSI\"},\n+ \t\t\t\t\ts\"cpu\":    {i: resource.int64Amount{value: 200, scale: -3}, s: \"200m\", Format: \"DecimalSI\"},\n- \t\t\t\t\ts\"memory\": {i: resource.int64Amount{value: 536870912}, Format: \"BinarySI\"},\n+ \t\t\t\t\ts\"memory\": {\n+ \t\t\t\t\t\ti:      resource.int64Amount{value: 644245094400, scale: -3},\n+ \t\t\t\t\t\ts:      \"644245094400m\",\n+ \t\t\t\t\t\tFormat: \"DecimalSI\",\n+ \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tRequests: core.ResourceList{\n- \t\t\t\t\ts\"cpu\":    {i: resource.int64Amount{value: 100, scale: -3}, s: \"100m\", Format: \"DecimalSI\"},\n+ \t\t\t\t\ts\"cpu\":    {i: resource.int64Amount{value: 200, scale: -3}, s: \"200m\", Format: \"DecimalSI\"},\n- \t\t\t\t\ts\"memory\": {i: resource.int64Amount{value: 536870912}, Format: \"BinarySI\"},\n+ \t\t\t\t\ts\"memory\": {\n+ \t\t\t\t\t\ti:      resource.int64Amount{value: 644245094400, scale: -3},\n+ \t\t\t\t\t\ts:      \"644245094400m\",\n+ \t\t\t\t\t\tFormat: \"DecimalSI\",\n+ \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tClaims: nil,\n  \t\t\t},\n  \t\t\tResizePolicy:  nil,\n  \t\t\tRestartPolicy: nil,\n  \t\t\t... // 13 identical fields\n  \t\t},\n  \t\t{Name: \"vttablet\", Image: \"infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/ape\"..., Command: {\"/scripts/vttablet.sh\"}, Ports: {{Name: \"vttabletport\", ContainerPort: 15100, Protocol: \"TCP\"}, {Name: \"vttabletgrpc\", ContainerPort: 16100, Protocol: \"TCP\"}}, ...},\n  \t\t{Name: \"lorry\", Image: \"docker.io/apecloud/kubeblocks-tools:0.9.0-beta.34\", Command: {\"lorry\", \"--port\", \"3501\", \"--grpcport\", ...}, Ports: {{Name: \"lorry-http-port\", ContainerPort: 3501, Protocol: \"TCP\"}, {Name: \"lorry-grpc-port\", ContainerPort: 50001, Protocol: \"TCP\"}}, ...},\n  \t\t{Name: \"config-manager\", Image: \"docker.io/apecloud/kubeblocks-tools:0.9.0-beta.34\", Command: {\"env\"}, Args: {\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bi\"..., \"/bin/reloader\", \"--log-level\", \"info\", ...}, ...},\n  \t},\n  \tEphemeralContainers: nil,\n  \tRestartPolicy:       \"Always\",\n  \t... // 28 identical fields\n  }\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227

➜ ~ k logs kubeblocks-7d87974fbb-hxhf8 -n kb-system >kb.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)

kb.txt
@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Jun 20, 2024
@ahjing99 ahjing99 added this to the Release 0.9.0 milestone Jun 20, 2024
@free6om free6om mentioned this issue Jun 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

Labels
kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.9.0
Development

Successfully merging a pull request may close this issue.

fix: test InPlacePodVerticalScaling all the time apecloud/kubeblocks
2 participants
@free6om @ahjing99