Add reserved-volume-attachments #1919
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
k8s-ci-robot
added
kind/bug
k8s-ci-robot
added
the
size/XL
Code Coverage Diff
|
gnufied
reviewed
Feb 2, 2024
| } | ||
|
|
||
| func (o *NodeOptions) AddFlags(fs *flag.FlagSet) { | ||
| fs.Int64Var(&o.VolumeAttachLimit, "volume-attach-limit", -1, "Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type.") | ||
| fs.Int64Var(&o.VolumeAttachLimit, "volume-attach-limit", -1, "Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes and overrides --reserved-volume-attachments. If not specified, the value is approximated from the instance type.") | ||
| fs.IntVar(&o.ReservedVolumeAttachments, "reserved-volume-attachments", -1, "Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. The total amount of volume attachments for a node is computed as: <nr. of attachments for corresponding instance type> - <number of NICs, if relevant to the instance type> - <reserved-volume-attachments value>. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.") |
There was a problem hiding this comment.
So you can't specify both. Was there any reason for doing that, other than perhaps both of them being redundant? I mean we could still allow specifying both - in which case limit is determined by volumeAttachLimit - reservedVolumeAttachment ?
There was a problem hiding this comment.
I think we should consider throwing an error if both are specified - there's no reason to ever use both so the presence of both options strongly indicates an invalid config and/or confused user.
|
Thank you @gnufied and @jsafrane for taking action on this. Apologies for the communication delay, but as of this morning, EBS folks are aligned with this PR's approach and want to merge this short-term fix ahead of v1.28.0. CC @torredil & @ConnorJC3. I will personally manually sanity-test this PR by Sunday, but from diff alone it looks great to me. Especially the clarity of documentation in node_options. Thank you again. Edit: Sanity checked with manual testing. Looks good. |
ConnorJC3
reviewed
Feb 5, 2024
There was a problem hiding this comment.
I think this is the superior approach to #1917 - the configuration is clear and more flexible for usecases like gardener (which now can pass in the exact number of extra volumes they intend to attach instead of relying on our buggy heuristic), and leaves open the option of changing the default behavior in a future release.
/lgtm
| } | ||
|
|
||
| func (o *NodeOptions) AddFlags(fs *flag.FlagSet) { | ||
| fs.Int64Var(&o.VolumeAttachLimit, "volume-attach-limit", -1, "Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes. If not specified, the value is approximated from the instance type.") | ||
| fs.Int64Var(&o.VolumeAttachLimit, "volume-attach-limit", -1, "Value for the maximum number of volumes attachable per node. If specified, the limit applies to all nodes and overrides --reserved-volume-attachments. If not specified, the value is approximated from the instance type.") | ||
| fs.IntVar(&o.ReservedVolumeAttachments, "reserved-volume-attachments", -1, "Number of volume attachments reserved for system use. Not used when --volume-attach-limit is specified. The total amount of volume attachments for a node is computed as: <nr. of attachments for corresponding instance type> - <number of NICs, if relevant to the instance type> - <reserved-volume-attachments value>. When -1, the amount of reserved attachments is loaded from instance metadata that captured state at node boot and may include not only system disks but also CSI volumes.") |
There was a problem hiding this comment.
I think we should consider throwing an error if both are specified - there's no reason to ever use both so the presence of both options strongly indicates an invalid config and/or confused user.
k8s-ci-robot
added
the
lgtm
AndrewSirenko
approved these changes
Feb 5, 2024
torredil
approved these changes
Feb 5, 2024
Exit with an error when both --volume-attach-limit and --reserved-volume-attachments are specified on the command line.
k8s-ci-robot
removed
the
lgtm
|
I added a commit that allows only one of --volume-attach-limit and --reserved-volume-attachments. |
|
/lgtm |
k8s-ci-robot
added
the
lgtm
AndrewSirenko
approved these changes
Feb 9, 2024
|
Thank you @jsafrane |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: torredil The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
This was referenced Feb 20, 2024
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
Mar 12, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
Mar 12, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
Apr 26, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
Apr 29, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
Apr 29, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.
jsafrane
added a commit
to jsafrane/aws-ebs-csi-driver
that referenced
this pull request
May 7, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume. This cherry-picks also kubernetes-sigs@f66f30a for a very related bugfix.
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/aws-ebs-csi-driver
that referenced
this pull request
May 9, 2024
Pick both commits in kubernetes-sigs#1919 to OCP. Introduce --reserved-volume-attachments cmd line option that disables loading list of volume attachments from instance metadata. Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong. --reserved-volume-attachments simply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume. This cherry-picks also kubernetes-sigs@f66f30a for a very related bugfix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an alternative approach to #1917
Is this a bug fix or adding new feature?
/kind bug
What is this PR about? / Why do we need it?
Introduce
--reserved-volume-attachmentscmd line option that disables loading list of volume attachments from instance metadata.Instance metadata has volume attachments that were present on instance boot, which may be different than the current nr. of attachments. In addition, it may include CSI volumes that were not drained before reboot, and thus the value may be completely wrong.
--reserved-volume-attachmentssimply reserves given nr. of attachments for any system disks. Typically it should be 1 for the root volume.#1917 introduces just boolean value that would reserve 1 attachment for the root disk, this PR allows users to set nr. of such attachments.
Note that the default value is
-1, i.e. the current heuristics. Existing CSI driver deployments won't see any change, until they explicitly specify--reserved-volume-attachments=X.What testing is done?
Manually, it indeed ignored multiple attached volumes and reported 26 available attachments with
--reserved-volume-attachments=1and 17 with--reserved-volume-attachments=10(both onm6i.xlarge).