Failing to extend the partition size

[osibanda]

Platform I'm building on:
We are running EKS with Kubernetes version 1.21.6 and platform version eks.4.

Problem statement
The issue is disk pressure warnings occur intermittently on a particular node in the cluster. We extended the volumes attached to that node and wish to extend the partition sizes. Running df -f returns the attached output df-output.xlsx.

What I expected to happen:
When I run lsblk -f I should get the filesystem type and the breakdown of the partitions. Thus we are not able to see the filesystem type. However, running xfs_growfs -d returns /dev/nvme0n1 is not a mounted XFS filesystem and resize2fs returns open: No such file or directory while opening directory /dev/nvme0n1. I have tried using the following:

• /dev/xvda (Volume name in AWS console)
• /dev/nvme0n1 (result from lsblk)
• /dev/nvme0n1p1
• As well as the mount directory

All to no avail.

What actually happened:
lsblk -f returns blank filesystem column as shown below:

NAME            FSTYPE LABEL UUID MOUNTPOINT
loop0                                     /.bottlerocket/rootfs/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
loop1                                     /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
nvme0n1
|-nvme0n1p1
|-nvme0n1p2
|-nvme0n1p3                         /.bottlerocket/rootfs/boot
|-nvme0n1p4
|-nvme0n1p5
|-nvme0n1p6
|-nvme0n1p7
|-nvme0n1p8
|-nvme0n1p9
|-nvme0n1p10
|-nvme0n1p11
`-nvme0n1p12                        /.bottlerocket/rootfs/var/lib/bottlerocket
nvme1n1
`-nvme1n1p1                         /.bottlerocket/rootfs/local
nvme2n1

lsblk returns:

NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0          7:0    0  280K  1 loop /.bottlerocket/rootfs/x86_64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
loop1          7:1    0 11.6M  1 loop /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
nvme0n1      259:0    0  160G  0 disk
|-nvme0n1p1  259:3    0    4M  0 part
|-nvme0n1p2  259:4    0    5M  0 part
|-nvme0n1p3  259:5    0   40M  0 part /.bottlerocket/rootfs/boot
|-nvme0n1p4  259:6    0  920M  0 part
|-nvme0n1p5  259:7    0   10M  0 part
|-nvme0n1p6  259:8    0   25M  0 part
|-nvme0n1p7  259:9    0    5M  0 part
|-nvme0n1p8  259:10   0   40M  0 part
|-nvme0n1p9  259:11   0  920M  0 part
|-nvme0n1p10 259:12   0   10M  0 part
|-nvme0n1p11 259:13   0   25M  0 part
`-nvme0n1p12 259:14   0   42M  0 part /.bottlerocket/rootfs/var/lib/bottlerocket
nvme1n1      259:1    0   50G  0 disk
`-nvme1n1p1  259:2    0   20G  0 part /.bottlerocket/rootfs/local
nvme2n1      259:15   0   20G  0 disk

How to reproduce the problem:
Log in to an EC2 instance which is a cluster node running Bottlerocket and switch to admin bash. Increase the partition size of any mounted volume.

[webern]

I'm sorry to hear this, let me get a bit more information and try to reproduce this issue:

• Region
• Your AMI ID and/or Bottlerocket version? (Are you using a custom-built AMI or one of our published AMIs?)
• Instance type
• Are you using any custom CSI driver?

on a particular node in the cluster

Do you mean that the problem seems to be isolated to a particular EC2 instance ID? Or do you mean that the problem is happening randomly at some frequency rate when spinning up new instances?

[webern]

I initially misunderstood the question. I am now trying to extend the data partition of a Bottlerocket instance and will let you know what I come up with.

[webern]

Ok, I was able to grow the data partition using the following procedure. I believe you may have had trouble because you had not used sheltie to enter the host namespaces (see instructions below).

Increase the disk size in the AWS console

• In the AWS console, identify the EBS volume labeled /dev/xvdb. If you ran Bottlerocket with default configurations, this should be 20G. (Note that we do not need to grow the 2GB OS partition, /dev/xvda.)
• Increase the size of the data volume in the EBS console.
• Wait and verify that the size increase has taken place (for example in the EC2 con

Increase the Partition size.

We can find the commands that Bottlerocket uses to grow the data partition here https://github.com/bottlerocket-os/bottlerocket/blob/d8698d9abef537a3dde4a39eff8a894a52e2a73d/packages/release/repart-local.service#L16..L20

Note that this service runs on boot, so you could simply reboot the instance. Also note that it is critical that you run exactly these commands, otherwise on reboot the partition could shrink and then the filesystem would be out of bounds and not mountable.

If you want to grow the partition without rebooting:

• Connect to the Bottlerocket instance using SSM.

  • aws ssm start-session --target [instance-id]
  • run enable-admin-container and wait a few seconds. Wait a bit for the admin container to be downloaded and start.
  • run apiclient exec admin bash to "enter" the admin container.
  • IMPORTANT Once in the admin container run sudo sheltie to enter the host namespaces.

• If we lsblk -l we should see nvme1n1 shows the increased space. Meanwhile the partition nvme1n1p1 needs to have its size increased.

• Run the commands that we found in repart-local.service

  • /usr/bin/systemd-repart --dry-run=no /dev/disk/by-partlabel/BOTTLEROCKET-DATA
  • /usr/lib/systemd/systemd-growfs /local

• Check lsblk -l and you should see the partition size has increased.

Alternatively, from the control container, you can run these commands through apiclient exec and sheltie non-interactively:

• apiclient exec admin sudo sheltie /usr/bin/systemd-repart —dry-run=no /dev/disk/by-partlabel/BOTTLEROCKET-DATA
• apiclient exec admin sudo sheltie /usr/lib/systemd/systemd-growfs /local

[webern]

@OscarSGit let us know if you need any additional help with this.

For the record, we're leaving this issue open because we want to add something to the documentation about this.

[osibanda]

@webern thank you for the suggestion, let me try it out and will update once I have feedback.