[Epic] Command to resize the disk

[jmazzitelli]

The initial size of the image is 15+1GB. This might not be enough for all usecases, so we need to find a way to allow the image to be resized on initial start (or when stopped). Note: Likely the command(s) will share functionality.

eg.

# crc start --disk-size 20GB

or

# crc config disk-size +5GB

---

(submitting this issue as per Slack response from Gerard Braad : "at the moment we do not support resizing of the disk image. please file an issue fotr this.")

I tried to install Maistra (Isito), its bookinfo demo, and Kiali, but am hitting disk resource limitations.

CRC should provide a mechanism to define the amount of disk space to assign to the VM image in order to run larger clusters.

[jmazzitelli]

Using qemu-img resize and virt-resize, I was able to make the crc disk image 40G putting the extra on /dev/sda3. Here's what the crc machine image reports:

$ sudo virt-filesystems --long -h --all -a $HOME/.crc/machines/crc/crc 
Name       Type        VFS      Label  MBR  Size  Parent
/dev/sda1  filesystem  unknown  -      -    1.0M  -
/dev/sda2  filesystem  ext4     boot   -    1.0G  -
/dev/sda3  filesystem  xfs      root   -    39G   -
/dev/sda1  partition   -        -      -    1.0M  /dev/sda
/dev/sda2  partition   -        -      -    1.0G  /dev/sda
/dev/sda3  partition   -        -      -    39G   /dev/sda
/dev/sda   device      -        -      -    40G   -

But when I log into the CRC VM, I see this:

[core@crc-jtskh-master-0 ~]$ sudo fdisk -l
Disk /dev/vda: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 17F91114-3EA4-46F3-A756-D8EA6DDBE05B

Device       Start      End  Sectors Size Type
/dev/vda1     2048     4095     2048   1M BIOS boot
/dev/vda2     4096  2101247  2097152   1G Linux filesystem
/dev/vda3  2101248 83883519 81782272  39G Linux filesystem

[core@crc-jtskh-master-0 ~]$ lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    252:0    0  40G  0 disk 
├─vda1 252:1    0   1M  0 part 
├─vda2 252:2    0   1G  0 part /boot
└─vda3 252:3    0  39G  0 part /sysroot

[core@crc-jtskh-master-0 ~]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
devtmpfs         7796036        0   7796036   0% /dev
tmpfs            7826572        0   7826572   0% /dev/shm
tmpfs            7826572     8988   7817584   1% /run
tmpfs            7826572        0   7826572   0% /sys/fs/cgroup
/dev/vda3       15715328 10646412   5068916  68% /sysroot
/dev/vda2         999320   138320    792188  15% /boot
tmpfs            1565312        0   1565312   0% /run/user/1000

Why does vda3 show as 39G (for /sysroot) from the lsblk command, but df shows vda3 (/sysroot) at still only 15G (15715328) which is at 68% used. This is with nothing installed in my cluster yet - this is just after a crc start

I'm going to install Istio and its bookinfo demo into my cluster and see what happens. I suspect /sysroot is going to fill up to 100% usage (or close to it).

[jmazzitelli]

with base Istio installed:

[core@crc-jtskh-master-0 ~]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
devtmpfs         7796036        0   7796036   0% /dev
tmpfs            7826572        0   7826572   0% /dev/shm
tmpfs            7826572    10540   7816032   1% /run
tmpfs            7826572        0   7826572   0% /sys/fs/cgroup
/dev/vda3       15715328 11819388   3895940  76% /sysroot
/dev/vda2         999320   138320    792188  15% /boot
tmpfs            1565312        0   1565312   0% /run/user/1000

There is no DiskPressure yet:

$ oc get node crc-jtskh-master-0  -o yaml | grep DiskPressure
    reason: KubeletHasNoDiskPressure
    type: DiskPressure

Then with its bookinfo demo installed:

[core@crc-jtskh-master-0 ~]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
devtmpfs         7796036        0   7796036   0% /dev
tmpfs            7826572        0   7826572   0% /dev/shm
tmpfs            7826572    11172   7815400   1% /run
tmpfs            7826572        0   7826572   0% /sys/fs/cgroup
/dev/vda3       15715328 13467996   2247332  86% /sysroot
/dev/vda2         999320   138320    792188  15% /boot
tmpfs            1565312        0   1565312   0% /run/user/1000

Now I have disk pressure:

$ oc get node crc-jtskh-master-0  -o yaml | grep DiskPressure
    reason: KubeletHasDiskPressure
    type: DiskPressure

And now I can see it purging files to help get disk usage below the 80% threshold:

[core@crc-jtskh-master-0 ~]$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
...
/dev/vda3       15715328 11928628   3786700  76% /sysroot
...

[jmazzitelli]

So, clearly my problem is that the additional room on the partition isn't making its way into the vda3 filesystem. You can see I enlarged the vda3 partition:

from fdisk -l:

/dev/vda3 2101248 83883519 81782272 39G Linux filesystem

and the filesystem even shows it is 39G from the lsblk command:

└─vda3 252:3 0 39G 0 part /sysroot

but for some reason that additional space is not recognized at runtime. From df:

/dev/vda3 15715328 13467996 2247332 86% /sysroot

I increased my disk space using these commands - if anyone knows what I am missing, please let me know. Unless there is a bug somewhere in the VM config?? I'm at a loss.

$ CRC_MACHINE_IMAGE="$HOME/.crc/machine/crc/crc"
$ crc stop
$ sudo qemu-img resize ${CRC_MACHINE_IMAGE} +24G
$ sudo cp ${CRC_MACHINE_IMAGE} ${CRC_MACHINE_IMAGE}.ORIGINAL
$ sudo virt-resize --expand /dev/sda3 ${CRC_MACHINE_IMAGE}.ORIGINAL ${CRC_MACHINE_IMAGE}

[cfergeau]

We can probably make the size of /dev/vda3 filesystem bigger on the image we ship. The image is thinly provisioned, so this should not have a noticeable impact on the tarball size.

[jmazzitelli]

FWIW: I do not have libguestfs-xfs installed and when I do virt-resize, I am getting this warning:

virt-resize: warning: unknown/unavailable method for expanding the xfs
filesystem on /dev/sda3

It is possible that this is why my filesystem is not getting the increased space. I will install that and test and see what happens.

[jmazzitelli]

We can probably make the size of /dev/vda3 filesystem bigger on the image we ship. The image is thinly provisioned, so this should not have a noticeable impact on the tarball size.

It would be preferable if crc provided an option to configure the partition sizes (particularly the vda3 partition) and have crc run the necessary cmds to do this resizing. Because whatever size you ship with, someone is going to want more :)

[jmazzitelli]

I installed libguestfs-xfs and that got me alittle further. I see it now trying to grow the filesystem during the virt-resize, but it gets this error:

virt-resize: error: libguestfs error: mount: mount exited with status 32: 
mount: wrong fs type, bad option, bad superblock on /dev/sda3,
       missing codepage or helper program, or other error

I am on RHEL 7.6. The CRC VM is Red Hat Enterprise Linux CoreOS release 4.1
however, it looks like it might be this:

https://bugzilla.redhat.com/show_bug.cgi?id=1671235

[cfergeau]

CoreOS 4.1 is based on RHEL8, so it's indeed going to hit the limitation mentioned in these bugs [ref]

With the release of RHEL8, both the XFS and ext4 filesystems have gained new
features and capabilities. XFS now has reflink capability for file-level
copy-on-write, as well as (optionally) a reverse mapping btree to map disk blocks
back to files for future enhanced repair capabilities. ext4 has been enhanced to
include metadata checksums for robust detection of on-disk corruption.

Each of these new features is read-only compatible with RHEL7, meaning that a RHEL7
kernel is able to mount a RHEL8 filesystem which contains these features in readonly
mode, but read/write mode is not possible.

As a result, certain virtual image manipulation tools such as guestfish, virt-customize,
or any other utility based in libguestfs will be unable to manipulate RHEL8 images
on a RHEL7 system if the filesystems within the image contain these new features.

While there is no workaround for this limitation, it is possible to disable the
new features at mkfs time, if the features are not required or desired. For XFS,
to make a RHEL7-compatible filesystem, use the "-m reflink=0" option at mkfs time.
For ext4, use "-O ^metadata_csum" at mkfs time.

[jmazzitelli]

CoreOS 4.1 is based on RHEL8, so it's indeed going to hit the limitation mentioned in these bugs [ref]

OK, then based on that... this is a high priority:

We can probably make the size of /dev/vda3 filesystem bigger on the image we ship. The image is thinly provisioned, so this should not have a noticeable impact on the tarball size.

Its high priority at least for anyone needing more than 15G in their cluster and they are using a Red Hat CSB machine or anyone using RHEL 7.x - they can't resize the image and thus hosed and unable to use CRC.

[jmazzitelli]

if the features are not required or desired. For XFS,
to make a RHEL7-compatible filesystem, use the "-m reflink=0" option at mkfs time.
For ext4, use "-O ^metadata_csum" at mkfs time.

Or the CRC team can see if those XFS features are really needed (I have no idea what these new featuers are) ... if not, the crc image's filesystems should be created with those options so they are compatible with RHEL 7.x systems.

[gbraad]

and have crc run the necessary cmds to do this resizing. Because whatever size you ship with, someone is going to want more :)

This is planned for later releases. We will first focus on the support of other hypervisors before venturing into supporting 'advanced' usecases. With advanced it is meant to say 'configuration changes' that are needed against the OOTB experience. I'll try to make this visible from the project kanbans at https://github.com/code-ready/crc/projects

[gbraad]

Original issue has been modified to mention possible resizing commands