Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate dockerhost-azure system to lower cost disks #3796

Open
sxa opened this issue Nov 4, 2024 · 0 comments
Open

Migrate dockerhost-azure system to lower cost disks #3796

sxa opened this issue Nov 4, 2024 · 0 comments

Comments

@sxa
Copy link
Member

sxa commented Nov 4, 2024

Currently dockerhost-azure-ubuntu2204-x64-1-intel is configured with three premium SSD LRS disks:
512GiB sdb for /var/lib/docker (Only 256GiB allocated, plus 16GiB swap partition, plus 60GiB Solaris partition)
60GiB sdd for Solaris (Defined as LVM)
256GiB sdc (Defined as LVM) with ~120GiB allocated to /home/jenkins.

This means we have half of sdc unused (/home/jenkins currently has 21GiB in use, 92GiB free) and a lot of the 512GiB disk also unused.

Suggest we replace the above as follows with disk type "Standard HDD" which is about a third of the cost:

  • 256GiB for /var/lib/docker
  • 256GiB 16GiB swap partition, then LVM split between Solaris (Two VMs currently using about half of the 50GiB FS) and /home/jenkins (alternate option: allocate this as a single non-LVM /home)
  • We could potentially replace the swap partition with a /swapfile - it's unlikely to be heavily used sine the machine has 64GiB of RAM.

Goal here should be:

  1. Create new disks
  2. Shutdown docker process and solaris VMs
  3. Copy all data from old disk to new disks
  4. Unmount everything
  5. Switch /etc/fstab to point to new disks
  6. Remount everything (or maybe reboot?)
  7. Start everything up again and make sure jenkins jobs work (particularly the Solaris build pipelines)
  8. Remove the old disks (maybe after a weekly EA build cycle.

We should also look at reducing the size of the second one if the above is successful, particularly if we can select an AMD system to complement the Intel one - a d8as_v4 with 8vCPU/32GiB RAM should be adequate looking at current usage of the second machine which doesn't have any Solaris VMs on it at present and a similar disk layout to the above (potentially with 128GiB instead of 256 for the second disk as it does not host Solaris images). This would likely reduce the cost of this machine by over 50%.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

1 participant