Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podvm: Understand and reduce podvm permutations #1890

Open
stevenhorsman opened this issue Jul 1, 2024 · 11 comments
Open

podvm: Understand and reduce podvm permutations #1890

stevenhorsman opened this issue Jul 1, 2024 · 11 comments
Assignees
Labels
podvm Related to podvm images

Comments

@stevenhorsman
Copy link
Member

stevenhorsman commented Jul 1, 2024

At the moment we have a matrix of 4 possible options for podvm (mkosi/packer) x (cloud-init/process-user-data). We then multiple this by base OSs too (ubuntu/fedora/rhel) (we will ignore OS version at the moment on the assumption we can sync on that?) and cloud-providers that can support it and it explodes quite a lot and becomes complicated to understand and test
We want to reduce this, so we can minimise differences and duplicated code. One possible plan is:

  1. Identify all the variants of podvm builds we have and track who is using what
  2. Try and switch all podvm builds to use mkosi
  3. Switch the CI to mkosi and then deprecate packer
  4. Later investigate removing cloud-init and getting process-user-data to read the data from there if possible?
@stevenhorsman stevenhorsman added the podvm Related to podvm images label Jul 1, 2024
@stevenhorsman
Copy link
Member Author

stevenhorsman commented Jul 1, 2024

Part 1 - Identify podvm builds

Note: I think we decided a while ago that mkosi didn't work so well with ubuntu, so we wanted to switch to a fedora-like stack and deprecate the Ubuntu based podvm builds upstream?

Base OS Architecture Cloud provider(s) mkosi/packer cloud-init/process-user-data Being used? Being tested Notes
Ubuntu amd64 aws packer cloud-init
Ubuntu amd64 azure packer cloud-init
Ubuntu amd64 docker packer cloud-init ✅ ?
Ubuntu amd64 ibmcloud packer cloud-init
Ubuntu amd64 libvirt packer cloud-init
Ubuntu amd64 powervs packer cloud-init
Ubuntu amd64 vsphere packer cloud-init Deprecate this?
Ubuntu s390x ibmcloud/libvirt packer cloud-init
Fedora amd64 aws mkosi process-user-data>
Fedora amd64 azure mkosi process-user-data>
Fedora amd64 docker mkosi? <cloud-init/process-user-data>
Fedora amd64 ibmcloud mkosi? cloud-init?
Fedora amd64 libvirt mkosi? cloud-init?
Fedora amd64 powervs mkosi/packer <cloud-init/process-user-data>
Fedora amd64 vsphere packer? cloud-init Deprecate this?
Fedora s390x ibmcloud/libvirt mkosi? cloud-init?
RHEL amd64 aws mkosi? <cloud-init/process-user-data> Any upstream testing?
RHEL amd64 azure mkosi? <cloud-init/process-user-data> Any upstream testing?
RHEL amd64 docker mkosi? <cloud-init/process-user-data> Any upstream testing?
RHEL amd64 ibmcloud mkosi? cloud-init? Any upstream testing?
RHEL amd64 libvirt mkosi? cloud-init? Any upstream testing?
RHEL amd64 powervs mkosi/packer <cloud-init/process-user-data> Any upstream testing?
RHEL amd64 vsphere packer cloud-init Deprecate this?
RHEL s390x ibmcloud/libvirt <mkosi/packer> cloud-init? Any upstream testing?

@mkulke
Copy link
Contributor

mkulke commented Jul 1, 2024

did you mean "Any downstream testing?"

maybe we can have a "being tested" column

mkosi_x86_64 should work on both AWS + Azure.

Afaik all the packer images use cloud-init?

@stevenhorsman
Copy link
Member Author

stevenhorsman commented Jul 1, 2024

did you mean "Any downstream testing?"

For RHEL I meant testing of the upstream podvm build, but that testing itself could be manual testing, or testing in a downstream environment (as I'm pretty confident we don't have any upstream automated testing for RHEL). We have some documentation for it though. I hope that helps clarify?

maybe we can have a "being tested" column

Will do

mkosi_x86_64 should work on both AWS + Azure.

These are both using process-user-data I believe and primarly fedora based in the upstream testing?

@mkulke
Copy link
Contributor

mkulke commented Jul 1, 2024

did you mean "Any downstream testing?"

From RHEL I meant testing of the upstream podvm build, but that testing itself could be manual testing, or testing in a downstream environment (as I'm pretty confident we don't have automated testing for RHEL). We have some documentation for it though. I hope that helps clarify?

not quite :) I guess we have either (automated) testing in the project or potentially "downstream" (e.g a vendor product that uses CAA). One could argue that untested images, if they are consumed and tested downstream should also be maintained downstream?

These are both using process-user-data I believe and primarly fedora based in the upstream testing?

yes. I think we can just check for "cloud-init" yes/no. cloud-init will not work on dm-verity protected root-fs's. so we could also just check for dm-verity yes/no?

@mkulke
Copy link
Contributor

mkulke commented Jul 1, 2024

none of the mkosi image is being tested atm, afaict

@mkulke
Copy link
Contributor

mkulke commented Jul 1, 2024

amd64 azure packer image is being tested

@stevenhorsman
Copy link
Member Author

not quite :) I guess we have either (automated) testing in the project or potentially "downstream" (e.g a vendor product that uses CAA). One could argue that untested images, if they are consumed and tested downstream should also be maintained downstream?

So the grey area that I was hinting at was for when pure upstream versions were tested internall. e.g. for ibmcloud, we tested the pure upstream version, but due to lack of publicly available resources those tests were done internally. I agree that if the versions are downstream then the downstream teams are responsible for maintenance (though we want to do our best to not break them, so it's potentially interesting). Sorry, I think I'm mostly overcomplicating an already complicated chart!

@stevenhorsman
Copy link
Member Author

stevenhorsman commented Jul 8, 2024

Part 2 - Identify To-Be of podvm builds

As the attempt to list out our "As-is" set of podvm images hasn't seemed to work, probably as it is too complex, maybe we should focus on out To-Be set and what we are aiming for in Step 2 of this work. My starting list of suggestions, based on quite a lot of ignorance is:

Base OS Architecture Cloud provider(s) mkosi/packer cloud-init/process-user-data Being used? Being tested? Notes
Fedora amd64 aws mkosi process-user-data
Fedora amd64 azure mkosi process-user-data
Fedora amd64 docker mkosi process-user-data
Fedora amd64 ibmcloud mkosi cloud-init Requires UEFI, which isn't available in all IBM Cloud regions
Fedora amd64 libvirt mkosi cloud-init
Fedora amd64 powervs mkosi <cloud-init/process-user-data>
Fedora s390x ibmcloud mkosi cloud-init
Fedora s390x libvirt mkosi cloud-init

It would be good to understand where we are - particularly with libvirt and ibmcloud to understand what gaps we have in just trying out podvm builds.

@wainersm
Copy link
Member

wainersm commented Jul 8, 2024

Hi @stevenhorsman !

What if we have an entry for Ubuntu/amd64/libvirt/packer/cloud-init for a while mkosi equivalent isn't running on CI?

@stevenhorsman
Copy link
Member Author

Hi @stevenhorsman !

What if we have an entry for Ubuntu/amd64/libvirt/packer/cloud-init for a while mkosi equivalent isn't running on CI?

So I'm just trying to track and understand if anyone has already tried these combination above for, the short-term goals. It isn't designed to say that we will remove the Ubuntu builds until we have replacements. Ideally if/when someone confirms that Fedora/amd64/libvirt/mkosi/cloud-init works then, we'd start a parallel builds in our CI to create that version of podvms.

@snir911
Copy link
Contributor

snir911 commented Jul 9, 2024

AFAIK RHEL is supported only by packer builds and for AWS, Azure and libvirt, and there's no upstream testing for RHEL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
podvm Related to podvm images
Projects
Status: 🏗 In progress
Development

No branches or pull requests

4 participants