Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Advanced partitioning customizations (COMPOSER-2355) #926

Open
wants to merge 37 commits into
base: main
Choose a base branch
from

Commits on Sep 19, 2024

  1. blueprint: partitioning customization

    - New structure that's more powerful than the existing
      FilesystemCustomization.
    - The general structure starts at 3 top-level objects:
        - Plain: customizations for plain disk partitioning, meaning
          partitions with traditional filesystem payloads (xfs, ext4, etc).
        - LVM: customizations for creating LVM layouts.  Supports one or
          more VolumeGroups.  Each VolumeGroup implies the creation of a
          partition with the appropriate type and payload.
        - Btrfs: customizations for creating btrfs layouts.  Supports one or
          more volumes and subvolumes for each volume.  Each btrfs volume
          implies the creation of a partition with the appropriate type and
          payload.
    - Partitioning.Filesystems is an array of the old
      FilesystemCustomization struct.
    - The new FilesystemCustomization struct supports a label and Type.
    - The LVCustomization (logical volume) embeds the
      FilesystemCustomization and adds a name for the LV.
    
    Unmarshal tests have been updated with new fields.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f39037a View commit details
    Browse the repository at this point in the history
  2. disk: NewCustomPartitionTable() function

    New function that creates a partition table from scratch based on the
    new partitioning customizations.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    805ece4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    68fb75a View commit details
    Browse the repository at this point in the history
  4. disk: more testing for NewCustomPartitionTable()

    Add one test for each general layout: plain, lvm, and btrfs.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    548e871 View commit details
    Browse the repository at this point in the history
  5. disk: move handling of the /boot partition

    Move the handling of the /boot partition before the plain partition
    handling so we don't have to find where to insert it (after BIOS boot
    and ESP, which are going to be added in a later commit, but before
    plain partitions).  Instead, we build the partition list sequentially as
    needed.
    
    Update tests.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    ae2d48b View commit details
    Browse the repository at this point in the history
  6. pkg/{distro,platform}: move BootMode to platform

    Move the BootMode enum from pkg/distro to pkg/platform.  We want to
    import BootMode to the disk package, which would create an import loop
    with distro.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    e0ba877 View commit details
    Browse the repository at this point in the history
  7. disk: add bios and esp partitions based on boot mode

    Add BIOS boot and ESP partitions based on boot mode.  The boot mode is
    passed as an argument to the NewCustomPartitionTable() function.
    
    Update tests.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    8b5653d View commit details
    Browse the repository at this point in the history
  8. disk: add root partition / lv / subvol if missing

    At the end of the partition table's creation, add a root partition if
    one wasn't created during the processing of the customizations.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    80ce76d View commit details
    Browse the repository at this point in the history
  9. disk: create root partition for empty customizations

    When the customizations argument is nil, initialise it with an empty
    struct and continue processing the function normally so that the root
    partition is created as well.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    766911f View commit details
    Browse the repository at this point in the history
  10. blueprint: include minsize in partitioning customization

    Add minsize to the top level of the partitioning customizations.
    In osbuild-composer, this value comes from a command line option through
    distro.ImageOptions.  The plan is to have that be a blueprint
    customization that osbuild-composer will resolve when it's defined in
    multiple places and pass it to images via the customization.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    7e9ca23 View commit details
    Browse the repository at this point in the history
  11. disk: add defaultType to NewCustomPartitionTable()

    Add a new argument, defaultType, to the NewCustomPartitionTable()
    function.  The purpose of the type is to determine which filesystem will
    be used for root and boot when they are added automatically.
    
    It will also be used to set the filesystem for any mountpoint that has
    no type defined.
    
    The default type should be determined by the distro definition.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    f88e6e2 View commit details
    Browse the repository at this point in the history
  12. disk: use defaultType when missing fs type customization

    When a custom partition or LogicalVolume does not specify a type, use
    the defaultType.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    397efb0 View commit details
    Browse the repository at this point in the history
  13. disk: define FSType enum for all supported types

    Add a new enum, FSType, for referring to all supported filesystem types.
    Use it as the default type argument to the NewCustomPartitionTable()
    function.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    9b75803 View commit details
    Browse the repository at this point in the history
  14. disk: relayout the partition table in NewCustomPartitionTable()

    Run a relayout() on the partition table before returning it from
    NewCustomPartitionTable() to set the start position and size for each
    partition as well as the total partition table size.
    
    Update tests.
    
    Partition sizes in tests changed to MiB (from plain bytes) so that the
    values make more sense when headers are added and values are rounded to
    the grain size (1 MiB).
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    4a689db View commit details
    Browse the repository at this point in the history
  15. disk: CustomPartitionTableOptions struct

    Move the options for the NewCustomPartitionTable() function to a struct.
    More will be added later and handling them all as one argument will make
    things simpler.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    51c44a5 View commit details
    Browse the repository at this point in the history
  16. disk: add PartitionTableType to CustomPartitionTableOptions

    Add an option for the partition table type to the
    CustomPartitionTableOptions.  The value must be one of "gpt", "dos", or
    "".  An empty string (or nil options) defaults to "gpt".
    
    The tests have been updated to explicitly specify "dos".  The relayout()
    method, which is affected by this, checks whether `pt.Type == "gpt"`,
    which means it was defaulting to "dos" when deciding to add a footer.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    986b64c View commit details
    Browse the repository at this point in the history
  17. disk: add "gpt", "dos", and "" types to tests

    Add examples of each valid value for the PartitionTableType (including
    unset) to the tests.
    
    More tests will be added later to make sure all combinations are
    covered.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    d7b9dd4 View commit details
    Browse the repository at this point in the history
  18. disk: generate UUIDs in NewCustomPartitionTable()

    Generate all (missing) UUIDs at the end of NewCustomPartitionTable()
    before returning the final PT.
    
    Update tests:
    - Move the random number generator initialisation inside the test loop
      so that each test uses its own generator.  This makes tests
      independent of the order they're run in.
    - Add generated UUIDs to the expected result.
    achilleas-k committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    24b83ca View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. disk: add RequiredSizes to CustomPartitionTableOptions

    Introduce the concept of required sizes from the existing
    NewPartitionTable() function into the NewCustomPartitionTable().
    
    RequiredSizes is necessary for setting a "sane default" (minimum) size
    for certain directories.  Unlike the old partition table creation
    function, there is no default value for the custom partition table
    generator.  Instead, image definitions will need to set their own
    defaults.
    
    Without this mechanism, automatically created root filesystems can end
    up being too small to fit the base OS packages or to have a usable
    system in the long run.  It's also needed when automatically creating
    root partitions or LVM Logical Volumes.  Without a required minimum, the
    root partition will only be the size of the leftover space on the image
    (the base image size minus the sum of the rest of the partitions, which
    can sometimes end up being 0) or LVM VolumeGroup.
    
    Add some required sizes to tests.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d844a8a View commit details
    Browse the repository at this point in the history
  2. blueprint: remove GetFilesystemsMinSize()

    This is leftover from the repo split.  It is implemented and used in
    osbuild-composer.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    4e7a7ee View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2a0bc1b View commit details
    Browse the repository at this point in the history
  4. blueprint: add CheckPartitioningPolicy()

    This is the equivalent to the CheckMountpointsPolicy() function for the
    new PartitioningCustomization.  It collects all the mountpoints from
    Plain, LVM, and Btrfs then compares them to the policy.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9a5f8dd View commit details
    Browse the repository at this point in the history
  5. distro/fedora: pass all customizations to getPartitionTable()

    Preparing the function to use the new PartitioningCustomization when
    set.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    92f089b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5da5eae View commit details
    Browse the repository at this point in the history
  7. distro/fedora: add minimum directory sizes to image types

    For all image types that don't specify a requiredPartitionSizes, add the
    defaults that are used in the NewPartitionTable() so that
    NewCustomPartitionTable() has similar behaviour.
    
    The only image type that explicitly doesn't specify minimum sizes is the
    iot-raw-image, which has a pre-existing empty map.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    bcfb63a View commit details
    Browse the repository at this point in the history
  8. distro/fedora: check partitioning customizations

    Add checks for Partitioning customizations in Fedora's checkOptions()
    function.  The function now returns an error if:
    - Partitioning is defined at the same time as Filesystem
    - Partitioning is defined for ostree image types (commit and container).
    - The mountpoints in Partitioning violate the mountpoint policies for
      the image type.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d50bf94 View commit details
    Browse the repository at this point in the history
  9. blueprint: validate partitioning customizations

    Add a Validate() method to PartitioningCustomization that returns an
    error if:
    1. Btrfs and LVM are both defined.
    2. Any mountpoint is invalid (empty string, relative path, or not
       "clean").
    3. Any mountpoint is defined more than once in the whole structure.
    4. There is more than one LVM Volume Group or more than one btrfs
       volume.
    5. Any LVM Logical Volume name is used more than once in the same
       VolumeGroup.
    6. Any Btrfs subvolume has an empty name.
    7. Any Btrfs subvolume name is used more than once in the same Btrfs
       volume.
    
    This function validates the customization itself in the absence of any
    distro or image-type specific policies.
    
    We may relax rule 3 in the future, at least in images if not in
    osbuild-composer, but for now, let's disallow it while keeping the
    customization structure flexible.
    
    val
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    84d94e0 View commit details
    Browse the repository at this point in the history
  10. disk: refactor LVM Logical Volume name generation

    The VolumeGroup.CreateLogicalVolume() function assumed that the given
    name was a mountpoint and would automatically escape it, append 'lv',
    and append numbers in case of collisions.  This made it impossible to
    use the helper function for LV creation with desired names.
    
    Refactor the function and modify all the calls so that now it uses the
    provided name without validating, escaping, or checking for collisions,
    and generates a name from the payload's mountpoint when one is not
    provided.
    
    The CreateMountpoint() function's functionality is not affected.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    d9fe555 View commit details
    Browse the repository at this point in the history
  11. disk: use vg.CreateLogicalVolume() in NewCustomPartitionTable()

    Use the CreateLogicalVolume() helper when creating LVs for the custom
    partition table.  This changes size calculations in tests because each
    LV is resized to be an integer number of extents, i.e. sizes are rounded
    up to 4 MiB multiples.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    541f95f View commit details
    Browse the repository at this point in the history
  12. disk: generate volume group names when unspecified

    When a volume group's name is not specified in the customizations, the
    custom partition table creator assigned the names vg00, vg01, etc.
    
    A new test is added with multiple volume groups to test this.  The
    scenario of the test is not supported by customizations, the blueprint
    package does not allow multiple volume groups, but the function can
    handle it so it's good to have it covered.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    c7c3486 View commit details
    Browse the repository at this point in the history
  13. test: add partitioning build configs

    One for each variant:
    - Plain
    - Plain + btrfs
    - Plain + lvm
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    837ce9b View commit details
    Browse the repository at this point in the history
  14. blueprint: add extra validation for mountpoints and fstypes

    Added extra validation for:
    - invalid filesystems for specific mountpoints (e.g. xfs on /boot/efi)
    - /boot or /boot/efi on lvm or btrfs
    - btrfs as a filesystem on plain
    
    Added tests to cover all new validation errors.
    Added an extra happy test for /boot and /boot/efi with automatic fs type
    selection (unset, empty strings).
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    9451486 View commit details
    Browse the repository at this point in the history
  15. disk: private resolver for CustomPartitionTableOptions.DefaultFSType

    A small function for resolving the filesystem type for a mountpoint with
    the global default.  Will return an error if both are unset.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    01b0ab7 View commit details
    Browse the repository at this point in the history
  16. disk: split NewCustomPartitionTable() into smaller functions

    Split the large NewCustomPartitionTable() function into smaller ones for
    readability and nicer handling of error conditions (no need to beak on a
    loop label).
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    5bf3416 View commit details
    Browse the repository at this point in the history
  17. disk: more custom partition table tests

    - Legacy boot mode.
    - UEFI boot mode.
    - Auto root creation on btrfs.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    f6fd14a View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    0266934 View commit details
    Browse the repository at this point in the history
  19. disk: reword invalid pt type error message

    Change the error message for invalid partition table types to match the
    other error messages from the NewCustomPartitionTable() function.
    
    Add a test.
    achilleas-k committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    6b1f037 View commit details
    Browse the repository at this point in the history