Possible evolution of the param feature

[jandubois]

Lima Param Evolution

I think the new param mechanism by @norio-nomura (#2498) is a great way to make our templates much more configurable.

I've been brainstorming on how to extend this basic mechanism. While I think some of the ideas below are good, I reserve the right to change my mind once I think some more about them. 😄

Additional attributes

I would like to attach additional attributes to parameters, so the current format should be as short-form:

param:
  KEY: VALUE

is going to be the same as:

param:
  KEY:
    default: VALUE

Validation

We are (trying to) validate all settings in lima.yaml as much as possible before we actually create an instance based on it. We should provide a mechanism to do the same for param values.

param:
  ROOTFUL:
    default: false
    validate: '^(true|false)$'

When you specify any other value, you will now get an error:

ERRO[0001] param.ROOTFUL must match regular expression `^(true|false)$`

Validation can be a list of choices

Instead of a regex you should be able to enumerate the valid choices. That is easier to read:

param:
  FRUIT:
    default: cherry
    validate: [apple, banana, cherry]

And allows us to generate better error messages:

ERRO[0003] param.FRUIT must be one of "apple", "banana", or "cherry"

Custom error messages

For more complex settings we may want to specify the error message outself:

param:
  K3S_VERSION:
    default: 1.26.8
    validate: '^(1\.[234]\d\.\d+|stable|latest)$'
    error: 'param.K3S_VERSION must be either a k3s semver value, "stable", or "latest"'

Default validation for numbers and boolean

As a convenience we can auto-generate validation and error message if the default is a boolean (true/false, or yes/no), or a number. For example:

param:
  ROOTFUL: false
  LOG_LEVEL: 2

would be expanded to

param:
  ROOTFUL:
    default: false
    validate: [true, false]
    error: 'param.ROOTFUL must be either "true" or "false"'
  LOG_LEVEL:
    default: 2
    validate: '^\d+$'
    error: 'param.LOG_LEVEL must be an integer'

Prompt for params in TTY mode

When we are creating a new instance while in --tty mode, we may want to ask for the value of some of the params.

param:
  ROOTFUL:
    default: no
    prompt: "Do you want to run docker in rootful mode?"

The validate setting will be auto-generate to [yes, no] because of the default value, and the error is then auto-generated from that enum.

Creating a new instance will prompt you:

$ limactl start template://docker
? Do you want to run docker in rootful mode?  (y/N) |

The optional prompting is why I proposed yes/no as alternate boolen values because they work better when asking a question interactively.

Adding if condition on list entries

The override.yaml and defaults.yaml mechanism has limited utility for lists because it has limited capabilities for overwriting (some entries can merge attributes when the entries have a unique key).

With the new param mechanism we could add if conditions to all lists (images, mounts, networks, portForwardings, provision scripts, probes, etc).

I could put this snippet into my override.yaml file to optionally mount an additional volume without having to manually edit the lima.yaml file:

mounts:
- location: /Volumes/USB
  if: .Param.MOUNT_USB
param:
  MOUNT_USB: false

When I need access to the USB volume I create the instance like this:

limactl start --set '.params.MOUNT_USB=true' template://docker

The value of the if attribute is a Go template expression. The entry will be ignored if the expression evaluates to the empty string, 0, false, or no.

Don't error if param from override.yaml or defaults.yaml is unused

This already works the way I want; I just want to confirm this is intentional, and not accidental: we only check that params declared in the template itself are used by the template. We don't check params declared in either override.yaml or defaults.yaml.

E.g. I may want to use rootful docker when I run limactl start template://docker, so would put this in override.yaml:

param:
  ROOTFUL: true

I don't want to see a warning about param.ROOTFUL being unused when I create a default instance with limactl start.

URL params on templates

We could add support for query parameters to template URLs:

limactl start --name rootful "template://docker?ROOTFUL=true"

It feels a bit weird though to put them on regular URLs:

limactl start "https://raw.githubusercontent.com/lima-vm/lima/master/examples/docker.yaml?ROOTFUL=true"

And even worse on plain filenames: docker.yaml?ROOTFUL=true.

It is also redundant with --set:

limactl start --name rootful --set .param.ROOTFUL=true template://docker

But then e.g. --rosetta is also redundant with --set .rosetta.enabled=true, so I don't know.

We could also add a --param option:

limactl start --name rootful --param ROOTFUL=true template://docker

Or (assuming we use Go parsing for boolean param):

limactl start --param MOUNT_USB template://docker

[jandubois]

I wrote the braindump above to show where I think we should go over time with the param feature.

However, it leaves us with the dilema of 2 conflicting goals:

1. Make our templates as short and simple as possible, so they are easy to understand and modify.
2. Make our templates so configurable that most users will never need to modify them.

Having too many templates that are mostly identical and just have minor configuration differences are eventually becoming confusing too (and harder to maintain with all the duplication).

I don't think there is a hard and fast rule, but I think if 2 templates share a significant amount of code/settings, then they may be candidates for combination.

For example we might have just a single template://docker, and the template://docker-rootful is replaced by setting a param on the main template.

This is all very new, and the docker templates are the first ones to make use of this mechanism. So please let's take some time and experiment with refactoring and simplifying things instead of immediately saying "this looks too complicated".

It may end up that having 2 templates remains the better choice, but let's give it some time to try different approaches.

[afbjorklund]

Previously we were using the approach of doing cutting/pasting and manual editing, in order to tweak the templates to what was needed (like changing parameters). And using something like "jinja2" or whatever, to generate templates with configuration. This keeps the implementation simple, but there is a lot of duplication (like the current images) and harder to upgrade and maintain.

This is all very new, and the docker templates are the first ones to make use of this mechanism. So please let's take some time and experiment with refactoring and simplifying things instead of immediately saying "this looks too complicated".

My fear was that the previously simple "docker" template (e.g. get.docker.com) is now turning into something horrible - like the "k8s" template which has way too many steps compared to the "k3s" template (mostly because upstream likes it that way). I also don't want the templates to turn into yet another yaml-language like Ansible, I liked the simple shell scripts (it's more like make)

But sure, will give it more looks. Just that it went straight into implementation mode, without any issue or discussion?

[norio-nomura]

Don't error if param from override.yaml or defaults.yaml is unused

This already works the way I want; I just want to confirm this is intentional, and not accidental: we only check that params declared in the template itself are used by the template. We don't check params declared in either override.yaml or defaults.yaml.

The --set command targets the instance's lima.yaml, so to avoid unnoticed errors from passing unused parameters (e.g., typos), a check for unused parameters is included. For _config/default.yaml and _config/override.yaml, I considered logging messages indicating that the parameters defined there were used, but decided not to implement this.
So, it's more accurate to say that there was no motivation to include checks for them, rather than intentionally leaving them unchecked.

[jandubois]

So, it's more accurate to say that there was no motivation to include checks for them, rather than intentionally leaving them unchecked.

Sorry, I worded this confusingly. I suspected that it might have been accidental, and I want to go on the record that going forward we should treat this as intentional.

That's why I mentioned it, even though it already works the way I want it to.

[afbjorklund]

Parameters (and using "templates") is one way, there could be other ways to simplify the provisioning.

images:
- ubuntu-24.04
mounts:
- default
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false

### ⬇️ talking about this part below

provision:
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log" in the guest

### ⬆️ talking about this part above

hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use lima-{{.Name}}
  docker run hello-world
  ------

For instance having more types of provisioners and probes, than always using script (the fallback).

It's not the same discussion, but something to keep in mind? e.g. "rootful" could be a top-level concept*.

* like it is in podman machine

EDIT: Added some whitespace

[afbjorklund]

One approach is to keep the scripts in separate files:

• Support script files in addition to inline scripts for provisioners. #2441

[afbjorklund]

Another is of course Ansible playbooks, usually overkill.

[jandubois]

One approach is to keep the scripts in separate files

I would like to keep scripts in separate files, even to just use shellcheck and shfmt on them, but I think it opens a can of worms:

• How do separate script files work with template: URLs? With https: URLs?
• How do you distribute templates? As a tarball, to include the scripts?
• When you create an instance, you now have to copy not just the YAML file, but also the script files into the instance directory.

At the end of that road lies something like helm charts, packaging multiple templates and scripts, and storing them in repositories etc, and I really don't want to go there.

---

I was however tempted to think about a simpler version of composable templates; I just didn't mention it yet because I didn't want to open Pandora's box of multi-file templates:

Assume we create templates for the base distros, that are parameterized by DISTRO_VERSION, like I've shown below.

Then we could maybe write a derived docker.yaml template like this:

param:
  DISTRO:
    default: ubuntu
    enum: [ubuntu, debian, opensuse, rocky, fedora]

  ROOTFUL:
    default: false
    prompt: Do you want to run docker in rootful mode?

basedOn: "template://{{.Param.DISTRO}}"

containerd:
  system: false
  user: false

# all the docker provisioning scripts and probes come here

And then you can create a rootful docker instance based on Rocky 8 with

limactl create --param DISTRO=rocky --param DISTRO_VERSION=8 --param ROOTFUL template://docker

The basedOn mechanism would conceptually insert a limayaml in the list of documents to be combined:

override.yaml
lima.yaml
basedon
default.yaml

However, I think we would need to store the merged content of lima.yaml and the basedOn template in $LIMA_HOME/$INSTANCE/lima.yaml, so on subsequent starts (or edits) the whole template is in a single file, and the instance is not modified just because the basedOn template has changed.

We (or users) can create additional templates for common combinations. E.g. if we don't want to delete rocky-8.yaml, then it can become

basedOn: template://rocky
param:
  DISTRO_VERSION: 8

I find this intriguing, but it probably needs more thoughts.

[jandubois]

If the user has their own template, that they want to use with template://docker, then they can enable it in their override.yaml file:

param:
  DISTRO:
    default: my-distro
    enum: [my-distro, ubuntu, debian, opensuse, rocky, fedora]

That way limactl start template://docker implicitly means:

limactl start --param DISTRO=my-distro template://docker

If basedOn can be chained, then the same would also work if they created a specific my-docker.yaml template:

param:
  DISTRO:
    default: my-distro
    enum: [my-distro]

basedOn: template://docker

and run it with limactl start template://my-docker.

This requires basedOn chaining (my-docker ← docker ← my-distro), but also that we only check if .Param.DISTRO is unused after the 3 documents have been merged.

[norio-nomura]

Validation

Personally, I feel that enhancing validation offers little benefit relative to the effort required.

Adding if condition on list entries

Rather than adding if:, I think it would be better to allow templates to be applied to location: and issue a warning instead of an error if it’s "", ignoring that entry.
In fact, I was considering writing something like this:

mounts:
- location: "{{.Dir}}/log"
  mountPoint: "/var/log"
  writable: true

If location is ignored when empty, then if: is unnecessary:

mounts:
- location: "{{if eq .Param.MOUNT_USB \"true\"}}/Volumes/USB{{end}}"
param:
  MOUNT_USB: false

[jandubois]

Personally, I feel that enhancing validation offers little benefit…

The validation is not really for users editing the template, but for the commandline usage. Assume we fold multiple distro versions into a single template:

param:
  DISTRO_VERSION:
    default: "24.04"
    enum: ["24.04", "23.10", "lts"]
images:
- location: "…/ubuntu-24.04.img"
  if: eq .Param.DISTRO_VERSION "24.04"
- location: "…/ubuntu-23.10.img"
  if: eq .Param.DISTRO_VERSION "23.10"
- location: "…/ubuntu-lts.img"
  if: eq .Param.DISTRO_VERSION "lts"

That way you can create a 23.10 Ubuntu instance with

limactl create --param DISTRO_VERSION=23.10 template://ubuntu

The validation exists to show a proper error message when the user specifies a setting that the template does not support:

$ limactl create --param DISTRO_VERSION=22.04 template://ubuntu
ERRO[0000] param.DISTRO_VERSION must be one of "24.04", "23.10", or "lts"

That is much better than failing with a message that no images have been defined.

…relative to the effort required.

I don't think the effort to implement validation is that big.

Rather than adding if:, I think it would be better to allow templates to be applied to location: and issue a warning instead of an error if it’s "", ignoring that entry.

I have no objection to applying host templates to location (and guest templates to mountPoint), but the if attribute would apply to all list type settings, not just mounts.

And I think it is more readable to decouple the condition from the value than having to make sure the template expression evaluates to an empty string if you want to omit this entry in a list.