[Question] extra args for minikube oci driver

[zhan9san]

What Happened?

There are many xxx driver only in minikube start --help.

For docker and podman drivers, could we unified them in one args, like --extra-args, and then pass this args to CreateContainerNode function as below.

minikube/pkg/drivers/kic/oci/oci.go

Lines 156 to 180 in 03b5bb7

	runArgs := []string{
	"-d", // run the container detached
	"-t", // allocate a tty for entrypoint logs
	// running containers in a container requires privileged
	// NOTE: we could try to replicate this with --cap-add, and use less
	// privileges, but this flag also changes some mounts that are necessary
	// including some ones docker would otherwise do by default.
	// for now this is what we want. in the future we may revisit this.
	"--privileged",
	"--security-opt", "seccomp=unconfined", // ignore seccomp
	"--tmpfs", "/tmp", // various things depend on working /tmp
	"--tmpfs", "/run", // systemd wants a writable /run
	// logs,pods be stroed on filesystem vs inside container,
	// some k8s things want /lib/modules
	"-v", "/lib/modules:/lib/modules:ro",
	"--hostname", p.Name, // make hostname match container name
	"--name", p.Name, // ... and set the container name
	"--label", fmt.Sprintf("%s=%s", CreatedByLabelKey, "true"),
	// label the node with the cluster ID
	"--label", p.ClusterLabel,
	// label the node with the role ID
	"--label", fmt.Sprintf("%s=%s", nodeRoleLabelKey, p.Role),
	// label th enode wuth the node ID
	"--label", p.NodeLabel,
	}

If we do so, some features can be easier implemented without code change, e.g.

Feature	With --extra-args	Without --extra-args
Run Minikube container with a static IP	--extra-args ip=192.168.10.3	add a parameter like --ip to minikube start
Run Minikube container in a subnet	--extra-args subnet=192.168.10.0/30	add a parameter like --subnet to minikube start #12315 #13730 #13766
Expose port in Minikube container	--extra-args publish=80:8080	add a parameter like --ports=[] to minikube start
etc	TBD	add a new parameter

I am not sure extra-args is a better design, and some validations may be lost.

Let me know what you think.

minikube start --help | grep only
      --dns-proxy=false: Enable proxy for NAT DNS requests (virtualbox driver only)
      --download-only=false: If true, only download and cache files for later use - don't install or start anything.
      --extra-disks=0: Number of extra disks created and attached to the minikube VM (currently only implemented for hyperkit and kvm2 drivers)
      --host-dns-resolver=true: Enable host resolver for NAT DNS requests (virtualbox driver only)
      --host-only-cidr='192.168.59.1/24': The CIDR to be used for the minikube VM (virtualbox driver only)
      --host-only-nic-type='virtio': NIC Type used for host only network. One of Am79C970A, Am79C973, 82540EM, 82543GC, 82545EM, or virtio (virtualbox driver only)
      --hyperkit-vpnkit-sock='': Location of the VPNKit socket used for networking. If empty, disables Hyperkit VPNKitSock, if 'auto' uses Docker for Mac VPNKit connection, otherwise uses the specified VSock (hyperkit driver only)
      --hyperkit-vsock-ports=[]: List of guest VSock ports that should be exposed as sockets on the host (hyperkit driver only)
      --hyperv-external-adapter='': External Adapter on which external switch will be created if no external switch is found. (hyperv driver only)
      --hyperv-use-external-switch=false: Whether to use external switch over Default Switch if virtual switch not explicitly specified. (hyperv driver only)
      --hyperv-virtual-switch='': The hyperv virtual switch name. Defaults to first found. (hyperv driver only)
      --kvm-hidden=false: Hide the hypervisor signature from the guest in minikube (kvm2 driver only)
      --kvm-network='default': The KVM default network name. (kvm2 driver only)
      --kvm-numa-count=1: Simulate numa node count in minikube, supported numa node count range is 1-8 (kvm2 driver only)
      --kvm-qemu-uri='qemu:///system': The KVM QEMU connection URI. (kvm2 driver only)
      --listen-address='': IP Address to use to expose ports (docker and podman driver only)
      --nat-nic-type='virtio': NIC Type used for nat network. One of Am79C970A, Am79C973, 82540EM, 82543GC, 82545EM, or virtio (virtualbox driver only)
      --nfs-share=[]: Local folders to share with Guest via NFS mounts (hyperkit driver only)
      --nfs-shares-root='/nfsshares': Where to root the NFS Shares, defaults to /nfsshares (hyperkit driver only)
      --no-kubernetes=false: If set, minikube VM/container will start without starting or configuring Kubernetes. (only works on new clusters)
      --no-vtx-check=false: Disable checking for the availability of hardware virtualization before the vm is started (virtualbox driver only)
      --ports=[]: List of ports that should be exposed (docker and podman driver only)
      --ssh-ip-address='': IP address (ssh driver only)
      --ssh-key='': SSH key (ssh driver only)
      --ssh-port=22: SSH port (ssh driver only)
      --ssh-user='root': SSH user (ssh driver only)
      --uuid='': Provide VM UUID to restore MAC address (hyperkit driver only)
      --vm=false: Filter to use only VM Drivers

Attach the log file

None

Operating System

Other

Driver

Docker

[RA489]

/kind support

[klaases]

Hi @zhan9san, we are always open to open to improving the minikube user experience.

In this case, it looks like having multiple persistent settings would be helpful, rather than entering numerous flags on minikube start.

Have you tried looking at the minikube config mechanism?
https://minikube.sigs.k8s.io/docs/commands/config/

With a minikube config, you can make many changes that will persist when starting minikube.

[zhan9san]

Hi @klaases

Thanks for your attention.

Sorry for no making myself understood. Let me highlight the key point.

If we use --extra-args,

1. There is no need to update the code when someone wants to customize the Minikube container's behavior. Instead, we only need to update the document showing how to implement the feature.
2. Reduce the code size

Besides, --extra-args would be configured in config file as well.

[afbjorklund]

In the original libmachine design, the flags were per driver and not global like in minikube.

So it was a side effect of forking and including the drivers, possibly also the cli lib used.

[klaases]

Hi @zhan9san, we currently allow for arguments and the configuration files. However, if we modify an --extra-args parameter, this might conceal some combination of arguments.

While this may add some convenience, we also find that it may add complexity.

1. Have you tried the config mechanism that was previously mentioned? This would be helpful in the sense that it allows for the arguments to be set as a default in a file, and therefore they no longer need to be passed as arguments when starting minikube.

2. Another suggestion would be to create a small bash script for your use case. With a script, you could customize the additional args that are needed in varying scenarios. So the list of args could be stored in the bash script, and they could be parameterized and called upon later.

I agree that having a large number of arguments can become unwieldily, and it makes sense to work on ways to more highly organize them.

[zhan9san]

@klaases

Thanks for your update. Please allow me to recap the issue.

I know the config mechanism, but it's a different scenario.

The key point is for how to implement new features in Minikube.

Assume features list in the first column is not supported in Minikube.

Let us compare how Minikube implement them with --extra-args and without --extra-args.

As we can see, if --extra-args is supported, the new features can be implemented without any code changes in Minikube. Instead, we have to create another pull requests to wrapper the parameters and pass it to docker daemon one by one.

Feature	With --extra-args	Without --extra-args
Run Minikube container with a static IP	--extra-args ip=192.168.10.3	add a parameter like --ip to minikube start
Run Minikube container in a subnet	--extra-args subnet=192.168.10.0/30	add a parameter like --subnet to minikube start #12315 #13730 #13766
Expose port in Minikube container	--extra-args publish=80:8080	add a parameter like --ports=[] to minikube start
etc	TBD	add a new parameter

Hope I express myself clearly.

[afbjorklund]

This is both a pro and a con, less "typing" of arguments also means more chance of breakage by not checking them.

It is similar to a programming language, where you have named parameters ("args") and pass in dictionaries ("kwargs")

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[mk46]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.