Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Start minikube on Chromebook Linux (Debian 12) failed #18026

Closed
xn01302 opened this issue Jan 23, 2024 · 22 comments
Closed

Start minikube on Chromebook Linux (Debian 12) failed #18026

xn01302 opened this issue Jan 23, 2024 · 22 comments
Labels
co/kvm2-driver KVM2 driver related issues kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@xn01302
Copy link

xn01302 commented Jan 23, 2024

What Happened?

Hi all.

start minikube on Chromebook Linux (Debian 12) with kvm2 driver failed. It's a problem with /dev/random.

mdersch@penguin:~$ minikube start --driver=kvm2
😄 minikube v1.32.0 on Debian 12.4 (amd64)
✨ Using the kvm2 driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing kvm2 VM for "minikube" ...
🤦 StartHost failed, but will try again: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')
🔄 Restarting existing kvm2 VM for "minikube" ...
😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

Attach the log file

logs.txt

Operating System

Other

Driver

KVM2
@afbjorklund
Copy link
Collaborator

You need some special settings, for running libvirt on ChromeOS

@afbjorklund afbjorklund added co/kvm2-driver KVM2 driver related issues kind/support Categorizes issue or PR as a support question. labels Jan 23, 2024
@xn01302
Copy link
Author

xn01302 commented Jan 23, 2024

Yes, this settings worked with Debian11. Now I have Debian12 and this strange /dev/random problem occurs.
I have done all these setting but no success.

@xn01302
Copy link
Author

xn01302 commented Jan 23, 2024

I'm switched back to Debian 11 (Bullseye). Everything works fine now. It's a problem with Debian 12. I don't know what exactly. The behavior is different.

@xn01302
Copy link
Author

xn01302 commented Jan 23, 2024

Please let me know if anybody has success with minikube driver=kvm2 and Debian12.

@caerulescens
Copy link

caerulescens commented Jan 27, 2024 •
edited
Loading

@xn01302 I will confirm from experience that this is not an issue with Debian. The kvm2 driver for minikube works great on Debian, and I use it everyday. I know the exact steps to configure KVM for minikube, and I performed these steps recently when I upgraded to Bookworm from Bullseye. You only need to follow first few steps in this guide after performing a vanilla install of Debian (see below).

  1. Install Debian
  2. (any other dependencies)
  3. sudo apt install qemu-system libvirt-daemon-system
  4. sudo adduser $USER libvirt
  5. Logout, then login to update user groups
  6. minikube start --driver=kvm2

Also, try deleting ~/.minikube and then try to start again; minikube will recreate ~/.minikube and download the kvm2 driver.

virsh version:

Compiled against library: libvirt 9.0.0
Using library: libvirt 9.0.0
Using API: QEMU 9.0.0
Running hypervisor: QEMU 7.2.7

minikube version:

minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

lsb_release -a:

Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

minikube start --driver=kvm2

😄  minikube v1.32.0 on Debian 12.4
✨  Using the kvm2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

@caerulescens
Copy link

caerulescens commented Jan 27, 2024 •
edited
Loading

@xn01302 You should install kvm like I said, and try: #14272 (comment)

@caerulescens caerulescens mentioned this issue Jan 28, 2024
Closed
@caerulescens
Copy link

/assign

@xn01302
Copy link
Author

xn01302 commented Feb 2, 2024

Okay, I will try this. Currently I'm working with Debian 11.

@caerulescens
Copy link

@xn01302 I can verify that the instructions work on Debian 11 too, so if you have issues, then it could be something else.

@caerulescens
Copy link

I think that @afbjorklund is correct because Chrome books end up having quite a few quirks with their security; although, I do not have much experience with running Debian on Chrome books.

@afbjorklund
Copy link
Collaborator

afbjorklund commented Feb 3, 2024 •
edited
Loading

libvirt has a command virt-host-validate qemu, which you can use to troubleshoot. The kvm/kvm2 driver uses libvirt.

Alternatively you can try with the qemu/qemu2 driver, which also supports kvm but does not use the libvirtd daemon.

@caerulescens
Copy link

caerulescens commented Feb 3, 2024 •
edited
Loading

@xn01302 Here's what a working configuration will look like using virt-host-validate qemu.

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (AMD Secure Encrypted Virtualization appears to be disabled in firmware.)

Using virt-host-validate without arguments will yield checking the Linux containers (LXC) configuration as well.

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (AMD Secure Encrypted Virtualization appears to be disabled in firmware.)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : FAIL (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS

@xn01302
Copy link
Author

xn01302 commented Feb 8, 2024

@caerulescens Sorry for delayed response. I'll try it. Have installed a additional container with debian 12 now. Give you feedback in a few days.

@vveisard
Copy link

vveisard commented Feb 10, 2024 •
edited
Loading

I am having the same issue on my Debian 12 install on ChromeOS. I've already tried editing /etc/libvirt/qemu.conf with the suggested settings

EDIT:
I tried the suggested process on a clean Debian 11 install and it worked.

@caerulescens caerulescens removed their assignment Feb 11, 2024
@caerulescens
Copy link

caerulescens commented Feb 11, 2024 •
edited
Loading

I don't have a Chromebook to debug this fix.

@xn01302
Copy link
Author

xn01302 commented Feb 12, 2024

I've tried again on my chromebook. I have created a new Debian12 container and installed minikube. Start process failed again with the same error message as before. Problem with /dev/random

minikube start --driver=kvm2
😄 minikube v1.32.0 on Debian 12.5 (amd64)
✨ Using the kvm2 driver based on user configuration
💾 Downloading driver docker-machine-driver-kvm2:
> docker-machine-driver-kvm2-...: 65 B / 65 B [---------] 100.00% ? p/s 0s
> docker-machine-driver-kvm2-...: 13.01 MiB / 13.01 MiB 100.00% 10.07 MiB
💿 Downloading VM boot image ...
> minikube-v1.32.1-amd64.iso....: 65 B / 65 B [---------] 100.00% ? p/s 0s
> minikube-v1.32.1-amd64.iso: 292.96 MiB / 292.96 MiB 100.00% 7.50 MiB p/
👍 Starting control plane node minikube in cluster minikube
💾 Downloading Kubernetes v1.28.3 preload ...
> preloaded-images-k8s-v18-v1...: 403.35 MiB / 403.35 MiB 100.00% 7.76 Mi
🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
🔥 Deleting "minikube" in kvm2 ...
🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')
🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ...
😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

@afbjorklund
Copy link
Collaborator

If it is a blocker, it is possibly to make the RNG block in the virt configuration more optional...

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng>

There are other blocks (such as GPU), that are optional and toggled with flags at runtime.

@afbjorklund afbjorklund added kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed kind/support Categorizes issue or PR as a support question. labels Feb 13, 2024
@norseto
Copy link

norseto commented Mar 12, 2024

I looked at https://gitlab.com/libvirt/libvirt/-/issues/556 and set namespaces=[] in /etc/libvirt/qemu.conf and was able to start minikube.
I hope this is helpful.

@xn01302
Copy link
Author

xn01302 commented Mar 17, 2024

Hi, thanks a lot. It works with namespaces = []. Great!!!

Best regards
Mirko

@caerulescens
Copy link

/close

@k8s-ci-robot
Copy link
Contributor

@caerulescens: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@caerulescens
Copy link

@xn01302 Awesome; close the issue when you can

@xn01302 xn01302 closed this as completed Jun 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
co/kvm2-driver KVM2 driver related issues kind/feature Categorizes issue or PR as related to a new feature. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@vveisard @afbjorklund @norseto @k8s-ci-robot @caerulescens @xn01302