Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minikube exits due to an error with PodSecurityPolicy admissions controller enabled #6686

Closed
Birdrock opened this issue Feb 19, 2020 · 7 comments · Fixed by #8454
Closed
Labels
area/addons kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.

Comments

@Birdrock
Copy link

When following the instructions here to enable PodSecurityPolicies with minikube, minikube exits due to an error.

A brief conversation in slack suggested this might be the result of a recent addons refactor.

The exact command to reproduce the issue:

➜ minikube start --extra-config=apiserver.enable-admission-plugins=PodSecurityPolicy

The full output of the command that failed:


😄 minikube v1.7.2 on Darwin 10.15.3
✨ Automatically selected the hyperkit driver. Other choices: virtualbox, docker (experimental)
🔥 Creating hyperkit VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.17.2 on Docker 19.03.5 ...
▪ apiserver.enable-admission-plugins=PodSecurityPolicy
🚀 Launching Kubernetes ...
🌟 Enabling addons: default-storageclass, storage-provisioner
⚠️ Enabling 'storage-provisioner' returned an error: running callbacks: [addon apply: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.17.2/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
serviceaccount/storage-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/storage-provisioner created

stderr:
Error from server (Forbidden): error when creating "/etc/kubernetes/addons/storage-provisioner.yaml": pods "storage-provisioner" is forbidden: no providers available to validate pod request
]
⌛ Waiting for cluster to come online ...

💣 Wait failed: apiserver never returned a pod list

😿 minikube is exiting due to an error. If the above message is not useful, open an issue:
👉 https://github.com/kubernetes/minikube/issues/new/choose

The full output of minikube logs:


➜ minikube logs
==> Docker <==
-- Logs begin at Wed 2020-02-19 18:48:50 UTC, end at Wed 2020-02-19 18:50:13 UTC. --
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875568468Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875625612Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875692313Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875778750Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875827210Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875862655Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875898709Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875936334Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.875970689Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876004079Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876109826Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876203749Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876490199Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876543306Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876598106Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876635873Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876669607Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876702302Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876738304Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876772094Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876808286Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876841585Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876876556Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876932133Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.876969229Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.877002897Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.877036324Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.877134739Z" level=info msg=serving... address="/var/run/docker/containerd/containerd-debug.sock"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.877219998Z" level=info msg=serving... address="/var/run/docker/containerd/containerd.sock"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.877285553Z" level=info msg="containerd successfully booted in 0.006910s"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.885614525Z" level=info msg="parsed scheme: "unix"" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.885654642Z" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.885671998Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.885678692Z" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.886481898Z" level=info msg="parsed scheme: "unix"" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.886509525Z" level=info msg="scheme "unix" not registered, fallback to default scheme" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.886526041Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock 0 }] }" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.886533374Z" level=info msg="ClientConn switching balancer to "pick_first"" module=grpc
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899887585Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899921084Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899928704Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_bps_device"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899932931Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_bps_device"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899936950Z" level=warning msg="Your kernel does not support cgroup blkio throttle.read_iops_device"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.899941065Z" level=warning msg="Your kernel does not support cgroup blkio throttle.write_iops_device"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.900081024Z" level=info msg="Loading containers: start."
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.954030807Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Feb 19 18:48:53 minikube dockerd[1909]: time="2020-02-19T18:48:53.988730348Z" level=info msg="Loading containers: done."
Feb 19 18:48:54 minikube dockerd[1909]: time="2020-02-19T18:48:54.004080049Z" level=info msg="Docker daemon" commit=633a0ea838 graphdriver(s)=overlay2 version=19.03.5
Feb 19 18:48:54 minikube dockerd[1909]: time="2020-02-19T18:48:54.004190464Z" level=info msg="Daemon has completed initialization"
Feb 19 18:48:54 minikube dockerd[1909]: time="2020-02-19T18:48:54.016293979Z" level=info msg="API listen on /var/run/docker.sock"
Feb 19 18:48:54 minikube systemd[1]: Started Docker Application Container Engine.
Feb 19 18:48:54 minikube dockerd[1909]: time="2020-02-19T18:48:54.016363813Z" level=info msg="API listen on [::]:2376"
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.539153619Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e3b309f2ded99dca2876e5e6124f12838e648a53c761cd9268feef9e22d47131/shim.sock" debug=false pid=3297
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.549195710Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/f207965f04723f93e15521101a3032bdfdab7ed94c135756665d71034429c2e3/shim.sock" debug=false pid=3310
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.563042103Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/16eb077abc4a82f7f4916dcbe2ce943658c67274aab28c3cabd91b4324e2904c/shim.sock" debug=false pid=3323
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.571691532Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ec0f6551d040a2decee4a6bc9195208167f6ee033940c5f8d5c0d864527966f2/shim.sock" debug=false pid=3327
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.730121753Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/6c68d336f0b8c7e5ac88dbc49f81c5381d8851a9068dab06c3f769f13f507133/shim.sock" debug=false pid=3492
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.794704791Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/686a156abf231e8cfe2a84041e47680c1e6221e77337dbf8400bb3c3ba57b736/shim.sock" debug=false pid=3509
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.796596581Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e9e374c1a8442943afc455613f2252f9bc7cbd47ceb761c3d7baf89ed45799d5/shim.sock" debug=false pid=3516
Feb 19 18:49:21 minikube dockerd[1909]: time="2020-02-19T18:49:21.807021288Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/3257a1bd320cf6b6db42ca0f205ae4608c065e5a48b1ccbc706f9543a32b0240/shim.sock" debug=false pid=3525

==> container status <==
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
3257a1bd320cf f52d4c527ef2f 52 seconds ago Running kube-scheduler 0 16eb077abc4a8
686a156abf231 41ef50a5f06a7 52 seconds ago Running kube-apiserver 0 ec0f6551d040a
e9e374c1a8442 303ce5db0e90d 52 seconds ago Running etcd 0 f207965f04723
6c68d336f0b8c da5fd66c4068c 52 seconds ago Running kube-controller-manager 0 e3b309f2ded99

==> dmesg <==
[Feb19 18:48] ERROR: earlyprintk= earlyser already used
[ +0.000000] You have booted with nomodeset. This means your GPU drivers are DISABLED
[ +0.000001] Any video related functionality will be severely degraded, and you may not even be able to suspend the system properly
[ +0.000000] Unless you actually understand what nomodeset does, you should reboot without enabling it
[ +0.005090] ACPI BIOS Warning (bug): Incorrect checksum in table [DSDT] - 0xC0, should be 0x1D (20180810/tbprint-177)
[ +3.105781] ACPI Error: Could not enable RealTimeClock event (20180810/evxfevnt-184)
[ +0.000001] ACPI Warning: Could not enable fixed event - RealTimeClock (4) (20180810/evxface-620)
[ +0.005599] platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
[ +1.648699] systemd[1]: Failed to bump fs.file-max, ignoring: Invalid argument
[ +0.005387] systemd-fstab-generator[1096]: Ignoring "noauto" for root device
[ +0.002586] systemd[1]: File /usr/lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[ +0.000002] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ +0.785661] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ +0.520083] vboxguest: loading out-of-tree module taints kernel.
[ +0.003207] vboxguest: PCI device not found, probably running on physical hardware.
[ +1.538038] systemd-fstab-generator[1877]: Ignoring "noauto" for root device
[ +0.116038] systemd-fstab-generator[1893]: Ignoring "noauto" for root device
[Feb19 18:49] systemd-fstab-generator[2629]: Ignoring "noauto" for root device
[ +0.623701] systemd-fstab-generator[2831]: Ignoring "noauto" for root device
[ +10.538281] kauditd_printk_skb: 65 callbacks suppressed
[ +6.817483] systemd-fstab-generator[3934]: Ignoring "noauto" for root device

==> kernel <==
18:50:13 up 1 min, 0 users, load average: 2.36, 0.82, 0.30
Linux minikube 4.19.88 #1 SMP Tue Feb 4 22:25:03 PST 2020 x86_64 GNU/Linux
PRETTY_NAME="Buildroot 2019.02.8"

==> kube-apiserver [686a156abf23] <==
W0219 18:49:23.721313 1 genericapiserver.go:404] Skipping API discovery.k8s.io/v1alpha1 because it has no resources.
W0219 18:49:23.728609 1 genericapiserver.go:404] Skipping API node.k8s.io/v1alpha1 because it has no resources.
W0219 18:49:23.741757 1 genericapiserver.go:404] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0219 18:49:23.744285 1 genericapiserver.go:404] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0219 18:49:23.767262 1 genericapiserver.go:404] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0219 18:49:23.786863 1 genericapiserver.go:404] Skipping API apps/v1beta2 because it has no resources.
W0219 18:49:23.786895 1 genericapiserver.go:404] Skipping API apps/v1beta1 because it has no resources.
I0219 18:49:23.792046 1 plugins.go:158] Loaded 11 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,TaintNodesByCondition,PodSecurityPolicy,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook,RuntimeClass.
I0219 18:49:23.792072 1 plugins.go:161] Loaded 8 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,RuntimeClass,ResourceQuota.
I0219 18:49:23.793241 1 client.go:361] parsed scheme: "endpoint"
I0219 18:49:23.793255 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }]
I0219 18:49:23.798910 1 client.go:361] parsed scheme: "endpoint"
I0219 18:49:23.798967 1 endpoint.go:68] ccResolverWrapper: sending new addresses to cc: [{https://127.0.0.1:2379 0 }]
I0219 18:49:26.201835 1 dynamic_cafile_content.go:166] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0219 18:49:26.202101 1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0219 18:49:26.202546 1 dynamic_serving_content.go:129] Starting serving-cert::/var/lib/minikube/certs/apiserver.crt::/var/lib/minikube/certs/apiserver.key
I0219 18:49:26.203006 1 secure_serving.go:178] Serving securely on [::]:8443
I0219 18:49:26.203047 1 apiservice_controller.go:94] Starting APIServiceRegistrationController
I0219 18:49:26.203052 1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0219 18:49:26.203092 1 tlsconfig.go:219] Starting DynamicServingCertificateController
I0219 18:49:26.205072 1 available_controller.go:386] Starting AvailableConditionController
I0219 18:49:26.205162 1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0219 18:49:26.205248 1 autoregister_controller.go:140] Starting autoregister controller
I0219 18:49:26.205328 1 cache.go:32] Waiting for caches to sync for autoregister controller
I0219 18:49:26.205715 1 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
I0219 18:49:26.205820 1 shared_informer.go:197] Waiting for caches to sync for cluster_authentication_trust_controller
I0219 18:49:26.206535 1 controller.go:81] Starting OpenAPI AggregationController
I0219 18:49:26.206650 1 crdregistration_controller.go:111] Starting crd-autoregister controller
I0219 18:49:26.206696 1 shared_informer.go:197] Waiting for caches to sync for crd-autoregister
I0219 18:49:26.206792 1 dynamic_cafile_content.go:166] Starting client-ca-bundle::/var/lib/minikube/certs/ca.crt
I0219 18:49:26.206891 1 dynamic_cafile_content.go:166] Starting request-header::/var/lib/minikube/certs/front-proxy-ca.crt
I0219 18:49:26.208571 1 crd_finalizer.go:263] Starting CRDFinalizer
I0219 18:49:26.252212 1 controller.go:85] Starting OpenAPI controller
I0219 18:49:26.252629 1 customresource_discovery_controller.go:208] Starting DiscoveryController
I0219 18:49:26.252757 1 naming_controller.go:288] Starting NamingConditionController
I0219 18:49:26.252827 1 establishing_controller.go:73] Starting EstablishingController
I0219 18:49:26.252926 1 nonstructuralschema_controller.go:191] Starting NonStructuralSchemaConditionController
I0219 18:49:26.252993 1 apiapproval_controller.go:185] Starting KubernetesAPIApprovalPolicyConformantConditionController
E0219 18:49:26.261794 1 controller.go:151] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.64.7, ResourceVersion: 0, AdditionalErrorMsg:
I0219 18:49:26.304850 1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0219 18:49:26.305323 1 cache.go:39] Caches are synced for AvailableConditionController controller
I0219 18:49:26.305569 1 cache.go:39] Caches are synced for autoregister controller
I0219 18:49:26.306211 1 shared_informer.go:204] Caches are synced for cluster_authentication_trust_controller
I0219 18:49:26.307538 1 shared_informer.go:204] Caches are synced for crd-autoregister
I0219 18:49:27.202055 1 controller.go:107] OpenAPI AggregationController: Processing item
I0219 18:49:27.202205 1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0219 18:49:27.202345 1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0219 18:49:27.211990 1 storage_scheduling.go:133] created PriorityClass system-node-critical with value 2000001000
I0219 18:49:27.214138 1 storage_scheduling.go:133] created PriorityClass system-cluster-critical with value 2000000000
I0219 18:49:27.214194 1 storage_scheduling.go:142] all system priority classes are created successfully or already exist.
I0219 18:49:27.481796 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0219 18:49:27.513301 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0219 18:49:27.639704 1 lease.go:224] Resetting endpoints for master service "kubernetes" to [192.168.64.7]
I0219 18:49:27.640314 1 controller.go:606] quota admission added evaluator for: endpoints
I0219 18:49:28.408793 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0219 18:49:28.740345 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0219 18:49:28.750864 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0219 18:49:28.988034 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0219 18:49:37.555356 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I0219 18:49:37.588347 1 controller.go:606] quota admission added evaluator for: replicasets.apps

==> kube-controller-manager [6c68d336f0b8] <==
E0219 18:49:37.642892 1 clusterroleaggregation_controller.go:180] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
E0219 18:49:37.644879 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.645516 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:37.663517 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:37.663566 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:37.663666 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:37.687601 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.687890 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:37.747240 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:37.747615 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:37.747450 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:37.766528 1 shared_informer.go:204] Caches are synced for stateful set
E0219 18:49:37.770723 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.771031 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.785818 1 shared_informer.go:204] Caches are synced for PVC protection
I0219 18:49:37.786280 1 shared_informer.go:204] Caches are synced for expand
I0219 18:49:37.791429 1 shared_informer.go:204] Caches are synced for persistent volume
I0219 18:49:37.886837 1 shared_informer.go:204] Caches are synced for attach detach
E0219 18:49:37.912257 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:37.912303 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:37.912538 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:37.934362 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.934612 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:37.938199 1 shared_informer.go:204] Caches are synced for endpoint
I0219 18:49:38.062640 1 shared_informer.go:204] Caches are synced for resource quota
I0219 18:49:38.091228 1 shared_informer.go:204] Caches are synced for garbage collector
I0219 18:49:38.091553 1 garbagecollector.go:138] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0219 18:49:38.096131 1 shared_informer.go:204] Caches are synced for resource quota
E0219 18:49:38.236221 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:38.236793 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:38.236974 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:38.257766 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:38.257821 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:38.881357 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:38.881447 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:38.881465 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:38.900279 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:38.900352 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:38.987268 1 shared_informer.go:197] Waiting for caches to sync for garbage collector
I0219 18:49:38.987326 1 shared_informer.go:204] Caches are synced for garbage collector
E0219 18:49:40.167346 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:40.167511 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:40.167591 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:40.183360 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:40.183579 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:42.731563 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:42.731584 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:42.731619 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:42.745993 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:42.746486 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:47.856636 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:47.856796 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:47.856876 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:47.869070 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:47.869131 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
E0219 18:49:58.101890 1 daemon_controller.go:965] pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:58.101977 1 daemon_controller.go:290] kube-system/kube-proxy failed with : pods "kube-proxy-" is forbidden: no providers available to validate pod request
I0219 18:49:58.102162 1 event.go:281] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"6f2ed941-c2ce-45cc-a8e1-3b667923453a", APIVersion:"apps/v1", ResourceVersion:"186", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "kube-proxy-" is forbidden: no providers available to validate pod request
E0219 18:49:58.112802 1 replica_set.go:534] sync "kube-system/coredns-6955765f44" failed with pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request
I0219 18:49:58.113185 1 event.go:281] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-6955765f44", UID:"14143c11-0e01-4987-80c3-3da45789cbf5", APIVersion:"apps/v1", ResourceVersion:"330", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: pods "coredns-6955765f44-" is forbidden: no providers available to validate pod request

==> kube-scheduler [3257a1bd320c] <==
I0219 18:49:22.920425 1 serving.go:312] Generated self-signed cert in-memory
W0219 18:49:23.481169 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0219 18:49:23.481420 1 configmap_cafile_content.go:102] unable to load initial CA bundle for: "client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" due to: configmap "extension-apiserver-authentication" not found
W0219 18:49:26.271766 1 authentication.go:348] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0219 18:49:26.272214 1 authentication.go:296] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0219 18:49:26.272266 1 authentication.go:297] Continuing without authentication configuration. This may treat all requests as anonymous.
W0219 18:49:26.272412 1 authentication.go:298] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
W0219 18:49:26.299428 1 authorization.go:47] Authorization is disabled
W0219 18:49:26.299476 1 authentication.go:92] Authentication is disabled
I0219 18:49:26.299485 1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251
I0219 18:49:26.300433 1 configmap_cafile_content.go:205] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0219 18:49:26.300548 1 shared_informer.go:197] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0219 18:49:26.300884 1 secure_serving.go:178] Serving securely on 127.0.0.1:10259
I0219 18:49:26.300896 1 tlsconfig.go:219] Starting DynamicServingCertificateController
E0219 18:49:26.304018 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0219 18:49:26.304133 1 reflector.go:153] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:246: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0219 18:49:26.304021 1 reflector.go:153] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0219 18:49:26.304438 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0219 18:49:26.304660 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0219 18:49:26.304677 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0219 18:49:26.304891 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0219 18:49:26.305135 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0219 18:49:26.305138 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0219 18:49:26.305436 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0219 18:49:26.305540 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0219 18:49:26.306314 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0219 18:49:27.305183 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0219 18:49:27.305955 1 reflector.go:153] k8s.io/kubernetes/cmd/kube-scheduler/app/server.go:246: Failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0219 18:49:27.307066 1 reflector.go:153] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: Failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0219 18:49:27.308713 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0219 18:49:27.309753 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0219 18:49:27.312140 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0219 18:49:27.312939 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0219 18:49:27.314941 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0219 18:49:27.317522 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0219 18:49:27.318293 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0219 18:49:27.320073 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0219 18:49:27.320843 1 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
I0219 18:49:28.401484 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0219 18:49:28.401808 1 leaderelection.go:242] attempting to acquire leader lease kube-system/kube-scheduler...
I0219 18:49:28.410307 1 leaderelection.go:252] successfully acquired lease kube-system/kube-scheduler

==> kubelet <==
-- Logs begin at Wed 2020-02-19 18:48:50 UTC, end at Wed 2020-02-19 18:50:13 UTC. --
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.868485 3943 kubelet.go:311] Watching apiserver
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.871943 3943 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.872011 3943 client.go:104] Start docker client with request timeout=2m0s
Feb 19 18:49:28 minikube kubelet[3943]: W0219 18:49:28.889585 3943 docker_service.go:563] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.889703 3943 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Feb 19 18:49:28 minikube kubelet[3943]: W0219 18:49:28.889807 3943 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Feb 19 18:49:28 minikube kubelet[3943]: W0219 18:49:28.891288 3943 hostport_manager.go:69] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Feb 19 18:49:28 minikube kubelet[3943]: W0219 18:49:28.891341 3943 hostport_manager.go:69] The binary conntrack is not installed, this can cause failures in network connection cleanup.
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.892183 3943 docker_service.go:255] Docker cri networking managed by kubernetes.io/no-op
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.898703 3943 docker_service.go:260] Docker Info: &{ID:VQCN:7GPY:2HS7:L5UY:7VNL:AKQ7:3CAK:SR5W:F224:I6LT:E7CY:YTSK Containers:8 ContainersRunning:8 ContainersPaused:0 ContainersStopped:0 Images:10 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:66 OomKillDisable:true NGoroutines:76 SystemTime:2020-02-19T18:49:28.892897083Z LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:4.19.88 OperatingSystem:Buildroot 2019.02.8 OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc0006d7c00 NCPU:2 MemTotal:2035298304 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:minikube Labels:[provider=hyperkit] ExperimentalBuild:false ServerVersion:19.03.5 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster: Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:b34a5c8af56e510852c35414db4c1f4fa6172339 Expected:b34a5c8af56e510852c35414db4c1f4fa6172339} RuncCommit:{ID:d736ef14f0288d6993a1845745d6756cfc9ddd5a Expected:d736ef14f0288d6993a1845745d6756cfc9ddd5a} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default] ProductLicense:Community Engine Warnings:[]}
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.900114 3943 docker_service.go:273] Setting cgroupDriver to cgroupfs
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914238 3943 remote_runtime.go:59] parsed scheme: ""
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914268 3943 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914287 3943 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 }] }
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914292 3943 clientconn.go:577] ClientConn switching balancer to "pick_first"
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914315 3943 remote_image.go:50] parsed scheme: ""
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914319 3943 remote_image.go:50] scheme "" not registered, fallback to default scheme
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914324 3943 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 }] }
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.914327 3943 clientconn.go:577] ClientConn switching balancer to "pick_first"
Feb 19 18:49:28 minikube kubelet[3943]: E0219 18:49:28.915146 3943 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Feb 19 18:49:28 minikube kubelet[3943]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.920682 3943 kuberuntime_manager.go:211] Container runtime docker initialized, version: 19.03.5, apiVersion: 1.40.0
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.927911 3943 server.go:1113] Started kubelet
Feb 19 18:49:28 minikube kubelet[3943]: E0219 18:49:28.928215 3943 kubelet.go:1302] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.928949 3943 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.931695 3943 server.go:143] Starting to listen on 0.0.0.0:10250
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.932159 3943 server.go:354] Adding debug handlers to kubelet server.
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.933855 3943 volume_manager.go:265] Starting Kubelet Volume Manager
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.941029 3943 desired_state_of_world_populator.go:138] Desired state populator starts to run
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.959916 3943 status_manager.go:157] Starting to sync pod status with apiserver
Feb 19 18:49:28 minikube kubelet[3943]: I0219 18:49:28.960038 3943 kubelet.go:1820] Starting kubelet main sync loop.
Feb 19 18:49:28 minikube kubelet[3943]: E0219 18:49:28.960136 3943 kubelet.go:1844] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.035597 3943 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Feb 19 18:49:29 minikube kubelet[3943]: E0219 18:49:29.062592 3943 kubelet.go:1844] skipping pod synchronization - container runtime status check may not have completed yet
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.076461 3943 kubelet_node_status.go:70] Attempting to register node minikube
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.085790 3943 kubelet_node_status.go:112] Node minikube was previously registered
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.085878 3943 kubelet_node_status.go:73] Successfully registered node minikube
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.089485 3943 cpu_manager.go:173] [cpumanager] starting with none policy
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.089534 3943 cpu_manager.go:174] [cpumanager] reconciling every 10s
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.089546 3943 policy_none.go:43] [cpumanager] none policy: Start
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.090670 3943 plugin_manager.go:114] Starting Kubelet Plugin Manager
Feb 19 18:49:29 minikube kubelet[3943]: E0219 18:49:29.268865 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(9c994ea62a2d8d6f1bb7498f10aa6fcf)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:29 minikube kubelet[3943]: E0219 18:49:29.268972 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(0ae6cf402f641e9b795a3aebca394220)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:29 minikube kubelet[3943]: E0219 18:49:29.271983 3943 kubelet.go:1662] Failed creating a mirror pod for "etcd-minikube_kube-system(b9fe2e6f6ce2d44f54873406cb711241)": pods "etcd-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.345908 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/b9fe2e6f6ce2d44f54873406cb711241-etcd-certs") pod "etcd-minikube" (UID: "b9fe2e6f6ce2d44f54873406cb711241")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.345969 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/a346b9d5328da2876bf84c8770df0f36-ca-certs") pod "kube-apiserver-minikube" (UID: "a346b9d5328da2876bf84c8770df0f36")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.345991 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/a346b9d5328da2876bf84c8770df0f36-usr-share-ca-certificates") pod "kube-apiserver-minikube" (UID: "a346b9d5328da2876bf84c8770df0f36")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346005 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/0ae6cf402f641e9b795a3aebca394220-ca-certs") pod "kube-controller-manager-minikube" (UID: "0ae6cf402f641e9b795a3aebca394220")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346018 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/0ae6cf402f641e9b795a3aebca394220-kubeconfig") pod "kube-controller-manager-minikube" (UID: "0ae6cf402f641e9b795a3aebca394220")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346031 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "usr-share-ca-certificates" (UniqueName: "kubernetes.io/host-path/0ae6cf402f641e9b795a3aebca394220-usr-share-ca-certificates") pod "kube-controller-manager-minikube" (UID: "0ae6cf402f641e9b795a3aebca394220")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346066 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/b9fe2e6f6ce2d44f54873406cb711241-etcd-data") pod "etcd-minikube" (UID: "b9fe2e6f6ce2d44f54873406cb711241")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346079 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/a346b9d5328da2876bf84c8770df0f36-k8s-certs") pod "kube-apiserver-minikube" (UID: "a346b9d5328da2876bf84c8770df0f36")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346091 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/0ae6cf402f641e9b795a3aebca394220-flexvolume-dir") pod "kube-controller-manager-minikube" (UID: "0ae6cf402f641e9b795a3aebca394220")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346102 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/0ae6cf402f641e9b795a3aebca394220-k8s-certs") pod "kube-controller-manager-minikube" (UID: "0ae6cf402f641e9b795a3aebca394220")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346115 3943 reconciler.go:209] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/9c994ea62a2d8d6f1bb7498f10aa6fcf-kubeconfig") pod "kube-scheduler-minikube" (UID: "9c994ea62a2d8d6f1bb7498f10aa6fcf")
Feb 19 18:49:29 minikube kubelet[3943]: I0219 18:49:29.346121 3943 reconciler.go:156] Reconciler: start to sync state
Feb 19 18:49:29 minikube kubelet[3943]: E0219 18:49:29.473963 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(a346b9d5328da2876bf84c8770df0f36)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:30 minikube kubelet[3943]: E0219 18:49:30.272901 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-apiserver-minikube_kube-system(a346b9d5328da2876bf84c8770df0f36)": pods "kube-apiserver-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:30 minikube kubelet[3943]: E0219 18:49:30.473425 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-scheduler-minikube_kube-system(9c994ea62a2d8d6f1bb7498f10aa6fcf)": pods "kube-scheduler-minikube" is forbidden: no providers available to validate pod request
Feb 19 18:49:30 minikube kubelet[3943]: E0219 18:49:30.671422 3943 kubelet.go:1662] Failed creating a mirror pod for "kube-controller-manager-minikube_kube-system(0ae6cf402f641e9b795a3aebca394220)": pods "kube-controller-manager-minikube" is forbidden: no providers available to validate pod request

The operating system version:

macOS Catalina Version 10.15.3 (19D76)

@medyagh
Copy link
Member

medyagh commented Feb 19, 2020

@Birdrock I am curious is this something that broke in 1.7.2 ? had u been using this feature before with minikube?

@Birdrock
Copy link
Author

@medyagh We have not tried pod security policies on any other version of minikube, so I do not know whether it is a regression. There was a suggestion in the slack conversation that it may be related to a recent refactor of addons, but I cannot offer any certainty whether it worked previously.

@sharifelgamal sharifelgamal added area/addons kind/bug Categorizes issue or PR as related to a bug. kind/documentation Categorizes issue or PR as related to documentation. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Feb 26, 2020
@ryysud
Copy link

ryysud commented Mar 12, 2020

I also faced same issue using minikube v1.8.1 .

@priyawadhwa priyawadhwa removed the kind/documentation Categorizes issue or PR as related to documentation. label Mar 30, 2020
@czunker
Copy link

czunker commented Apr 2, 2020

Hi,

I also had the problem with v1.9.0. But this workaround from #3818 helped.

@colinhoglund
Copy link
Contributor

colinhoglund commented Apr 3, 2020

I'm also having this issue in 1.9.0.

This used to work by following the instructions in https://minikube.sigs.k8s.io/docs/tutorials/using_psp/#tutorial. However, this method does not appear to work after the removal of addonmanager.

The only workaround I have found is by manually applying the PSP during minikube start once the control plane is available.

@sharifelgamal
Copy link
Collaborator

I can confirm this is still an issue in minikube 1.10.1

@knrt10
Copy link

knrt10 commented Jun 22, 2020

Sill issue in v1.11.0, but above PR will fix the issue most probably

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/addons kind/bug Categorizes issue or PR as related to a bug. priority/backlog Higher priority than priority/awaiting-more-evidence.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants