apiserver: stopped - connection to the server x:8443 was refused #3649

kamilgregorczyk · 2019-02-11T12:28:07Z

Minikube is constantly reseting/crashing/doing something with kubernetes api server. After running minikube, increasing cores to 8 and ram to 20 gb, installing helm and istio I'm getting random crashes which are super frustrating here are minikube logs: https://pastebin.com/fiPNnjz9

➜  ~ minikube status
host: Running
kubelet: Running
apiserver: Stopped
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

The connection to the server 192.168.99.100:8443 was refused - did you specify the right host or port?

Environment:

Minikube version: minikube version: v0.33.1

OS: MacOS
VM Driver: VirtualBox
ISO version: file:///Users/kgregorczyk/.minikube/cache/iso/minikube-v0.33.1.iso"
Install tools: helm & istio from helm

The text was updated successfully, but these errors were encountered:

tstromberg · 2019-02-12T23:14:02Z

Hey @kamilgregorczyk - sorry about minikube not working out for you so far. I suspect that the apiserver is being evicted, due to running out of resources for the footprint you are attempting to deploy, but we don't do a good job of showing it. Do you mind outputting some logs for me, and perhaps steps to replicate? It'd really help us to stabilize this.

minikube logs

kubectl get pods --all-namespaces

and:

minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

Thanks so much for your bug report!

fabianbaier · 2019-02-13T06:47:52Z

Same thing happens to me when running skaffold dev on a 6 core 8gb minikube cluster with hyperkit.

kubectl get pods --all-namespaces

Won't work as the apiserver is not running.

minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'

Returns

$ minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'
"docker logs" requires exactly 1 argument.
See 'docker logs --help'.

Usage:  docker logs [OPTIONS] CONTAINER

Fetch the logs of a container

When looking further into by minikube ssh in a healthy minikube, then running skaffold dev and have the issue replicate I am starting to see eviction messages in journalctl

Feb 13 07:16:43 minikube kubelet[2410]: E0213 07:16:43.769888    2410 reflector.go:251] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to watch *v1.Pod: Get https://localhost:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dminikube&resourceVersion=17030&timeoutSeconds=347&watch=true: dial tcp 127.0.0.1:8443: connect: connection refused
Feb 13 07:16:44 minikube kubelet[2410]: I0213 07:16:44.041581    2410 eviction_manager.go:563] eviction manager: pod kube-apiserver-minikube_kube-system(87f41e2e0629c3deb5c2239e08d8045d) is evicted successfully
Feb 13 07:16:44 minikube kubelet[2410]: I0213 07:16:44.041623    2410 eviction_manager.go:187] eviction manager: pods kube-apiserver-minikube_kube-system(87f41e2e0629c3deb5c2239e08d8045d) evicted, waiting for pod to be cleaned up
.
.
.
Feb 13 07:17:14 minikube kubelet[2410]: W0213 07:17:14.043001    2410 eviction_manager.go:392] eviction manager: timed out waiting for pods kube-apiserver-minikube_kube-system(87f41e2e0629c3deb5c2239e08d8045d) to be cleaned up
.
.
.
Feb 13 07:18:30 minikube kubelet[2410]: E0213 07:18:30.460108    2410 event.go:147] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-apiserver-minikube.1582db2c05e7b870", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-apiserver-minikube", UID:"87f41e2e0629c3deb5c2239e08d8045d", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers{kube-apiserver}"}, Reason:"Killing", Message:"Killing container with id docker://kube-apiserver:Need to kill Pod", Source:v1.EventSource{Component:"kubelet", Host:"minikube"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf110ef6f539ea70, ext:141711610374, loc:(*time.Location)(0x71d3440)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf110ef6f539ea70, ext:141711610374, loc:(*time.Location)(0x71d3440)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}' (retry limit exceeded!)

Why do we see evictions with pod kube-apiserver-minikube_kube-system(87f41e2e0629c3deb5c2239e08d8045d) is evicted successfully ?

Whats also maybe helpful is

Feb 13 07:14:22 minikube kubelet[2410]: E0213 07:14:22.330199    2410 kubelet.go:1308] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Feb 13 07:14:22 minikube kubelet[2410]: E0213 07:14:22.481301    2410 eviction_manager.go:243] eviction manager: failed to get summary stats: failed to get node info: node "minikube" not found

and

Feb 13 07:15:42 minikube kubelet[2410]: W0213 07:15:42.542247    2410 eviction_manager.go:329] eviction manager: attempting to reclaim ephemeral-storage
Feb 13 07:15:42 minikube kubelet[2410]: W0213 07:15:42.760098    2410 eviction_manager.go:414] eviction manager: unexpected error when attempting to reduce ephemeral-storage pressure: wanted to free 9223372036854775807 bytes, but freed 989661055 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ee867562a37d (must be forced) - image is being used by stopped container ca5d61bdb610, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 557d1c801957 (must be forced) - image is being used by stopped container 9ed6ed528ad3, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 3d7d78396776 (must be forced) - image is being used by stopped container a5c4e5c32fe6, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 77a332912e0d (must be forced) - image is being used by stopped container ff8ecbd50c79, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 213ee1099b0c (must be forced) - image is being used by stopped container 133abcf6c34a, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 1a8de823737d (must be forced) - image is being used by stopped container a6432e7fc825, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 52d894fca6d4 (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete bc8fb6e6e49d (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete a3040c587f52 (must be forced) - image is being used by stopped container 72a3783f5edf, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete c7c0e41e93f1 (must be forced) - image is being used by stopped container 3759185213a0, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 579404a7b407 (must be forced) - image is being used by stopped container c99f4a3c89e3, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 5a02f920193b (cannot be forced) - image has dependent child images, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete d4353dbbc50f (must be forced) - image is being used by stopped container d4c86f39d0b2, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete fe52e4bf5bc2 (must be forced) - image is being used by stopped container 02b7b443fe10, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 6e60582cf5d2 (must be forced) - image is being used by stopped container 1853de35a8e1, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete ed266c958895 (must be forced) - image is being used by stopped container 0fefcdb949fe, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 8cc61adda887 (must be forced) - image is being used by stopped container b096504f5e9b]
Feb 13 07:15:42 minikube kubelet[2410]: I0213 07:15:42.802271    2410 eviction_manager.go:358] eviction manager: pods ranked for eviction: kube-addon-manager-minikube_kube-system(5c72fb06dcdda608211b70d63c0ca488), kube-scheduler-minikube_kube-system(9729a196c4723b60ab401eaff722982d), kube-controller-manager-minikube_kube-system(a27cf7498262006bec6c9e79ae3ebd16), kube-apiserver-minikube_kube-system(87f41e2e0629c3deb5c2239e08d8045d), etcd-minikube_kube-system(75f0900b110d44d8e0930ffb50187ab7), trader-postgres-7687cd6f9c-td8nx_default(881670d9-2f4f-11e9-a92c-52406b8570cc), storage-provisioner_kube-system(35904133-2f2a-11e9-831d-52406b8570cc), trader-backend-f5dd485-mlg98_default(88c87f50-2f4f-11e9-a92c-52406b8570cc), coredns-86c58d9df4-pjz2f_kube-system(34bcf7d5-2f2a-11e9-831d-52406b8570cc), coredns-86c58d9df4-9brvs_kube-system(34bbf9fd-2f2a-11e9-831d-52406b8570cc), fabianbaier-prometheus-5dd7b6f9b7-sqs6f_default(91d739d4-2f4f-11e9-a92c-52406b8570cc), trader-scheduler-55d6df78ff-gnr7r_default(fa7e2756-2f53-11e9-a92c-52406b8570cc)

Is that maybe the clue? So we run out of storage and the eviction manager starts to evict pods, and the apiserver happens to be one of many? I think that would make sense...somehow...

I just don't get why storage is an issue:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.9G  638M  3.2G  17% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   17M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G  140K  3.9G   1% /tmp
/dev/vda1        16G   12G  2.5G  83% /mnt/vda1
$ sudo du -h -d 1 /var/lib/docker
20K	/var/lib/docker/plugins
4.0K	/var/lib/docker/runtimes
72K	/var/lib/docker/buildkit
12G	/var/lib/docker/overlay2
648K	/var/lib/docker/containerd
4.0K	/var/lib/docker/swarm
5.1M	/var/lib/docker/containers
2.3M	/var/lib/docker/volumes
18M	/var/lib/docker/image
20K	/var/lib/docker/builder
4.0K	/var/lib/docker/trust
4.0K	/var/lib/docker/tmp
108K	/var/lib/docker/network
12G	/var/lib/docker

I should have at least 4G available for my docker images.

kamilgregorczyk · 2019-02-13T09:19:38Z

@tstromberg You might be right, I had istio running. I can't give you logs as I deleted my VM and moved to GCP with my experiments. I did:

Run minikube
Install helm with tiller account
Stop minikube and assign 8 cores and 21 gb of ram
Start minikube
Install istio with all the features helm install install/kubernetes/helm/istio --name istio --namespace istio-system --set prometheus.service.nodePort.enabled=true --set tracing.enabled=true --set tracing.service.type=NodePort --set kiali.enabled=true --set grafana.enabled=true --set grafana.service.type=NodePort

and after some time I started getting these dropped connections

fabianbaier · 2019-02-13T22:09:54Z

I was able to reproduce the issue. So it seems by default I have in /dev/vda1 16G and if that hits a threshold (around 12G) it sets off the eviction of those vital processes.

To replicate try this:

$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.9G  638M  3.2G  17% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  8.8M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G   16K  3.9G   1% /tmp
/dev/vda1        16G  859M   14G   6% /mnt/vda1
$ sudo fallocate -l 12G /mnt/vda1/12gb
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          3.9G  638M  3.2G  17% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  8.8M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           3.9G   20K  3.9G   1% /tmp
/dev/vda1        16G   13G  1.6G  90% /mnt/vda1

The output of kubectl get events -w looks like this:

$ kubectl get events -w
LAST SEEN   TYPE     REASON                    KIND   MESSAGE
46s         Normal   NodeHasSufficientMemory   Node   Node minikube status is now: NodeHasSufficientMemory
46s         Normal   NodeHasNoDiskPressure     Node   Node minikube status is now: NodeHasNoDiskPressure
46s         Normal   NodeHasSufficientPID      Node   Node minikube status is now: NodeHasSufficientPID
18s         Normal   RegisteredNode            Node   Node minikube event: Registered Node minikube in Controller
16s         Normal   Starting                  Node   Starting kube-proxy.
0s    Warning   ImageGCFailed   Node   failed to garbage collect required amount of images. Wanted to free 1671698841 bytes, but freed 0 bytes
E0213 14:06:27.209682    5918 streamwatcher.go:109] Unable to decode an event from the watch stream: http2: server sent GOAWAY and closed the connection; LastStreamID=63, ErrCode=NO_ERROR, debug=""

And sometimes like this:

.
.
0s    Warning   EvictionThresholdMet   Node   Attempting to reclaim ephemeral-storage
0s    Normal   NodeHasDiskPressure   Node   Node minikube status is now: NodeHasDiskPressure
0s    Warning   EvictionThresholdMet   Node   Attempting to reclaim ephemeral-storage
0s    Warning   EvictionThresholdMet   Node   Attempting to reclaim ephemeral-storage
0s    Warning   EvictionThresholdMet   Node   Attempting to reclaim ephemeral-storage

This is when the apiserver got evicted. When then running kubectl get pods I get:

The connection to the server 192.168.64.9:8443 was refused - did you specify the right host or port?

mayankpundir27 · 2019-06-13T16:49:15Z

mayankpundir27 · 2019-06-13T16:49:53Z

minikube is constanty crashing. Please give some solutions.

linkavich14 · 2019-07-08T02:15:37Z

What was the solution ?
My /dev/vda1 is only 18% and keeps crashing

irrgit · 2019-07-09T13:58:18Z

Had the same happen to me recently, minikube was running with vm-driver=none . It was up for over 28 days without issue, so I am not exactly sure what caused it to come down. Stopping and starting it again fixed the problem but it would be nice to know what to do to prevent it from happening again in the future.

NanXuejiao · 2019-07-25T03:34:42Z

I faced the same thing, after restarting one my own pod, minikube apiserver turned to Stopped, serval times waiting for a long time apiserver restart well with "Running" status, but many times it always is Stopped. The following is its log.

$ minikube ssh 'docker logs $(docker ps -a -f name=k8s_kube-api --format={{.ID}})'
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0725 08:34:13.148972       1 server.go:557] external host was not specified, using 192.168.99.101
I0725 08:34:13.149096       1 server.go:146] Version: v1.13.4
I0725 08:34:13.640528       1 initialization.go:91] enabled Initializers feature as part of admission plugin setup
I0725 08:34:13.641385       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0725 08:34:13.641459       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0725 08:34:13.642174       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0725 08:34:13.642301       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0725 08:34:13.772310       1 master.go:228] Using reconciler: lease
W0725 08:34:14.812254       1 genericapiserver.go:338] Skipping API batch/v2alpha1 because it has no resources.
W0725 08:34:14.960597       1 genericapiserver.go:338] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0725 08:34:14.968874       1 genericapiserver.go:338] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0725 08:34:14.980357       1 genericapiserver.go:338] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0725 08:34:15.074086       1 genericapiserver.go:338] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2019/07/25 08:34:15 log.go:33: [restful/swagger] listing is available at https://192.168.99.101:8443/swaggerapi
[restful] 2019/07/25 08:34:15 log.go:33: [restful/swagger] https://192.168.99.101:8443/swaggerui/ is mapped to folder /swagger-ui/

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        934M     0  934M   0% /dev
tmpfs           996M     0  996M   0% /dev/shm
tmpfs           996M   22M  974M   3% /run
tmpfs           996M     0  996M   0% /sys/fs/cgroup
tmpfs           996M  300K  996M   1% /tmp
/dev/sda1        17G  2.7G   14G  17% /mnt/sda1
/Users          466G  122G  345G  27% /Users

jpninanjohn · 2019-10-09T03:28:26Z

You might be right, I had istio running. I can't give you logs as I deleted my VM and moved to GCP with my experiments. I did:

I also faced this issue after I installed istio. I had to delete my minikube and create it again. Fingers crossed it wont happen again.

prabsdubey · 2020-10-25T06:27:33Z

After installing , apiserver is not coming up(apiserver: Stopped)
$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Stopped
kubeconfig: Configured

prabs@LAPTOP-HQ5LK73I ~
$ pwd
/home/prabs

prabs@LAPTOP-HQ5LK73I ~
$ kubectl
kubectl controls the Kubernetes cluster manager.

Find more information at:
https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
create Create a resource from a file or from stdin.
expose Take a replication controller, service, deployment or pod and
expose it as a new Kubernetes Service
run Run a particular image on the cluster
set Set specific features on objects

Basic Commands (Intermediate):
explain Documentation of resources
get Display one or many resources
edit Edit a resource on the server
delete Delete resources by filenames, stdin, resources and names, or by

resources and label selector

Deploy Commands:
rollout Manage the rollout of a resource
scale Set a new size for a Deployment, ReplicaSet or Replication
Controller
autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController

Cluster Management Commands:
certificate Modify certificate resources.
cluster-info Display cluster info
top Display Resource (CPU/Memory/Storage) usage.
cordon Mark node as unschedulable
uncordon Mark node as schedulable
drain Drain node in preparation for maintenance
taint Update the taints on one or more nodes

Troubleshooting and Debugging Commands:
describe Show details of a specific resource or group of resources
logs Print the logs for a container in a pod
attach Attach to a running container
exec Execute a command in a container
port-forward Forward one or more local ports to a pod
proxy Run a proxy to the Kubernetes API server
cp Copy files and directories to and from containers.
auth Inspect authorization

Advanced Commands:
diff Diff live version against would-be applied version
apply Apply a configuration to a resource by filename or stdin
patch Update field(s) of a resource using strategic merge patch
replace Replace a resource by filename or stdin
wait Experimental: Wait for a specific condition on one or many
resources.
convert Convert config files between different API versions
kustomize Build a kustomization target from a directory or a remote url.

Settings Commands:
label Update the labels on a resource
annotate Update the annotations on a resource
completion Output shell completion code for the specified shell (bash or
zsh)

Other Commands:
alpha Commands for features in alpha
api-resources Print the supported API resources on the server
api-versions Print the supported API versions on the server, in the form of
"group/version"
config Modify kubeconfig files
plugin Provides utilities for interacting with plugins.
version Print the client and server version information

Usage:
kubectl [flags] [options]

Use "kubectl --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all
commands).

prabs@LAPTOP-HQ5LK73I ~
$ minikube
minikube provisions and manages local Kubernetes clusters optimized for
development workflows.

Basic Commands:
start Starts a local Kubernetes cluster
status Gets the status of a local Kubernetes cluster
stop Stops a running local Kubernetes cluster
delete Deletes a local Kubernetes cluster
dashboard Access the Kubernetes dashboard running within the minikube
cluster
pause pause Kubernetes
unpause unpause Kubernetes

Images Commands:
docker-env Configure environment to use minikube's Docker daemon
podman-env Configure environment to use minikube's Podman service
cache Add, delete, or push a local image into minikube

Configuration and Management Commands:
addons Enable or disable a minikube addon
config Modify persistent configuration values
profile Get or list the current profiles (clusters)
update-context Update kubeconfig in case of an IP or port change

Networking and Connectivity Commands:
service Returns a URL to connect to a service
tunnel Connect to LoadBalancer services

Advanced Commands:
mount Mounts the specified directory into minikube
ssh Log into the minikube environment (for debugging)
kubectl Run a kubectl binary matching the cluster version
node Add, remove, or list additional nodes

Troubleshooting Commands:
ssh-key Retrieve the ssh identity key path of the specified cluster
ip Retrieves the IP address of the running cluster
logs Returns logs to debug a local Kubernetes cluster
update-check Print current and latest version number
version Print the version of minikube

Other Commands:
completion Generate command completion for a shell

Use "minikube --help" for more information about a given command.