[BUG] Error waiting for apiserver (macOS M1)

[murphye]

General information

• OS: macOS
• Hypervisor:
• Did you run crc setup before starting it (Yes)?
• Running CRC on: Laptop

CRC version

OpenShift version: 4.14.7
Podman version: 4.4.4

CRC status

CRC VM:          Running
OpenShift:       Unreachable (v4.14.7)
RAM Usage:       1.061GB of 33.59GB
Disk Usage:      18.95GB of 32.68GB (Inside the CRC VM)
Cache Usage:     38.3GB

CRC config

- consent-telemetry                     : no
- cpus                                  : 8
- memory                                : 32768

Host Operating System

ProductName:		macOS
ProductVersion:		14.2.1
BuildVersion:		23C71

Steps to reproduce

1. crc start

Expected

Successful startup

Actual

Failed startup

Logs

crc --log-level=debug start
DEBU CRC version: 2.31.0+6d23b6                   
DEBU OpenShift version: 4.14.7                    
DEBU Podman version: 4.4.4                        
DEBU Running 'crc start'                          
DEBU Total memory of system is 68719476736 bytes  
DEBU No new version available. The latest version is 2.31.0 
DEBU Checking file: /Users/eric/.crc/machines/crc/.crc-exist 
INFO Using bundle path /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64.crcbundle 
INFO Checking if running as non-root              
INFO Checking if crc-admin-helper executable is cached 
INFO Checking if running on a supported CPU architecture 
DEBU GOARCH is arm64 GOOS is darwin               
INFO Checking if crc executable symlink exists    
INFO Checking minimum RAM requirements            
DEBU Total memory of system is 68719476736 bytes  
INFO Checking if running emulated on Apple silicon 
INFO Checking if vfkit is installed               
INFO Checking if old launchd config for tray and/or daemon exists 
INFO Checking if crc daemon plist file is present and loaded 
DEBU Running 'bash -c launchctl list | grep com.redhat.crc.daemon | awk '{print $1}'' 
DEBU Checking file: /Users/eric/.crc/machines/crc/.crc-exist 
INFO Loading bundle: crc_vfkit_4.14.7_arm64...    
DEBU Checking file: /Users/eric/.crc/machines/crc/.crc-exist 
INFO Starting CRC VM for openshift 4.14.7...      
DEBU Updating CRC VM configuration                
DEBU kern.osproductversion is: 14.2.1             
DEBU Running /usr/local/crc/vfkit --cpus 8 --memory 32768 --kernel /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/vmlinuz-5.14.0-70.80.1.el9_0.aarch64 --initrd /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/initramfs-5.14.0-70.80.1.el9_0.aarch64.img --kernel-cmdline console=hvc0 ignition.platform.id=qemu $ignition_firstboot ostree=/ostree/boot.1/rhcos/b7705dc4c78fc1b8bdb29112494acb3eb8bf8faa30dc193ac0d524f4471a49d3/0 root=UUID=a1d0a565-661a-4732-8fbe-8242d1f4e28c rw rootflags=prjquota boot=UUID=9b15c22e-c6df-400d-99aa-077c7552627b systemd.unified_cgroup_hierarchy=1 cgroup_no_v1="all" psi=1
 --device virtio-serial,logFilePath=/Users/eric/.crc/machines/crc/vfkit.log --device virtio-fs,sharedDir=/Users/eric,mountTag=dir0 --device virtio-rng --device virtio-blk,path=/Users/eric/.crc/machines/crc/crc.img --device virtio-vsock,port=1024,socketURL=/Users/eric/.crc/tap.sock,listen --timesync vsockPort=1234 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="&{8 32768 /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/vmlinuz-5.14.0-70.80.1.el9_0.aarch64 console=hvc0 ignition.platform.id=qemu $ignition_firstboot ostree=/ostree/boot.1/rhcos/b7705dc4c78fc1b8bdb29112494acb3eb8bf8faa30dc193ac0d524f4471a49d3/0 root=UUID=a1d0a565-661a-4732-8fbe-8242d1f4e28c rw rootflags=prjquota boot=UUID=9b15c22e-c6df-400d-99aa-077c7552627b systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=\"all\" psi=1\n /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/initramfs-5.14.0-70.80.1.el9_0.aarch64.img {[] false} vsockPort=1234 [virtio-serial,logFilePath=/Users/eric/.crc/machines/crc/vfkit.log virtio-fs,sharedDir=/Users/eric,mountTag=dir0 virtio-rng virtio-blk,path=/Users/eric/.crc/machines/crc/crc.img virtio-vsock,port=1024,socketURL=/Users/eric/.crc/tap.sock,listen] none://  false}" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="boot parameters: &{VmlinuzPath:/Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/vmlinuz-5.14.0-70.80.1.el9_0.aarch64 KernelCmdLine:console=hvc0 ignition.platform.id=qemu $ignition_firstboot ostree=/ostree/boot.1/rhcos/b7705dc4c78fc1b8bdb29112494acb3eb8bf8faa30dc193ac0d524f4471a49d3/0 root=UUID=a1d0a565-661a-4732-8fbe-8242d1f4e28c rw rootflags=prjquota boot=UUID=9b15c22e-c6df-400d-99aa-077c7552627b systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=\"all\" psi=1\n InitrdPath:/Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/initramfs-5.14.0-70.80.1.el9_0.aarch64.img}" 
DEBU time="2024-01-20T19:30:19-08:00" level=info  
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="virtual machine parameters:" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="\tvCPUs: 8" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="\tmemory: 32768 MiB" 
DEBU time="2024-01-20T19:30:19-08:00" level=info  
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Adding virtio-serial device (logFile: /Users/eric/.crc/machines/crc/vfkit.log)" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Adding virtio-fs device" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Adding virtio-rng device" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Adding virtio-blk device (imagePath: /Users/eric/.crc/machines/crc/crc.img)" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Adding virtio-vsock device" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="virtual machine is running" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Exposing vsock port 1024 on /Users/eric/.crc/tap.sock (listening)" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="Setting up host/guest time synchronization" 
DEBU time="2024-01-20T19:30:19-08:00" level=info msg="waiting for VM to stop" 
DEBU Waiting for machine to be running, this may take a few minutes... 
DEBU retry loop: attempt 0                        
DEBU Machine is up and running!                   
INFO CRC instance is running with IP 127.0.0.1    
DEBU Waiting until ssh is available               
DEBU retry loop: attempt 0                        
DEBU Running SSH command: exit 0                  
DEBU Using ssh private keys: [/Users/eric/.crc/machines/crc/id_ecdsa /Users/eric/.crc/cache/crc_vfkit_4.14.7_arm64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output:     
INFO CRC VM is running                            
DEBU Using root access: disable core user password 
DEBU Running SSH command: sudo passwd --lock core 
DEBU SSH command results: err: <nil>, output: Locking password for user core.
passwd: Success 
DEBU Running SSH command: cat /home/core/.ssh/authorized_keys 
DEBU SSH command results: err: <nil>, output: ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAAvNN8K7vOfjBRopSetRVok1FM+BC3v+zRDRQQ0FDAfaKE0LOL5x8lalCcrRegGVzdDujqun/oc7+L0BIt4SrHnSACJn59OpDyS4QSz/8DxsBryPN9I3YeCgIH+THNGq4xoCN0sIvc2JbGdPpTqEVflGVRQBWpbveAsG75ql9fK0TEU6Q== 
DEBU Using root access: Get device id             
DEBU Running SSH command: sudo /usr/sbin/blkid -t TYPE=xfs -o device 
DEBU SSH command results: err: <nil>, output: /dev/vda4 
DEBU Using root access: Growing /dev/vda4 partition 
DEBU Running SSH command: sudo /usr/bin/growpart /dev/vda 4 
DEBU SSH command results: err: Process exited with status 1, output: NOCHANGE: partition 4 is size 63961055. it cannot be grown 
DEBU No free space after /dev/vda4, nothing to do 
DEBU Configuring shared directories               
INFO Configuring shared directories               
DEBU Using root access: Making / mutable          
DEBU Running SSH command: sudo chattr -i /        
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Creating /Users/eric      
DEBU Running SSH command: sudo mkdir -p /Users/eric 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Making / immutable again  
DEBU Running SSH command: sudo chattr +i /        
DEBU SSH command results: err: <nil>, output:     
DEBU Mounting tag dir0 at /Users/eric             
DEBU Using root access: Mounting /Users/eric      
DEBU Running SSH command: sudo mount -o context="system_u:object_r:container_file_t:s0" -t virtiofs dir0 /Users/eric 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: make root Podman socket accessible 
DEBU Running SSH command: sudo chmod 777 /run/podman/ /run/podman/podman.sock 
DEBU SSH command results: err: <nil>, output:     
DEBU Running '/usr/local/crc/crc-admin-helper-darwin rm api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing downloads-openshift-console.apps-crc.testing canary-openshift-ingress-canary.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing' 
DEBU Running '/usr/local/crc/crc-admin-helper-darwin add 127.0.0.1 api.crc.testing oauth-openshift.apps-crc.testing console-openshift-console.apps-crc.testing downloads-openshift-console.apps-crc.testing canary-openshift-ingress-canary.apps-crc.testing default-route-openshift-image-registry.apps-crc.testing' 
DEBU Creating /etc/resolv.conf with permissions 0644 in the CRC VM 
DEBU Running SSH command: <hidden>                
DEBU SSH command succeeded                        
DEBU retry loop: attempt 0                        
DEBU Running SSH command: host -R 3 foo.apps-crc.testing 
DEBU SSH command results: err: <nil>, output: foo.apps-crc.testing has address 192.168.127.2 
INFO Check internal and public DNS query...       
DEBU Running SSH command: curl --head quay.io     
DEBU SSH command results: err: <nil>, output: HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Sun, 21 Jan 2024 03:30:30 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Location: https://quay.io:443/
 
INFO Check DNS query from host...                 
DEBU api.crc.testing resolved to [127.0.0.1]      
INFO Verifying validity of the kubelet certificates... 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-01-03T15:06:46+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-01-03T15:06:58+00:00 
DEBU Running SSH command: date --date="$(sudo openssl x509 -in /etc/kubernetes/static-pod-resources/kube-apiserver-certs/configmaps/aggregator-client-ca/ca-bundle.crt -noout -enddate | cut -d= -f 2)" --iso-8601=seconds 
DEBU SSH command results: err: <nil>, output: 2025-01-03T15:10:24+00:00 
INFO Starting kubelet service                     
DEBU Using root access: Executing systemctl daemon-reload command 
DEBU Running SSH command: sudo systemctl daemon-reload 
DEBU SSH command results: err: <nil>, output:     
DEBU Using root access: Executing systemctl start kubelet 
DEBU Running SSH command: sudo systemctl start kubelet 
DEBU SSH command results: err: <nil>, output:     
INFO Waiting for kube-apiserver availability... [takes around 2min] 
DEBU retry loop: attempt 0                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 1                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 2                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 3                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 4                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 5                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 6                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 7                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU Get "https://api.crc.testing:6443/api/v1/nodes?limit=500": dial tcp 192.168.127.2:6443: connect: connection refused - error from a previous attempt: read tcp 192.168.127.2:40136->192.168.127.2:6443: read: connection reset by peer 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 8                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 9                        
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 10                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 11                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 12                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 13                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 14                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 15                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 16                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 17                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 18                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 19                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 20                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 21                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 22                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 23                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 24                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 25                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 26                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 124, output:  
DEBU                                              
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 - sleeping 1s 
DEBU retry loop: attempt 27                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU Get "https://api.crc.testing:6443/api/v1/nodes?limit=500": dial tcp 192.168.127.2:6443: connect: connection refused - error from a previous attempt: read tcp 192.168.127.2:42522->192.168.127.2:6443: read: connection reset by peer 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 28                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 29                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 30                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 31                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 32                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 33                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 34                       
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 

...

DEBU retry loop: attempt 124                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 125                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 126                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 127                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 128                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 129                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 130                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 131                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 132                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 133                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 134                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 135                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 136                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 137                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 138                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU retry loop: attempt 139                      
DEBU Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig 
DEBU SSH command results: err: Process exited with status 1, output:  
DEBU The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port? 
DEBU error: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 - sleeping 1s 
DEBU RetryAfter timeout after 140 tries           
Error waiting for apiserver: Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x3)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1

Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x3)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 (x17)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x3)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 (x29)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x3)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1
 (x42)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 124
 (x3)
Temporary error: ssh command error:
command : timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig
err     : Process exited with status 1

[anjannath]

@murphye from the logs it seems you are trying to start an existing crc instance, if you can afford to lose it, could you please try to cleanup and then start again from a clean env, that might fix the issue for you..

$ crc cleanup 
$ crc setup
$ crc start

[murphye]

Yes that resolved the issue. Thanks.

It would be good to have error handling to prevent this scenario, whatever caused it.

[anjannath]

@murphye from the logs it seems you are trying to start an existing crc instance, if you can afford to lose it, could you please try to cleanup and then start again from a clean env, that might fix the issue for you..

$ crc cleanup 
$ crc setup
$ crc start

[murphye]

Yes that resolved the issue. Thanks.

It would be good to have error handling to prevent this scenario, whatever caused it.