Cannot delete pods completely with kubernetes

[ddongchen]

Description of problem

I tested the cri-o with clearcontainers runtime on kubernetes, and found the pod cannot be deleted completely.
(replace this text with the list of steps you followed)

1. Using kubernetes to create the pod
   [xxx]# ./cluster/kubectl.sh get pod
   NAME READY STATUS RESTARTS AGE
   pod-test 1/1 Running 0 11m

we can also see from crictl:
[xxx]# crictl pods
PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
17cd228356cf7 11 minutes ago SANDBOX_READY pod-test default 0
[xxx]# crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
9c25758ceb0a3 xxx 9 minutes ago CONTAINER_RUNNING pod-t 0

2. Deleting the pod from kubernetes:
   [xxx]# ./cluster/kubectl.sh delete pod pod-test
   pod "pod-test" deleted
   [xxx]# ./cluster/kubectl.sh get pod
   No resources found.

The pod is deleted from kubernetes successfully, while in fact, the pod is in "NotReady" state.
[xxx]# crictl pods
PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
17cd228356cf7 14 minutes ago SANDBOX_NOTREADY pod-test default 0
[xxx]# crictl ps -a
CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT

We delete the pod using "criclt", and failed.
[xxx]# crictl rmp 17c
FATA[0005] removing the pod sandbox failed: rpc error: code = Unknown desc = failed to delete container k8s_POD_pod-test_default_08acddcb-ff60-11e7-b62e-f0921cee3497_0 in pod sandbox 17cd228356cf7b872d316a5b2e0f0f4dccc0981646b80bcd919ff7380c186014: /usr/bin/cc-runtime delete --force 17cd228356cf7b872d316a5b2e0f0f4dccc0981646b80bcd919ff7380c186014 failed: failed to connect to proxy after 5s: dial unix /run/virtcontainers/pods/17cd228356cf7b872d316a5b2e0f0f4dccc0981646b80bcd919ff7380c186014/proxy.sock: connect: connection refused
(exit status 1)

3. if we just use "crictl" to stop the pod, it's successful.
   [xxx]# crictl pods
   PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT
   08182b3a31ad9 2 minutes ago SANDBOX_READY pod-test default 0
   [xxx]# crictl ps -a
   CONTAINER ID IMAGE CREATED STATE NAME ATTEMPT
   bb65bbe26e757 xx 9 seconds ago CONTAINER_RUNNING pod-t 0
   [xxx]# crictl stop bb65
   bb65
   [xxx]# crictl rm bb65
   bb65
   [xxx]# crictl stopp 0818
   Stopped sandbox 0818
   [xxx]# crictl rmp 0818
   Removed sandbox 0818
   [xxx]# crictl pods
   PODSANDBOX ID CREATED STATE NAME NAMESPACE ATTEMPT

4. I check the log from "crio" daemon, there is not so much differences. While i check the runtime log, i found the differences:
   (1) When using "crictl", the last step is to destroy vm, while the cri-o does not have these cmd:

Jan 22 16:23:19 cc-proxy[17435]: time="2018-01-22T16:23:19.245629005+08:00" level=info msg="hyper(cmd="destroypod", data="")" client=13 name=cc-proxy pid=17435 source=proxy
.....
Jan 22 16:23:19 cc-runtime[18637]: time="2018-01-22T16:23:19+08:00" level=info msg="Stopping VM"
......
time="2018-01-22T16:23:19.382775035+08:00" level=info msg="UnregisterVM()" client=14 name=cc-proxy pid=17435 source=proxy

(2) I found there is "stopping and stopped shims", while in cri-o, i just not found "stopped shims":

Jan 22 16:23:19 cc-runtime[18637]: time="2018-01-22T16:23:19+08:00" level=info msg="Stopping shim" shim-pid=17473 source=virtcontainers subsystem=shim
Jan 22 16:23:19 d cc-runtime[18637]: time="2018-01-22T16:23:19+08:00" level=info msg="Stopped shims" pod-id=5de6e8195caf73c1e9691a1d1503e1b40f211b873694aaa538a30eb1fb103662 shim-count=1 source=virtcontainers subsystem=pod

I'm not sure if the issue should be cri-o or clear containers. I also open the issue for cri-o(cri-o/cri-o#1279).

5. Version
   [xxxx]# cc-runtime cc-env
   [Meta]
   Version = "1.0.6"

[Runtime]
Debug = false
[Runtime.Version]
Semver = "3.0.12"
Commit = "be7d1cebb5e8afbb2a9329264d4163adc70b0362-dirty"
OCI = "1.0.0-dev"
[Runtime.Config]
Path = "/usr/share/defaults/clear-containers/configuration.toml"

[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
Path = "/usr/local/bin/qemu-system-x86_64"
Debug = false

[Image]
Path = "/usr/share/clear-containers/clear-19790-containers.img"

[Kernel]
Path = "/usr/share/clear-containers/vmlinuz-4.9.60-82.1.container"
Parameters = ""

[Proxy]
Type = "ccProxy"
Version = "Version: 3.0.12+git.3c6daa6"
Path = "/usr/libexec/clear-containers/cc-proxy"
Debug = true

[Shim]
Type = "ccShim"
Version = "shim version: 3.0.12 (commit: d01f9a7)"
Path = "/usr/libexec/clear-containers/cc-shim"
Debug = true

[Agent]
Type = "hyperstart"
Version = "<>"

[Host]
Kernel = "3.10.0-327.22.2.el7.x86_64"
CCCapable = true
[Host.Distro]
Name = "CentOS Linux"
Version = "7"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz"

[xxx]# docker --version
Docker version 17.09.1-ce, build 19e2cf6

[xxx]# uname -a
Linux 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Expected result

(replace this text with an explanation of what you thought would happen)

Actual result

(replace this text with details of what actually happened)

---

(replace this text with the output of the cc-collect-data.sh script, after you have
reviewed its content to ensure it does not contain any private information).

[chavafg]

This might be related with #571