none: tunnel doesn't cleanup routes

[tstromberg]

While inspecting go/bugbear/logs/Linux_Integration_Tests_none/builds/1850 - I noticed that the tunnel doesn't seem to be cleaning up after itself. Here is where the tunnel gets created:

>> out/e2e-linux-amd64 exited with 0 at Thu Dec 6 17:54:37 UTC 2018
...
++ status=success
++ echo 'minikube: SUCCESS'
minikube: SUCCESS
++ echo '>> Cleaning up after ourselves ...'
>> Cleaning up after ourselves ...
++ sudo -E out/minikube-linux-amd64 tunnel --cleanup
++ sudo -E out/minikube-linux-amd64 delete
++ true
++ cleanup_stale_routes
++ local 'show=netstat -rn -f inet'
++ local 'del=sudo route -n delete'
+++ uname
++ [[ Linux == \L\i\n\u\x ]]
++ show='ip route show'
++ del='sudo ip route delete'
+++ ip route show
+++ awk '{ print $1 }'
+++ grep 10.96.0.0
++ local troutes=10.96.0.0/12
++ for route in ${troutes}
++ echo 'WARNING: deleting stale tunnel route: 10.96.0.0/12'
WARNING: deleting stale tunnel route: 10.96.0.0/12
++ sudo ip route delete 10.96.0.0/12

The cancellation of "minikube tunnel", "minikube tunnel --cleanup" and "minikube delete" calls all should have arguably deleted the route. Here's what the tunnel process that added the tunnel:

type=CWD msg=audit(1544118759.794:3358): cwd="/home/jenkins/workspace/Linux_Integration_Tests_none"
type=EXECVE msg=audit(1544118759.794:3358): argc=6 a0="/home/jenkins/workspace/Linux_Integration_Tests_none/out/minikube-linux-amd64" a1="tunnel" a2="--alsologtostderr" a3="-v" a4="8" a5="--logtostderr"
type=SYSCALL msg=audit(1544118759.794:3358): arch=c000003e syscall=59 success=yes exit=0 a0=c42018e410 a1=c42001c980 a2=c42035e800 a3=0 items=2 ppid=7952 pid=9256 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="minikube-linux-" exe="/home/jenkins/workspace/Linux_Integration_Tests_none/out/minikube-linux-amd64" key="root-commands"

Here are all the commands 'minikube-linux-amd64 tunnel' executed. Notice the route add, but not route delete:

ausearch  -ts 12/6/2018 17:40:00 -te 12/6/2018 17:54:37 -m EXECVE -pp 9256 | egrep -o "argc.*" | sort -u
argc=4 a0="netstat" a1="-nr" a2="-f" a3="inet"
argc=4 a0="sudo" a1="/bin/bash" a2="-c" a3=7375646F2073797374656D63746C2069732D616374697665206B7562656C657420263E2F6465762F6E756C6C202626206563686F202252756E6E696E6722207C7C206563686F202253746F7070656422
argc=7 a0="sudo" a1="ip" a2="route" a3="add" a4="10.96.0.0/12" a5="via" a6="10.128.0.3"

minikube tunnel --cleanup didn't seem to run any commands:

# ausearch  -ts 12/6/2018 17:40:00 -te 12/6/2018 17:54:37 -m EXECVE -pp 16902 
----
time->Thu Dec  6 17:54:37 2018
type=PROCTITLE msg=audit(1544118877.161:7036): proctitle=6F75742F6D696E696B7562652D6C696E75782D616D6436340074756E6E656C002D2D636C65616E7570
type=PATH msg=audit(1544118877.161:7036): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=144244 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1544118877.161:7036): item=0 name="out/minikube-linux-amd64" inode=14025032 dev=08:01 mode=0100755 ouid=1004 ogid=1005 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1544118877.161:7036): cwd="/home/jenkins/workspace/Linux_Integration_Tests_none"
type=EXECVE msg=audit(1544118877.161:7036): argc=3 a0="out/minikube-linux-amd64" a1="tunnel" a2="--cleanup"
type=SYSCALL msg=audit(1544118877.161:7036): arch=c000003e syscall=59 success=yes exit=0 a0=55ae07cc10a8 a1=55ae07cb3068 a2=55ae07cc8250 a3=7f17c3c6b82c items=2 ppid=16902 pid=16904 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="minikube-linux-" exe="/home/jenkins/workspace/Linux_Integration_Tests_none/out/minikube-linux-amd64" key="root-commands"

# ausearch  -ts 12/6/2018 17:40:00 -te 12/6/2018 17:54:37 -m EXECVE -pp 16904
<no matches>

minikube delete didn't run any route commands:

# ausearch  -ts 12/6/2018 17:40:00 -te 12/6/2018 17:54:37 -m EXECVE -pp 16917
----
time->Thu Dec  6 17:54:37 2018
type=PROCTITLE msg=audit(1544118877.253:7043): proctitle=6F75742F6D696E696B7562652D6C696E75782D616D6436340064656C657465
type=PATH msg=audit(1544118877.253:7043): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=144244 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1544118877.253:7043): item=0 name="out/minikube-linux-amd64" inode=14025032 dev=08:01 mode=0100755 ouid=1004 ogid=1005 rdev=00:00 nametype=NORMAL
type=CWD msg=audit(1544118877.253:7043): cwd="/home/jenkins/workspace/Linux_Integration_Tests_none"
type=EXECVE msg=audit(1544118877.253:7043): argc=2 a0="out/minikube-linux-amd64" a1="delete"
type=SYSCALL msg=audit(1544118877.253:7043): arch=c000003e syscall=59 success=yes exit=0 a0=55afcbf670a8 a1=55afcbf59068 a2=55afcbf6cbc0 a3=7fd05476d82c items=2 ppid=16917 pid=16918 auid=1004 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="minikube-linux-" exe="/home/jenkins/workspace/Linux_Integration_Tests_none/out/minikube-linux-amd64" key="root-commands"
root@kvm-integration-slave:/home/tstromberg# ausearch  -ts 12/6/2018 17:40:00 -te 12/6/2018 17:54:37 -m EXECVE -pp 16918
<no matches>

After tunnel creation, the only commands I see calling "netstat" or "ip" is the looping call to netstat from the original tunnel process:

id=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=1 comm="netstat" exe="/bin/netstat" key="root-commands"

And the commands I call in common.sh to cleanup the tunnel:

type=EXECVE msg=audit(1544118877.337:7047): argc=3 a0="ip" a1="route" a2="show"
type=EXECVE msg=audit(1544118877.341:7050): argc=5 a0="sudo" a1="ip" a2="route" a3="delete" a4="10.96.0.0/12"
type=EXECVE msg=audit(1544118877.345:7054): argc=4 a0="ip" a1="route" a2="delete" a3="10.96.0.0/12"

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

Closing as stale.

[Chugarah]

Hi, I had this problem aswell on Win 10 Minikube 1.4.0. Tried with
minikube tunnel --cleanup --profile "ECE-Minikube-Wordpress""
and it didnt worked.

If someone else have this problem I managed to do this manually in Powershell terminal as administrator

To display your routes and found default minikube network "10.96.0.0"
route PRINT

Deleting it
route -p delete 10.96.0.0

Now I can start minikube tunnel --profile "ECE-Minikube-Wordpress" as usual and no erros apear :).