Enable gvisor addon in minikube #3399

priyawadhwa · 2018-12-04T01:18:09Z

This PR adds the code for enabling gvisor in minikube. It adds the pod
that will run when the addon is enabled, and the code for the image
which will run when this happens.

When gvisor is enabled, the pod will download runsc and the
gvisor-containerd-shim. It will replace the containerd config.toml and
restart containerd.

When gvisor is disabled, the pod will be deleted by the addon manager.
This will trigger a pre-stop hook which will revert the config.toml to
it's original state and restart containerd.

TODO:

Add integration test
Add docs
Fail sanely when using a runtime that isn't containerd

…ainerd

This PR adds the code for enabling gvisor in minikube. It adds the pod that will run when the addon is enabled, and the code for the image which will run when this happens. When gvisor is enabled, the pod will download runsc and the gvisor-containerd-shim. It will replace the containerd config.toml and restart containerd. When gvisor is disabled, the pod will be deleted by the addon manager. This will trigger a pre-stop hook which will revert the config.toml to it's original state and restart containerd.

k8s-ci-robot · 2018-12-04T01:18:15Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: priyawadhwa
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: dlorenc

If they are not already assigned, you can assign the PR to them by writing /assign @dlorenc in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tstromberg · 2018-12-04T01:56:27Z

@minikube-bot OK to test

hyperkit issue seems interesting, but more likely due to the recently merged #3332

deploy/addons/gvisor/gvisor-containerd-shim.toml

pkg/gvisor/enable.go

docs/gvisor.md

ianlewis · 2018-12-04T14:14:35Z

docs/gvisor.md

+To run a pod in gVisor, add this annotation to the Kubernetes yaml:
+
+```
+io.kubernetes.cri.untrusted-workload: "true"


This is because we are still using containerd 1.1 right?

deploy/addons/gvisor/gvisor-config.toml

deploy/addons/gvisor/config.toml

docs/gvisor.md

ianlewis · 2018-12-04T14:32:47Z

deploy/addons/gvisor/gvisor-pod.yaml

+  namespace: kube-system
+  labels:
+    addonmanager.kubernetes.io/mode: Reconcile
+spec: 


nit: Consider setting terminationGracePeriodSeconds to something sane but less than 30s (an upper bound for what the preStop hook should take to run). AFAICT deleting the pod will take the full 30s since we are just sleeping for 1y

I ended up removing the prestop hook because it wasn't consistently running without errors; I've changed the code to intercept the SIGTERM kill signal and disable gvisor then.

Ok. that should work

Instead of checking in default config.toml, save it at /tmp/config.toml on the node upon enable and copy it back upon disable. Also, instead of using the prestop hook, intercept the SIGTERM kill signal upon pod termination, disable gvisor, and then exit with code 0. This should work better because now we will be able to see the logs from disabling, and because the prestop hook wouldn't consistenly run the disable command and clean up the pod correctly.

balopat · 2018-12-04T22:04:58Z

@minikube-bot OK to test

When enabling gvisor, first validate that the container runtime is containerd.

…isoraddon

Added integration test which follows these steps: 1. enable gvisor 2. make sure untrusted workload runs correctly 3. disable gvisor 4. make sure untrusted workload results in FailedCreateSandboxEvent event I also added a link to the iso url for starting containerd until the integration tests start using the new version of the iso.

test/integration/util/util.go

cmd/minikube/cmd/config/validations.go

deploy/addons/gvisor/README.md

pkg/gvisor/enable.go

tstromberg · 2018-12-06T04:46:03Z

@minikube-bot OK to test

We have to fix the test failures first

also remove extra logs

balopat · 2018-12-07T22:10:44Z

@minikube-bot OK to test

priyawadhwa · 2018-12-07T22:12:16Z

@minikube-bot OK to test

pkg/minikube/constants/constants.go

priyawadhwa · 2018-12-07T22:32:48Z

@minikube-bot OK to test

balopat · 2018-12-07T23:26:01Z

Known flakes: #3381
#3421
Merging.

Priya Wadhwa added 6 commits December 3, 2018 12:10

Add integration test to make sure minikube starts and stops with cont…

04c1215

…ainerd

Skip testing containerd if using none driver

01ad85e

add newline

8c7290f

Fixed imports

e385449

Add skip message when using none driver and containerd

40ad9e7

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 4, 2018

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Dec 4, 2018

k8s-ci-robot requested review from balopat and dlorenc December 4, 2018 01:18

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Dec 4, 2018

Added doc for enabling gvisor in minikube

f885f4b

priyawadhwa force-pushed the gvisoraddon branch from 85b7a4d to f885f4b Compare December 4, 2018 01:56

ianlewis reviewed Dec 4, 2018

View reviewed changes

priyawadhwa force-pushed the gvisoraddon branch from 47738c8 to ac963e2 Compare December 4, 2018 21:13

Priya Wadhwa added 4 commits December 4, 2018 14:09

Simplify gvisor and check for containerd runtime

c894943

When enabling gvisor, first validate that the container runtime is containerd.

small improvements

f9915ac

Merge branch 'integration' of github.com:priyawadhwa/minikube into gv…

81948f0

…isoraddon

priyawadhwa changed the title ~~[WIP] Enable gvisor addon in minikube~~ Enable gvisor addon in minikube Dec 5, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 5, 2018

Priya Wadhwa added 3 commits December 5, 2018 11:27

Fixed merge conflict

fd1f375

Show gvisor pod logs upon test failure

7bee870

Additional logs

765cc93

balopat reviewed Dec 5, 2018

View reviewed changes

test/integration/util/util.go Outdated Show resolved Hide resolved

balopat reviewed Dec 5, 2018

View reviewed changes

cmd/minikube/cmd/config/validations.go Outdated Show resolved Hide resolved

Code review comments and added logs

6e6020f

priyawadhwa force-pushed the gvisoraddon branch from 095bfc6 to 6e6020f Compare December 5, 2018 23:51

Priya Wadhwa added 3 commits December 5, 2018 15:58

Remove default runsc debug as it's too verbose

1c01497

change iso url

830f8f8

More logs

2ec2e53

fvoznika reviewed Dec 6, 2018

View reviewed changes

deploy/addons/gvisor/README.md Show resolved Hide resolved

deploy/addons/gvisor/README.md Show resolved Hide resolved

pkg/gvisor/enable.go Outdated Show resolved Hide resolved

Priya Wadhwa added 2 commits December 6, 2018 11:06

Code review comments and additional logs

a3cb54c

Merge branch 'master' of github.com:kubernetes/minikube into gvisoraddon

420eb32

priyawadhwa force-pushed the gvisoraddon branch 2 times, most recently from 022122a to adf2b61 Compare December 6, 2018 19:52

delete minikube in test before starting with containerd

0e4a28c

also remove extra logs

priyawadhwa force-pushed the gvisoraddon branch from adf2b61 to 0e4a28c Compare December 6, 2018 19:55

Priya Wadhwa added 3 commits December 6, 2018 13:06

delete after finishing up containerd

c976e11

Add minikube User-Agent to requests for shim and runsc

1073ea0

Only delete minikube if there is already one running

af6d30b

Merge branch 'master' of github.com:kubernetes/minikube into gvisoraddon

fdfa42a

fvoznika reviewed Dec 7, 2018

View reviewed changes

pkg/minikube/constants/constants.go Outdated Show resolved Hide resolved

update gvisor url

c7e5341

balopat self-requested a review December 7, 2018 23:26

balopat approved these changes Dec 7, 2018

View reviewed changes

balopat merged commit 8f128a7 into kubernetes:master Dec 7, 2018

priyawadhwa deleted the gvisoraddon branch December 7, 2018 23:28

grahamwhaley mentioned this pull request May 28, 2019

kernel: Add config for tc u32 filter and mirred action #4340

Merged

illume mentioned this pull request Jun 8, 2022

Addon: add headlamp kinvolk/minikube#7

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable gvisor addon in minikube #3399

Enable gvisor addon in minikube #3399

priyawadhwa commented Dec 4, 2018 •

edited

Loading

k8s-ci-robot commented Dec 4, 2018

tstromberg commented Dec 4, 2018

ianlewis Dec 4, 2018

ianlewis Dec 4, 2018

priyawadhwa Dec 4, 2018

ianlewis Dec 6, 2018

balopat commented Dec 4, 2018

tstromberg commented Dec 6, 2018

balopat commented Dec 7, 2018

priyawadhwa commented Dec 7, 2018

priyawadhwa commented Dec 7, 2018

balopat commented Dec 7, 2018

Enable gvisor addon in minikube #3399

Enable gvisor addon in minikube #3399

Conversation

priyawadhwa commented Dec 4, 2018 • edited Loading

k8s-ci-robot commented Dec 4, 2018

tstromberg commented Dec 4, 2018

ianlewis Dec 4, 2018

Choose a reason for hiding this comment

ianlewis Dec 4, 2018

Choose a reason for hiding this comment

priyawadhwa Dec 4, 2018

Choose a reason for hiding this comment

ianlewis Dec 6, 2018

Choose a reason for hiding this comment

balopat commented Dec 4, 2018

tstromberg commented Dec 6, 2018

balopat commented Dec 7, 2018

priyawadhwa commented Dec 7, 2018

priyawadhwa commented Dec 7, 2018

balopat commented Dec 7, 2018

priyawadhwa commented Dec 4, 2018 •

edited

Loading