latest vsock stable fixes in upstream kernel result in kata failures

[ericwadams]

Description of problem

I have 3 Clear Linux systems that I let auto update. I noticed late last week that 2 of them failed to run a kata container after updating. Today my personal system also failed to run a kata container. Error message is "docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown."

Expected result

The container would be created

Actual result

OCI error message

---

I tried to run the script at https://github.com/kata-containers/runtime/blob/master/data/kata-collect-data.sh.in but it said it couldn't find the runtime. I put the output of kata-env instead but this happens across 3 different systems only after Clear Linux updated.

$ sudo ./katacollect.sh
Password:
ERROR: katacollect.sh: cannot find runtime '@RUNTIME_NAME@'

$ sudo kata-runtime kata-env
[Meta]
  Version = "1.0.20"
[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = true
  DisableNewNetNs = false
  Path = "/usr/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.5.0"
    Commit = ""
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/usr/share/defaults/kata-containers/configuration-qemu.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.0\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/usr/bin/kata-qemu-lite-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.5.0_agent_a581aebf473.img"

[Kernel]
  Path = "/usr/share/kata-containers/vmlinuz-4.14.106-14.container"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.5.0"
  Path = "/usr/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.5.0"
  Path = "/usr/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "5.0.2-717.native"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = false
  [Host.Distro]
    Name = "Clear Linux OS"
    Version = "28310"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Core(TM) i7-6770HQ CPU @ 2.60GHz"

[Netmon]
  Version = "kata-netmon version 1.5.0"
  Path = "/usr/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

[jodh-intel]

Hi @eadamsintel - thanks for reporting.

Hmm, either the problem is occurring randomly or it might imply we need additional Kata CI tests in Clear?

By the way, that file isn't designed to be run - the .in file is the template. When you build the runtime, that template gets converted into a valid shell script called kata-collect-data.sh. And that script is supposed to be packaged with the other Kata binaries.

If you happen to know the release when this stopped working, that would help to narrow the issue down.

One final point - Clear isn't using the latest version of Kata (see https://github.com/clearlinux-pkgs/kata-runtime/blob/master/Makefile).

/cc @bryteise.

Related: #1370.

[ericwadams]

Using the Clearlinux release notes as a guide I downgraded to 28240 which works. When I upgrade to 28310 it quits working. According to the release notes the only change is the kata kernel went from 4.14.105-13 -> 4.14.106-14 but the runtime is still 1.5.0

[ericwadams]

As an FYI I downgraded to 28300. Kata works ok. When upgrading back to version 28310 Kata breaks.

[jodh-intel]

Any thoughts on this @bryteise?

[bryteise]

I'm a little confused as to how a patch version change to an LTS kernel is causing a breakage. I have updated Kata which should make it to the afternoon build but it seems odd that we didn't see this crash on our end.

[jodh-intel]

Ditto. I'm also confused by https://github.com/clearlinux-pkgs/linux-kata/blob/master/Makefile since the version in there is 4.14.90 (for the tip)... and yet we're talking here about older releases using 4.14.105 and 4.14.106...?

I think we need more info on what changed kernel-wise and where those changes are (.config). Of course other things will have changed between these two Clear Linux releases including the host kernel (5.0.1-312 -> 5.0.2-313). Is it possible to downgrade the host kernel to rule that out?

We probably want input from @jcvenegas and @devimc but they are not around today.

Any thoughts @amshinde, @sboeuf?

I've also raised kata-containers/ci#132.

[bryteise]

It doesn't appear to be related to the host kernel (I tried to downgrade there and saw the same issue).

@jodh-intel The Makefile in that case isn't used for the URL and has gotten out of date, if you look at the spec file you'll see the use of 106 as the kernel sources.

[sboeuf]

@jodh-intel nothing from the top of my head.

/cc @mcastelino as he's using CL with Kata on a regular basis I believe

[bryteise]

The only change between 105 and 106 is whatever changed in the kernel sources, no config, the rest of the changes between versions don't have any obvious problem children. May need to bisect between 105 and 106 but first we can try running 28300 but sneaking in the 28310 kata kernel.

[bryteise]

So I have tried using the 105 kernel in 28310 and the bug doesn't show. Using the 106 kernel in 28300 has the bug so the issue definitely seems to stem from a change in the 4.14 LTS between 105 and 106.

[jodh-intel]

@bryteise - thanks for looking into this! Could you possibly post a kernel config diff so we can compare that to the known good configs in?:

• https://github.com/kata-containers/packaging/tree/master/kernel/configs

[bryteise]

I tracked it down to a specific commit related to vsocks but I see 107 just got released with some vsock changes so I'm going to try that and see if it fixes. If that doesn't work then I'll post hopefully useful information =).

[grahamwhaley]

thx @bryteise - if you happen to have a ref to the commit(s), we have a vsocks person here on the kata repos. /cc @stefanha