[Feature Request] Adding vhost-net support to firecracker

[majek]

Hi,

Firecracker doesn't have fast enough networking for many users. Firecracker networking benchmarks claiming tens of Gbps use TCP and use network offloads (TSO). However, often for real users these assumptions can't be met. For example, QUIC uses UDP, and when using XDP in the guest, the offloads must be disabled.

Under such conditions - UDP and no offloads - firecracker network performance is dismal. We can get only 1-2Gbps. This is really understandable once you realize how the tap code works.

Other kvm-based virtualization systems can do much better, they typically do this by utilizing vhost-net host kernel acceleration.

I'm considering working on adding vhost-net support to firecracker. Vhost-net is a host kernel API, exposing tap device in a format compatible with virtio-net guest driver. With shared memory and ioctl's dance it's possible to greatly speed up the data path for networking. Host vhost-net kernel thread would copy network data directly into the kvm guest memory. It's possible to set it up in such a way that firecracker process is not touching neither the data nor interrupts when doing networking. This saves loads of context switches, greatly improves latency and reduces CPU usage.

It doesn't solve the scalability issue - a naive vhost-net implementation would still only be single-queue and use one vCPU in the guest. This is a separate issue.

Vhost-net is a well established API. From the previous discussions it feels the firecracker community was not in favor of vhost-net. Vhost-net exposes the host kernel, so it extends the attack surface against the host. However, over the years vhost-net matured, the implementation strengthened and given its benefits it might be a good time to reconsider it.

I think with vhost-net we should be able to increase the network performance for UDP + disabled offloads from 1-2Gbps to 7-8Gbps. Users of vhost-net would see a reduced CPU usage for networking-heavy virtual machines and improved latency.

When looking at adding vhost-net to firecracker I had some concerns:

(1) It's unclear how to support MMDS with vhost-net. Is that a problem?

(2) It's unclear how to deal with networking rate limits.

I think both of these features could be supported by doing ebpf on the host side. I'm actually surprised by software rate-limit implementation done in firecracker since it could be done much more efficiently in bpf. I'm not looking at vsock at this stage.

I would like to understand if:

• the maintainers are generally open for adding optional vhost-net support

• there are any extra conditions that must be met to consider the vhost-net code

In a perfect world I would like to have a vhost-net networking option in firecracker. Users prioritizing perceived security could use traditional data path, users prioritizing lower cpu usage and better network scalability could opt-in for vhost-net.

Marek

[acatangiu]

I guess the answer to whether vhost-net or vhost-block will be supported by firecracker comes down to this past discussion, and in particular this:

The main focus is on maintaining the Firecracker security barrier. I.e. Firecracker must control all data exchanged between the guest and the host.

and

Vhost-net exposes the host kernel, so it extends the attack surface against the host. However, over the years vhost-net matured, the implementation strengthened and given its benefits it might be a good time to reconsider it.

Doing a shallow search of vhost-related CVEs does show a downward trend over the years, so it has indeed matured - but the crux of the matter of having considerably more attack surface is still a pain point - also this "extra" attack surface is directly to host kernel so it's bypassing most other defense-in-depth layers of firecracker (de-privileged process, cgroups, namespaces, seccomp, etc) - so the risk while small has higher potential of big impact.

This is not to say I am personally against vhost-* emulation - it is indeed faster and more efficient. I am simply offering some background and context for the decision-makers.

[bchalios]

Hi Marek,

Thanks for getting in touch and the input, I think you are raising very valid points. On our side we know that IO performance in Firecracker is not what it could be and we have recently started internal discussions on defining the problems and sketching potential solutions. We plan to actively work on it during the second part of the year.

vhost is definitely one of the ways to go. While vhost-net is the easiest option, we have also been considering
vhost-user. The latter, apart from promising performance advantages would give us a lot more flexibility in selecting networking back-ends more easily (no changes would be needed in Firecracker itself).

We will keep the community posted on updates on our side, but feedback at this point from you and others that are interested in this is valuable.

Some remarks on your comments:

Vhost-net is a well established API. From the previous discussions it feels the firecracker community was not in favor of vhost-net. Vhost-net exposes the host kernel, so it extends the attack surface against the host. However, over the years vhost-net matured, the implementation strengthened and given its benefits it might be a good time to reconsider it.

The security aspect of vhost-net is still important for us. vhost-net exposes the host kernel to guests, without Firecracker intermediation, so we need to think about it very carefully.

Under such conditions - UDP and no offloads - firecracker network performance is dismal. We can get only 1-2Gbps. This is really understandable once you realize how the tap code works.

It sounds that this might be orthogonal to vhost-net or no vhost-net discussion. Do you maybe have throughput numbers for UDP workloads where you just disable offloading in the Firecracker emulation code?

(1) It’s unclear how to support MMDS with vhost-net. Is that a problem?

At the moment MMDS is supported from Firecracker, so any re-architecture should take it into account. BPF is one way, another would be to decouple MMDS from the emulation of the Network device all together.

I would like to understand if:

• the maintainers are generally open for adding optional vhost-net support

We are definitely open to discussion at this point for finding a proper solution for this.

there are any extra conditions that must be met to consider the vhost-net code

Rate limiting is a must have. MMDS too, at the time being. More requirements might occur once we dive deeper in the problem.

[DemiMarie]

One option would be vhost-user + DPDK + Vector Packet Processing.

[kalyazin]

Hi @majek. While considering support for vhost-net in Firecracker, we would like to gain confidence in the following areas.

over the years vhost-net matured, the implementation strengthened

One of the major concerns is vhost-net involves kernel in the virtqueue processing. If something goes wrong, the blast radius may be larger (potentially affecting other microVMs/tenants) compared to processing in userspace where only a single microVM/tenant would be affected. Do you have specific data that convinces you that vhost-net is sufficiently safe to use in a multi-tenant environment?

It's unclear how to deal with networking rate limits.

We would certainly need to keep the capability of rate limiting in the network device. Do you have some concrete ideas or references of how this can be implemented via eBPF or is it just an assumption?

It's unclear how to support MMDS with vhost-net. Is that a problem?

We would like to decouple MMDS from the networking stack. It probably does not make much sense to bind it to vhost-net, but we would need to have an alternative way of supporting it.

there are any extra conditions that must be met to consider the vhost-net code

It would also be very helpful to estimate the potential performance gain for various workloads. Have you been doing some analysis you could share that let you obtain the numbers you provided?

the maintainers are generally open for adding optional vhost-net support

If the concerns above are resolved, we would certainly be looking into supporting vhost-net.

Thanks for opening this discussion!

[xmarcalx]

Hi guys,

do you have any follow up on the previous questions? Are you still interested in this feature?

[majek]

Hi, a quick update. We have a hacky local firecracker fork with vhost networking. We're working on making the code upstreamable, it's harder than anticipated. We intend to: provide convincing benchmarks, provide code that could be used to bootstrap discussion. Long term we are likely to add multiqueue on top of that (which is non trivial considering mmio limitations).

[Noah-Kennedy]

A question for maintainers: would you prefer that this be implemented via a new device, or by putting multiple backends in the net device and selecting which one to use based on a setting in the vm config?

[kalyazin]

Hi @Noah-Kennedy . The path we would like to follow at the moment for the cases similar to this one is:

• keep a single API endpoint for all devices of the kind (eg /network-interfaces).
• use backend-specific parameters to differentiate devices within a kind (eg vhost object in the network interface properties to contain vhost-net-specific arguments)
• use a separate structure for a device within a kind (eg src/vmm/src/devices/virtio/net/vhost/*). This may require moving the existing device one level down in the directory structure.

We are also thinking about introducing device subtypes similar to types to facilitate differentiation between devices within the kind.

We are hoping to publish a PR that would illustrate these points in a week or two.

[kalyazin]

Hi @Noah-Kennedy . There is a draft PR for adding a vhost-user block device: #4069 . Please let us know if a similar approach is not easily applicable to the vhost-net device for some reason.

[DemiMarie]

One option would be vhost-user + DPDK + Vector Packet Processing.

To elaborate on this:

An alternative approach to getting rid of context switches is to not switch contexts at all. Instead of having the networking stack on the same core as the user workload, run it on a different core, and rely on either asynchronous cross-core notifications or polling. DPUs take this to the extreme by running the networking code on completely different hardware.

[kalyazin]

Hi @majek . While our team are looking at the prerequisite PR, would you mind sharing your thoughts on my question earlier in the thread:

One of the major concerns is vhost-net involves kernel in the virtqueue processing. If something goes wrong, the blast radius may be larger (potentially affecting other microVMs/tenants) compared to processing in userspace where only a single microVM/tenant would be affected. Do you have specific data that convinces you that vhost-net is sufficiently safe to use in a multi-tenant environment?

While discussing internally, there have multiple opinions been voiced favouring the existing virtio net device or a potential vhost-user-net implementation over vhost-net for that exact reason.