virtio-net and iPXE interoperability

[chschnell]

Networking in iPXE (and its derivates like netboot.xyz) works with v86's ne2k network device, but unfortunately fails with virtio. To reproduce, simply download CD-ROM ipxe.iso and boot it in v86 (I recommend to also look at netboot.xyz.iso, it's a pure network-based installer for a bunch of 32-bit OSes useable for v86).

While the iPXE guest correctly detects v86's virtio-net device (and also prints a MAC address with the expected v86 prefix of 00:22:15), traffic does not work. Using tcpdump, I can see a DHCP DISCOVER request coming from the iPXE guest through the virtio-net device, and I can also see a DHCP OFFER reply directed back to the iPXE guest (which means the request was OK). For some reason, the iPXE guest does not accept the DHCP OFFER and sends the next DHCP DISCOVER after a short timeout, and then fails. Inbound guest traffic does not work correctly.

The DHCP exchange works fine when using the iPXE guest with v86's ne2k network device instead of virtio. Unfortunately, ne2k shows stability problems under heavy load and is also not reliably usable here.

I'd like to ask for any ideas, starting points or debugging methods to understand and fix this interoperability issue in virtio. The network stack in iPXE has a different origin than, for example, the Linux kernel, so it cannot be excluded that this is actually a bug in iPXE (which would also be a valuable finding).

Details

• Version of v86: latest-102-g2b162b1e
• Browser/OS: Firefox/Windows 10
• Tested with different network backends (wsnic and bellenottelling/websockproxy)

iPXE console output using "ne2k" network device

SeaBIOS (version rel-1.16.2-0-gea1b7a0)
Booting from DVD/CD...

ISOLINUX 6.04 20200816 ETCD Copyright (C) 1994-2015 H. Peter Anvin et al
iPXE boot image
Loading ipxe.lkrn... ok
iPXE initialising devices...
WARNING: Using legacy NIC wrapper on 00:22:15:e0:02:f6

iPXE 1.21.1+ (g77cc3e) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI NFS TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT

net0: 00:22:15:e0:02:f6 using ne on ISA0300 (Ethernet) [open]
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 00:22:15:e0:02:f6)...... ok
net0: 192.168.86.2/255.255.255.0 gw 192.168.86.1
net0: fe80::222:15ff:fee0:2f6/64
Nothing to boot: No such file or directory (https://ipxe.org/2d03e13b)
No more network devices

iPXE>

tcpdump of iPXE using "ne2k" network device

$ sudo tcpdump -vvn -i wsbr0 "udp port 67 or udp port 68"
tcpdump: listening on wsbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
09:14:56.098014 IP (tos 0x0, ttl 64, id 512, offset 0, flags [none], proto UDP (17), length 413)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:d8:19:df, length 385, xid 0x81c11a5c, secs 8, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:d8:19:df
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.5.208.65.6.0.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:d8:19:df
09:14:56.098132 IP (tos 0xc0, ttl 64, id 11020, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2ea3 -> 0xd652!] BOOTP/DHCP, Reply, length 303, xid 0x81c11a5c, secs 8, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:d8:19:df
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Offer
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1
09:14:58.129401 IP (tos 0x0, ttl 64, id 769, offset 0, flags [none], proto UDP (17), length 425)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:d8:19:df, length 397, xid 0x81c11a5c, secs 18, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:d8:19:df
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Request
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.5.208.65.6.0.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:d8:19:df
            Server-ID (54), length 4: 192.168.86.1
            Requested-IP (50), length 4: 192.168.86.2
09:14:58.262409 IP (tos 0xc0, ttl 64, id 11199, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2ea3 -> 0xd348!] BOOTP/DHCP, Reply, length 303, xid 0x81c11a5c, secs 18, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:d8:19:df
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: ACK
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1

iPXE console output using "virtio" network device

SeaBIOS (version rel-1.16.2-0-gea1b7a0)
Booting from DVD/CD...

ISOLINUX 6.04 20200816 ETCD Copyright (C) 1994-2015 H. Peter Anvin et al
iPXE boot image
Loading ipxe.lkrn... ok
iPXE initialising devices...

iPXE 1.21.1+ (g77cc3e) -- Open Source Network Boot Firmware -- https://ipxe.org
Features: DNS HTTP iSCSI NFS TFTP SRP AoE ELF MBOOT PXE bzImage Menu PXEXT

net0: 00:22:15:1f:ce:b4 using virtio-net on 0000:00:0a.0 (Ethernet) [open]
  [Link:up, TX:0 TXE:0 RX:0 RXE:0]
Configuring (net0 00:22:15:1f:ce:b4)................. No configuration methods succeeded (https://ipxe.org/040ee119)
No more network devices

iPXE>

tcpdump of iPXE using "virtio" network device

$ sudo tcpdump -vvn -i wsbr0 "udp port 67 or udp port 68"
tcpdump: listening on wsbr0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:33:58.505098 IP (tos 0x0, ttl 64, id 358, offset 0, flags [none], proto UDP (17), length 413)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:70:ef:c0, length 385, xid 0xcafd0c0b, secs 4, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.1.26.244.16.65.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:70:ef:c0
18:33:58.505185 IP (tos 0xc0, ttl 64, id 5523, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2e9d -> 0xc5fd!] BOOTP/DHCP, Reply, length 303, xid 0xcafd0c0b, secs 4, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Offer
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1
18:33:59.270070 IP (tos 0x0, ttl 64, id 716, offset 0, flags [none], proto UDP (17), length 413)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:70:ef:c0, length 385, xid 0xcafd0c0b, secs 8, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.1.26.244.16.65.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:70:ef:c0
18:33:59.270171 IP (tos 0xc0, ttl 64, id 5565, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2e9d -> 0xc5f9!] BOOTP/DHCP, Reply, length 303, xid 0xcafd0c0b, secs 8, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Offer
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1
18:34:01.307837 IP (tos 0x0, ttl 64, id 836, offset 0, flags [none], proto UDP (17), length 413)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:70:ef:c0, length 385, xid 0xcafd0c0b, secs 12, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.1.26.244.16.65.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:70:ef:c0
18:34:01.307932 IP (tos 0xc0, ttl 64, id 5844, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2e9d -> 0xc5f5!] BOOTP/DHCP, Reply, length 303, xid 0xcafd0c0b, secs 12, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Offer
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1
18:34:05.371198 IP (tos 0x0, ttl 64, id 1058, offset 0, flags [none], proto UDP (17), length 413)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 00:22:15:70:ef:c0, length 385, xid 0xcafd0c0b, secs 16, Flags [none] (0x0000)
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Discover
            MSZ (57), length 2: 1472
            ARCH (93), length 2: 0
            NDI (94), length 3: 1.2.1
            Vendor-Class (60), length 32: "PXEClient:Arch:00000:UNDI:002001"
            User-Class (77), length 4:
              instance#1: [ERROR: invalid option]
            Parameter-Request (55), length 24:
              Subnet-Mask (1), Default-Gateway (3), Domain-Name-Server (6), LOG (7)
              Hostname (12), Domain-Name (15), RP (17), MTU (26)
              NTP (42), Vendor-Option (43), Vendor-Class (60), TFTP (66)
              BF (67), Unknown (119), Unknown (128), Unknown (129)
              Unknown (130), Unknown (131), Unknown (132), Unknown (133)
              Unknown (134), Unknown (135), Unknown (175), Unknown (203)
            Unknown (175), length 51: 177.5.1.26.244.16.65.235.3.1.21.1.23.1.1.39.1.1.34.1.1.19.1.1.17.1.1.25.1.1.41.1.1.16.1.2.33.1.1.21.1.1.24.1.1.27.1.1.18.1.1
            Client-ID (61), length 7: ether 00:22:15:70:ef:c0
18:34:05.371307 IP (tos 0xc0, ttl 64, id 6309, offset 0, flags [none], proto UDP (17), length 331)
    192.168.86.1.67 > 192.168.86.2.68: [bad udp cksum 0x2e9d -> 0xc5f1!] BOOTP/DHCP, Reply, length 303, xid 0xcafd0c0b, secs 16, Flags [none] (0x0000)
          Your-IP 192.168.86.2
          Server-IP 192.168.86.1
          Client-Ethernet-Address 00:22:15:70:ef:c0
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message (53), length 1: Offer
            Server-ID (54), length 4: 192.168.86.1
            Lease-Time (51), length 4: 86400
            RN (58), length 4: 43200
            RB (59), length 4: 75600
            Subnet-Mask (1), length 4: 255.255.255.0
            BR (28), length 4: 192.168.86.255
            Default-Gateway (3), length 4: 192.168.86.1
            Domain-Name (15), length 9: "v86.local"
            Domain-Name-Server (6), length 4: 192.168.86.1

[basicer]

My virtio code has always had this bug with two directional transport queues. The OS sends empty buffers into the receive queue, and the device is expected to pop them from the queue as it writes data into it. The problem is, because they are ring buffers, the buffer looking full and the buffer looking empty both results in the start and end pointers being at the same location. The code works around this by returning the last buffer that its sent, so the rest of the virtio code thinks theres entries in the queue. This works on linux as linux give a bunch of buffers all in one notification. ipxe notifies once per buffer that it sends, and so the hack just drops each buffer as soon as it gets it :(

[chschnell]

The problem is, because they are ring buffers, the buffer looking full and the buffer looking empty both results in the start and end pointers being at the same location.

I'm only half way through understanding the virtio code, but I'm getting there.

So maybe I'm wrong for not knowing better yet, but wouldn't a naive buffer counter in VirtQueue solve this? Initialize it to zero, increment when receiving a buffer from the guest, decrement when passing one to the guest. Then it wouldn't be neccessary to look at the queue pointers to discern an empty from a full queue, counter == 0 would indicate an empty queue.

I'm asking because this is one of the traditional solutions to this recurring problem with ring buffers.

[basicer]

That is the traditional solution. I thought that wouldn't work because the value of read pointer is set by the OS and we just read it out of system memory. But as it turns out the virtio spec says that value is the unwrapped value. I have a patch for this almost ready to go.

Im still messing with it though because its not enough to get iPXE to netboot for some reason. With the fix it gets an IP address now, but then just stops sending packets shortly after.

[basicer]

Okay, looks like whats actually happening here, is when ipxe reads the MTU information from the device, it reads it as two bytes instead of as a single word. There's a few bugs in v86 that stop this from working.

The first is that addr isn't actually passed to shim that adapts an 8bit read to a 16bit value on capability struct only registers the handler to the first byte. The second is that the shim function expects to be passed the actual address of the read, but the io function doesn't forward it.

[basicer]

With #1278 netboot.xyz seems to work with virito-net.

[chschnell]

With #1278 netboot.xyz seems to work with virito-net.

Yes, I can confirm that your PR fixes it, awesome!

To test it I installed Debian 12 into a blank 2G hda using netboot.xyz, works perfectly.

So far I could not make my iSCSI-experiment boot with iPXE, I'll look into that tomorrow, I will then also try to NFS-boot with iPXE. I don't think my iSCSI-stuff has anything to do with virtio-net though, netboot.xyz being based on iPXE is proof enough.

So, thank you very much!

[chschnell]

The problem with iPXE and iSCSI was that I had never initialized the MBR of the iSCSI disk. So installing grub, running grub-install and setting a proper kernel command line fixed it.