Troubleshooting IPv6 HTTP(S) deployment

[Kangie]

Hi Team,

I'm trying to get the most basic deployment of Confluent going with two servers directly connected via ethernet cables (no switches or anything in the way).

Our desired configuration will use HTTP boot over UEFI, so I am attempting to set this up without DNS or DHCP in the first instance to trial node deployment (and image management) before we scale out to one of our smaller clusters.

I have been able to define my node, automatically discover it's MAC, and assign that MAC to the the defined node (I have plugged in a second interface to make sure the shared ILO port is not the issue here). My deployment interface(s) (not defined anywhere...) have IPv6 enabled and link-local (FE80) addresses.

root@headnode:~# nodediscover list
 Node| Model| Serial|                                 UUID|       Mac Address|       Type| Current IP Addresses
-----|------|-------|-------------------------------------|------------------|-----------|---------------------
   t1|      |       | 36334c44-4730-xxxx-yyyy-4e50ax373134| xx:xx:xx:d2:b4:c0| pxe-client|                     
   t1|      |       | 36334c44-4730-xxxx-yyyy-4e50ax373134| xx:xx:xx:d2:b4:c1| pxe-client|                     

I have defined the client node to boot only from HTTP(S) UEFI options.

I see the following via tcpdump of the deployment interface (limited by node MAC) when the node attempts to HTTP boot off that port:

tcpdump -i eno2 -nnnnn  ether host xx:xx:xx:d2:b4:c1
. . .
16:38:04.802851 IP6 fe80::2267:7cff:fed2:b4e1.547 > fe80::2267:7cff:fed2:b4c1.546: dhcp6 [|dhcp6]
16:38:08.951124 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:38:09.749800 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor solicitation, who has fe80::2267:7cff:fed2:b4e1, length 32
16:38:09.749839 IP6 fe80::2267:7cff:fed2:b4e1 > fe80::2267:7cff:fed2:b4c1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4e1, length 24
16:38:15.169719 IP6 fe80::2267:7cff:fed2:b4e1 > fe80::2267:7cff:fed2:b4c1: ICMP6, neighbor solicitation, who has fe80::2267:7cff:fed2:b4c1, length 32
16:38:15.169851 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4c1, length 32
16:38:17.950750 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:38:34.950175 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:40:25.807578 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:40:29.943707 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:40:38.943350 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:40:55.942762 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:42:46.786935 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:42:50.936298 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:42:59.935936 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:43:16.935357 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:45:07.779679 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:45:07.780750 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4c1, length 32
16:45:07.780774 IP6 fe80::2267:7cff:fed2:b4e1.547 > fe80::2267:7cff:fed2:b4c1.546: dhcp6 [|dhcp6]
16:45:11.928902 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:45:12.727582 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor solicitation, who has fe80::2267:7cff:fed2:b4e1, length 32
16:45:12.727606 IP6 fe80::2267:7cff:fed2:b4e1 > fe80::2267:7cff:fed2:b4c1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4e1, length 24
16:45:20.928531 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:45:37.927937 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:45:37.928998 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4c1, length 32
16:45:37.929023 IP6 fe80::2267:7cff:fed2:b4e1.547 > fe80::2267:7cff:fed2:b4c1.546: dhcp6 [|dhcp6]
16:45:42.925999 IP6 fe80::2267:7cff:fed2:b4c1 > fe80::2267:7cff:fed2:b4e1: ICMP6, neighbor solicitation, who has fe80::2267:7cff:fed2:b4e1, length 32
16:45:42.926021 IP6 fe80::2267:7cff:fed2:b4e1 > fe80::2267:7cff:fed2:b4c1: ICMP6, neighbor advertisement, tgt is fe80::2267:7cff:fed2:b4e1, length 24
16:47:28.784301 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:47:32.922494 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:47:41.921126 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit
16:47:58.920540 IP6 fe80::2267:7cff:fed2:b4c1.546 > ff02::1:2.547: dhcp6 solicit

Confluent appears to be listening on the DHCP6 port:

root@headnode:~# ss -tulpn
Netid           State            Recv-Q           Send-Q                     Local Address:Port                       Peer Address:Port           Process                                                                                                                                           
udp             UNCONN           0                0                                0.0.0.0:44939                           0.0.0.0:*               users:(("avahi-daemon",pid=920,fd=14))                                                                                                           
udp             UNCONN           0                0                             127.0.0.54:53                              0.0.0.0:*               users:(("systemd-resolve",pid=910,fd=16))                                                                                                        
udp             UNCONN           0                0                          127.0.0.53%lo:53                              0.0.0.0:*               users:(("systemd-resolve",pid=910,fd=14))                                                                                                        
udp             UNCONN           0                0                                0.0.0.0:67                              0.0.0.0:*               users:(("confluent",pid=1395,fd=15))                                                                                                             
udp             UNCONN           0                0                                0.0.0.0:427                             0.0.0.0:*               users:(("confluent",pid=1395,fd=14))                                                                                                             
udp             UNCONN           0                0                                0.0.0.0:631                             0.0.0.0:*               users:(("cups-browsed",pid=2055,fd=7))                                                                                                           
udp             UNCONN           0                0                                0.0.0.0:1900                            0.0.0.0:*               users:(("confluent",pid=1395,fd=26))                                                                                                             
udp             UNCONN           0                0                                0.0.0.0:4011                            0.0.0.0:*               users:(("confluent",pid=1395,fd=19))                                                                                                             
udp             UNCONN           0                0                                0.0.0.0:5353                            0.0.0.0:*               users:(("avahi-daemon",pid=920,fd=12))                                                                                                           
udp             UNCONN           0                0                                   [::]:41150                              [::]:*               users:(("avahi-daemon",pid=920,fd=15))                                                                                                           
udp             UNCONN           0                0                                      *:59932                                 *:*               users:(("confluent",pid=1395,fd=20))                                                                                                             
udp             UNCONN           0                0                                   [::]:427                                [::]:*               users:(("confluent",pid=1395,fd=13))                                                                                                             
udp             UNCONN           0                0                                      *:547                                   *:*               users:(("confluent",pid=1395,fd=16))                                                                                                             
udp             UNCONN           0                0                                   [::]:1900                               [::]:*              

And the node is primed for deployment:

# nodedeploy t1
t1: pending: ubuntu-24.04.1-x86_64-default (node authentication armed)

I never see any response from my Confluent head node telling this node where to boot. I assume this is all supposed to happen over Layer 2 IPv6 "magic" based on the sparse confluent docs that exist. What have I missed here?

Edit:

Head Node Details:

• Ubuntu 24.04
• Confluent installed via Apt:

confluent-server/unknown,now 3.11.1-1 all [installed,automatic]
  confluent systems management server

• No firewall enabled on the host.

[jjohnson42]

Anything in /var/log/confluent/events?

confluent_selfcheck -n t1

[Kangie]

Apologies for the delay; long weekend here

/var/log/confluent/events does shed a bit of light on the situation:

Oct 04 15:09:16 {"info": "Unable to provide IPv6 boot services to t1, no viable IPv6 configuration on interface index \"5\" to respond through."}

As for the selfcheck:

root@head-node:~# confluent_selfcheck -n t1
OS Deployment: Initialized
Confluent UUID: Consistent
Web Server: Running
Web Certificate: OK
Checking web download: OK
Checking web API access: OK
TFTP Status: TFTP failure, PXE will not work, though media and HTTP boot can still work. (Example resolution: osdeploy initialize -p)
SSH root user public key: OK
Checking SSH Certificate authority: OK
Checking confluent SSH automation key: OK
Checking for blocked insecure boot: OK
Checking IPv6 enablement: OK
Performing node checks for 't1'
Checking node attributes in confluent...
Checking network configuration for t1
t1 may not have any viable IP network configuration (check name resolution (DNS or hosts file) and/or net.*ipv4_address, and verify that the deployment server addresses and subnet mask/prefix length are accurate)
No issues detected with attributes of t1
Checking name resolution: Name resolution failed for node, it is normally a good idea for the node name to resolve to an IP

Thanks for that pointer, I'll try and work out what it wants. FWIW the deployment interface '5' has a FE80 address which the docs have lead me to believe is all that we require to provision.

5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 20:67:aa:bb:b4:e3 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f3
    inet 192.168.1.2/24 brd 192.168.1.255 scope global noprefixroute eno4
       valid_lft forever preferred_lft forever
    inet6 fe80::2267:7cff:febb:b4e3/64 scope link
       valid_lft forever preferred_lft forever

[Kangie]

Tracking this a touch further, we are in fact failing early in reply_dhcp6 and therefore are not responding to the DHCP6 requests and tell the node where to boot.

confluent/confluent_server/confluent/discovery/protocols/pxe.py

Lines 635 to 639 in 3a0218c

	def reply_dhcp6(node, addr, cfg, packet, cfd, profile, sock):
	myaddrs = netutil.get_my_addresses(addr[-1], socket.AF_INET6)
	if not myaddrs:
	log.log({'info': 'Unable to provide IPv6 boot services to {0}, no viable IPv6 configuration on interface index "{1}" to respond through.'.format(node, addr[-1])})
	return

The question is... why. I might quickly add a quick log of all of addr.

Not very helpful:

Oct 08 16:11:45 {"info": ["fe80::2267:7cff:xxyy:b4c0", 546, 0, 5]}
Oct 08 16:11:45 {"info": "Unable to provide IPv6 boot services to t1, no viable IPv6 configuration on interface index \"5\" to respond through."}

I can't quite see what's wrong here, any suggestions?

[jjohnson42]

For IPv6 boot, you need a more 'real' IPv6 address, like a ULA address. (the standard way is to generate 40 random bits after fd and then you have a /48 you can allocate from.). For example:
fde9:44ff:33a2::/64

Then you could have the deployment server be:
fde9:44ff:33a2::1/64

You'd further want IPv6 addresses corresponding to the nodes, e.g.:
fde9:44ff:33a2::a:1
Just any addresses that would fit in that subnet.

LLA addresses (fe80::) can work for some things, for example, for connecting to BMCs over standard network protocols. Broadly speaking, some things simply require a non-LLA IPv6 address. Since they usually require a scope index added to the address and would otherwise be too ambiguous in multi-homed systems, they are too awkward for a lot of software to accomodate, and so a number of standards exclude them from everyday use. IIRC, UEFI is one of those contexts that do not implement pure LLA operation. You also can't use LLA in DNS or /etc/hosts.

[Kangie]

That would do it. I'm happy to call this a docs issue:

IPv6 configuration

Deployment interfaces must have IPv6 enabled, with at least an automatic fe80:: address. Generally this is default network interface configuration. IPv6 need only be enabled, it need not be given any address manually, by DHCP, or by route advertisements, the automatic fe80:: addresses suffice.

then

DHCP (optional)

on https://hpc.lenovo.com/users/documentation/confluentosdeploy.html

Gives the impression that all that is required is a LLA.

I'll spin up some DHCP4 stuff and get on with my deployment. Do you want to close off this ticket or leave it open to track the docs updates?