UDP native socket server reply not received by client

[anecdata]

Firmware

Adafruit CircuitPython 6.2.0-beta.4 on 2021-03-18; FeatherS2 with ESP32S2  # also tested on beta.2

Code/REPL

CircuitPython native UDP Server code & CPython UDP Client code are in the comments below.

Behavior

I've been testing cross-compatibility of CPython, CircuitPython ESP32SPI, and CircuitPython ESP32-S2 sockets, both TCP and UDP. All supported combinations work except CircuitPython ESP32-S2 UDP Server... none of the three types of clients receive the server's reply to the client-initiated message. It seems to be related to the port number.

TCP

For comparison, let's look first at the working TCP case. Notice in the CircuitPython TCP Server output below that the server accepts a connection from a remote (IP, PORT), receives data from the client, then echoes it back to the client.

CircuitPython TCP Server output:

code.py output:
Connecting to Wifi
Self IP 192.168.6.198
Server ping 192.168.6.198 0.0 ms
Create TCP Server socket ('192.168.6.198', 5010)
Listening
Accepting connections
Accepted from ('192.168.5.32', 51687)
Received packet bytearray(b'Hello, world') 12 bytes from ('192.168.5.32', 51687)
Sent bytearray(b'Hello, world') 12 bytes to ('192.168.5.32', 51687)

TCP Client output (any of the three types):

Create TCP Client Socket
Connecting
Sent 12 bytes
Received b'Hello, world'

Checking the tcpdump view of the transaction, we see that the client outgoing port (59337) does not match the CircuitPython-server-printed port (51687), BUT the dump shows the reply going back to the proper originating port. It works..

16:20:53.930013 IP: (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->ad81)!)
    192.168.5.32.59337 > 192.168.6.198.5010: Flags [S], cksum 0x8d69 (incorrect -> 0xe628), seq 956746751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 678730438 ecr 0,sackOK,eol], length 0
16:20:54.015444 IP: (tos 0x0, ttl 255, id 56508, offset 0, flags [none], proto TCP (6), length 44)
    192.168.6.198.5010 > 192.168.5.32.59337: Flags [S.], cksum 0xee99 (correct), seq 9048109, ack 956746752, win 2880, options [mss 1440], length 0
...

UDP

Notice in the CircuitPython UDP Server output below that the server (waits until it) receives data from a remote (IP, PORT), then echoes it back to the client.

CircuitPython UDP Server output:

code.py output:
Connecting to Wifi
Self IP 192.168.6.198
Server ping 192.168.6.198 0.0 ms
Create UDP Server socket ('192.168.6.198', 5000)
Received bytearray(b'Hello, world') 12 bytes from ('192.168.5.32', 20727)
Sent bytearray(b'Hello, world') 12 bytes to ('192.168.5.32', 20727)

UDP Client output (any of the three types):

Create UDP Client Socket
Sent 12 bytes
Traceback (most recent call last):
  File "udp_client_CPython.py", line 29, in <module>
    size, addr = s.recvfrom_into(buf)
socket.timeout: timed out

Checking the tcpdump view of the transaction, we see that the client outgoing port (63312) does not match the CircuitPython-server-printed port (20727). In this case the dump shows the reply going back to the CircuitPython-printed port (20727), BUT this does not work... the reply is never received by the client.

16:03:06.414501 IP: (tos 0x0, ttl 64, id 7958, offset 0, flags [none], proto UDP (17), length 40, bad cksum 0 (->ce78)!)
    192.168.5.32.63312 > 192.168.6.198.5000: UDP, length 12
16:03:06.522949 IP: (tos 0x0, ttl 255, id 56498, offset 0, flags [none], proto UDP (17), length 40)
    192.168.6.198.5000 > 192.168.5.32.20727: UDP, length 12
16:03:06.523016 IP: (tos 0x0, ttl 64, id 64454, offset 0, flags [none], proto ICMP (1), length 56, bad cksum 0 (->f1c7)!)
    192.168.5.32 > 192.168.6.198: ICMP 192.168.5.32 udp port 20727 unreachable, length 36
	(tos 0x0, ttl 255, id 56498, offset 0, flags [none], proto UDP (17), length 40)
    192.168.6.198.5000 > 192.168.5.32.20727: UDP, length 12

I've tried longer timeouts, catching client exceptions and retrying, etc. None of the three types of clients receive the reply back from the native UDP server. All three types of clients do receive replies back from CPython UDP servers (ESP32SPI servers are not supported).

[anecdata]

CircuitPython ESP32-S2 UDP Socket Server example code:

import wifi
import socketpool
import ipaddress
import time
from secrets import secrets

TIMEOUT = None
HOST = ""  # see below; "127.0.0.1" and "localhost" don't work
PORT = 5000
MAXBUF = 256

# connect to wifi
print("Connecting to Wifi")
wifi.radio.connect(secrets["ssid"], secrets["password"])
pool = socketpool.SocketPool(wifi.radio)

print("Self IP", wifi.radio.ipv4_address)
HOST = str(wifi.radio.ipv4_address)
server_ipv4 = ipaddress.ip_address(pool.getaddrinfo(HOST, PORT)[0][4][0])
print("Server ping", server_ipv4, wifi.radio.ping(server_ipv4), "ms")

# make socket
print("Create UDP Server socket", (HOST, PORT))
s = pool.socket(pool.AF_INET, pool.SOCK_DGRAM)
s.settimeout(TIMEOUT)
s.bind((HOST, PORT))

buf = bytearray(MAXBUF)
while True:
    size, addr = s.recvfrom_into(buf)
    print("Received", buf[:size], size, "bytes from", addr)

    size = s.sendto(buf[:size], addr)
    print("Sent", buf[:size], size, "bytes to", addr)

[anecdata]

CPython UDP Socket Client example code:

#!/usr/bin/env python3
import time
import socket


TIMEOUT = 5
HOST = "192.168.6.198"  # edit as needed
PORT = 5000
MAXBUF = 256


buf = bytearray(MAXBUF)
while True:
    print("Create UDP Client Socket")
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.settimeout(TIMEOUT)

    size = s.sendto(b"Hello, world", (HOST, PORT))
    print("Sent", size, "bytes")

    size, addr = s.recvfrom_into(buf)
    print("Received", buf[:size], size, "bytes from", addr)

    s.close()

    time.sleep(5)

[anecdata]

@hierophect seems something is odd about the client port number, TCP vs. UDP. I looked at the code but nothing jumped out at me.

tcpdump command line I'm using to capture the packets (macOS):
sudo tcpdump -i en0 -vns 0 host 192.168.6.198 # edit IP address as needed to match server

Some sources mention optionally using bind at the client. This doesn't change the behavior, other than that the tcpdump originating port from the client will be the bound port. But the reply still goes back to the server-printed port and the client still does not receive the reply.

[DavePutz]

Looks like the byte-order of the port number is the issue. i.e. 63312 is 0xF750 and 20727 is 0x50F7. Not sure which side isn't using network byte-order, though...

[anecdata]

Good catch!

[anecdata]

I think this may be the issue, in Socket.c sendto():
dest_addr.sin_port = htons(port);

A cursory glance at the lwip layer indicates that the port byte order is mostly managed there. However, bind() and connect() also switch the port byte order in common-hal/socketpool/Socket.c and they work fine (bind is a local port though, and I don't think it's supposed to use network order. Using bind() or not in ESP32-S2 UDP Server doesn't seem to make a difference).

Not flipping the port byte order in sendto() makes ESP32-S2 UDP Server work, but I don't understand the code layers well enough to know fully why all of these byte order changes (or not) are the way they are. Tested with ESP32-S2 UDP Server + CPython UDP Client.

Also, sendto is intended for UDP (but does work with TCP), so I wonder if this affects anything in sendto():

    const struct addrinfo hints = {
        .ai_family = AF_INET,
        .ai_socktype = SOCK_STREAM,
    };

But changing it to SOCK_DGRAM didn't seem to change behavior. Not sure what that's really for.

some time later...

This works:
https://github.com/adafruit/circuitpython/blob/main/tests/circuitpython-manual/socketpool/datagram/ntp.py
as an ESP32-S2 UDP client, implying sendto() is fine.

The CPython UDP client above works with this CPython UDP server:

#!/usr/bin/env python3
import time
import socket

HOST = ""
PORT = 5000
TIMEOUT = None
MAXBUF = 256

print("Create UDP Server Socket")
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.settimeout(TIMEOUT)

s.bind((HOST, PORT))

buf = bytearray(MAXBUF)
while True:
    size, addr = s.recvfrom_into(buf)
    print("Received", buf[:size], size, "bytes from", addr)

    size = s.sendto(buf[:size], addr)
    print("Sent", buf[:size], size, "bytes to", addr)

[anecdata]

Hypothesis No. 2: recvfrom_into() needs the port byte order switched. The TCP case works with recv_into() but not with recvfrom_into(). Swapping the port byte order in CircuitPython UDP server code between recv and send does work:

    size, addr = s.recvfrom_into(buf)
    print("Received", buf[:size], size, "bytes from", addr)

    addr = (addr[0], htons(addr[1]))  #test

    size = s.sendto(buf[:size], addr)
    print("Sent", buf[:size], size, "bytes to", addr)

[anecdata]

Closed by #4465