VXLAN EVPN with a different IP and UDP ports other than 4789?

[stevefan1999-personal]

I'm not sure when EVPN routes are advertised would the port and IP address be taken account for:

 ip link add vxlan0 type vxlan \
    id 123 \
    dstport 9999 \
    local 192.168.8.100 \
    dev vmbr0 \
    l2miss \
    l3miss \
    nolearning

The problem is I'm running behind NAT, so not only 4789 can't be used (but port forwarding is available), but also that my "local" address can be...different.

My home network always assigned me a floating IP via DHCP, so I need to find a way to let other EVPN peers to dynamically "reassign" my upstream IP address to fix it to a correct VTEP endpoint, which I think there is no documentations for this scenario.

For example, I have a home NAT router with IP address 192.0.2.1/24, server A at 198.51.100.1/24 and server B at 203.0.113.1/24,
I have a virtual server V1 behind my NAT router with IP address 192.168.10.1/24, and another virtual server V2 at 192.168.10.2/24.
Now, both V1 and V2 do not have a public IP and will connect to different VXLAN (take for example, one will go to internet and the other goes to DN42), so port 4789 can't be used.
Instead, I will have to setup a port forwarding on my NAT router to open two UDP ports for VXLAN traffic, and the topology looks like this:

Home router 192.0.2.1:2345 <-> V1 192.168.10.1:3456
On V1:

 ip link add vxlan0 type vxlan \
    id 123 \
    dstport 3456 \
    local 192.168.10.1 \
    dev vmbr0 \
    l2miss \
    l3miss \
    nolearning

Home router 192.0.2.1:3456 <-> V2 192.168.10.2:4567
On V2:

 ip link add vxlan0 type vxlan \
    id 124 \
    dstport 4567 \
    local 192.168.10.2 \
    dev vmbr0 \
    l2miss \
    l3miss \
    nolearning

V1, server A and server B will join each other via VXLAN on id 123 and V2 will only join server A on id 124.

And then I will setup VXLAN EVPN over BGP on V1 and V2 respectively (also using port forwarding). The problem is, would other peers know that it should send VXLAN packets to V1 via 192.0.2.1:2345 and V2 via 192.0.2.1:3456 as well? If so, what should I do to set it?

For example, server A and B should intelligently know that it should connect to V1 on 192.0.2.1:2345 rather than on 192.168.10.1:3456, and on V1 I will still see source packet from 198.51.100.1:4789 or 203.0.113.1:4789 respectively in V1 and V2 since they have a stable static IP and VXLAN port.

I was able to statically configure VXLAN between my servers by doing this on V1&V2:

bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 198.51.100.1 port 4789
bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 203.0.113.1 port 4789

But on server A it should be:

bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 203.0.113.1 port 4789
bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 192.0.2.1 port 2345 # This goes to V1
bridge fdb append 00:00:00:00:00:00 dev vxlan1 dst 192.0.2.1 port 3456 # This goes to V2

on server B it should be:

bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 198.51.100.1 port 4789
bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 192.0.2.1 port 2345 # This goes to V1

(Appendix: https://man7.org/linux/man-pages/man8/bridge.8.html)

This painfully works, and to get a better appraoch I should do VXLAN EVPN over BGP instead of using an ad-hoc DHT to store the VXLAN address-port information. (This is what I raised in neocturne/fastd#34, but I used fastd + BATMAN for a semi-working DMEVPN, but the performance is not good even with kernel offloading)

Of course, I know this problem can be solved by using a layer 3 VPN where each peer has an isolated IP address that I can safely listen to 4789. In theory, I could have just slap in Wireguard to each peer with a static IP address and run BGP on top of it. In practice, I already did this to some extent in DN42, but the encryption and UDP overhead is not ideal. (You could say that I want Wireguard, but without the encryption and I have a L2 connectivity plus I want it automatically routed for HA, so a VXLAN EVPN is the most suitable.)

(Alas, I have a simplified scenario where the port mappings can be 1:1 mapped, so the remaining problem is to confirm that I can change the ports other than 4789, the IANA assigned port number for VXLAN)

I think I can still use 4789, but I will have to setup IP masquerade on my home router to rewrite the IP and port numbers which is not feasible on a cheap Cortex-A9 router. I also can't open my home router to BGP due to legal reasons.