Transparently proxying UDP traffic with pf?

[Orum]

This is a similar symptom to my earlier issue (#1473), but as the circumstances have drastically changed I thought I'd open a new issue. To summarize the changes, I am no longer running sslocal on the host (Linux/iptables) machine; it's instead being run on my FreeBSD router which uses pf.

Obviously, this requires different firewall rules to work, but there is little if any documentation I could find for transparently proxying via pf within the shadowsocks-rust project. What I've done here is largely guesswork, but this is the rule I came up with to transparently proxy both TCP & UDP to sslocal (and then on to ssserver):
rdr on $int_if inet proto { tcp, udp } to !<private> -> 127.0.0.1 port 1080

sslocal is being run on the router via the following command:
sslocal -b 127.0.0.1:1080 -U --protocol redir -s 192.168.x.y:8388 -m none --tcp-redir pf --udp-redir pf

...and finally, ssserver is run with -U -m none -b 192.168.x.y:8388

This "works" in the sense that both TCP & UDP traffic are being redirected to sslocal (unlike in #1473 where only TCP traffic was ever reaching sslocal), which is then sending the traffic on to sserver. However, once again, only TCP traffic is being fully proxied correctly.

UDP traffic does arrive at sslocal without any doubt, as it shows up when running sslocal with -vvv like so (timestamps removed for brevity, and the host system's address being substituted here with simply <host>):

TRACE  tokio-runtime-worker ThreadId(05) shadowsocks::relay::udprelay::proxy_socket: crates/shadowsocks/src/relay/udprelay/proxy_socket.rs:235: UDP server client send to 127.0.0.1:1080, control: UdpSocketControlData { client_session_id: 7043831170210357480, server_session_id: 0, packet_id: 1, user: None }, payload length 96 bytes, packet length 103 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::redir::udprelay: crates/shadowsocks-service/src/local/redir/udprelay/mod.rs:307: received UDP packet from <host>:44849, destination 127.0.0.1:1080, length 96 bytes
TRACE  tokio-runtime-worker ThreadId(05) shadowsocks_service::local::net::udp::association: crates/shadowsocks-service/src/local/net/udp/association.rs:433: udp relay <host>:44849 -> 127.0.0.1:1080 (proxied) with 96 bytes

This traffic is then forwarded on to ssserver, which I can verify by watching outgoing traffic on the router via tcpdump. After they arrive at ssserver though, they fail to reach the internet, so clearly something isn't configured correctly.

My best guess as to what is happening here, based on the logs, is that sslocal thinks the destination of the UDP packet is 127.0.0.1 (i.e. the address that sslocal is running on and where incoming UDP traffic on the router is redirected to), and fails to retrieve the original destination address before pf's rdr rule took effect. This wouldn't surprise me as my rdr rule is complete guesswork, so I assume I'm missing something.

If that is indeed the problem, what should the rdr rule look like to transparently proxy UDP traffic with pf? Or am I barking up the wrong tree, and is the problem due to something else entirely?

[zonyitoo]

I have never tested on FreeBSD, so this project supports FreeBSD is still "theoretically".

I have tested pf on macOS, which supports TCP perfectly just like you did on FreeBSD. But on macOS, UDP's destination must be retrieved from ioc_getstates. This API is far different from FreeBSD's definition, so I think FreeBSD doesn't use the same mechanism to get UDP packets' destination when redirected by pf.

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

The 127.0.0.1:1080 in your case was exactly the value of msg_name returned from recv_dest_from.

[Orum]

In your last PR, you said on FreeBSD UDP's destination could be retrieved from msg_name returned by recvfrom. Is that correct? I couldn't see any document from FreeBSD talking about it.

That wasn't me, so you must be thinking of someone else. While I know how to write rules for pf fairly well, I've never programmed anything to interface with it, so I'm unsure of how the API is supposed to be used.

The good news is that you can access the man pages online. There's also a forum, mailing lists, and a Discord server, all of which feature some form of development/programming discussion.

I'm interested in learning as well, but I'm still learning rust, so it may be some time until I am of use.

[zonyitoo]

Glad to know that you are familiar with FreeBSD and pf. Could you find any doc / references about how to get the "original destination address" of UDP packets?

On Linux, it is quite straight forward: https://man7.org/linux/man-pages/man7/ip.7.html (IP_RECVORIGDSTADDR).

On macOS, I got it from another post: (?) Deleted.

A famous tools mitmproxy only supports TCP on FreeBSD: https://docs.mitmproxy.org/stable/howto-transparent/ .

So it is quite hard to find a reference about how FreeBSD handles UDP redirects and how to get the original destination address programmatically.

[zonyitoo]

And you could also try the ipfw: https://forums.freebsd.org/threads/retrieving-the-destination-port-in-an-ipfwed-udp-packet.37545/

[zonyitoo]

@Orum Do you still interest in making this Project working on FreeBSD? What's the current status?

[Orum]

I am still very interested in making it work. However, I think the best way to do so at this point is to just read through FreeBSD's source code. As such, I'm familiarizing myself with it, but it's going to take some time as I have other obligations and projects demanding my attention, so it's on the back burner for me right now. This is probably the best, or at least, surest way to figure out how to get the address (assuming it's even possible at present) from pf.

Additionally, I'd need to learn much more about rust before I contribute anything other than information to this project, as my knowledge of it is spotty at best. But, if I do figure out the FreeBSD side of the picture, perhaps someone here can handle that side of things once they have the information. I'm also not only willing, but happy to test things to see if they work once they're implemented.

[ge9]

semigodking/redsocks#200
FYI, I succeeded in running transparent proxy in FreeBSD with redsocks+SOCKS5 (sorry they are completely unrelated to shadowsocks) and ipfw, but not with pf. When I tested with pf, the UDP destination address was obtained as the transparent proxy's listen address, just as @Orum mentioned.
As I mentioned in the link, in OpenBSD, where pf originates from, using divert-to rule of pf worked perfectly. divert rule seems not ported to FreeBSD's pf.

[Orum]

FreeBSD and OpenBSD diverged in their rules syntax quite a few years back when the latter made some changes that broke older rule sets, IIRC. I've stuck with FreeBSD, which might have an equivalent of divert-to (or not), but I don't really know what it does as I never bothered to keep up with OBSD's syntax as I don't use it.

I also don't really want to switch to ipfw, or any other firewall, as I really value simplicity and clarity of pf's rules' syntax. So many other firewalls are hard to decipher, which is undesirable to say the least when it comes to security.

In any case I will try and dedicate some time to reading through and understanding the pf/bpf code in the future, but that will be some time from now.

[zonyitoo]

@ge9 Have you tried shadowsocks-rust with ipfw? What problem did you see?

[ge9]

I tested it and it worked, but there were a problem.
shadowsocks-rust uses IPv4-mapped IPv6 address to return message to IPv4 clients:

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs

Line 199 in ac67a08

peer_addr = SocketAddr::new(v4_peer_addr.ip().to_ipv6_mapped().into(), v4_peer_addr.port());

However, FreeBSD disables IPv4-mapped IPv6 address by default (according to scala-native/scala-native#3630). If I set sysctl net.inet6.ip6.v6only=0, then it worked.
Except for this, normal transparent proxy setting will work.

FYI, My settings are following:

• Initial setup (the machine already had 10.0.2.15/24)

#!/bin/sh
kldload ipfw
fwcmd=ipfw
ifconfig em0 alias 10.0.2.25 netmask 0xffffff00
$fwcmd add 100 allow all from any to any via lo0
$fwcmd add 500 fwd 127.0.0.1,22222 tcp from 10.0.2.25 to any
$fwcmd add 600 fwd 127.0.0.1,22222 udp from 10.0.2.25 to any
$fwcmd add 700 allow ip from any to any

• sslocal -b 127.0.0.1:22222 -U --protocol redir -s 192.168.1.7:1080 --tcp-redir pf --udp-redir pf -m none
  • (specifying pf seems fine)
• ssserver -U -s 192.168.1.7:1080 -m none (on another PC)
  • note that specifying 0.0.0.0:1080 here doesn't work

[zonyitoo]

There is a SUPPORT_IPV6_TRANSPARENT flag that indicates whether the current platform supports IPv6, if it is true, then shadowsocks-rust will always convert IPv4 to IPv4-mapped-IPv6 and uses only one socket for sending back.

This is always working for Linux, because dual-stack socket was enabled by default for most distribution of Linux.

Is there a way to test if dual-stack is enabled on FreeBSD? By reading sysctl net.inet6.ip6.v6only=0 should be one possible option.

[zonyitoo]

Could you please help to run the test program on FreeBSD and see if any of these syscalls returned errors?
https://stackoverflow.com/questions/30184377/how-to-detect-if-dual-stack-socket-is-supported

[zonyitoo]

cc @madeye , FreeBSD doesn't support IPv4-mapped-IPv6 by default. So my previous (years ago) PR that uses IPv6 sockets for sending back UDP packets won't be able to run properly on FreeBSD.

[ge9]

This is the result of the python test program.

Traceback (most recent call last):
  File "/root/test.py", line 4, in <module>
    s.connect(('::ffff:169.254.1.1', 53))
OSError: [Errno 22] Invalid argument

If I set sysctl net.inet6.ip6.v6only=0 and commented out s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, 1), it worked. Otherwise, it failed.

In the shadowsocks, this is the error.

2024-06-17T03:45:10.334750916+09:00  WARN udp failed to send back 68 bytes to client 10.0.2.25:44550, from target 3.132.228.249:3478 (proxied), error: Invalid argument (os error 22)

This originates from

shadowsocks-rust/crates/shadowsocks-service/src/local/redir/udprelay/mod.rs

Line 203 in ac67a08

inbound.send_to(data, peer_addr).await.map(|n| {

.