TCP Keepalive not forwarded to upstream origin server.

[SkyperTHC]

It appears that wiretap does not send empty PSH packets (tcp keepalive) to check if the Origin-host still has the port open.

I'm using https://thc.org/segfault/wireguard with the wiretap v0.3.0 (--simple branch) with WIRETAP_SIMPLE=true ./wiretap_linux_amd64 serve --ipv4-relay 192.168.0.1 --ipv6-relay fd::1 --allowed 192.168.0.1/28,fd::1/125

The Exit Node is a Linux x86_64 running wiretap (WT-EXIT).
The origin host runs nmap -n -Pn -sT -p1-512 --open scanme.nmap.org

The Origin host sends a SYN to a host on the Internet that does not exist:

┌──(EXIT:Dirt)(root💀sf-BiologyMetal)-[~]
└─# nmap -n -Pn -sT -T5  -p80 --open 30.31.32.33

Observer on the Exit Node the state SYN-SENT for a long time even that the app (nmap) has long exited:

SYN-SENT 0      1       51.83.131.42:53150    30.31.32.33:80    users:(("wiretap_linux_a",pid=125616,fd=14))

The same bug (eg. lag of empty PSH for TCP KeepAlive between origin-host and WT-Exit) can be observed when starting an app on the upstream origin-server and not sending a FIN when killing the app (or that FIN drops on the wireguard/udp-leg):

In this case wiretap keeps the connection from the Exit Node to the target in ESTB "forever" even that the upstream app on the origin-server no longer has this connection open.

There are TCP Keepalive messages between wiretap and the target but there are no TCP keepalive between the wiretap and the upstream origin server.

A proposed solution would be to send an empty PSH (e.g. TCP KeepAlive) every 60 seconds from WT-EXIT to the upstream origin host and watch out for any returning RST or ICMP Dest unreachable.

[luker983]

Thanks for reporting this.

I added a timeout to how long the server will wait for the handshake to complete when first checking if the remote service exists. Right now it defaults to 1 second, but can be changed with --conn-timeout (08673a9 in tcp-fixes).

I added a separate timeout to how long the server will wait for the client to actually complete its handshake after the server has verified the remote service exists. This also defaults to 1 second and can be changed with the --catch-timeout flag. (da1d233 in tcp-fixes)

Once the initial handshake is complete, the server should be aware when the origin connection closes and also close the connection to the target.

[SkyperTHC]

Thanks for reporting this.

I added a timeout to how long the server will wait for the handshake to complete when first checking if the remote service exists. Right now it defaults to 1 second, but can be changed with --conn-timeout (08673a9 in tcp-fixes).

(I assume server is the host running Wiretap EXIT to the Internet). (The WG jargon of 'relay' and 'client' doesnt always make sense when connections are established the opposite direction :>)

I better default timeout would be >15 seconds for most use cases. Linux default is 180 sec and Windows is 72 sec to complete a TCP 3-way. Completing a TCP 3-way within 1 second works 99% of the time but once there is packet-drop then the 1sec-timeout wont allow for the TCP stack to re-transmit the SYN before wiretap's 1 sec timeout kicks in.

I added a separate timeout to how long the server will wait for the client to actually complete its handshake after the server has verified the remote service exists. This also defaults to 1 second and can be changed with the --catch-timeout flag. (da1d233 in tcp-fixes)

I would recommend 5 seconds. A default of 1 second would not work if Wiretap client <-> server are further than 500ms apart.

Once the initial handshake is complete, the server should be aware when the origin connection closes and also close the connection to the target.

That's my point: The Wiretap Server is not aware. The origin-host (running nmap or masscan) might 'give up on that port' without sending any FIN or RST to the WT-EXIT. The WT-EXIT however will keep the TCP open 'forever' to the destination on the Internet.

I'm sorry if my explanation was so bad. I'll try again with a proposed solution and 1 TCP connection:

1. If there is no packet on the Origin-host <=> WT-EXIT for 60 seconds then the WT-EXIT shall send an empty TCP-PSH (aka TCP Keepalive) to the Origin-host every 5 seconds.
2. If the WT-EXIT receives a PSH then all is in order (reset time back to 60 seconds - packet was received).
3. Otherwise kill the WT-EXIT <-> Internet TCP connection
4. Also kill the WT-EXIT <-> Internet connection if WT-EXIT receives a RST, FIN or ICMP unreachable from the origin-host.

[luker983]

I better default timeout would be >15 seconds for most use cases. Linux default is 180 sec and Windows is 72 sec to complete a TCP 3-way. Completing a TCP 3-way within 1 second works 99% of the time but once there is packet-drop then the 1sec-timeout wont allow for the TCP stack to re-transmit the SYN before wiretap's 1 sec timeout kicks in.

How about 5 seconds and the timer gets reset every SYN?

I would recommend 5 seconds. A default of 1 second would not work if Wiretap client <-> server are further than 500ms apart.

I'll change this to a 5 second default

That's my point: The Wiretap Server is not aware. The origin-host (running nmap or masscan) might 'give up on that port' without sending any FIN or RST to the WT-EXIT. The WT-EXIT however will keep the TCP open 'forever' to the destination on the Internet.

Ah, sorry for the misunderstanding. Looking back into it it seems that the current connections between origin-host and WT-EXIT do have keepalives but they don't start until the connection has been idle for 2 hours. And then keepalives start sending every 1m15s. Does this line up with your testing or did you have a connection that remained alive for longer than 2 hours?

If that does match your testing, I can reduce the keepalive idle timer to something more reasonable like 2 minutes and then keep the current keepalive interval of 1m15s?

EDIT: Sorry, misread the code. Keepalive was in-fact disabled. But enabling it sets the 2hour idle default before sending keepalives. So the question of sane defaults for keepalive idle and interval timers still stands.

[SkyperTHC]

2h also happens to be linux's default TCP KEEPALIVE timer - which is a bit of a coincident and I like to make sure we are talking about the problem with TCP KEEPALIVE were there is no real TCP connection (from the Kernel's perspective):

I'm using this command on the origin server:

masscan --interface wgExit --source-ip 192.168.0.3 --rate 1 --banners -p31337 30.31.32.33

(It's a quick way of sending a TCP SYN and nothing else. No RST. No FIN).

From the kernel's perspective there is no real TCP connection between Origin Server and the EXIT NODE (wiretap) because the TCP SYN was send a raw packet - like most scanners do. The only 'real' TCP connection is from Exit Node (wiretap) to Target (30.31.32.33).

tcp        0      1 1.2.3.4:45560    30.31.32.33:31337       SYN_SENT    16334/wiretap

There are two distinct possibilities:
A. The target never answers.
B. The target (30.31.32.33:31337) answers with a SYN/ACK.
C. Same a B but the origin replies with a RST

Scenario A: Wiretap's outgoing connect(2) to the target fails after 180 seconds (the default linux kernel timeout). Wiretap could have given up much earlier. Your solution to use 5 sec timeout should work. This scenario is considered fixed.

Scenario B: Change the target 30.31.32.33:31337 to any target that has that port open. Wiretap's connect(2) to the target succeeds. WT sends the SYN/ACK to the Origin Server. The Origin Server replies with ACK. The Linux kernel on the Exit Node (wiretap) correctly starts its TCP KEEPALIVE between the Exit Node (wiretap) and the target but WT does not send a TCP KEEPALIVE to the Origin Server. In this scenario (masscan) the Origin Server has long given up on the port while WT keeps the connection between WT and Target in ESTABLISHED forever (I tested for 3h and then stopped the test - connection to target was still alive when it should not have been).

The proposed solution is for WT to send an empty PSH (aka TCP keepalive) between WT and the Origin Server (masscan; not the target) and if there is no reply received after 3 TCP keepalives (each 5 seconds) then close the TCP connection between WT and the Target.

Scenario C can be triggered like so:

masscan --interface wgExit --source-ip 192.168.0.3 --rate 1 -p31338 94.177.9.11

(A dirty way to send a SYN and a RST if a SYN/ACK is received)

13:01:58.160355 IP 192.168.0.3.56518 > 94.177.9.11.31338: Flags [S], seq 1925581689, win 1024, length 0
13:01:58.175219 IP 94.177.9.11.31338 > 192.168.0.3.56518: Flags [S.], seq 3985737959, ack 1925581690, win 27584, options [mss 1380], length 0
13:01:58.175257 IP 94.177.9.11.31338 > 192.168.0.3.56518: Flags [S.], seq 3985737959, ack 1925581690, win 27584, options [mss 1380], length 0
13:01:59.160371 IP 192.168.0.3.56518 > 94.177.9.11.31338: Flags [R], seq 1925581690, win 1200, length 0

The Origin (masscan) sent the RST to WT but WT keeps the TCP connection to the Target (94.117.9.11) open forever:

ESTAB  0      0      192.145.44.201:48580    94.177.9.11:31338 users:(("wiretap",pid=16334,fd=12))

[luker983]

Yeah, 2 hours was chosen to be in sync with the Linux implementation. But it likely wasn't enabled when you were testing. Scenario B should be fixed in this commit: 01b5b21

I lowered the defaults quite a bit. Keepalives start sending after keepaliveIdleDefault which is 1 minute, and then keepaliveCountDefault is set to 2 (2 defaults must be dropped before connection is closed). keepaliveIntervalDefault has been left at 1minute 15 seconds. So after 1 minute the connection will drop if the origin server responds to Keepalive with RST/Unreachable, or after 3m30s if all keepalives are ignored. Every 5 seconds seems fairly chatty and I imagine this is an uncommon scenario? Correct me if I'm wrong.

[SkyperTHC]

Your fix for Scenario B seems to be working. (Oddly, when testing v0.2.1 the connection stayed alive beyond 2h (i tested for 3h10 and then stopped the test - but WT's connection to the target was still open).

Linux's default for KeepAliveIdle is 7200 and Interval is 75. I recommend to set keepaliveIntervalDefault to a value lower than keepaliveIdleDefault, such as 10.

Yes, I agree that 5 seconds would be fairly chatty. The concern however are port scanning via WT: At the default rate of 10,000 ports/sec and 'timeout by keep-alive' of 210 seconds (3m30s) would mean 2,100,000 open tcp connections by WT when the scanner software does not have any tcp connection open at all.

A compromise would be:

1. Start with keepaliveIdleDefault=5, keepaliveIntervalDefault=5
2. Set keepaliveIdleDefault=60, keepaliveIntervalDefault =10 when tcp payload (from origin to target) is received.

[luker983]

Keepalives shouldn't be coming into play for port scans.

• The catch/conn timers of 5 seconds will kill any SYN scan connections.
• Connection scans should be closed by the scanner itself (nmap -sT does this)

The keepalives are just there for when an origin server hasn't properly closed it's connection, which should be an exceptional case.

[SkyperTHC]

Keepalives shouldn't be coming into play for port scans.

• The catch/conn timers of 5 seconds will kill any SYN scan connections.

yes, you are right. That case is covered. I was thinking along a different path and did not make this clear. I'm sorry. It now appears to be an edge case but it's worth mentioning:

I think that --conn-timeout and --catch-timeout are only effective before the 3-way is completed. Thereafter WT will kill the connection after 3m30sec (death-by-keepalive 😅) because the origin-host (masscan) will not reply to empty PSH.

The catch is:

1. There are IP ranges on the Internet that have all ports (64k) on all IP ranges "open" (some kind of smart firewall). Eventually and within a few seconds WT will have 2,500,000 connections to hold open until death-by-keepalive kicks in.

• Connection scans should be closed by the scanner itself (nmap -sT does this)

nmap does and so does masscan. In both cases is it actually the Linux Kernel which sees that there is no actual TCP connection on the origin (raw TCP was used instead of connect(2)) and sends an RST. This wont work if masscan uses a spoofed ip (which it does when scanning for banners). In this case masscan will always reply to any PSH with a valid ACK - forever. Any keepalive will always be answered with a correct reply.

For a 'normal' router (running WG) that would not be a problem because no 'state' is kept when forwarding packets. Not so for WT - it has to keep state and the state will blow (I think WT dies far before the 2,500,000. Last I tested it dies at 80,000 open TCP connections on a 2GB router.....

The keepalives are just there for when an origin server hasn't properly closed it's connection, which should be an exceptional case.

[luker983]

Alright, sounds like there is a use-case for more frequent keepalives. Added flags to configure this, but left the defaults on the order of minutes. Commit 0bd8d00