Test SO_LINGER options for incoming and outgoing connections

[bryancall]

By default we enable SO_LINGER on incoming connections and turn it off on out going connections. Looking at other proxies Nginx has an option to set SO_LINGER only on connection that timeout before closing the socket.

Should we change our configuration option to 1 by default? Should we only set SO_LINGER based on how we close the connection? I would like to know what other companies defaults are. At Yahoo we enable SO_LINGER on both the incoming and outgoing sockets and we are seeing problem with VIPs keeping track of socket connection after the connection is closed.

I setup 2 ATS servers and changed the configuration of the flags between the servers:

testing this configuration options:
proxy.config.net.sock_option_flag_in
proxy.config.net.sock_option_flag_out

hera   | zues
tier 1 | tier 2
fout   | fin    | result (see below)
---------------------------------
5      | 5      | (1) <-- this is the default at Yahoo
1      | 5      | (2) <-- this is the default in Apache
1      | 1      | (3)
5      | 1      | (4)


config on both servers:
proxy.config.http.keep_alive_no_activity_timeout_in: 30
proxy.config.http.keep_alive_no_activity_timeout_out: 25
proxy.config.http.keep_alive_enabled_in: 1
proxy.config.http.keep_alive_enabled_out: 1

1:
09:57:08.022414 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423762389 ecr 3882926414], length 0
09:57:33.487745 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423787854 ecr 3882926414], length 24
09:57:33.487858 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 0
09:57:33.487972 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 24
09:57:33.487995 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [.], ack 17427, win 11, options [nop,nop,TS val 1423787854 ecr 3882951880], length 0
09:57:33.488015 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 0
09:57:33.488078 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17428, win 11, options [nop,nop,TS val 1423787854 ecr 3882951880], length 0

*** saw this a few time with no FINs sent ***
09:57:53.047158 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423807413 ecr 3882971439], length 0
09:58:18.201827 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423832568 ecr 3882971439], length 24
09:58:18.202044 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1423832568 ecr 3882971439], length 0
09:58:18.202118 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42960: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882996594 ecr 1423832568], length 24
09:58:18.202199 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [R], seq 505152777, win 0, length 0

2:
09:31:53.623626 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1422247990 ecr 3881412016], length 0
09:32:19.203163 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1422273569 ecr 3881412016], length 24
09:32:19.203236 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3881437595 ecr 1422273569], length 0
09:32:19.203383 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [F.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1422273570 ecr 3881437595], length 0
09:32:19.203391 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3881437596 ecr 1422273569], length 24
09:32:19.203393 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3881437596 ecr 1422273569], length 0
09:32:19.203486 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845298, win 0, length 0
09:32:19.203510 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845298, win 0, length 0
09:32:19.203538 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [.], ack 1196, win 503, options [nop,nop,TS val 3881437596 ecr 1422273570], length 0
09:32:19.203560 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845299, win 0, length 0

3:
09:44:50.515840 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [.], ack 18940, win 10, options [nop,nop,TS val 1423024882 ecr 3882188908], length 0
09:45:15.845300 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 18940, win 11, options [nop,nop,TS val 1423050211 ecr 3882188908], length 24
09:45:15.845451 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882214238 ecr 1423050211], length 0
09:45:15.845490 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [F.], seq 1195, ack 18940, win 11, options [nop,nop,TS val 1423050212 ecr 3882188908], length 0
09:45:15.845634 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [P.], seq 18940:18964, ack 1196, win 503, options [nop,nop,TS val 3882214238 ecr 1423050212], length 24
09:45:15.845714 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [R], seq 1029790287, win 0, length 0
09:45:15.845734 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [F.], seq 18964, ack 1196, win 503, options [nop,nop,TS val 3882214238 ecr 1423050212], length 0
09:45:15.845755 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [R], seq 1029790287, win 0, length 0

4:
09:49:34.440847 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423308807 ecr 3882472833], length
09:49:59.666017 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423334032 ecr 3882472833], length 24
09:49:59.666124 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 0
09:49:59.666187 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 24
09:49:59.666217 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1423334032 ecr 3882498058], length 0
09:49:59.666252 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 0
09:49:59.666304 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [R], seq 332745245, win 0, length 0

[randall]

I think we should disable SO_LINGER ( at least with its default value of 0).

Trying to fix the propagation of proxy.config.net.sock_option_flag_in to client connections (#8463), with this defaulted to on, results in autest failures all over the place. From talking with knowledgeable people along with readings, this sounds like it's really bad to be using.

[c-taylor]

The abortive close approach to setting with timeout 0 just before close() is a great mitigation for 'misbehaving' clients.

• Timed out
• Invalid data
• Rate limited
  etc.

I can see a reusable function that performs an abortive close being worthwhile. However the default should be to not set outside of the above.

I don't believe the need to (write->drain->close) as a one shot is a common use case, and also should not be a default.

[bryancall]

Related PRs:
#8784
#8623
#8463

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.