Test SO_LINGER options for incoming and outgoing connections

[bryancall]

By default we enable SO_LINGER on incoming connections and turn it off on out going connections. Looking at other proxies Nginx has an option to set SO_LINGER only on connection that timeout before closing the socket.

Should we change our configuration option to 1 by default? Should we only set SO_LINGER based on how we close the connection? I would like to know what other companies defaults are. At Yahoo we enable SO_LINGER on both the incoming and outgoing sockets and we are seeing problem with VIPs keeping track of socket connection after the connection is closed.

I setup 2 ATS servers and changed the configuration of the flags between the servers:

testing this configuration options:
proxy.config.net.sock_option_flag_in
proxy.config.net.sock_option_flag_out

hera   | zues
tier 1 | tier 2
fout   | fin    | result (see below)
---------------------------------
5      | 5      | (1) <-- this is the default at Yahoo
1      | 5      | (2) <-- this is the default in Apache
1      | 1      | (3)
5      | 1      | (4)


config on both servers:
proxy.config.http.keep_alive_no_activity_timeout_in: 30
proxy.config.http.keep_alive_no_activity_timeout_out: 25
proxy.config.http.keep_alive_enabled_in: 1
proxy.config.http.keep_alive_enabled_out: 1

1:
09:57:08.022414 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423762389 ecr 3882926414], length 0
09:57:33.487745 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423787854 ecr 3882926414], length 24
09:57:33.487858 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 0
09:57:33.487972 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 24
09:57:33.487995 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [.], ack 17427, win 11, options [nop,nop,TS val 1423787854 ecr 3882951880], length 0
09:57:33.488015 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42958: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3882951880 ecr 1423787854], length 0
09:57:33.488078 enp4s0 Out IP hera.bryancall.com.42958 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17428, win 11, options [nop,nop,TS val 1423787854 ecr 3882951880], length 0

*** saw this a few time with no FINs sent ***
09:57:53.047158 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423807413 ecr 3882971439], length 0
09:58:18.201827 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423832568 ecr 3882971439], length 24
09:58:18.202044 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1423832568 ecr 3882971439], length 0
09:58:18.202118 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42960: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882996594 ecr 1423832568], length 24
09:58:18.202199 enp4s0 Out IP hera.bryancall.com.42960 > zues.bryancall.com.pharos: Flags [R], seq 505152777, win 0, length 0

2:
09:31:53.623626 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1422247990 ecr 3881412016], length 0
09:32:19.203163 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1422273569 ecr 3881412016], length 24
09:32:19.203236 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3881437595 ecr 1422273569], length 0
09:32:19.203383 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [F.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1422273570 ecr 3881437595], length 0
09:32:19.203391 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3881437596 ecr 1422273569], length 24
09:32:19.203393 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3881437596 ecr 1422273569], length 0
09:32:19.203486 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845298, win 0, length 0
09:32:19.203510 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845298, win 0, length 0
09:32:19.203538 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42934: Flags [.], ack 1196, win 503, options [nop,nop,TS val 3881437596 ecr 1422273570], length 0
09:32:19.203560 enp4s0 Out IP hera.bryancall.com.42934 > zues.bryancall.com.pharos: Flags [R], seq 4294845299, win 0, length 0

3:
09:44:50.515840 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [.], ack 18940, win 10, options [nop,nop,TS val 1423024882 ecr 3882188908], length 0
09:45:15.845300 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 18940, win 11, options [nop,nop,TS val 1423050211 ecr 3882188908], length 24
09:45:15.845451 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882214238 ecr 1423050211], length 0
09:45:15.845490 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [F.], seq 1195, ack 18940, win 11, options [nop,nop,TS val 1423050212 ecr 3882188908], length 0
09:45:15.845634 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [P.], seq 18940:18964, ack 1196, win 503, options [nop,nop,TS val 3882214238 ecr 1423050212], length 24
09:45:15.845714 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [R], seq 1029790287, win 0, length 0
09:45:15.845734 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42942: Flags [F.], seq 18964, ack 1196, win 503, options [nop,nop,TS val 3882214238 ecr 1423050212], length 0
09:45:15.845755 enp4s0 Out IP hera.bryancall.com.42942 > zues.bryancall.com.pharos: Flags [R], seq 1029790287, win 0, length 0

4:
09:49:34.440847 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [.], ack 17403, win 10, options [nop,nop,TS val 1423308807 ecr 3882472833], length
09:49:59.666017 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [P.], seq 1171:1195, ack 17403, win 11, options [nop,nop,TS val 1423334032 ecr 3882472833], length 24
09:49:59.666124 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [.], ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 0
09:49:59.666187 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [P.], seq 17403:17427, ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 24
09:49:59.666217 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [R.], seq 1195, ack 17403, win 11, options [nop,nop,TS val 1423334032 ecr 3882498058], length 0
09:49:59.666252 enp4s0 In  IP zues.bryancall.com.pharos > hera.bryancall.com.42948: Flags [F.], seq 17427, ack 1195, win 503, options [nop,nop,TS val 3882498058 ecr 1423334032], length 0
09:49:59.666304 enp4s0 Out IP hera.bryancall.com.42948 > zues.bryancall.com.pharos: Flags [R], seq 332745245, win 0, length 0

[randall]

I think we should disable SO_LINGER ( at least with its default value of 0).

Trying to fix the propagation of proxy.config.net.sock_option_flag_in to client connections (#8463), with this defaulted to on, results in autest failures all over the place. From talking with knowledgeable people along with readings, this sounds like it's really bad to be using.

[c-taylor]

The abortive close approach to setting with timeout 0 just before close() is a great mitigation for 'misbehaving' clients.

• Timed out
• Invalid data
• Rate limited
  etc.

I can see a reusable function that performs an abortive close being worthwhile. However the default should be to not set outside of the above.

I don't believe the need to (write->drain->close) as a one shot is a common use case, and also should not be a default.

[bryancall]

Related PRs:
#8784
#8623
#8463

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

[bryancall]

In current master SO_LINGER is opt-in and off by default. The abortive close (SO_LINGER with a linger value of 0) is applied only when the SOCK_OPT_LINGER_ON flag (bit 0x4) is set in proxy.config.net.sock_option_flag_in or sock_option_flag_out (see src/iocore/net/UnixConnection.cc and src/iocore/net/Server.cc, and the flag definition in include/iocore/net/NetVCOptions.h). Both of those settings default to 0x1, which enables TCP_NODELAY only, so linger is not enabled by the default configuration. The concern raised here (that propagating proxy.config.net.sock_option_flag_in to client connections would turn linger on by default) does not apply, since the default flag value does not include the linger bit, and the discussion converged on keeping the abortive close as an opt-in mitigation rather than removing it. Since the behavior now matches that outcome, I am closing this. Please reopen if you want the linger option removed entirely rather than left opt-in.