clients not being cleaned up properly

[UriHendler]

When the TCP connection is dropped from the client side, Tile38 still keeps the client in its list of connections.

Here's an example when using tile38-cli:

1. Start new client
   The Tile38 log shows the connection:
   2017/02/09 09:32:22 [DEBU] opened connection: 1.2.3.4:3086

2. Let the client be idle for a minute or two. This causes the connection to timeout (not sure where this is set).

3. Send a command.

4. Client gives a TCP error message and exits, but the Tile38 log does not show an error or "closed connection" message. (At this point, the TCP port has been released to the OS, as expected.)

geofences.mydomain.test:9854> hooks geoFence*
read tcp 192.168.0.23:3086->tile38-server-ip-here:9854: wsarecv: A connection attempt failed because the connected party did not
properly respond after a period of time, or established connection failed because connected host has failed to respond.

5. Start another client.
6. Run the CLIENT LIST command and the old client will still be there.

tile38-cli -h geofences.mydomain.test -p 9854

geofences.mydomain.test:9854> client list
id=57 addr=1.2.3.4:3086 name= age=740 idle=684

To check/fix:

1. It seems the TCP connection timeout is set on the Tile38 side. Where/how is this configured?

2. Are the clients being cleaned up when the TCP connection is dropped?

[tidwall]

I tried your instructions but I haven't been able to reproduce. I waited 10 minutes.

There aren't any timeouts defined on the Tile38 server-side. It's pretty basic actually. Just a listen/accept pattern.

Could you provide some details on your environment? If I can reproduce, then I can watch the packets on wire and see which side it's coming from.

Regardless of which side it's coming from though, perhaps we could set up keepalives on the TCP connections on the server. This would allow for the server to discover when a client drops off. The downside is that under-the-hood there's a little more traffic being sent around.

[tidwall]

In fact, I think we should just add keepalives.

[tidwall]

I'm marking this issue as a bug, because the server should be responsible for managing it's connections, and TCP keepalives were created for this purpose.

TODO: Add TCP keepalives after the server accepts a tcp connection. Set the default period to 300 seconds. Add a config option to define the period duration.

[tidwall]

I just pushed an update to the master branch which should address this issue. Please test again and let me know. Thanks.

[tidwall]

@UriHendler I ran a few tests pulling over physical networks using cat-5 by hot unplugging cables and watching timeouts. The keepalive code seems to behave well.

I'm closing this as fixed for now. Feel free to reopen if you see any problems.

Thanks for you help!