[ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8302->10.254.0.7:8302: sendto: operation not permitted

[prologic]

HI,

I'm getting some errors in the seed and the nodes that I don't yet fully understand.

I'm using this docker stack / docker-compose.yml and spinning up the service(s) with:

$ docker stack deploy -c consul.yml consul

Seed logs:

==> Starting Consul agent...
==> WARNING: Expect Mode enabled, expecting 3 servers
==> Consul agent running!
           Version: 'v0.8.3'
           Node ID: 'af95ca51-bce1-b19b-e3c5-b8bf57df02f1'
         Node name: 'seed'
        Datacenter: 'dc1'
            Server: true (bootstrap: false)
       Client Addr: 127.0.0.1 (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 10.254.0.7 (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false
             Atlas: <disabled>

==> Log data will now stream in as it occurs:

    2017/06/07 06:41:28 [INFO] raft: Initial configuration (index=0): []
    2017/06/07 06:41:28 [INFO] raft: Node at 10.254.0.7:8300 [Follower] entering Follower state (Leader: "")
    2017/06/07 06:41:28 [INFO] serf: EventMemberJoin: seed 10.254.0.7
    2017/06/07 06:41:28 [INFO] serf: EventMemberJoin: seed.dc1 10.254.0.7
    2017/06/07 06:41:28 [INFO] consul: Adding LAN server seed (Addr: tcp/10.254.0.7:8300) (DC: dc1)
    2017/06/07 06:41:28 [INFO] consul: Handled member-join event for server "seed.dc1" in area "wan"
    2017/06/07 06:41:35 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:41:36 [WARN] raft: no known peers, aborting election
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:41:57 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:41:57 [INFO] consul: Handled member-join event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:41:57 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:41:57 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:41:58 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:41:59 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:42:02 [ERR] agent: coordinate update error: No cluster leader
==> Failed to check for updates: Get https://checkpoint-api.hashicorp.com/v1/check/consul?arch=amd64&os=linux&signature=b7827419-5378-9b71-0bcc-67ea0039da7d&version=0.8.3: dial tcp: lookup checkpoint-api.hashicorp.com on 127.0.0.11:53: read udp 127.0.0.1:58949->127.0.0.11:53: i/o timeout
    2017/06/07 06:42:07 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 1s...
    2017/06/07 06:42:09 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:42:09 [ERR] memberlist: Push/Pull with f88dacfafd73 failed: dial tcp 10.255.0.16:8301: i/o timeout
    2017/06/07 06:42:14 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:14 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:42:18 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 2s...
    2017/06/07 06:42:19 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:42:24 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:24 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:42:27 [INFO] serf: EventMemberJoin: f88dacfafd73.dc1 10.255.0.16
    2017/06/07 06:42:27 [INFO] consul: Handled member-join event for server "f88dacfafd73.dc1" in area "wan"
    2017/06/07 06:42:28 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:42:28 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:42:30 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:42:30 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 4s...
    2017/06/07 06:42:32 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:42:35 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:42:35 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:42:36 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:42:41 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:42:44 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 8s...
    2017/06/07 06:42:56 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:42:57 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:42:58 [INFO] serf: EventMemberJoin: 1ae40b55c20b.dc1 10.255.0.10
    2017/06/07 06:42:58 [INFO] consul: Handled member-join event for server "1ae40b55c20b.dc1" in area "wan"
    2017/06/07 06:43:01 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:43:02 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 16s...
    2017/06/07 06:43:03 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:43:06 [INFO] memberlist: Marking 6d158c5b5562.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:06 [INFO] serf: EventMemberFailed: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:43:06 [INFO] consul: Handled member-failed event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:43:09 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:43:10 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:43:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:43:15 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:15 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:43:19 [ERR] memberlist: Push/Pull with f88dacfafd73 failed: dial tcp 10.255.0.16:8301: i/o timeout
    2017/06/07 06:43:20 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:43:21 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:43:26 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:43:28 [ERR] consul: Failed to confirm peer status for 6d158c5b5562: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout. Retrying in 32s...
    2017/06/07 06:43:31 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:43:32 [ERR] memberlist: Push/Pull with 1ae40b55c20b.dc1 failed: dial tcp 10.255.0.10:8302: i/o timeout
    2017/06/07 06:43:36 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:36 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:43:38 [INFO] serf: attempting reconnect to 1ae40b55c20b 10.255.0.10:8301
    2017/06/07 06:43:41 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:43:42 [INFO] memberlist: Suspect f88dacfafd73.dc1 has failed, no acks received
    2017/06/07 06:43:46 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:43:46 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:43:47 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:43:53 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:02 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:44:08 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: rpc error: lead thread didn't get connection. Retrying in 1s...
    2017/06/07 06:44:12 [INFO] memberlist: Marking f88dacfafd73.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:44:12 [INFO] serf: EventMemberFailed: f88dacfafd73.dc1 10.255.0.16
    2017/06/07 06:44:12 [INFO] consul: Handled member-failed event for server "f88dacfafd73.dc1" in area "wan"
    2017/06/07 06:44:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:44:15 [INFO] memberlist: Suspect f88dacfafd73.dc1 has failed, no acks received
    2017/06/07 06:44:18 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:44:19 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 2s...
    2017/06/07 06:44:22 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:44:24 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:28 [INFO] serf: attempting reconnect to 6d158c5b5562.dc1 10.255.0.13:8302
    2017/06/07 06:44:29 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:44:29 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:44:29 [ERR] memberlist: Push/Pull with 6d158c5b5562 failed: dial tcp 10.255.0.13:8301: i/o timeout
    2017/06/07 06:44:31 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 4s...
    2017/06/07 06:44:34 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:44:37 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:44:45 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 8s...
    2017/06/07 06:44:48 [INFO] memberlist: Suspect 1ae40b55c20b.dc1 has failed, no acks received
    2017/06/07 06:44:51 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:44:53 [INFO] serf: EventMemberJoin: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:44:53 [INFO] serf: EventMemberJoin: f88dacfafd73 10.255.0.16
    2017/06/07 06:44:58 [INFO] serf: attempting reconnect to 6d158c5b5562 10.255.0.13:8301
    2017/06/07 06:45:01 [INFO] serf: EventMemberJoin: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:45:01 [INFO] consul: Handled member-join event for server "6d158c5b5562.dc1" in area "wan"
    2017/06/07 06:45:02 [ERR] memberlist: Push/Pull with 1ae40b55c20b.dc1 failed: dial tcp 10.255.0.10:8302: i/o timeout
    2017/06/07 06:45:03 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 16s...
    2017/06/07 06:45:04 [INFO] memberlist: Suspect 1ae40b55c20b has failed, no acks received
    2017/06/07 06:45:09 [INFO] memberlist: Marking 1ae40b55c20b as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:09 [INFO] serf: EventMemberFailed: 1ae40b55c20b 10.255.0.10
    2017/06/07 06:45:09 [ERR] memberlist: Push/Pull with 1ae40b55c20b failed: dial tcp 10.255.0.10:8301: i/o timeout
    2017/06/07 06:45:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:45:13 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:45:14 [INFO] memberlist: Suspect f88dacfafd73 has failed, no acks received
    2017/06/07 06:45:18 [INFO] memberlist: Marking 1ae40b55c20b.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:18 [INFO] serf: EventMemberFailed: 1ae40b55c20b.dc1 10.255.0.10
    2017/06/07 06:45:18 [INFO] consul: Handled member-failed event for server "1ae40b55c20b.dc1" in area "wan"
    2017/06/07 06:45:19 [INFO] memberlist: Marking f88dacfafd73 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:19 [INFO] serf: EventMemberFailed: f88dacfafd73 10.255.0.16
    2017/06/07 06:45:21 [INFO] memberlist: Suspect 1ae40b55c20b.dc1 has failed, no acks received
    2017/06/07 06:45:25 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:45:27 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:45:28 [INFO] serf: attempting reconnect to f88dacfafd73.dc1 10.255.0.16:8302
    2017/06/07 06:45:29 [ERR] consul: Failed to confirm peer status for f88dacfafd73: rpc error: failed to get conn: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout. Retrying in 32s...
    2017/06/07 06:45:30 [INFO] memberlist: Marking 6d158c5b5562 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:45:30 [INFO] serf: EventMemberFailed: 6d158c5b5562 10.255.0.13
    2017/06/07 06:45:36 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:45:37 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:45:38 [INFO] serf: attempting reconnect to 6d158c5b5562 10.255.0.13:8301
    2017/06/07 06:45:56 [INFO] memberlist: Suspect 6d158c5b5562.dc1 has failed, no acks received
    2017/06/07 06:46:01 [ERR] agent: coordinate update error: No cluster leader
    2017/06/07 06:46:01 [INFO] consul: Found expected number of peers, attempting bootstrap: 10.254.0.7:8300,10.255.0.13:8300,10.255.0.16:8300
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 1ae40b55c20b (Addr: tcp/10.255.0.10:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server f88dacfafd73 (Addr: tcp/10.255.0.16:8300) (DC: dc1)
    2017/06/07 06:46:01 [INFO] consul: Removing LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:04 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2017/06/07 06:46:04 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 2
    2017/06/07 06:46:10 [WARN] raft: Election timeout reached, restarting election
    2017/06/07 06:46:10 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 3
    2017/06/07 06:46:13 [INFO] serf: EventMemberJoin: 6d158c5b5562 10.255.0.13
    2017/06/07 06:46:13 [INFO] consul: Adding LAN server 6d158c5b5562 (Addr: tcp/10.255.0.13:8300) (DC: dc1)
    2017/06/07 06:46:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:14 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:14 [ERR] agent: failed to sync remote state: No cluster leader
    2017/06/07 06:46:18 [INFO] serf: attempting reconnect to f88dacfafd73 10.255.0.16:8301
    2017/06/07 06:46:19 [WARN] raft: Election timeout reached, restarting election
    2017/06/07 06:46:19 [INFO] raft: Node at 10.254.0.7:8300 [Candidate] entering Candidate state in term 4
    2017/06/07 06:46:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:20 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:25 [INFO] memberlist: Suspect 6d158c5b5562 has failed, no acks received
    2017/06/07 06:46:26 [INFO] memberlist: Marking 6d158c5b5562.dc1 as failed, suspect timeout reached (0 peer confirmations)
    2017/06/07 06:46:26 [INFO] serf: EventMemberFailed: 6d158c5b5562.dc1 10.255.0.13
    2017/06/07 06:46:26 [INFO] consul: Handled member-failed event for server "6d158c5b5562.dc1" in area "wan"
==> Caught signal: terminated
    2017/06/07 06:46:28 [INFO] agent: requesting shutdown
    2017/06/07 06:46:28 [INFO] consul: shutting down server
    2017/06/07 06:46:28 [WARN] serf: Shutdown without a Leave
    2017/06/07 06:46:28 [ERR] dns: error starting tcp server: accept tcp 127.0.0.1:8600: use of closed network connection
    2017/06/07 06:46:28 [WARN] serf: Shutdown without a Leave
    2017/06/07 06:46:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.16:8300 10.255.0.16:8300}: dial tcp 10.254.0.7:0->10.255.0.16:8300: i/o timeout
    2017/06/07 06:46:29 [ERR] raft: Failed to make RequestVote RPC to {Voter 10.255.0.13:8300 10.255.0.13:8300}: dial tcp 10.254.0.7:0->10.255.0.13:8300: i/o timeout
    2017/06/07 06:46:29 [INFO] agent: shutdown complete

Sample node logs:

    2017/06/07 06:54:49 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.254.0.7:8301: sendto: operation not permitted
    2017/06/07 06:54:49 [ERR] memberlist: Push/Pull with seed.dc1 failed: dial tcp 10.254.0.7:8302: getsockopt: no route to host
    2017/06/07 06:54:51 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.255.0.13:8301: sendto: operation not permitted
    2017/06/07 06:54:52 [ERR] memberlist: Failed to send ping: write udp 10.255.0.10:8301->10.254.0.7:8301: sendto: operation not permitted
    2017/06/07 06:54:52 [ERR] memberlist: Push/Pull with seed failed: dial tcp 10.254.0.7:8301: getsockopt: no route to host

[lnshi]

@prologic

The problem is related to three parts:

1. ur -bind={{ GetInterfaceIP "eth0" }}
2. ur ports port mapping
3. how docker routing mesh works

1 make all consul traffic will go through physical IP eth0 and the corresponding port
2 u just map the container port to physical host also
with 3, here

You can rely on the routing mesh. When you publish a service port, the swarm makes the service accessible at the target port on every node, regardless of whether there is a task for the service running on that node or not. This is less complex and is the right choice for many types of services.

u can see how many services are trying to use one port in ur each physical node, so : port conflicting.

solution, map the port only to the host which the service is running, which is the host mode

ports:
  - target: 65301
    published: 65301
    mode: host

don't forget use above syntax with docker-compose version 3.2

[slackpad]

Thanks @lnshi for fielding this one! @prologic can we close this out?

[prologic]

Sure! I haven't actually gone back to try spinning this up yet; I may find some time this weekend -- Can we keep this open so others i the community can comment and maybe put up a final solution / working compose v3 spec? James Mills / prologic E: prologic@shortcircuit.net.au W: prologic.shortcircuit.net.au

…

On Sat, Jun 24, 2017 at 3:14 PM, James Phillips ***@***.***> wrote: Thanks @lnshi <https://github.com/lnshi> for fielding this one! @prologic <https://github.com/prologic> can we close this out? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#60 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOv-ouKGvVSBIKnicoUdpaZ7y5OQCrHks5sHYo1gaJpZM4NySBL> .

[prologic]

@lnshi I just had a chance to review your comments and I'm a bit confused to say the least. What's the point of using an overlay network if I'm just going to bind the conteiner's ports to the host? What am I missing here? The whole point of my (half baked) consul.yml spec above is to spin up a consul cluster as a docker service for things that need a distributed kvstore.

[prologic]

Finally got this working:

consul.yml:

version: '3.2'

#customise this with options from
#https://www.consul.io/docs/agent/options.html

services:
  seed:
    hostname: seed
    image: consul:latest
    deploy:
      restart_policy:
        condition: none  #we do not want this to be restarted on timeout (see entrypoint options below)
      replicas: 1
      placement:
        constraints:
          - "node.labels.access == temp"
          - "node.labels.access != consul"
    environment:
      - "CONSUL_LOCAL_CONFIG={\"disable_update_check\": true}"
      - "CONSUL_BIND_INTERFACE=eth0"
    entrypoint:
      - timeout     #this seed fires up the cluster after which it is no longer needed
      - -sTERM      #this is the same signal as docker would send on a scale down / stop
      - -t300       #terminate after 5 mins
      - consul
      - agent
      - -server
      - -bootstrap-expect=3
      - -data-dir=/tmp/consuldata
      - -bind={{ GetInterfaceIP "eth0" }}
    networks:
      - "consul"

  cluster:
    image: consul:latest
    depends_on:
      - "seed"
    deploy:
      mode: global                                      ##this will deploy to all nodes that
      placement:
        constraints:
          - "node.labels.access == consul"            ##have the consul label
          - "node.labels.access != temp"
    environment:
      - "CONSUL_LOCAL_CONFIG={\"disable_update_check\": true}"
      - "CONSUL_BIND_INTERFACE=eth0"
      - "CONSUL_HTTP_ADDR=0.0.0.0"
    entrypoint:
      - consul
      - agent
      - -server
      - -data-dir=/tmp/consuldata
      - -bind={{ GetInterfaceIP "eth0" }}
      - -client=0.0.0.0
      - -retry-join=seed:8301
      - -ui                                              ##assuming you want the UI on
    networks:
      - "consul"
    ports:
      - "8500:8500"
      - "8600:8600"

networks:
  consul:
    driver: overlay

Usage:

$ docker stack deploy -c consul.yml consul
$ docker stack ls
NAME    SERVICES
consul  2

$ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                   PORTS
7x6s8vxl8csc        consul_cluster      global              3/3                 consul:latest           *:8500->8500/tcp,*:8600->8600/tcp
cl4f9ob98oji        consul_seed         replicated          1/1                 consul:latest