Thanos doesn't always respect the /etc/hosts file

[paulcdejean]

Thanos is giving me the error:

"level=error ts=2018-07-30T20:49:38.431204001Z caller=cluster.go:215 component=cluster msg="none of the peers was can be reached" peerType=source knownPeers=thanosquery.reviewpush.internal:10910 err="1 error occurred:\n\n* Failed to join 10.1.202.126: dial tcp 10.1.202.126:10910: i/o timeout""

However when I run this go program:

package main

import (
        "net"
        "fmt"
)

func main() {
        fmt.Println(net.LookupHost("thanosquery.reviewpush.internal"))
}

I get this output: [127.0.0.1] <nil>

root@monitoring-staging-10-1-209-175:~# thanos --version
thanos, version 0.1.0-rc.2 (branch: master, revision: ecfce89)
build user: root@2a8d986bc659
build date: 20180727-18:43:29
go version: go1.10.3

root@monitoring-staging-10-1-209-175:~# prometheus --version
prometheus, version 2.3.2 (branch: HEAD, revision: 71af5e29e815795e9dd14742ee7725682fa14b7b)
build user: root@5258e0bd9cc1
build date: 20180712-14:02:52
go version: go1.10.3

[bwplotka]

We can look through what resolver is actually used and why to support /etc/host files

[bwplotka]

Feel free to propose a PR if you have solution

[xjewer]

related hashicorp/consul#1507

hashicorp/memberlist uses firstly it's own lookup https://github.com/hashicorp/memberlist/blob/bc4f35373700bc93b12d01b6fdd91f74c369c088/memberlist.go#L341, which looks for /etc/resolv.conf

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.