Open
Description
openedon Dec 16, 2020
error msg:
2020/12/16 04:02:17 maxprocs: Updating GOMAXPROCS=1: using minimum allowed GOMAXPROCS
{"level":"info","ts":1608091337.2560472,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1608091337.2560928,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1608091337.2562084,"caller":"flags/admin.go:121","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1608091337.2562501,"caller":"flags/admin.go:127","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1608091337.256265,"caller":"flags/admin.go:113","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"fatal","ts":1608091342.4766536,"caller":"command-line-arguments/main.go:74","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available","stacktrace":"main.main.func1\n\tcommand-line-arguments/main.go:74\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tcommand-line-arguments/main.go:133\nruntime.main\n\truntime/proc.go:204"}
external es 7.10.1 yaml file(es operator 1.3.1):
https://www.elastic.co/guide/en/cloud-on-k8s/1.3/k8s-deploy-eck.html
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: jaeger-storage
spec:
version: 7.10.1
http:
tls:
selfSignedCertificate:
subjectAltNames:
- dns: jaeger-storage-es-http.observability.svc.cluster.local
- dns: jaeger-storage-es-http.observability.svc
- dns: jaeger-storage-es-http
nodeSets:
- name: default
count: 1
config:
node.roles:
- master
- data
node.store.allow_mmap: false
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: xxxjaeger yaml file:
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
name: tds-jaeger-dev
spec:
strategy: production
collector:
maxReplicas: 3
resources:
limits:
cpu: 100m
memory: 128Mi
storage:
type: elasticsearch
options:
es:
# server-urls: https://jaeger-storage-es-http:9200
# server-urls: https://jaeger-storage-es-http.observability.svc:9200
server-urls: https://jaeger-storage-es-http.observability.svc.cluster.local:9200
tls: "true"
tls.ca: /es/certificates/ca.crt
tls.cert: /es/certificates/tls.crt
tls.key: /es/certificates/tls.key
secretName: jaeger-secret
volumeMounts:
- name: certificates
mountPath: /es/certificates/
readOnly: true
volumes:
- name: certificates
secret:
secretName: jaeger-storage-es-http-certs-internalI have tried all these urls, but still got the same error
# server-urls: https://jaeger-storage-es-http:9200
# server-urls: https://jaeger-storage-es-http.observability.svc:9200
server-urls: https://jaeger-storage-es-http.observability.svc.cluster.local:9200
I have tried the same config follow this issuehttps://github.com//issues/496, but still not work.
The error tell healthcheck timeout, so I use curl to test as follow
curl -X HEAD -u "elastic:fZdf2d80D238g0mEr03CDM3e" -k "https://172.21.10.94:9200" -v
* About to connect() to 172.21.10.94 port 9200 (#0)
* Trying 172.21.10.94...
* Connected to 172.21.10.94 (172.21.10.94) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=jaeger-storage-es-http.observability.es.local,OU=jaeger-storage
* start date: 12月 16 03:39:31 2020 GMT
* expire date: 12月 16 03:49:31 2021 GMT
* common name: jaeger-storage-es-http.observability.es.local
* issuer: CN=jaeger-storage-http,OU=jaeger-storage
* Server auth using Basic with user 'elastic'
> HEAD / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzpmWmRmMmQ4MEQyMzhnMG1FcjAzQ0RNM2U=
> User-Agent: curl/7.29.0
> Host: 172.21.10.94:9200
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 557
<
this way(-X HEAD), the response code is 200, but just got stuck, the request can't finish
curl --head -u "elastic:fZdf2d80D238g0mEr03CDM3e" -k "https://172.21.10.94:9200" -v
* About to connect() to 172.21.10.94 port 9200 (#0)
* Trying 172.21.10.94...
* Connected to 172.21.10.94 (172.21.10.94) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=jaeger-storage-es-http.observability.es.local,OU=jaeger-storage
* start date: 12月 16 03:39:31 2020 GMT
* expire date: 12月 16 03:49:31 2021 GMT
* common name: jaeger-storage-es-http.observability.es.local
* issuer: CN=jaeger-storage-http,OU=jaeger-storage
* Server auth using Basic with user 'elastic'
> HEAD / HTTP/1.1
> Authorization: Basic ZWxhc3RpYzpmWmRmMmQ4MEQyMzhnMG1FcjAzQ0RNM2U=
> User-Agent: curl/7.29.0
> Host: 172.21.10.94:9200
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
content-type: application/json; charset=UTF-8
< content-length: 557
content-length: 557
<
* Connection #0 to host 172.21.10.94 left intact
this way(--head) finish successfully.
So I'm wondering if the problem is with healcheck method processing in es.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment