Expose option to enable TLS when sniffing an Elasticsearch Cluster #2263

jennynilsen · 2020-05-28T16:28:51Z

Jaeger uses the default scheme set by the olivere client (which is http) when sniffing an Elasticsearch cluster without the option to change it. This makes it impossible to use sniffing with a TLS Elasticsearch cluster.

The scheme can be set using the SetScheme client option
https://pkg.go.dev/github.com/olivere/elatic/v7\?tab\=doc\#SetScheme

This change exposes that client option as a boolean command line option: --es.sniffer-tls-enabled

Signed-off-by: nilsenj jennynilsen@rentalcars.com

Which problem is this PR solving?

In our use case, jaeger-collector needs to connect to Elasticsearch running in Kubernetes from outside the cluster. By using sniffing, we can use the ingress for the initial node discovery and then send traffic directly to the elasticsearch nodes without it needing to go through the ingress.
In our case, when using ingress only (without sniffing) jaeger represents a substantial proportion of the ingress traffic. When we switched to using sniffing we were able to reduce the load on our ingress 10x.
If Elasticsearch is running with TLS enabled, then we need a way to tell jaeger-collector to use https when connecting to the discovered nodes.

Short description of the changes

Jaeger uses the default scheme set by the olivere client (which is http) when sniffing an Elasticsearch cluster without the option to change it. This makes it impossible to use sniffing with a TLS Elasticsearch cluster. The scheme can be set using SetScheme client option https://pkg.go.dev/github.com/olivere/elatic/v7\?tab\=doc\#SetScheme This change exposes that client option as a boolean command line option: --es.sniffer-tls-enabled Signed-off-by: nilsenj <jennynilsen@rentalcars.com>

pavolloffay · 2020-05-28T16:53:36Z

What --es.server-urls were you using with TLS configuration? We are using TLS with ES in our operator and there wasn't this problem https://github.com/jaegertracing/jaeger-operator/blob/b7ce28f171a416945d881cf7c018280baa4aae01/pkg/storage/elasticsearch.go#L22.

jennynilsen · 2020-05-29T12:35:42Z

The operator does not use sniffing. It connects using the service address which only works from within the cluster.
We want to connect to an Elasticsearch cluster in a different Kubernetes cluster and so --es.server-urls points to the address of an nginx ingress controller.

yurishkuro

please run make lint

make go-lint
make[1]: Entering directory `/home/travis/gopath/src/github.com/jaegertracing/jaeger'
Running go lint...
Lint Failures
pkg/es/config/config.go:49:2: struct field SnifferTlsEnabled should be SnifferTLSEnabled
plugin/storage/es/options.go:34:2: const suffixSnifferTlsEnabled should be suffixSnifferTLSEnabled
make[1]: *** [go-lint] Error 1

pavolloffay · 2020-06-02T08:30:32Z

pkg/es/config/config.go

@@ -288,6 +292,11 @@ func (c *Configuration) getConfigOptions(logger *zap.Logger) ([]elastic.ClientOp
 		// we don' have a valid token to do the check ad if we don't disable the check the service that
 		// uses this won't start.
 		elastic.SetHealthcheck(!c.AllowTokenFromContext)}
+	if c.SnifferTlsEnabled {
+		options = append(options, elastic.SetScheme("https"))


Could we derive the scheme from es.server-urls? That way we don't have to add new flag.

Using the scheme from es-server-urls would work for us because both ingress and the elasticsearch nodes are using https. So, I'm happy to make this change. However, strictly speaking the two things are independent of each other. The point of using sniffing is that we can use ingress/loadbalancer for the initial node discovery then connect to the actual elasticsearch nodes for sending traffic. Are you concerned about somebody that has an https enabled ingress/loadbalancer but uses http for the elasticsearch nodes?

I thought that people using TLS would normally protect all endpoints. If that is not the case we can merge your patch that adds an additional flag to control TLS just for sniffing.

The PR looks good I have suggested two edits. Once that is done we can merge it.

pavolloffay · 2020-06-02T15:58:42Z

plugin/storage/es/options.go

@@ -95,6 +96,7 @@ func NewOptions(primaryNamespace string, otherNamespaces ...string) *Options {
 				CreateIndexTemplates: true,
 				Version:              0,
 				Servers:              []string{defaultServerURL},
+				SnifferTlsEnabled:    false,


can be removed false is default

pavolloffay · 2020-06-02T15:59:05Z

pkg/es/config/config.go

@@ -288,6 +292,11 @@ func (c *Configuration) getConfigOptions(logger *zap.Logger) ([]elastic.ClientOp
 		// we don' have a valid token to do the check ad if we don't disable the check the service that
 		// uses this won't start.
 		elastic.SetHealthcheck(!c.AllowTokenFromContext)}
+	if c.SnifferTlsEnabled {
+		options = append(options, elastic.SetScheme("https"))
+	} else {


http is the default the else branch can be removed.

pavolloffay · 2020-06-02T16:04:51Z

pkg/es/config/config.go

@@ -288,6 +292,11 @@ func (c *Configuration) getConfigOptions(logger *zap.Logger) ([]elastic.ClientOp
 		// we don' have a valid token to do the check ad if we don't disable the check the service that
 		// uses this won't start.
 		elastic.SetHealthcheck(!c.AllowTokenFromContext)}
+	if c.SnifferTlsEnabled {
+		options = append(options, elastic.SetScheme("https"))


I thought that people using TLS would normally protect all endpoints. If that is not the case we can merge your patch that adds an additional flag to control TLS just for sniffing.

The PR looks good I have suggested two edits. Once that is done we can merge it.

Signed-off-by: nilsenj <jennynilsen@rentalcars.com>

codecov · 2020-06-12T15:37:05Z

Codecov Report

Merging #2263 into master will increase coverage by 0.04%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2263      +/-   ##
==========================================
+ Coverage   96.15%   96.19%   +0.04%     
==========================================
  Files         219      218       -1     
  Lines       10652    10692      +40     
==========================================
+ Hits        10242    10285      +43     
+ Misses        354      351       -3     
  Partials       56       56

Impacted Files	Coverage Δ
plugin/storage/es/options.go	`100.00% <100.00%> (ø)`
cmd/query/app/static_handler.go	`84.16% <0.00%> (-2.68%)`	⬇️
plugin/storage/es/factory.go	`100.00% <0.00%> (ø)`
plugin/storage/badger/options.go	`100.00% <0.00%> (ø)`
plugin/storage/memory/factory.go	`100.00% <0.00%> (ø)`
plugin/storage/cassandra/options.go	`100.00% <0.00%> (ø)`
plugin/storage/es/dependencystore/schema.go
plugin/storage/badger/factory.go	`98.16% <0.00%> (+0.03%)`	⬆️
cmd/query/app/server.go	`92.20% <0.00%> (+0.31%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e46f873...a298a3f. Read the comment docs.

yurishkuro

Thank you!

pavolloffay · 2020-06-15T10:08:58Z

@jennynilsen would you like to add your company to https://github.com/jaegertracing/jaeger/blob/master/ADOPTERS.md or comment on #207?

jennynilsen requested a review from a team as a code owner May 28, 2020 16:28

jennynilsen requested a review from jpkrohling May 28, 2020 16:28

pavolloffay added the storage/elasticsearch label May 28, 2020

yurishkuro reviewed May 29, 2020

View reviewed changes

pavolloffay reviewed Jun 2, 2020

View reviewed changes

pavolloffay removed the request for review from jpkrohling June 2, 2020 15:47

pavolloffay reviewed Jun 2, 2020

View reviewed changes

Removing default options and correcting lint failures

a298a3f

Signed-off-by: nilsenj <jennynilsen@rentalcars.com>

yurishkuro approved these changes Jun 12, 2020

View reviewed changes

yurishkuro merged commit b46738c into jaegertracing:master Jun 12, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose option to enable TLS when sniffing an Elasticsearch Cluster #2263

Expose option to enable TLS when sniffing an Elasticsearch Cluster #2263

jennynilsen commented May 28, 2020

pavolloffay commented May 28, 2020

jennynilsen commented May 29, 2020 •

edited

Loading

yurishkuro left a comment

pavolloffay Jun 2, 2020

jennynilsen Jun 2, 2020

pavolloffay Jun 2, 2020

pavolloffay Jun 2, 2020

pavolloffay Jun 2, 2020

pavolloffay Jun 2, 2020

codecov bot commented Jun 12, 2020

yurishkuro left a comment

pavolloffay commented Jun 15, 2020

Expose option to enable TLS when sniffing an Elasticsearch Cluster #2263

Expose option to enable TLS when sniffing an Elasticsearch Cluster #2263

Conversation

jennynilsen commented May 28, 2020

Which problem is this PR solving?

Short description of the changes

pavolloffay commented May 28, 2020

jennynilsen commented May 29, 2020 • edited Loading

yurishkuro left a comment

Choose a reason for hiding this comment

pavolloffay Jun 2, 2020

Choose a reason for hiding this comment

jennynilsen Jun 2, 2020

Choose a reason for hiding this comment

pavolloffay Jun 2, 2020

Choose a reason for hiding this comment

pavolloffay Jun 2, 2020

Choose a reason for hiding this comment

pavolloffay Jun 2, 2020

Choose a reason for hiding this comment

pavolloffay Jun 2, 2020

Choose a reason for hiding this comment

codecov bot commented Jun 12, 2020

Codecov Report

yurishkuro left a comment

Choose a reason for hiding this comment

pavolloffay commented Jun 15, 2020

jennynilsen commented May 29, 2020 •

edited

Loading