Enable IAM based auth to ES for AWS clients (#465) #1036

wesleyk · 2018-08-30T20:55:43Z

Allows clients to specify AWS IAM configuration
details for their connection to ElasticSearch.
We allow this to enable an additional layer
of security for those clients running on AWS.

More details on connecting to ElasticSearch
on AWS here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html

Signed-off-by: Wesley Kim wesley@squareup.com

wesleyk · 2018-08-30T21:23:23Z

Hmm could use some help here, I'm getting a build failure that looks unrelated to the ES changes I made:

cmd/ingester/app/consumer/consumer_test.go:73:7: cannot use partitionConsumerWrapper literal (type *partitionConsumerWrapper) as type cluster.PartitionConsumer in send:
	*partitionConsumerWrapper does not implement cluster.PartitionConsumer (missing InitialOffset method)
cmd/ingester/app/consumer/consumer_test.go:141:21: cannot use partitionConsumerWrapper literal (type *partitionConsumerWrapper) as type cluster.PartitionConsumer in field value:
	*partitionConsumerWrapper does not implement cluster.PartitionConsumer (missing InitialOffset method)
make: *** [lint] Error 2

EDIT: ah, it's due to the dependency being updated likely. I'll limit my glide update to only introduce the relevant AWS deps.

EDIT2: the update looks to have been required; the failure was due to a test that implemented one of its interfaces as a fake. I went ahead and updated that test.

codecov · 2018-08-30T21:54:43Z

Codecov Report

Merging #1036 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1036   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files         139     139           
  Lines        6415    6423    +8     
======================================
+ Hits         6415    6423    +8

Impacted Files	Coverage Δ
plugin/storage/es/options.go	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ddf9b22...4499321. Read the comment docs.

Allows clients to specify jaeger to use AWS IAM auth for their connection to ElasticSearch. We allow this to enable an additional layer of security for those clients running on AWS. More details on connecting to ElasticSearch on AWS here: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-vpc.html AWS configuration and credentials are expected to be available at either ~/.aws/credentials or ~/.aws/config. More details on configuration here: https://docs.aws.amazon.com/sdk-for-go/api/aws/session/#Session Signed-off-by: Wesley Kim <wesley@squareup.com>

yurishkuro · 2018-09-05T03:20:35Z

pkg/es/config/config.go

 	"github.com/pkg/errors"
+	"github.com/sha1sum/aws_signing_client"


I need to consult our lawyers if embedding crypto capability creates an export issue for Jaeger binaries.

yurishkuro · 2018-09-05T03:22:23Z

pkg/es/config/config.go

+
+	// if AWS IAM is enabled, instantiate the elastic client with a signing client created via the AWS SDK
+	if c.AwsIamConfig.Enabled {
+		sess, err := session.NewSession()


why is a live session needed here? Looks like you're only accessing Config from it.

yurishkuro · 2018-09-05T03:23:34Z

plugin/storage/es/options.go

+	flagSet.Bool(
+		nsConfig.namespace+suffixAwsIamEnabled,
+		nsConfig.AwsIamConfig.Enabled,
+		"Whether to connect to AWS ElasticSearch with IAM authentication. Requires the proper .aws/ files to be setup to connect.")


should there be an option with the directory name of these aws files? Relying on the working directory seems brittle.

Yeah, agreed. I think I'm actually leaning back to preferring that clients explicitly pass in the credential values that we require, rather than implicitly loading them from a file. Furthermore, see discussion #465 (comment) as to why we can't actually directly use the session created from such file.

I can go back to accepting the appropriate credential params.

cc @jmwaniki

Before I push over theses changes, here's the commit that has the params explicitly passed in, that I'm now in favor in. Mind taking a peak at that? wesleyk@b3eb1a1

@wesleyk
Regarding why a new session was needed, the credentials are retrieved from AWS STS when creating a new session using the instance IAM role. It uses the endpoint provider credentials.
See this: https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/endpointcreds/

I have a solution I've been testing that forces the credentials to be refreshed before expiring and it's worked for 2 days without a problem. That solution involves using the ec2RoleCreds
see: https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/ec2rolecreds/
with an expiry window. We can discuss further and see if it's a good solution

Thanks!

Heads up, I was out for a bit. Will take a closer look at this sometime this week!

Also, @jmwaniki feel free to push out another change if you have something locally.

yurishkuro · 2018-09-05T03:24:17Z

plugin/storage/es/options_test.go

@@ -50,7 +52,7 @@ func TestOptionsWithFlags(t *testing.T) {
 		"--es.sniffer=true",
 		"--es.max-span-age=48h",
 		"--es.num-shards=20",
-		"--es.num-replicas=10",


why is this one removed?

My bad, reverting!

yurishkuro · 2018-09-05T03:26:50Z

Good job!

I'd rather not depend on aws libs, but don't see any another way, until we implement #422. So I'm ok with it for now, later we can refactor into a plugin.

yurishkuro · 2018-10-30T14:48:09Z

@wesleyk please opine on #1139 (comment)

yurishkuro · 2018-11-16T20:38:47Z

@wesleyk I am going to close this on the assumption that auth is now addressed by #1139. Feel free to reopen the discussion if it does not address your needs.

wesleyk · 2018-11-16T21:23:01Z

@yurishkuro sounds good, will re-open if so.

lucasw · 2020-02-14T14:14:10Z

(Coming from https://gitter.im/jaegertracing/Lobby?at=5e4601d9cd2d737bb0781032)

I've looked through #1139 but I'm not clear on how to use it with aws elasticsearch- it leads to #1863, I can follow up there if successful. @wesleyk can you provide more details on how it was solved for you? Does it only apply to VPC configurations?

My instance is set to public access and I can get in remotely with a IAM access and secret key and sigv4.

The only reference to tls I see in the aws es configuration is in reference to ES cluster node-node communication. https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/ntn.html

The other command line flag for collector that looks promising is the --es.token_file- what is the format of that file, can I generate it with aws sts get-session-token and then have a script alter the format? Or make a python script that used aws_requests_auth to output a file?

From https://github.com/jaegertracing/jaeger/blob/master/pkg/es/config/config.go#L352 it looks like the token file file should be a raw string, not json or the text format, but I run into failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available after altering the get-session-token json to only have the value of SessionToken in it.

emyes · 2020-04-30T12:56:16Z

Hi All,
Good day.
I'm trying to use AWS Elastic search as my datasource to Jaeger Collector. I'm able to connect without IAM configuration in access policies . With IAM configuration enabled , i'm not able to connect. Please find logs from jaeger-collector container below

{
"level": "fatal",
"ts": 1588237836.2432656,
"caller": "collector/main.go:70",
"msg": "Failed to init storage factory",
"error": "failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available",
"stacktrace": "main.main.func1\n\tgithub.com/jaegertracing/jaeger@/cmd/collector/main.go:70\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/cobra@v0.0.3/command.go:762\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/cobra@v0.0.3/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/cobra@v0.0.3/command.go:800\nmain.main\n\tgithub.com/jaegertracing/jaeger@/cmd/collector/main.go:126\nruntime.main\n\truntime/proc.go:203"

I couldn't find any proper documentation about usage of Jaeger with AWS Elastic Search IAM enabled.
Any help on this would be appreciated @jpkrohling @yurishkuro . I'm not sure whether it is good thing to tag you guys directly in this comment. Sorry if i did anything wrong.

ovigio · 2020-04-30T21:52:06Z

Hello @emyes, I would suggest to explore a way to use a proxy to sign your requests to ES, have this proxy run along side the collector as a side car and configure the collector to call ES through it. AWS offers a way to run containers in vpc mode, which will give each container an ENI and allow them to talk to each other through localhost. You'll have to find a way to have the proxy that will sign the requests to ES be up before the collector starts up.

…

On Thu, Apr 30, 2020, 8:56 AM SathishKumar ***@***.***> wrote: Hi All, Good day. I'm trying to use AWS Elastic search as my datasource to Jaeger Collector. I'm able to connect without IAM configuration in access policies . With IAM configuration enabled , i'm not able to connect. Please find logs from jaeger-collector container below { "level": "fatal", "ts": 1588237836.2432656, "caller": "collector/main.go:70", "msg": "Failed to init storage factory", "error": "failed to create primary Elasticsearch client: health check timeout: no Elasticsearch node available", "stacktrace": "main.main.func1\n\ tgithub.com/jaegertracing/jaeger@***@***.******@***.******@***.***/command.go:800\nmain.main\n\tgithub.com/jaegertracing/jaeger@/cmd/collector/main.go:126\nruntime.main\n\truntime/proc.go:203 <http://tgithub.com/jaegertracing/jaeger@***@***.******@***.******@***.***/command.go:800%5Cnmain.main%5Cn%5Ctgithub.com/jaegertracing/jaeger@/cmd/collector/main.go:126%5Cnruntime.main%5Cn%5Ctruntime/proc.go:203> " I couldn't find any proper documentation about usage of Jaeger with AWS Elastic Search IAM enabled. Any help on this would be appreciated. .! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1036 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC67U2Y3M6WQ73EAEBNBQFLRPFYQDANCNFSM4FSQ2FDA> .

emyes · 2020-05-01T08:19:36Z

@jmwaniki : Thanks for your suggestions. Appreciated. I will have a look at it. I thought this ticket is something to bring AWS signing capability in Jaeger which has been Implemented but not having enough documentation to use it

tjjjwxzq · 2020-10-13T10:58:07Z

Hi, I wonder if there's been any plans to re-look this feature. I don't think the sister-PR #1139 necessarily addresses all the possible use-cases that one might have with IAM authentication.

For example, on my current project we are running an ElasticSearch cluster on AWS with Fine-grained access control. How we're authenticating with this cluster with other services, such as fluentd, is via IAM roles, which makes the integration smooth and easy. As it is with Jaeger, it looks like we have no other option but create another elasticsearch local user just for Jaeger and explicitly supply those user credentials to Jaeger config so it can be used in the basic auth headers, otherwise when Jaeger controller starts up its Elastisearch healthchecks simply die with a 401 error.

vmw-web · 2024-02-21T19:16:10Z

I am Looking to enable IAM authentication on the Jaeger collector. Is this currently supported? Please direct me to the configuration options.

wesleyk requested review from black-adder, jpkrohling, pavolloffay, vprithvi and yurishkuro as code owners August 30, 2018 20:55

wesleyk force-pushed the wesley-aws-signing branch 2 times, most recently from 213e52e to 0b23f4b Compare August 30, 2018 21:10

wesleyk force-pushed the wesley-aws-signing branch from 0b23f4b to b3eb1a1 Compare August 30, 2018 21:46

wesleyk force-pushed the wesley-aws-signing branch from b3eb1a1 to 4499321 Compare August 31, 2018 17:49

yurishkuro reviewed Sep 5, 2018

View reviewed changes

yurishkuro mentioned this pull request Oct 25, 2018

Fixed "Using Client Certificate for Elastic Search Authentication" #1139

Merged

yurishkuro closed this Nov 16, 2018

sohan mentioned this pull request Apr 16, 2021

Support AWS Elasticsearch IAM authentication by signing requests #465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable IAM based auth to ES for AWS clients (#465) #1036

Enable IAM based auth to ES for AWS clients (#465) #1036

wesleyk commented Aug 30, 2018

wesleyk commented Aug 30, 2018 •

edited

Loading

codecov bot commented Aug 30, 2018 •

edited

Loading

yurishkuro Sep 5, 2018

yurishkuro Sep 5, 2018

yurishkuro Sep 5, 2018

wesleyk Sep 5, 2018

wesleyk Sep 5, 2018

ovigio Sep 6, 2018

wesleyk Sep 24, 2018

yurishkuro Sep 5, 2018

wesleyk Sep 5, 2018

yurishkuro commented Sep 5, 2018

yurishkuro commented Oct 30, 2018

yurishkuro commented Nov 16, 2018

wesleyk commented Nov 16, 2018

lucasw commented Feb 14, 2020 •

edited

Loading

emyes commented Apr 30, 2020 •

edited

Loading

ovigio commented Apr 30, 2020 via email •

edited

Loading

emyes commented May 1, 2020

tjjjwxzq commented Oct 13, 2020

vmw-web commented Feb 21, 2024

		"github.com/pkg/errors"
		"github.com/sha1sum/aws_signing_client"

Enable IAM based auth to ES for AWS clients (#465) #1036

Enable IAM based auth to ES for AWS clients (#465) #1036

Conversation

wesleyk commented Aug 30, 2018

wesleyk commented Aug 30, 2018 • edited Loading

codecov bot commented Aug 30, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yurishkuro commented Sep 5, 2018

yurishkuro commented Oct 30, 2018

yurishkuro commented Nov 16, 2018

wesleyk commented Nov 16, 2018

lucasw commented Feb 14, 2020 • edited Loading

emyes commented Apr 30, 2020 • edited Loading

ovigio commented Apr 30, 2020 via email • edited Loading

emyes commented May 1, 2020

tjjjwxzq commented Oct 13, 2020

vmw-web commented Feb 21, 2024

wesleyk commented Aug 30, 2018 •

edited

Loading

codecov bot commented Aug 30, 2018 •

edited

Loading

lucasw commented Feb 14, 2020 •

edited

Loading

emyes commented Apr 30, 2020 •

edited

Loading

ovigio commented Apr 30, 2020 via email •

edited

Loading