Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

Closed
flyerjayden opened this issue Mar 16, 2023 · 19 comments
Closed

Kafka Exporter w/ AWS_MSK_IAM Mechanism Not Working #19747

flyerjayden opened this issue Mar 16, 2023 · 19 comments

Comments

@flyerjayden
Copy link

Component(s)

exporter/kafka

What happened?

Description

I am getting following error message when authentication mechanism is set to AWS_MSK_IAM:

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Steps to Reproduce

Deploy OpenTelemetry collector using following config:

exporters:
  kafka:
    topic: {{ kafka_topic_name }}
    brokers:
      - {{ broker1_host }}:9098
      - {{ broker2_host}}:9098
    auth:
      sasl:
        username: {{ aws_access_key }}
        password: {{ aws_secret_key }}
        mechanism: AWS_MSK_IAM
        aws_msk:
          region: us-west-2
          broker_addr: {{ broker1_host }}:9098

Expected Result

OpenTelemetry collector sends traces to MSK topic

Actual Result

Error message:

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Collector version

0.73

Environment information

Environment

OS: EKS 1.24 / Amazon Linux 2
Compiler(if manually compiled): Used Docker image from Docker hub

OpenTelemetry Collector configuration

exporters:
  kafka:
    topic: {{ kafka_topic_name }}
    brokers:
      - {{ broker1_host }}:9098
      - {{ broker2_host}}:9098
    auth:
      sasl:
        username: {{ aws_access_key }}
        password: {{ aws_secret_key }}
        mechanism: AWS_MSK_IAM
        aws_msk:
          region: us-west-2
          broker_addr: {{ broker1_host }}:9098

Log output

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/16 06:28:18 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

Additional context

It seems Sarama (https://github.com/Shopify/sarama/blob/7dbf0b5d85538641227a30a281f1a54e18a94cb7/broker.go#L64) does not support (and they are not planning to support) AWS_MSK_IAM mechanism.

If anyone was able to configure Kafka exporter with IAM, your advices would be very appreciated.

@flyerjayden flyerjayden added bug Something isn't working needs triage New item requiring triage labels Mar 16, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@atoulme
Copy link
Contributor

atoulme commented Mar 17, 2023

I think we have an open PR for this. Would you like to help test it out or review it?

@flyerjayden
Copy link
Author

Definitely I would love to help!

@atoulme
Copy link
Contributor

atoulme commented Mar 17, 2023

I was wrong sorry, it’s for another component: #19650 (review)

I bet we can reuse the logic though.

@atoulme atoulme added enhancement New feature or request and removed bug Something isn't working needs triage New item requiring triage labels Mar 17, 2023
@flyerjayden
Copy link
Author

Do you know which component has that implementation? Commit that you shared is about README.md update.

@atoulme
Copy link
Contributor

atoulme commented Mar 18, 2023

I will look into it for you. Give me a bit.

@atoulme
Copy link
Contributor

atoulme commented Mar 18, 2023

Well, I'll be... no this is for the Kafka exporter. Check out the latest README and see how this is now documented. It should work for your use case, or there is a bug.

@flyerjayden
Copy link
Author

Error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)
2023/03/19 02:00:16 collector server run finished with error: failed to build pipelines: failed to create "kafka" exporter, in pipeline "traces/*": kafka: invalid configuration (The SASL mechanism configuration is invalid. Possible values are `OAUTHBEARER`, `PLAIN`, `SCRAM-SHA-256`, `SCRAM-SHA-512` and `GSSAPI`)

It does not work.

@flyerjayden
Copy link
Author

flyerjayden commented Mar 19, 2023

AWS_MSK_ISM is not one of mechanism that sarama library supports:

authentication.go line 125

case "AWS_MSK_IAM":
		saramaConfig.Net.SASL.SCRAMClientGeneratorFunc = func() sarama.SCRAMClient {
			return awsmsk.NewIAMSASLClient(config.AWSMSK.BrokerAddr, config.AWSMSK.Region, saramaConfig.ClientID)
		}
		saramaConfig.Net.SASL.Mechanism = awsmsk.Mechanism

sarama@v1.38.1/broker.go line 63:

// SASLMechanism specifies the SASL mechanism the client uses to authenticate with the broker
type SASLMechanism string

const (
	// SASLTypeOAuth represents the SASL/OAUTHBEARER mechanism (Kafka 2.0.0+)
	SASLTypeOAuth = "OAUTHBEARER"
	// SASLTypePlaintext represents the SASL/PLAIN mechanism
	SASLTypePlaintext = "PLAIN"
	// SASLTypeSCRAMSHA256 represents the SCRAM-SHA-256 mechanism.
	SASLTypeSCRAMSHA256 = "SCRAM-SHA-256"
	// SASLTypeSCRAMSHA512 represents the SCRAM-SHA-512 mechanism.
	SASLTypeSCRAMSHA512 = "SCRAM-SHA-512"
	SASLTypeGSSAPI      = "GSSAPI"
	// SASLHandshakeV0 is v0 of the Kafka SASL handshake protocol. Client and
	// server negotiate SASL auth using opaque packets.
	SASLHandshakeV0 = int16(0)
	// SASLHandshakeV1 is v1 of the Kafka SASL handshake protocol. Client and
	// server negotiate SASL by wrapping tokens with Kafka protocol headers.
	SASLHandshakeV1 = int16(1)
	// SASLExtKeyAuth is the reserved extension key name sent as part of the
	// SASL/OAUTHBEARER initial client response
	SASLExtKeyAuth = "auth"
)

@michaelsaah
Copy link

here's the PR: #5763

@MovieStoreGuy did you ever get around to testing this? also playing with it and seeing the same error from sarama as above.

@michaelsaah
Copy link

hmmm so perhaps this comment means the implementation is known to be broken?

IBM/sarama#1985 (comment)

@MovieStoreGuy
Copy link
Contributor

Sorry, yes I had tried to run and validate this however, how the client is configured, it wasn't possible to do so.

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@TaoYang526
Copy link

@MovieStoreGuy Any updates on this? I saw the same issue.
IIUIC, AWS_MSK_ISM should be an internal mechanism and has not been supported by sarama yet, according to updates on SCRAMClientGeneratorFunc by this mechanism, maybe the saramaConfig.Net.SASL.Mechanism should be sarama.SASLTypeSCRAMSHA512 or sarama.SASLTypeSCRAMSHA256 ?

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Sep 18, 2023
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 17, 2023
@bpalermo
Copy link

Should we leverage this? https://github.com/aws/aws-msk-iam-sasl-signer-go

@koh-satoh-wpg
Copy link

@flyerjayden did you happen to find any workaround?

Otherwise I believe this is the only way to go 🤔

Should we leverage this? https://github.com/aws/aws-msk-iam-sasl-signer-go

@jnavarro86
Copy link

I have built and deployed this version and it is working for me using iam roles.

MovieStoreGuy added a commit that referenced this issue Dec 4, 2024
…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (#32500)

**Description:** 
This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to
kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL
Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This
mechanism is added because the "AWS_MSK_IAM" is not working in our
cluster and also in this
[issue](#19747).
We added an new mechanism instead of replace the existing one because we
want to keep the backward compatibility just in case someone is using
"AWS_MSK_IAM".

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** 

[19747](#19747)

**Testing:** 
We built the images and tested the SASL mechanism in our team.
We added related unit tests.

**Documentation:** 
We updated the kafkaexporter and kafakreciever README on the SASL
mechanism.

---------

Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>
shivanthzen pushed a commit to shivanthzen/opentelemetry-collector-contrib that referenced this issue Dec 5, 2024
…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (open-telemetry#32500)

**Description:** 
This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to
kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL
Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This
mechanism is added because the "AWS_MSK_IAM" is not working in our
cluster and also in this
[issue](open-telemetry#19747).
We added an new mechanism instead of replace the existing one because we
want to keep the backward compatibility just in case someone is using
"AWS_MSK_IAM".

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** 

[19747](open-telemetry#19747)

**Testing:** 
We built the images and tested the SASL mechanism in our team.
We added related unit tests.

**Documentation:** 
We updated the kafkaexporter and kafakreciever README on the SASL
mechanism.

---------

Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>
ZenoCC-Peng pushed a commit to ZenoCC-Peng/opentelemetry-collector-contrib that referenced this issue Dec 6, 2024
…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (open-telemetry#32500)

**Description:** 
This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to
kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL
Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This
mechanism is added because the "AWS_MSK_IAM" is not working in our
cluster and also in this
[issue](open-telemetry#19747).
We added an new mechanism instead of replace the existing one because we
want to keep the backward compatibility just in case someone is using
"AWS_MSK_IAM".

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** 

[19747](open-telemetry#19747)

**Testing:** 
We built the images and tested the SASL mechanism in our team.
We added related unit tests.

**Documentation:** 
We updated the kafkaexporter and kafakreciever README on the SASL
mechanism.

---------

Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>
sbylica-splunk pushed a commit to sbylica-splunk/opentelemetry-collector-contrib that referenced this issue Dec 17, 2024
…S_MSK_IAM_OAUTHBEARER" to kafkaexporter (open-telemetry#32500)

**Description:** 
This PR added the SASL mechanism "AWS_MSK_IAM_OAUTHBEARER" to
kafkaexporter and kafkareceiver. This mechanism use the AWS MSK IAM SASL
Signer for Go https://github.com/aws/aws-msk-iam-sasl-signer-go. This
mechanism is added because the "AWS_MSK_IAM" is not working in our
cluster and also in this
[issue](open-telemetry#19747).
We added an new mechanism instead of replace the existing one because we
want to keep the backward compatibility just in case someone is using
"AWS_MSK_IAM".

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

**Link to tracking Issue:** 

[19747](open-telemetry#19747)

**Testing:** 
We built the images and tested the SASL mechanism in our team.
We added related unit tests.

**Documentation:** 
We updated the kafkaexporter and kafakreciever README on the SASL
mechanism.

---------

Co-authored-by: Sean Marciniak <30928402+MovieStoreGuy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

8 participants