wip: redpanda serverless demo

[glenngillen]

No description provided.

[glenngillen]

This is the most useful difference between this and the docker version, for now.

If you sign up for a Redpanda account you can start a free serverless cluster. You'll also need to create a user, (I think) add an ACL that grants everything, and then in the API settings section you'll see that they support SASL_SSL and SCRAM-SHA-256/512 as the protocol and mechanism. These settings need to be passed into the kafka CLI tools and AIUI this is how you do that.

[glenngillen]

So the problem I currently have is I can't get the Redpanda Serverless cluster working. I've gone all the way back and tried to manually change the steps from our previous managed kafka guide. Unfortunately trying to start the consumer throws up the following error:

$ export JAVA_HOME=/opt/homebrew/opt/openjdk
kafka-console-consumer --topic demo \
  --bootstrap-server localhost:4000 \
  --consumer.config ./kafka.config

[2024-04-18 14:00:12,988] WARN [Consumer clientId=console-consumer, groupId=console-consumer-98168] Connection to node -1 (localhost/127.0.0.1:4000) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
[2024-04-18 14:00:12,989] WARN [Consumer clientId=console-consumer, groupId=console-consumer-98168] Bootstrap broker localhost:4000 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
[2024-04-18 14:00:15,719] WARN [Consumer clientId=console-consumer, groupId=console-consumer-98168] Connection to node -1 (localhost/127.0.0.1:4000) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient)
[2024-04-18 14:00:15,719] WARN [Consumer clientId=console-consumer, groupId=console-consumer-98168] Bootstrap broker localhost:4000 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
^CProcessed a total of 0 messages

As the error states, this appears to be a problem with TLS/authentication. I've either got it misconfigured, or the node/relay is interfering with expectations the kafka console tools have about how that works. Any ideas?

[glenngillen]

Thought I should break out the actual suggested causes in the error message incase it's something other than the one I suggested:

(1) Authentication failed due to invalid credentials with brokers older than 1.0.0,
(2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic),
(3) Transient network issue. (org.apache.kafka.clients.NetworkClient)