Support KRaft mode for EmbeddedKafkaBroker

[vladimirfx]

Expected Behavior

The embedded Kafka broker should be able to run in KRaft mode.

Current Behavior

There is no option to run Kafka in KRaft mode nor to completely disable Zookeeper.

Context

KRaft mode becomes production ready from Kafka 3.3.0 and only available mode from Kafka 4+. So support for KRaft is becoming a requirement in 2023. But it is relevant just now for EmbeddedKafkaBroker because it is relative simplicity to run and minimize footprint by removing ZK.

[artembilan]

Feel free to contribute this feature.
Or let's wait until it really becomes and official feature (3.3.0 GA), so we would be good to drop any ZK support we have in the embedded broker so far.

[garyrussell]

We are now on Kafka 3.3.1 so this can be implemented; contributions are welcome.

[govi20]

If no one is working on it, I would like to give it a shot.

[artembilan]

@govi20 ,

no one as far as I am aware.

Thank you for help proposal!

[eddumelendez]

Hi 👋🏽, sorry for hijacking the thread but I think this can be useful information. Testcontainers Kafka module version 1.18.0 offers support for Kraft mode. Look at this test

https://github.com/testcontainers/testcontainers-java/blob/main/modules/kafka/src/test/java/org/testcontainers/containers/KafkaContainerTest.java#L136-L145

[garyrussell]

I also found this apache/kafka#13375 which could be used as a basis for the changes needed here.

[garyrussell]

I made some progress, but this is blocked by https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-15092

[garyrussell]

Unblocked; the necessary class is in server-common...-test.jar.