Add support for Reactor Kafka

[thepaep]

As said in reactor/reactor-kafka#100 (comment) native support for Reactor Kafka would be nice to lots of users. Spring Boot currently supports WebFlux, but auto-configuration and more capabilities aren't supported yet with Reactor Kafka. As others said in different issues, I also think the connection between Reactor Kafka and Spring Boot should be more documented and supported since it's the most basic usage of Reactor Kafka. There is currently no official way of using Reactor Kafka with Spring Boot, which is pretty odd.

[lynch19]

+1

[lynch19]

@snicoll @simonbasle @garyrussell any help with this?

Can someone describe what features are currently missing on this subject and how should they be implemented (what functionalities should be added and where), so it'll be easier for contributors to help with this?

[roger751]

+1

[snicoll]

@lynch19 @roger751 please use the reaction on the original description rather than +1 comments like this.

[garyrussell]

@lynch19 reactor-kafka has two fundamental properties objects

https://github.com/reactor/reactor-kafka/blob/main/src/main/java/reactor/kafka/receiver/ReceiverOptions.java

https://github.com/reactor/reactor-kafka/blob/main/src/main/java/reactor/kafka/sender/SenderOptions.java

Each with a number of properties.

In addition, creation of these takes a Map<String, Object> of regular Kafka Properties (similar to the ...properties extension for the spring-kafka auto configuration).

There are several levels of Boot auto configuration that would be useful, with the MVP being the auto-configuration of these two beans using application properties.

[AntonLGVS]

I'm using the reactive kafka in my applications. The Reactor kafka provides everything you need to work. Or does it means support in the form of annotations?
For example:

@ReactiveKafkaListener(topic = "smth", autoCommit=true)
public Mono<Void> consumer(DTO dto) {
    log.info("received dto: " + dto);
    return service.call(dto).then();
}

[garyrussell]

No; not at all; this is just about configuring the sender and receiver options via Boot properties.

Such a mechanism would belong in Spring for Apache Kafka (spring-kafka), but there are currently no plans to do so.

[garyrussell]

I would suggest something like package org.springframework.boot.autoconfigure.reactor.kafka in the spring-boot-autoconfigure project.

We can leverage the existing KafkaProperties for the consumer and producer properties passed into the receiver and sender options.

[wilkinsona]

This has been requested previously (#18751) but was declined as the Reactor team advised us against adding support. IIRC, this was due to the status of Reactor's Kafka support at the time and things may well have moved on since then.

@simonbasle, what's you take on this now please? Has this moved on sufficiently in the last couple of years that this is now worth considering again?

[simonbasle]

@wilkinsona with the broad changes in 1.3.x late 2020, reactor-kafka has been made more stable and maintainable. thanks to @garyrussell and @artembilan, the project is more actively maintained. thus I would defer to gary and artem regarding that decision, but it definitely looks better than back in 2019

[garyrussell]

I concur; it is in much better shape now, thanks to some significant work by Sergie back then. Also, due to community requests, the Spring Cloud Stream team are likely to incorporate it in the next major release and basic auto configuration of the sender and receiver properties will make things easier there too.

[almogtavor]

@garyrussell what will it exactly demand? Only ReactiveKafkaAutoConfiguration, ReactiveKafkaAnnotationDrivenConfiguration (that will use KafkaProperties\ ReactiveKafkaProperties), ReceiverOptionsCustomizer and SenderOptionsCustomizer classes? Or is there anything else that needs to be noted?

We can leverage the existing KafkaProperties for the consumer and producer properties passed into the receiver and sender options.

Do you mean that we should use the existing KafkaProperties class, right?