Kafka message sending fails with 'class SslBundleSslEngineFactory could not be found'

[kst1980]

Spring Boot Kafka SSL Issue on EKS Deployment using Spring boot 3.4.3(tried 3.3.5 and 3.4.2)
We are facing an issue where the Kafka producer works fine in local environments (IntelliJ, mvn spring-boot:run, and java -jar). However, after deploying to EKS, we encounter the following exception:

Exception occurred: org.apache.kafka.common.config.ConfigException: Invalid value org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory for configuration ssl.engine.factory.class: Class org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory could not be found.. Stack Trace: [org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:778), org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:531), org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:114), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:134), org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:643), org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:295), org.springframework.kafka.core.DefaultKafkaProducerFactory.createRawProducer(DefaultKafkaProducerFactory.java:944), org.springframework.kafka.core.DefaultKafkaProducerFactory.createKafkaProducer(DefaultKafkaProducerFactory.java:826), org.springframework.kafka.core.DefaultKafkaProducerFactory.doCreateProducer(DefaultKafkaProducerFactory.java:793), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:768), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:762), org.springframework.kafka.core.KafkaTemplate.getTheProducer(KafkaTemplate.java:976), org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:828), org.springframework.kafka.core.KafkaTemplate.observeSend(KafkaTemplate.java:805), org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:608),

Issue Details:
Kafka producer initialization delayed: Instead of initializing at application startup, it only starts when sending a message, leading to the SslBundleSslEngineFactory class not being found.
Temporary workaround: We manually create the producer in a @PostConstruct method inside a KafkaConfig class.

@Slf4j
@Configuration
@RequiredArgsConstructor
public class KafkaConfig {
    private final ProducerFactory<String, String> producerFactory;

    @PostConstruct
    public void initializeProducer() {
        try {
            Producer<String, String> producer = producerFactory.createProducer(); // Workaround to initialize Kafka producer
            log.info("Kafka Producer initialized successfully: {}", producer);
        } catch (Exception e) {
            log.error("Failed to initialize Kafka Producer", e);
        }
    }
}

Spring Boot Configuration (application.yml)

spring:
  application:
    name: test
  ssl:
    bundle:
      pem:
        kafkaCert:
          keystore:
            certificate: classpath:KafkaKeystoreCert.pem
            private-key: classpath:KafkaKeystoreKey.pem
          truststore:
            certificate: classpath:KafkaTruststorePem.pem
  kafka:
    bootstrap-servers: ${KafkaBootstrapServers}
    ssl:
      bundle: kafkaCert
    security:
      protocol: SSL
    client-id: client-id
    producer:
      retries: 3
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Request for a Proper Solution
Do we have a proper fix for this issue instead of the workaround?

Is there a missing dependency or classpath issue specific to the environment?
Are there configurations that need adjustment to ensure Kafka SSL settings load correctly?
Any best practices for ensuring SslBundleSslEngineFactory is properly available at runtime?

[mhalbritter]

This could be a classloader issue. It does work locally with SSL enabled, too? Or is SSL only enabled in EKS?

Please take the time to provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.

[kst1980]

Yes, it works locally with the same configuration and SSL enabled, but I read that it might be functioning correctly locally due to the JVM.

I have added the following dependency, which provides org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory, and verified its presence in the image after packaging:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-autoconfigure</artifactId>
</dependency>

Since the Spring Kafka producer is lazy-initialized, it throws a ClassNotFoundException after the application startup.

[mhalbritter]

Please provide a sample to reproduce.

[kst1980]

kafka-producer.zip - sample