Added Alias support and external resource feature changes

[jjain1259]

Problem

The existing Elasticsearch Sink Connector only supports auto creation of indexes and datastreams based on Kafka topic name. This approach enforces connector-defined naming conventions and limits customers from integrating with pre-existing Elasticsearch resources. Additionally, the connector previously lacked support for writing to aliases.

Solution (INIT-8745)

This PR introduces a new feature: External Topic to Resource Mappings. This enhancement allows users to explicitly map Kafka topics to existing Elasticsearch resources—such as indices, data streams, and aliases (both index and data stream aliases).

New Configuration Parameters:

• external.resource.usage

  • Type: String
  • Default: DISABLED
  • Valid Values: INDEX, DATASTREAM, ALIAS_INDEX, ALIAS_DATASTREAM, DISABLED
  • Purpose: Defines the type of Elasticsearch resource to target for writes.

• topic.to.external.resource.mapping

  • Type: List (comma-separated topic:resource pairs)
  • Default: ""
  • Purpose: Provides explicit mappings between Kafka topics and Elasticsearch resources (e.g., orders:index-orders, logs:logs-ds).

Additional Notes:

• All mapped resources must be pre-created in Elasticsearch before connector startup.
• Each topic must map to exactly one resource (strict one-to-one mapping).
• Enhanced validation logic to support and verify aliases during connector initialization.

Does this solution apply anywhere else?

• yes
• no

If yes, where?

Test Strategy

Testing done:

• Unit tests
• Integration tests
• System tests
• Manual tests

Scenarios Validated:

• Writing to pre-created indices.
• Writing to pre-created data streams.
• Writing to aliases (both index and data stream).
• Connector behavior when configured with non-existent or invalid resources.
• Corresponding KDP tests written for the above scenarios (to be merged post-review).

Release Plan

• Will be performing a new release: 15.1.x
• The change is backward compatible; default behavior remains unchanged (external.resource.usage=DISABLED).
• Changes were tested in the devel environment by deploying the new image to an existing production-exposed connector.
• The feature will be released in both Confluent Cloud and Confluent Platform distributions.

[confluent-cla-assistant]

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

[sonarqube-confluent]

Analysis Details

7 Issues

• 0 Bugs
• 2 Vulnerabilities
• 5 Code Smells

Coverage and Duplications

• 95.50% Coverage (90.60% Estimated after merge)
• No duplication information (0.00% Estimated after merge)

Project ID: kafka-connect-elasticsearch

View in SonarQube