Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka topics management #3

Merged
merged 12 commits into from
Sep 3, 2024
Merged

Kafka topics management #3

merged 12 commits into from
Sep 3, 2024

Conversation

iyourshaw
Copy link
Collaborator

Updates the kafka-setup service to use the jikkou resource-as-code framework to create Kafka topics.

The kafka-setup service can now be configured with the environment variables described in the updated README to create groups of topics for the ODE applications: jpo-ode, jpo-geojsonconverter, jpo-conflictmonitor, and jpo-deduplicator.

New/updated files in the /kafka folder:

  • kafka-topics-values.yaml - Configuration file containing topics for all applications
  • kafka-topics-template.jinja - A template to create resource definitions for the topics.
  • application.conf, jikkouconfig - Jikkou configuration files
  • kafka_init.sh - Initialization script that runs the Jikkou CLI to validate the resource definitions and create topics
  • Dockerfile.jikkou - Runs the init script in a container.

This PR also adds a health check to the kafka service in docker-compose-kafka.yml, and adds dependencies to the other containers in that compose so that they wait for Kafka to fully start up before running.

Tested by running Docker Compose with the kafka_full profile, and observing the topics created in the Kafka UI.

Future Development

Worth looking into for future updates:

  • Jikkou also has the ability to manage Kafka Connect and Schema Registry and to be run as a long-running API service. For example it could simplify creating connectors, and monitor their health and restart them as needed.
  • Could look into integrating the Jikkou tool into a Kubernetes init container and/or Gihub Actions for use in production.

Copy link
Collaborator

@Michael7371 Michael7371 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, just one comment.

kafka/kafka-topics-values.yaml Outdated Show resolved Hide resolved
@Michael7371 Michael7371 merged commit 849e009 into develop Sep 3, 2024
@Michael7371 Michael7371 deleted the kafka-topics-management branch September 3, 2024 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants