NOTE: This project was formally known as Kafka Topology Builder, old versions of this project can still be found under that name.
JulieOps helps you automate the management of your things within Apache Kafka, from Topics, Configuration to Metadata but as well Access Control, Schemas. More items are plan, check here for details.
One of the typical questions while building an Apache Kafka infrastructure is how to handle topics, configurations and the required permissions to use them (Access Control List).
The JulieOps cli, in close collaboration with git and Jenkins (CI/CD) is here to help you setup an organised and automated way of managing your Kafka Cluster.
We recommend taking time to read the docs. There's quite a bit of detailed information about GitOps, Apache Kafka and how this project can help you automate the common operational tasks.
You might be wondering what is the usual workflow to implement this approach:
Action: As a user, part of a developer team (for example), I like to have some changes in Apache Kafka.
Change Request: As a user:
- Go to the git repository where the topology is described
- Create a new branch
- Perform the changes need
- Make a pull request targeting master branch
Approval process: As an ops admin, I can:
- Review the pull request (change request) initiated by teams
- Request changes when need
- Merge the requests.
Considerations:
- Using webhooks, the git server (github, gitlab or bitbucket) will inform the CI/CD system changes had happened and the need to apply them to the cluster.
- All changes (git push) to master branch are disabled directly. Changes only can happen with a pull request. Providing a Change Management mechanism to fit into your org procedures.
If you are using the JulieOps tool, or plan to use it in your project? might be you have encounter a bug? or a challenge? need a certain future? feel free to reach out into our gitter community.
What can you achieve with this tool:
- Support for multiple access control mechanisms:
- Traditional ACLs
- Role Bases Access Control as provided by Confluent
- Automatically set access control rules for:
- Kafka Consumers
- Kafka Producers
- Kafka Connect
- Kafka Streams applications ( microservices )
- KSQL applications
- Schema Registry instances
- Confluent Control Center
- KSQL server instances
- Manage topic naming with a topic name convention
- Including the definition of projects, teams, datatypes and for sure the topic name
- Some of the topics are flexible defined by user requirements
- Allow for creation, delete and update of:
- topics, following the topic naming convention
- Topic configuration, variables like retention, segment size, etc
- Acls, or RBAC rules
- Service Accounts (Experimental feature only available for now in Confluent Cloud)
- Manage your cluster schemas.
- Support for Confluent Schema Registry
Out of the box support for Confluent Cloud and other clouds that enable you to use the AdminClient API.
This tool is available in multiple formats:
- As a Docker image, available from docker hub
- As an RPM package, for the RedHat alike distributions
- As a DEB package, for Debian based distros
- Directly as a fat jar (zip/tar.gz)
- As a fat jar.
The latest version are available from the releases page.
This is how you can run the tool directly as a docker image:
docker run purbon/kafka-topology-builder:latest julie-ops-cli.sh --help
Parsing failed cause of Missing required options: topology, brokers, clientConfig
usage: cli
--brokers <arg> The Apache Kafka server(s) to connect
to.
--clientConfig <arg> The client configuration file.
--dryRun Print the execution plan without
altering anything.
--help Prints usage information.
--overridingClientConfig <arg> The overriding AdminClient
configuration file.
--plans <arg> File describing the predefined plans
--quiet Print minimum status update
--topology <arg> Topology config file.
--validate Only run configured validations in
your topology
--version Prints useful version information.If you install the tool as rpm, you will have available in your $PATH the julie-ops-cli.sh. You can run this script with the same options observed earlier, however you will need to be using, or be in the group, for the user julie-kafka.
An example topology should look like this (in yaml format):
context: "context"
source: "source"
projects:
- name: "foo"
consumers:
- principal: "User:app0"
- principal: "User:app1"
streams:
- principal: "User:App0"
topics:
read:
- "topicA"
- "topicB"
write:
- "topicC"
- "topicD"
connectors:
- principal: "User:Connect1"
topics:
read:
- "topicA"
- "topicB"
- principal: "User:Connect2"
topics:
write:
- "topicC"
- "topicD"
topics:
- name: "foo" # topicName: context.source.foo.foo
config:
replication.factor: "2"
num.partitions: "3"
- name: "bar" # topicName: context.source.foo.bar
config:
replication.factor: "2"
num.partitions: "3"
- name: "bar"
topics:
- name: "bar" # topicName: context.source.bar.bar
config:
replication.factor: "2"
num.partitions: "3"more examples can be found at the example/ directory.
Also, please check, the documentation in the docs for extra information and examples on managing ACLs, RBAC, Principales, Schemas and many others.
If you're having problems with JulieOps I would recommend lookup up two main sources of information:
- The project issues tracker. Highly possible others might have had your problem before.
- Our always work in progress troubleshooting guide
Interested on contributing back? might be have an idea for a great future? or wanna fix a bug? Check our contributing doc for guidance.
The project is build using Java and Maven, so both are required if you aim to build the tool from scratch. The minimum version of Java supported is Java 8, note it soon will be deprecated here, it is only keep as supported for very legacy environments.
It is recommended to run JulieOps with Java 11 and an open JDK version.
If you are interested on building a release artifact from the source code, check our release doc for guidance.
Nightly builds as well as release builds are regularly available from the Actions in this project.
Nightly release build are available as well from here.